You describe the experiment. An LLM proposes a structured protocol and visual checking instructions. After your approval, a deterministic executor moves the simulated xArm 7 using known model geometry. Sensors and a VLM check each critical operation.
The LLM never commands joints or invents motion coordinates. The VLM never measures exact microlitres from an image. Those responsibilities belong to known model geometry, robot feedback and the liquid ledger.
The simulation starts ready with robot, tool and camera geometry from its MuJoCo model. There is no calibration step or saved profile to manage. Station poses are still measured from rendered ArUco images at the beginning of every experiment.
Fill a 96-well plate with four different liquids. Dispense 100 µL into each well, assigning 24 wells to each liquid. Change tips when switching liquids.
This example compiles to 201 critical actions: one localization, four tip pickups, 96 aspirations, 96 dispenses and four tip ejections. Completion adds a final audit event.
The hosted application calls Gemini to interpret your instructions. User-facing Liquid 1–4 map to source indices 0–3. Wells are named A1–H12. The result is parsed through a strict Pydantic schema; unsupported requests and invalid outputs are rejected rather than executed.
A SHA-256 digest covers the exact protocol, including its visual checklists. Starting with a changed protocol and an old approval hash is rejected.
The action compiler uses explicit rules to create the same action sequence for the same validated protocol. Model responses can vary, but an approved sequence is not rewritten by the LLM during execution.
{
"transfers": [{"source": 0, "well": "A1", "volume": 100}],
"tip_policy": "per_source",
"verification": "strict_vlm",
"verification_checks": {
"locate": "Check visible fixture presence…",
"pickup": "Check that a tip is visibly attached…",
"aspirate": "Check gross interaction at the source…",
"dispense": "Check tip presence and interaction at the plate…",
"eject": "Check that the pipette is now bare…"
}
}The abbreviated strings above illustrate the structure. The actual LLM-generated instructions are included in the protocol you review.
Each station has three unique 32 mm ArUco tags fixed to supported pedestals on its cartridge. Their surveyed corners span the station and use different heights. They are one rigid constellation, not three unrelated pose estimates.
T_base_station = T_base_camera × T_camera_station p_base_target = T_base_station × p_station_target
| Station | Required IDs | Local targets |
|---|---|---|
| 96-well plate | 0, 1, 2 | 8 × 12 wells at 9 mm pitch |
| Tip rack | 3, 4, 5 | 8 × 12 tips at 9 mm pitch |
| Four-source rack | 6, 7, 8 | Tube centres at 32 mm spacing |
The fit rejects RMS reprojection above 0.8 px, any corner above 1.5 px and weak pose observability. Independent simulation geometry additionally rejects errors above 3 mm or 0.02 rad; truth is a gate, never a replacement pose. Logs include all tag IDs, view count and full pose.
Localization and motion authorization are separate. The detector estimates tilted/elevated stations in all six degrees of freedom. Pipetting is commissioned for securely seated, near-horizontal cartridges at the deck height; a changed height/tilt requires a new fixture model and validation. Movement after survey invalidates targets. Three tags improve evidence but do not prove an accuracy specification by themselves. OpenCV rigid marker boards.
The model uses the MuJoCo Menagerie xArm 7 chain, joint limits and meshes. A deterministic inverse-kinematics seed selects a consistent solution. The hosted ROS 2 / MoveIt 2 / Pilz service plans PTP transit and LIN approach/retraction trajectories; the offline adapter uses quintic joint trajectories. Position actuators follow them in MuJoCo at a 2 ms physics timestep, with gravity/Coriolis compensation.
The arm withdraws from the interaction point, rises to a 220 mm clearance plane, transfers across the deck and approaches its destination. Short 20 mm entry/withdrawal segments reduce the lateral arc near a well. Planned paths are checked before execution; contacts, joint positions and joint speeds are also checked while stepping the physics.
The simulated disposable tip is 50 mm long and tapers from a 2.5 mm proximal radius to a 0.5 mm distal radius. Stereo visual servoing measures the distal tip from RGB images before each transfer. No measured TCP offset is fitted or stored. Read the feedback loop and visibility limits.
The well rim is at fixture-local z=20 mm and the bottom at z=10 mm. The dispense target is z=17 mm: nominally 3 mm inside the well.
The tapered envelope is checked against actual fixture geometry throughout collision preflight and physics stepping. Open-well walls use an analytic guard because a convex mesh collision hull would incorrectly fill the cavity. The attached tip also participates in MuJoCo contacts with solid fixtures. An unsafe approach stops before liquid is dispensed.
MuJoCo handles the robot and scene; a separate state ledger handles liquid amounts. Aspiration subtracts from a source and loads the pipette. Dispensing empties that load into the designated well. Each operation checks tip presence, liquid identity, capacity, available volume and execution order.
source volumes + well volumes + pipette load = 40,000 µL
Visible liquid levels track this ledger. Aspiration targets follow the simulated source surface with 4 mm immersion. Tip pickup removes the selected rack tip visually; ejection adds a discarded tip to the bin. Tip retention is an explicit simulated adapter, not a physical press-fit model.
The white ring on a well means liquid has been dispensed. The Wells verified counter advances only after its verification succeeds. If a VLM check fails after liquid transfer, the ledger retains the transferred volume and the well is not counted as verified.
A separate continuous VLM hazard supervisor watches execution and the idle hosted workcell. Deterministic whole-arm table/collision guards run independently at every physics step.
The planning LLM generates separate checking instructions for localization, pickup, aspiration, dispensing and ejection. These instructions are approved with the experiment, then sent to a separate VLM call after each critical operation.
Measured robot pose, tip state, source and pipette volumes, well contents, well-entry clearances and liquid conservation. These checks establish exact identities, quantities and geometric tolerances.
Actual rendered overhead, wrist and pipette-inspection images. Pickup/ejection also include a before-image. The VLM cross-checks visible fixture presence, gross interaction, attached/bare tip state and visible anomalies.
Sensor evidence is labeled explicitly for the VLM. It must not claim to have observed motion, liquid flow or a dispensing stroke from still images. Exact well coordinates and microlitre amounts are not inferred visually.
The dashboard shows the actual result, reason, observations, model identity, latency and checking prompt. A compact readout remains next to the camera. The full history and transmitted images are retained with the experiment records. Offline local mode is explicitly labeled and does not fabricate VLM results.
| View | Purpose |
|---|---|
| Workcell | Watch the full arm and its movements between stations. |
| Station detail | Inspect the plate, rack and nearby end-effector motion. |
| Top view | See fixture arrangement and the robot from above. |
| Tip close-up | Watch the tapered tip approach, enter and withdraw from a well. |
| Robot-camera thumbnail | See the wrist-camera view used for localization. |
The main view receives a continuous JPEG stream, independently of the slower status/thumbnail updates. Switching views explicitly aborts the previous request. The server tracks active camera subscriptions and renders them fairly, so different viewers do not lock one another onto a stale camera. Connections are retried after a stream error.
The slider selects 0.25×–4× motion playback, with 1× as the default. Lower settings make insertion easier to watch. Physics timesteps and the approved action sequence remain unchanged. Rendering targets up to 15 fps per view; actual throughput depends on CPU load, network speed and the number of simultaneous views. The displayed fps is measured from received, decoded frames. The pose naturally holds during VLM review or pause; that is not a camera freeze.
Pause holds at a motion checkpoint; Resume continues. Stop requests a position hold and preserves the ledger. A critical failure or inconclusive review also stops the executor. The simulation stop is not a certified hardware emergency stop.
Automatic retries are disabled because an action may already have transferred liquid before its verification failed. Inspect the error, receipt and report first. Use New run to start with fresh simulated inventory. Do not assume repeating an uncertain transfer is safe.
The state sequence is idle → running ↔ paused → completed, stopped or failed. Starting requires a fresh idle workstation and the matching approved protocol hash. Service restarts create a new idle simulation; they do not silently resume an interrupted physical sequence.
The hosted application runs on the pipette-lab Google Compute Engine VM in europe-west2-a (London), project pipette-lab-cst-20261103. It does not depend on your laptop remaining online. Docker restart policies and the VM automatic-restart setting keep the service running across ordinary restarts; cloud outages and maintenance can still interrupt it.
Caddy serves HTTPS and forwards requests to FastAPI. One worker owns MuJoCo and its rendering context; browser requests receive state, camera frames and controls through the API. Vertex AI calls use the VM service account. Keep your access token private; it grants control of the shared simulated workstation.
| Module | Input → output |
|---|---|
ai.py | Natural language → protocol; images/checklist/sensors → VLM result |
protocol.py | Protocol JSON → validated schema, digest and deterministic action list |
perception.py | Camera image + known sensor geometry → accepted fixture transforms |
scene.py | Deck layout → robot, supported fixtures, tags, labware and cameras |
simulation.py | Localized targets → trajectories, physics, clearance checks and images |
engine.py | Approved actions → skill execution, ledger, verification and receipts |
api.py | Browser commands ↔ worker state, reports and camera streams |
web/ | Experiment review, controls, live visualization and this documentation |
Reports, event logs, protocols and images live in /opt/pipette-lab/data/<run-id>/ on the VM. Download the current report through the dashboard. Installation instructions and tests are in the GitHub repository.
This is an actuator-driven MuJoCo demonstrator. Liquid transfer and tip retention are logical adapters with synchronized visuals; fluid dynamics, press-fit forces and physical pipette calibration are not simulated. The modular deck is a rigid simulation design, not manufacturing-qualified CAD.
Automated tests cover full 96-well execution in both deck layouts, conservation of liquid, tip policy, fixture relocation, invalid protocols, authorization, inconclusive verification and deliberately misaligned insertion. Passing these checks does not guarantee zero errors or certify a real robot.
A hardware adapter would need a UFactory driver, validated stereo feedback and camera/hand-eye calibration, independent metrology, measured pipette actuation, fixture sensing, emergency stops and contamination validation before physical execution.