Observe the tip. Correct its position.

The robot no longer fits or stores a measured flange-to-tip offset. Two fixed cameras measure the visible tip before every aspiration and dispense. The controller uses fresh measured error to make bounded translations.

What runs at each transfer

1. Three ArUco tags per station and three wrist views with known camera geometry establish the station’s 6D pose. Surveyed labware geometry defines the intended opening.
2. A conservative CAD tool envelope guides a coarse approach above the target. The pipette remains vertical; this controller corrects three-dimensional position, not tool orientation. The tip must be straight and have the commissioned shape.
3. Two actual 3840 × 2880 RGB measurement images locate the blue simulated tip silhouette. A robust shaft-centreline fit across many image rows avoids relying on one shaded endpoint row. Stereo projection from the model reconstructs the distal centre. A known 0.5 mm distal radius corrects the silhouette rim’s perspective; no flange offset is estimated.
4. The controller calculates error = target − observed tip. It requests a differential translation of 0.85 × error, capped at 1 mm, through the same guarded MoveIt/Pilz LIN path.
5. Fresh images close the loop. Two observations within 0.25 mm are required. No previous tip’s alignment is reused.
6. A bounded vertical insertion preserves this alignment. Independent geometry and actuation checks must pass before the liquid ledger changes.

This is a settled, position-based visual feedback loop, not high-rate image streaming into joint commands. See Chaumette & Hutchinson, Visual Servo Control, Part I for the distinction between image-based and position-based visual control. The LLM supplies protocol intent and review checklists; it never chooses corrections.

Visibility and hard limits

Each observation must be no more than 5 seconds old with no intervening robot motion. The stereo ray angle must exceed 20°, and each reprojection residual must be ≤1.2 px. Missing, ambiguous or clipped tips stop the operation. The correction envelope is 3 mm initial error, 1 mm per correction, 5 mm total travel and 10 observations. Divergence stops motion.

The well or liquid can hide an immersed tip. The system therefore aligns above the opening, then executes a guarded vertical stroke: 9 mm at a well and at most 32 mm at a source. This final stroke is explicitly logged as potentially occluded; it is not claimed to have direct visual endpoint feedback. The simulator checks actual tip position, well-wall/bottom clearance, contacts and liquid conservation independently.

Whole-arm table bounds and collision checks remain active at every 2 ms physics step. Continuous VLM hazard review remains an additional, slower safeguard. Neither a VLM nor this optical loop replaces a physical protective circuit.

Known simulation geometry

The MuJoCo model supplies robot kinematics, the nominal tool dimensions and camera intrinsics/extrinsics at startup. No user calibration, error injection, fitted joint offsets, fixture corrections or saved profiles are used. ArUco image processing still measures each station’s pose, and stereo image processing still measures the tip before each transfer.

The UI shows current error, observation count, target, insertion mode and measured evidence. Reports preserve camera hashes, image files, pixel coordinates, reconstruction quality, corrections and independent transfer checks. Model-provided geometry is identified as MUJOCO_MODEL in the report.

Real-world requirements

Use two rigid, synchronized global-shutter cameras with overlapping views of every approach point, calibrated intrinsics/distortion and extrinsics relative to the deck. Lock focus and mounts. Select optics from the required field of view and measured pixel-to-millimetre uncertainty. Transparent tips need validated backlit edge detection; the simulation’s blue-tip color thresholds are not a physical detector.

Validate all tip lots, seating variation, lighting, payloads and temperatures. Measure error independently at corners and interior wells. For occluded immersion, add a validated depth/force or other local sensor and a stopping-distance budget. No hardware driver, transparent-tip detector or physical accuracy certification is included. See the parts and setup guide.