From a pen stroke to a robot action: introducing Datumwell Handwriting

A short writing task asks a robot to connect several things at once: a phrase, the strokes that form it, the pen's motion, and what the camera can actually see. A finished image only shows the result. A useful training-data representation also needs to show the observation and command at each step.

Datumwell Handwriting is a research release of 200 synthetic robot-writing episodes, with three synchronized camera views and planned joint targets in LeRobot v3. It is inspired by the marker-on-paper “Write hello world” task in Artificial Analysis AA-Robotics v0.1. The release includes other short phrases, so “hello world” is an inspectable example rather than the entire collection.

Start with ordered handwriting

The handwriting begins with UJI Pen Characters, Version 2: recorded isolated characters with ordered x/y samples and pen strokes. Those ordered samples provide a path to follow, rather than requiring a model to infer stroke order from a finished font image.

We normalize coordinate scales, clean and resample strokes, compose characters into short phrases, and lay them out inside the paper's black border. Source stroke order is retained. Letter spacing, scale, placement and timing can vary deterministically. The resulting phrases are synthetic compositions of recorded characters; they are not human recordings of natural connected phrase writing.

The collection covers 30 phrases, 84 base layouts and 51 anonymous source writers. Variants remain with their source split, and test phrases are held out. There are 107 training, 21 validation and 72 test episodes. Two hundred episodes should not be read as two hundred independent human handwriting samples.

Turn the path into a simulated execution

Pen-down strokes become paper-plane tip paths, with synthetic timing and lifted transitions between strokes. The pen is already held rigidly. There is no grasping phase, slipping pen, or measured human pressure signal.

Inverse kinematics and joint-space planning produce five absolute joint-position targets. A torque-driven MuJoCo controller then follows those references, while the simulation records actual joint state, pen-tip position and contact diagnostics. In the export, action is the planned joint reference in radians; observation.state is the measured simulated joint position. They are different quantities.

The published appearance version replays the 200 saved executions and renders their original camera timestamps. Cream tabletop, brown tray, taped white paper, black arm and a paneled room follow the visual reference. A second oblique camera joins the first oblique and overhead views. Numeric trajectories remain byte-identical to the previous simulation export; this rendering pass is not 200 additional physics executions.

Keep the camera and command on one clock

The LeRobot v3 package contains 1,232,056 numeric rows at 50 Hz and 600 H.264 videos at 640 × 480. Native images were captured at approximately 2.994 Hz. The 50 fps videos repeat the most recent native image causally; they do not provide 50 new visual observations per second.

Each image uses the saved robot state and contact ink available at that moment. Future strokes are not painted into earlier observations. Image-source timestamps, source indices, image age and synthetic camera calibration are retained for inspection. In the browser example, changing the camera preserves the playback position while joint state and target stay tied to the same row.

For a policy input, start with the instruction, current or past camera frames, and measured state. Keep action labels, future frames, final ink diagnostics and planned-target diagnostics out of the observation. The feature policy documents which retained fields are suitable inputs and which are audit information.

Inspect quality before choosing a use

Release checks cover all 200 numeric files, all 600 video formats and durations, causal frame mappings, unchanged numeric hashes, split isolation and sampled image/video correspondence. The official JavaScript LeRobot reader was used to read all 200 episodes. A native Python LeRobot/Torch read-through was not executed.

Auxiliary OCR on crops of the final overhead-camera observations matched 170 of 200 target phrases after case and whitespace normalization. Thirty episodes remain flagged. OCR is not a human legibility certificate, and flagged samples should be reviewed or filtered for the intended experiment.

There are also material physical limits. The paper is a miniature 136.5 × 96.2 mm plane; the tray, tape and room are visual geometry, not a reconstructed collision environment. AA dimensions, camera calibration and contact parameters were not measured. Planned motion bounds pass their checks, but actual simulated joint acceleration exceeds the configured 1.5 rad/s² threshold. These are not hardware-ready or acceleration-compliant demonstrations.

Useful starting points include checking an observation/action loader, studying causal visual sampling, comparing handwriting layouts, and developing small synthetic VLM/VLA data experiments. Any claim about learned performance, transfer to another robot, or real-world writing would require separate evaluation.

Download, preview, and trace the source

The approximately 1.86 GB Hugging Face release includes Parquet data, three-view MP4 observations, metadata, a lightweight loader, a decision example, quality reports and file checksums. The project page offers a synchronized example before downloading. The data card records the immutable v0.3.0-aa-style-rerender release tag and loading instructions.

UJI handwriting retains its CC BY 4.0 attribution to Prat, Castro, Llorens, Marzal and Vilar; the UCI source record documents the original data. The example loader is MIT licensed. Robot appearance derives from Hiwonder/NexArm; an applicable rendered-output distribution/reuse license has not been established, and no manufacturer permission is asserted. Public availability is not a blanket license grant for the composite data.

Raw robot meshes, URDF/Xacro/model XML files and AA reference footage are excluded from the package. Our annotations, packaging and new scene geometry are distinguished from third-party appearance. Read RIGHTS.md, NOTICE and the quality limitations together.

Also from Datumwell: Introducing EmbodiedTrace describes the separate recorded-observation dataset and its generated weak labels.