Introducing EmbodiedTrace: an inspectable path from observation to action

A robot clip can be fascinating to watch. For a team developing an embodied agent, the next questions are practical: what did the system observe, which action should be predicted, and how can the example be inspected before it enters a training pipeline?

Datumwell EmbodiedTrace is a concrete starting point for that conversation. The public preview connects visual observations, generated task-oriented annotations, and structured robot actions across 100 episodes. You can explore the representation in a browser, inspect one decision in Python, and assess whether a tailored pilot would fit your requirements.

Make the decision inspectable

Try the representation / Episode 001

Put the marker in the pot

Four selected checkpoints from one 20-decision episode. Choose a decision and a camera; open the studio for the full sequence.

01 / ApproachRecorded frame 002 / CloseRecorded frame 5003 / MoveRecorded frame 8604 / OpenRecorded frame 125

CameraExterior camera 1Exterior camera 2Wrist camera

GENERATED / WEAK LABELS

Approach / align with marker

Concise rationale · generated

Recorded gripper closure is 0.00 (0 open, 1 closed). Align the end effector for a candidate interaction with marker. Maintain closure toward 0.00 while requesting the labeled pose. Object contact and task completion require observation.

Action description · generated

Request 9.19 cm of EEF translation (+X 4.90 cm, +Y 3.57 cm, −Z 6.90 cm), 7.1° orientation change, and gripper closure 0.00.

Recorded-command EEF target

Metres · robot-base frame
Quaternion XYZW · 0 open, 1 closed
Rounded display; download retains precision.
Derived target not separately executed.

move_eef(position_m=[0.43237, 0.10918, 0.48090], quaternion_xyzw=[-0.98832, -0.03401, -0.11712, 0.09141], gripper_closure=0.000)

The unit of inspection is a decision inside an episode. A task instruction provides context when available. The observation includes the current robot pose, gripper state, and three camera references. A generated subgoal and concise rationale offer annotations to examine, followed by a structured end-effector action label.

Later recorded feedback stays separate. That matters because an observed next frame is different from information available when selecting an action. Keeping those fields distinct gives developers a clearer starting point for constructing their own training inputs and targets.

An end-effector pose action describes a target position and orientation for the robot's working end, alongside a gripper command. The package documents metres, robot-base coordinates, quaternion XYZW order, and gripper semantics. Full decision records preserve numeric precision; the interactive example is rounded for readability.

Three representation choices

Keep the action structured. An action description helps a person inspect an example, while a machine-readable pose target provides an explicit output interface. Both are available together, so evaluators can distinguish the textual annotation from the numeric command label.

Keep the sequence visible. Decisions are linked to episode frames and past context. The studio lets you move between decisions, switch cameras, and inspect recorded feedback. Its downloadable episode JSON is an inspection export; the Hugging Face split JSONL files and optional original JPEG archives form the data package.

Make annotation status explicit. Subgoals, rationales, and action descriptions are generated labels. They are suggestions for inspection, not recovered human thoughts or proof of object contact, grasp retention, or task completion. The label provenance and limitations remain accessible in the dataset documentation.

What the sample includes

The release contains 100 episodes, 4,081 structured decisions, and 1,288 generated subgoal segments, with three synchronized camera views. Tasks include transfers and a smaller selection of presses, wipes, articulated-object interactions, and multi-step instructions.

Instructions are available for 77 episodes; 23 lack task instructions. Recorded episode metadata includes 81 successes and 19 failures. Those source outcomes do not establish that every generated subgoal succeeded. Missing instructions remain explicit rather than being filled with invented task descriptions.

The sample is designed for inspecting a representation. It does not establish production task coverage or generalization performance. Its train/validation/test split holds out episode IDs, rather than guaranteeing unseen scenes, operators, or tasks.

Start with the question your team needs to answer

For an AI lead, a useful first check is whether the structured action format fits the intended model and embodiment. For a data lead, the questions may concern task distribution, observation coverage, required annotations, and review criteria. For procurement, the next step is understanding what a scoped evaluation would contain.

A practical first pass can be short:

1. Open an episode relevant to your work and inspect the footage alongside its generated annotations.
2. Load one decision and compare the action interface with your own system.
3. List the tasks, inputs, annotations, and acceptance criteria that a tailored pilot would need.

No model-performance improvement is claimed by this demo. Derived sparse action targets have not been separately executed on a robot. Embodiment compatibility and an appropriate action adapter require evaluation. Generated semantic labels should be reviewed before using them for reasoning supervision.

Questions for the next pilot

Which task phases are meaningful for your system? What evidence should support an annotation? Which action representation can your model predict and your adapter consume? How should reviewers handle uncertainty and missing context?

These questions define the work more usefully than a larger episode count alone. A pilot discussion can establish the desired sample format, deliverables, and evaluation criteria before an engagement. The public release lets you bring a concrete example to that conversation.

For loading examples, schema, source attribution, licensing, and detailed limitations, read the EmbodiedTrace dataset card.