Hebbian-Robotics / Hebbian-Robotics/hflow
lerobot export: frame task_index survives task-list collapse, shipping frames that reference a nonexistent task
- Dominant language
- Python
- Stars
- 269
- Forks
- 150
- Avg merge
- 8h 18m
- Merged PRs (30d)
- 246
Description
The exporter's contract is explicit: "feature schema, dtypes, shapes, and frame timing match the source exactly," and "failures fail before any publishable output is written" (examples/lerobot/export.py:10-25). For task labels, neither holds, and the failure is silent.
## What breaks
v3 data frames carry `task_index`, a per-frame pointer into the episode's task list. `_write_v3_repository` copies frame rows with `SELECT *` (export.py:357-381) and rewrites only `episode_index`, `frame_index`, and `index`, leaving `task_index` untouched. But the published per-episode `tasks` column is replaced with the single provenance task, `[sel.task]` (export.py:415). On a multi-task episode, the task table collapses to one entry while every frame pointer survives. Half the exported frames then name a task that does not exist in the dataset the exporter just shipped.
## Why it matters
Task labels are the training signal. An exported dataset whose action frames point at a wrong or nonexistent language label silently corrupts the downstream model. The exporter prints SUCCESS and records sha256 provenance over the contradictory result, so nothing downstream can tell.
## Why it is invisible
`_validate_v3` (export.py:481-548) checks windows, lengths, and file references, but never cross-checks that copied `task_index` values resolve against the task list the export itself rewrote. The float32-only feature filter (export.py:438-442) also drops `task_index` from output info.json, so a schema-aware consumer cannot catch the skew either. No test covers it: the fixtures in tests/test_lerobot_export.py build single-task episodes whose data parquets carry no `task_index` column at all.
## Repro
Synthetic v3 source, episode `tasks = ["pick cup", "place cup"]`, frame `task_index = 0,0,1,1`, run through the real `export.export()`:
```
export: SUCCESS (no refusal raised)
published episode tasks: [(0, ['pick cup'])]
published frame rows (frame_index, task_index): [(0, 0), (1, 0), (2, 1), (3, 1)]
task_index in features: False
```
The published task table holds one entry; frames reference index 1.
## On scope
Even if current corpora are single-task, the v3 format supports per-frame subtask labeling and the exporter claims v3 fidelity, so this is a spec-robustness gap. Context: the loss has an upstream half, the importer collapses `task = tasks[0]` when stamping provenance (src/hflow/importers/lerobot.py:815). The export is where the contradiction ships, because it is the only side that both copies the pointers and rewrites the table.
## Fix direction
Publish the source episode's full task list (the materialized archive already carries it, export.py:207-211) and rewrite per-frame `task_index` against the published table; or refuse loudly when a copied frame references a task the output does not publish, naming episode, frame, and index, in the style of the existing "expected N data rows, found M" refusal (export.py:368-372).
## Definition of done
- A multi-task synthetic source exports with every frame's `task_index` resolving to the same task string it resolved to at the source.
- An export whose frames reference an unpublished task raises before any destination is written, naming the episode and index.
- Deleting the new consistency check flips a test red (mutation proof).
- The existing export suite stays green.
Contributor guide
Research direction
Start with _write_v3_repository in examples/lerobot/export.py:357-381 and the task-list handling at 415, then read _validate_v3 at 481-548 and the fixtures in tests/test_lerobot_export.py. Add multi-task and unpublished-task cases, run the existing export suite, and verify that published frame task_index values resolve to the source task strings or fail before any destination is written.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 74/100