Hebbian-Robotics / Hebbian-Robotics/hflow
[Bug]: transform: declared-but-empty camera channel is dropped, merging distinct sources and erasing provenance
- Dominant language
- Python
- Stars
- 269
- Forks
- 150
- Avg merge
- 8h 18m
- Merged PRs (30d)
- 246
Description
### Version or commit
Current main
### Environment
Ubuntu 24.04 (WSL2), Python 3.12, x86_64
### Minimal reproduction
Using synthetic data, no real robot recordings:
1. Build a source with two channels: a declared camera (CompressedImage) channel with zero messages, and one state channel with messages.
2. Run the canonical transform on it.
3. Build a second source identical to the first but with the empty camera declaration removed.
4. Observe: both canonicals are byte-identical, so both produce the same episode_id.
### Expected behavior
The declared-but-empty camera channel is either preserved in the canonical (registered with zero messages, matching non-camera channels) or ingest fails loudly with a named refusal. required_topics reports the declared topic as present. Two sources differing only by the empty declaration produce distinct episode_ids.
### Actual behavior
The empty camera channel is silently dropped with only a logger.warning. The canonical has no trace of it. Because the drop happens before content hashing, the two sources collapse to byte-identical canonicals and share one episode_id. Provenance names a topic with no channel.
### Additional context
Root cause: transform treats declared-but-empty as absent for camera channels only. transform.py:673-678 logs a warning and skips the channel; transform.py:815-817 never registers it. Non-camera channels are registered unconditionally when empty (transform.py:830-846). The content hash (catalog.content_episode_id) is computed over the canonical bytes after the drop, so the declaration never enters the episode_id.
Contradiction: checks.py:803-804 states "A declared channel makes its topic present even when it contains no messages," but transform violates this for cameras.
Why it matters: content-addressing collapses, so two distinct recordings merge into one episode_id, and provenance no longer names what was recorded. Both break the trustworthy-evidence promise.
Fix direction: preserve the declaration by registering an empty canonical channel (matching the non-camera path), or refuse loudly via SourceNotConforming naming the topic (matching transform.py:574-578). Never log-only.
Definition of done:
- Empty camera declaration survives in the canonical or ingest fails with a named refusal.
- required_topics absent-vs-empty distinction holds.
- Two sources differing only by the empty declaration produce distinct episode_ids.
- Provenance group/map contains no topic without a corresponding channel.
Contributor guide
Research direction
Start in transform.py:673-678 and 815-817, comparing the camera path with the unconditional empty-channel registration at 830-846. Read checks.py:803-804 and the refusal path at transform.py:574-578, then inspect how catalog.content_episode_id hashes the canonical. Done means the declaration is preserved or rejected by name, required_topics distinguishes absent from empty, episode IDs differ, and provenance has no unmatched topic.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 76/100