Hebbian-Robotics / Hebbian-Robotics/hflow

[Bug]: transform: declared-but-empty camera channel is dropped, merging distinct sources and erasing provenance

Open
#528 2 comments 0 reactions 0 assignees View on GitHub
bug help wanted
Dominant language
Python
Stars
269
Forks
150
Avg merge
8h 18m
Merged PRs (30d)
246

Description

### Version or commit

Current main

### Environment

Ubuntu 24.04 (WSL2), Python 3.12, x86_64

### Minimal reproduction

Using synthetic data, no real robot recordings:
1. Build a source with two channels: a declared camera (CompressedImage) channel with zero messages, and one state channel with messages.
2. Run the canonical transform on it.
3. Build a second source identical to the first but with the empty camera declaration removed.
4. Observe: both canonicals are byte-identical, so both produce the same episode_id.

### Expected behavior

The declared-but-empty camera channel is either preserved in the canonical (registered with zero messages, matching non-camera channels) or ingest fails loudly with a named refusal. required_topics reports the declared topic as present. Two sources differing only by the empty declaration produce distinct episode_ids.

### Actual behavior

The empty camera channel is silently dropped with only a logger.warning. The canonical has no trace of it. Because the drop happens before content hashing, the two sources collapse to byte-identical canonicals and share one episode_id. Provenance names a topic with no channel.

### Additional context

Root cause: transform treats declared-but-empty as absent for camera channels only. transform.py:673-678 logs a warning and skips the channel; transform.py:815-817 never registers it. Non-camera channels are registered unconditionally when empty (transform.py:830-846). The content hash (catalog.content_episode_id) is computed over the canonical bytes after the drop, so the declaration never enters the episode_id.

Contradiction: checks.py:803-804 states "A declared channel makes its topic present even when it contains no messages," but transform violates this for cameras.

Why it matters: content-addressing collapses, so two distinct recordings merge into one episode_id, and provenance no longer names what was recorded. Both break the trustworthy-evidence promise.

Fix direction: preserve the declaration by registering an empty canonical channel (matching the non-camera path), or refuse loudly via SourceNotConforming naming the topic (matching transform.py:574-578). Never log-only.

Definition of done:
- Empty camera declaration survives in the canonical or ingest fails with a named refusal.
- required_topics absent-vs-empty distinction holds.
- Two sources differing only by the empty declaration produce distinct episode_ids.
- Provenance group/map contains no topic without a corresponding channel.

Contributor guide

Open the contributing guide

Research direction

Start in transform.py:673-678 and 815-817, comparing the camera path with the unconditional empty-channel registration at 830-846. Read checks.py:803-804 and the refusal path at transform.py:574-578, then inspect how catalog.content_episode_id hashes the canonical. Done means the declaration is preserved or rejected by name, required_topics distinguishes absent from empty, episode IDs differ, and provenance has no unmatched topic.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.