Hebbian-Robotics / Hebbian-Robotics/hflow
[Bug]: camera_video publishes scratch-bound artifact URIs as durable, causing silent byte-replacement or hard deletion on sync rewrite
- Dominant language
- Python
- Stars
- 269
- Forks
- 150
- Avg merge
- 8h 23m
- Merged PRs (30d)
- 249
Description
### Version or commit
Current main
### Environment
Ubuntu 24.04 (WSL2), Python 3.12, x86_64
### Minimal reproduction
Using a local-root App with `camera_video` registered:
1. Run the full pipeline for generation N. The `video:` artifact URI is published pointing inside `scratch/` (e.g., `.../scratch/wrist_cam_compressed.mp4`).
2. Bump the pipeline version and run a `{SYNC, META}` reprocess (generation N+1). The canonical is rewritten, and the scratch directory is rmtree'd and rebuilt.
3. Query `measurements_latest` for the generation N episode's `video:` artifact URI.
### Expected behavior
The durable catalog URI for a published artifact must point to a stable, content-bound location (like `run_dir/media/` or `run_dir/artifacts/`). If the underlying bytes change generations, the URI must either be invalidated, or the digest must change, refusing to serve unverified bytes.
### Actual behavior
The publish loop (`app.py:2686-2701`) accepts the scratch-cached remux as a durable artifact.
- On a `{SYNC, META}` reprocess, the scratch directory is wiped and rebuilt. The generation N artifact URI now silently resolves to generation N+1's bytes. The server's `is_uri_servable` check still passes (the file "exists"), so a downstream reviewer gets the wrong pixels with zero errors.
- On a forced sync-only reprocess (where the labels/media stages don't run), the generation N artifact URI resolves to a missing file (404), even though the catalog claims the artifact exists.
Executed proof:
[N] episode_id=0c12703341c7e653
[N] artifact URI: .../scratch/wrist_cam_compressed.mp4 (exists: True, sha=6818cf269ee5)
[N+1] (pipeline bump) artifact URI now resolves to: sha=3d86896cddb6 (identical: False)
[N+2] (sync-only rewrite) artifact file exists: False
Note: The built-in contact sheet, stored under `run_dir/media/`, survives the `shutil.rmtree(scratch_dir)` correctly. Only the enrichment publish path fails this boundary.
### Additional context
Root cause: The publish loop binds a durable catalog URI to a path inside the scratch directory's lifecycle, and scratch has no content identity of its own. This violates the contract in `steps.py:405` which explicitly documents `ep.workdir` as scratch. Additionally, `test_camera_video.py:66` has a flawed assertion (`is_relative_to(data_root)`) that allowed this to slip through because `scratch/` is technically inside the data root.
Why it matters: The published `video:` artifact is exactly what a browser scrubber or human reviewer opens to trust a curation cut. After any reprocess, the reviewer's video may be a different generation's pixels (silently accepted) or missing. The evidence chain is broken.
Fix direction: In the publish loop, refuse to record artifact URIs under the scratch subtree. Stage such artifacts into `run_dir/artifacts/` (like `media/`) and publish from there. Additionally, bind every published artifact to its own digest so a replaced file is detectable by a reader.
Definition of done:
- Regression test: after a `{SYNC,META}` reprocess, the generation-N artifact URI hashes to the exact bytes episode N recorded, or is explicitly refused.
- Mutation proof (a): drop the `shutil.rmtree(scratch_dir)` guard → test must fail on the deleted-file branch.
- Mutation proof (b): skip staging into `artifacts/` → test must fail on the silent-replacement branch.
- `test_camera_video.py:66` corrected to assert the URI is *not* under `scratch/`.
Contributor guide
Research direction
Start in app.py:2686-2701 and read steps.py:405 to understand the scratch-workdir contract. Run the camera_video reproduction and inspect test_camera_video.py:66, then add regression coverage for both deleted files and silent byte replacement after reprocessing. Done means published URIs are outside scratch, remain content-stable or are refused, and the corrected assertion passes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100