Hebbian-Robotics / Hebbian-Robotics/hflow

[Bug]: camera_video publishes scratch-bound artifact URIs as durable, causing silent byte-replacement or hard deletion on sync rewrite

Open
#545 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
269
Forks
150
Avg merge
8h 23m
Merged PRs (30d)
249

Description

### Version or commit

Current main

### Environment

Ubuntu 24.04 (WSL2), Python 3.12, x86_64

### Minimal reproduction

Using a local-root App with `camera_video` registered:
1. Run the full pipeline for generation N. The `video:` artifact URI is published pointing inside `scratch/` (e.g., `.../scratch/wrist_cam_compressed.mp4`).
2. Bump the pipeline version and run a `{SYNC, META}` reprocess (generation N+1). The canonical is rewritten, and the scratch directory is rmtree'd and rebuilt.
3. Query `measurements_latest` for the generation N episode's `video:` artifact URI.

### Expected behavior

The durable catalog URI for a published artifact must point to a stable, content-bound location (like `run_dir/media/` or `run_dir/artifacts/`). If the underlying bytes change generations, the URI must either be invalidated, or the digest must change, refusing to serve unverified bytes.

### Actual behavior

The publish loop (`app.py:2686-2701`) accepts the scratch-cached remux as a durable artifact.
- On a `{SYNC, META}` reprocess, the scratch directory is wiped and rebuilt. The generation N artifact URI now silently resolves to generation N+1's bytes. The server's `is_uri_servable` check still passes (the file "exists"), so a downstream reviewer gets the wrong pixels with zero errors.
- On a forced sync-only reprocess (where the labels/media stages don't run), the generation N artifact URI resolves to a missing file (404), even though the catalog claims the artifact exists.

Executed proof:
[N] episode_id=0c12703341c7e653
[N] artifact URI: .../scratch/wrist_cam_compressed.mp4 (exists: True, sha=6818cf269ee5)
[N+1] (pipeline bump) artifact URI now resolves to: sha=3d86896cddb6 (identical: False)
[N+2] (sync-only rewrite) artifact file exists: False

Note: The built-in contact sheet, stored under `run_dir/media/`, survives the `shutil.rmtree(scratch_dir)` correctly. Only the enrichment publish path fails this boundary.

### Additional context

Root cause: The publish loop binds a durable catalog URI to a path inside the scratch directory's lifecycle, and scratch has no content identity of its own. This violates the contract in `steps.py:405` which explicitly documents `ep.workdir` as scratch. Additionally, `test_camera_video.py:66` has a flawed assertion (`is_relative_to(data_root)`) that allowed this to slip through because `scratch/` is technically inside the data root.

Why it matters: The published `video:` artifact is exactly what a browser scrubber or human reviewer opens to trust a curation cut. After any reprocess, the reviewer's video may be a different generation's pixels (silently accepted) or missing. The evidence chain is broken.

Fix direction: In the publish loop, refuse to record artifact URIs under the scratch subtree. Stage such artifacts into `run_dir/artifacts/` (like `media/`) and publish from there. Additionally, bind every published artifact to its own digest so a replaced file is detectable by a reader.

Definition of done:
- Regression test: after a `{SYNC,META}` reprocess, the generation-N artifact URI hashes to the exact bytes episode N recorded, or is explicitly refused.
- Mutation proof (a): drop the `shutil.rmtree(scratch_dir)` guard → test must fail on the deleted-file branch.
- Mutation proof (b): skip staging into `artifacts/` → test must fail on the silent-replacement branch.
- `test_camera_video.py:66` corrected to assert the URI is *not* under `scratch/`.

Contributor guide

Open the contributing guide

Research direction

Start in app.py:2686-2701 and read steps.py:405 to understand the scratch-workdir contract. Run the camera_video reproduction and inspect test_camera_video.py:66, then add regression coverage for both deleted files and silent byte replacement after reprocessing. Done means published URIs are outside scratch, remain content-stable or are refused, and the corrected assertion passes.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.