Hebbian-Robotics / Hebbian-Robotics/hflow
Real-corpus friction (Egocentric-10K/100K): no WebDataset ingest path; the video adapter's JPEG detour costs 2.3x landing size and a double lossy transcode for H.265 sources
- Dominant language
- Python
- Stars
- 269
- Forks
- 150
- Avg merge
- 8h 18m
- Merged PRs (30d)
- 246
Description
## Context
I ran HFlow against the shape of `builddotai/Egocentric-10K` and `builddotai/Egocentric-100K` (the largest real egocentric factory corpora on Hugging Face, and the same source the `examples/egocentric` quickstart shard comes from). Both corpora are WebDataset-style tars of ~180 s H.265/MP4 clips (1080p30 for 10K, 456x256 for 100K) plus per-clip JSON sidecars and per-worker `intrinsics.json`.
The finding: **there is no first-party ingestion path for this corpus shape, and the only available video-to-MCAP adapter detours through JPEG, which costs roughly 2.3x landing size and a double lossy transcode for H.265 sources.**
## What exists today
- `src/hflow/importers/` contains only the LeRobot v3 converter. WebDataset-style tars have no importer; users must write their own converter (the documented escape hatch) or reach for `hflow.testing.write_video_episode`.
- `write_video_episode` is explicitly a demo/test helper, and it decodes the source video and re-encodes **every frame as JPEG** `sensor_msgs/CompressedImage` (`_build_video_camera_messages` in `src/hflow/testing.py`). The canonical transform then re-encodes that JPEG into in-band H.264.
So for an H.265 corpus the media is transcoded **HEVC -> JPEG -> H.264** (two lossy steps, one avoidable) even though H.264 is the canonical target and ffmpeg decodes HEVC directly.
## Measurements
Profile-matched HEVC media (ffmpeg `testsrc2`, same resolution/fps/codec as the corpora; the real footage is gated so I could not download it for this test). Apple Silicon macOS, ffmpeg 8.1.1 from Homebrew, hflow at `78a6b1f`.
30 s @ 1920x1080 @ 30 fps (Egocentric-10K profile):
| Step | Result |
| --- | --- |
| Source HEVC bytes (same span) | ~7 MB |
| `write_video_episode` landing MCAP (JPEG) | **55.3 MB** (~8x source bitrate) |
| Canonical episode H.264 media bytes | 24.5 MB (landing is 2.3x this) |
| Adapter wall time | 1.8 s |
| Cold full pipeline (`app.test`) | **12.0 s (~2.5x slower than realtime)** |
| of which `camera_frame_stats` decode | 8.1 s |
60 s @ 456x256 @ 30 fps (Egocentric-100K profile): cold full pipeline 2.9 s (~48 ms per source-second) -- the small profile is fine; the cost is concentrated at 1080p.
At 2.5x-slower-than-realtime, a single process QC pass over Egocentric-10K's 10,000 hours is on the order of a machine-year; the per-episode unit cost is what matters, and most of it is avoidable decode/transcode overhead plus JPEG round-tripping.
## Why it matters
1. **Disk**: wrapping a real corpus through the JPEG detour multiplies landing storage by ~2.3x versus what the canonical stage needs, before any pipeline run.
2. **Quality**: every real frame passes through two lossy re-encodes; `camera_frame_stats` and any downstream consumer measure artifacts of the JPEG step, not just of the source.
3. **Awkwardness**: the JSON sidecars (factory/worker/duration/fps/codec) map naturally onto `episode/v1` metadata records and `intrinsics.json` onto calibration attachments, but there is no example or helper wiring a WebDataset tar into that, so every team rediscovers the adapter path on their own.
## Suggestions (non-prescriptive)
- A first-party WebDataset-to-canonical importer (or a documented example next to `examples/egocentric`) that transcodes HEVC -> in-band H.264 **directly**, skipping the JPEG landing detour.
- Or, if the JPEG path is intentional for test isolation, a note in the docs saying so and pointing real-corpus users at the converter how-to, so the demo helper does not become the de facto ingestion path.
Happy to split any of this into smaller scoped issues if maintainers prefer.
Contributor guide
Research direction
Start by reading src/hflow/importers/, src/hflow/testing.py, and the _build_video_camera_messages path, then inspect examples/egocentric and the documented converter escape hatch. The issue presents both a first-party WebDataset importer and a documentation-only alternative, so the intended scope needs maintainer agreement; done should be a decided ingestion or documentation path with coverage for the described corpus shape and its media handling.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 42/100