Hebbian-Robotics / Hebbian-Robotics/hflow

Real-corpus friction (Egocentric-10K/100K): no WebDataset ingest path; the video adapter's JPEG detour costs 2.3x landing size and a double lossy transcode for H.265 sources

Open
#287 9 comments 0 reactions 0 assignees View on GitHub
advanced enhancement
Dominant language
Python
Stars
269
Forks
150
Avg merge
8h 18m
Merged PRs (30d)
246

Description

## Context

I ran HFlow against the shape of `builddotai/Egocentric-10K` and `builddotai/Egocentric-100K` (the largest real egocentric factory corpora on Hugging Face, and the same source the `examples/egocentric` quickstart shard comes from). Both corpora are WebDataset-style tars of ~180 s H.265/MP4 clips (1080p30 for 10K, 456x256 for 100K) plus per-clip JSON sidecars and per-worker `intrinsics.json`.

The finding: **there is no first-party ingestion path for this corpus shape, and the only available video-to-MCAP adapter detours through JPEG, which costs roughly 2.3x landing size and a double lossy transcode for H.265 sources.**

## What exists today

- `src/hflow/importers/` contains only the LeRobot v3 converter. WebDataset-style tars have no importer; users must write their own converter (the documented escape hatch) or reach for `hflow.testing.write_video_episode`.
- `write_video_episode` is explicitly a demo/test helper, and it decodes the source video and re-encodes **every frame as JPEG** `sensor_msgs/CompressedImage` (`_build_video_camera_messages` in `src/hflow/testing.py`). The canonical transform then re-encodes that JPEG into in-band H.264.

So for an H.265 corpus the media is transcoded **HEVC -> JPEG -> H.264** (two lossy steps, one avoidable) even though H.264 is the canonical target and ffmpeg decodes HEVC directly.

## Measurements

Profile-matched HEVC media (ffmpeg `testsrc2`, same resolution/fps/codec as the corpora; the real footage is gated so I could not download it for this test). Apple Silicon macOS, ffmpeg 8.1.1 from Homebrew, hflow at `78a6b1f`.

30 s @ 1920x1080 @ 30 fps (Egocentric-10K profile):

| Step | Result |
| --- | --- |
| Source HEVC bytes (same span) | ~7 MB |
| `write_video_episode` landing MCAP (JPEG) | **55.3 MB** (~8x source bitrate) |
| Canonical episode H.264 media bytes | 24.5 MB (landing is 2.3x this) |
| Adapter wall time | 1.8 s |
| Cold full pipeline (`app.test`) | **12.0 s (~2.5x slower than realtime)** |
| of which `camera_frame_stats` decode | 8.1 s |

60 s @ 456x256 @ 30 fps (Egocentric-100K profile): cold full pipeline 2.9 s (~48 ms per source-second) -- the small profile is fine; the cost is concentrated at 1080p.

At 2.5x-slower-than-realtime, a single process QC pass over Egocentric-10K's 10,000 hours is on the order of a machine-year; the per-episode unit cost is what matters, and most of it is avoidable decode/transcode overhead plus JPEG round-tripping.

## Why it matters

1. **Disk**: wrapping a real corpus through the JPEG detour multiplies landing storage by ~2.3x versus what the canonical stage needs, before any pipeline run.
2. **Quality**: every real frame passes through two lossy re-encodes; `camera_frame_stats` and any downstream consumer measure artifacts of the JPEG step, not just of the source.
3. **Awkwardness**: the JSON sidecars (factory/worker/duration/fps/codec) map naturally onto `episode/v1` metadata records and `intrinsics.json` onto calibration attachments, but there is no example or helper wiring a WebDataset tar into that, so every team rediscovers the adapter path on their own.

## Suggestions (non-prescriptive)

- A first-party WebDataset-to-canonical importer (or a documented example next to `examples/egocentric`) that transcodes HEVC -> in-band H.264 **directly**, skipping the JPEG landing detour.
- Or, if the JPEG path is intentional for test isolation, a note in the docs saying so and pointing real-corpus users at the converter how-to, so the demo helper does not become the de facto ingestion path.

Happy to split any of this into smaller scoped issues if maintainers prefer.

Contributor guide

Open the contributing guide

Research direction

Start by reading src/hflow/importers/, src/hflow/testing.py, and the _build_video_camera_messages path, then inspect examples/egocentric and the documented converter escape hatch. The issue presents both a first-party WebDataset importer and a documentation-only alternative, so the intended scope needs maintainer agreement; done should be a decided ingestion or documentation path with coverage for the described corpus shape and its media handling.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.