ADR-079 P8 follow-up: PCK@20 3.0% → ≥35% requires more paired data + multi-room framing
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 94.4k
- Forks
- 12.5k
- Avg merge
- 21h 27m
- Merged PRs (30d)
- 43
Description
Status (after today's cog v0.0.1 ship — see PRs #642, #643, #644)
ADR-079 P7 (data collection), P8 (alignment + train + eval) and cog packaging end-to-end all ran today. The pipeline is validated and a signed cog-pose-estimation@0.0.1 binary is live at gs://cognitum-apps/cogs/{arm,x86_64}/, installed on cognitum-v0. The remaining work for a useful model is data-bound.
| Lever | Current (v0.0.1) | Target |
|---|---|---|
| Paired sample count | 1,077 | 30,000+ (multiple 30-min sessions × full-body framing) |
| Camera framing | torso-up at desk (avg n_visible 14.3/17) | full-body, varied movements, multiple rooms |
| Avg detection confidence | 0.476 | ≥ 0.7 |
| Training epochs | 400 (Candle CUDA, 2.1 s on RTX 5080) | 1000+ if needed (still seconds on the GPU) |
| PCK@20 | 3.0% | ≥ 35% |
| PCK@50 | 18.5% | ≥ 60% |
| MPJPE (normalized) | 0.093 | < 0.05 |
What the v0.0.1 numbers tell us
Per-joint PCK@50 ranks show the model is learning where the camera lets it:
r_hip 76.9% ← excellent (right side most consistently in frame)
r_knee 35.2%
l_hip 27.3%
l_elbow 26.4%
l_wrist 24.1%
l_knee 20.8%
r_shoulder 19.9%
...
nose 5.1% ← essentially random (face joints at desk-level zoom)
l_ankle 7.9%
r_ankle 9.3%
The asymmetry is a direct reading of the seated-at-desk camera framing — not a model defect. CSI at 56 subcarriers × 20 frames carries enough spatial info for proximal joints with consistent visibility; it doesn't carry enough for fine-grained extremities. More data won't fix that subcarrier-density bottleneck for fingertips / face, but multi-room full-body data will solve it for the 11 joints that today already show some signal.
Suggested data-collection plan
- 3 × 30-min sessions with the camera backed up so head→ankles fits in frame. Different rooms (or different times of day for the same room) to give the model spatial diversity. Vary movements: walk pattern, arm raises, sit/stand transitions, squats, reaches, lying down.
- Re-run
scripts/align-ground-truth.js(now streaming-loader-safe per #641) to produce a multi-session paired set. - Train via the existing Candle pipeline on ruvultra's RTX 5080. Expected wall time: still well under a minute even for 30K samples / 1000 epochs.
- Re-evaluate. PCK@20 should approach the 35% target if the framing + variety land.
Optimizations available within the pipeline (do not require new data)
- LoRA cross-environment fine-tune (per ADR-079 P9). Today's encoder was random-initialized because the HF presence encoder's MLP architecture didn't match; with multi-room data we can train a real shared encoder first and then per-room LoRA adapters.
- Subcarrier attention weighting was already enabled (top-5: [33, 47, 50, 19, 16]).
- Stoer-Wagner min-cut multi-person separation enabled.
Artifacts shipped today (for context)
- Signed v0.0.1 binaries:
gs://cognitum-apps/cogs/arm/cog-pose-estimation-arm+.../x86_64/.... - Trained model:
models/wifi-densepose-pretrained.safetensors→pose_v1.safetensors(507 KB) +pose_v1.onnx(12 KB). - Benchmark log:
docs/benchmarks/pose-estimation-cog.md. - Live install:
/var/lib/cognitum/apps/pose-estimation/on cognitum-v0.
Acceptance criteria for closing this issue
- Multi-room paired dataset ≥ 30K samples at avg conf ≥ 0.7 produced.
- PCK@20 ≥ 35% on a held-out time window from a different session.
- PCK@50 ≥ 60%.
- Per-joint PCK@50 ≥ 30% for at least 13 of 17 joints (face joints can lag).
- Re-release
cog-pose-estimation@0.1.0with the new weights (no code change required — same Candle inference path, just better weights).
🤖 Generated with claude-flow
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with scripts/align-ground-truth.js, the existing Candle training pipeline, and docs/benchmarks/pose-estimation-cog.md. Produce a multi-room paired dataset of at least 30K samples, retrain and evaluate on a held-out session, then verify the listed PCK and per-joint targets before re-releasing cog-pose-estimation@0.1.0 with the new weights.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, machine-learning, rust
- Domain
- data-engineering, machine-learning, release
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100