atomantic / atomantic/PortOS

Watch for an Apple Silicon path to FastVideo's FastH3 8-Step V2 checkpoint

Open
#7,418 0 comments 0 reactions 0 assignees View on GitHub
area:local-models area:media blocked effort:medium model:medium plan planner:opus-5 reference-watch
Dominant language
JavaScript
Stars
38
Forks
31
Avg merge
22m
Merged PRs (30d)
988

Description

## Watch item

Upstream shipped **FastH3 8-Step V2** on 2026-09-15 (`FastVideo/FastVideo-FastH3-8-Step-V2` on Hugging Face, plus hao-ai-lab/FastVideo PR #1852). It is **CUDA-only**. There is nothing for PortOS to adopt yet — file this so we notice when an Apple Silicon path appears.

## Evidence that V2 has no Apple Silicon path

- **PR #1852 touched zero MLX/Metal files.** Changed: `fastvideo/configs/pipelines/minimax_h3.py`, `fastvideo/pipelines/basic/minimax_h3/minimax_h3_pipeline.py`, `.../stages/minimax_h3_denoising.py`, docs, `examples/inference/basic/basic_fasth3_8step.py`, tests. No `mlx/` or `metal/` path, no `mlx` in any filename.
- **The HF repo ships no MLX weights.** ~147.8 GB of plain diffusers safetensors (transformer ~66.2 GB, text encoder ~68.6 GB, VAE ~10.4 GB, audio VAE ~605 MB). No `mlx_dit.safetensors` / `mlx_*.json`, unlike the `-MLX-INT4/INT6/INT8` siblings we already ship.
- **The model card names a CUDA-only dependency:** "requires FastVideo's VSA-H3 attention backend" — sparse attention at 0.8 sparsity, tile 64, distributed as CUDA 13 kernel wheels, validated on 4x B200. Our MLX path is dense; running a checkpoint distilled *for* 80% sparsity through a dense attention kernel is not a drop-in.
- **No framework "v2" exists.** Latest PyPI `fastvideo` is 0.2.1 (2026-08-28); newest git tag is `v0.2.0`. The only V2 artifact is the checkpoint above.

## PortOS is already current on Apple Silicon

Nothing is behind. Everything upstream has published for Metal is shipped:

- FastMetal 1.3B / 5B / 14B QAD (upstream 2026-08-19, download sizes corrected in #5871, migration `336-fastmetal-download-size-names.js`)
- FastH3 Preview v1 Dense Data-Free MLX INT8 / INT6 / INT4 (upstream 2026-09-01, migrations `333-fasth3-dense-datafree-mlx.js` / `334-fasth3-upstream-source-mlx.js`)
- `data.reference/media-models.json` carries `disclosure.reviewedAt: "2026-09-02"` on every `runtime: 'fastvideo'` row — one day after the last Apple Silicon drop.
- `scripts/setup-image-video.sh` installs FastVideo from git `main` (`FASTVIDEO_PIN="${FASTVIDEO_PIN:-main}"`, `uv pip install -e '.[mlx]'`), so runner-side MLX fixes arrive without a PortOS change.

## Trigger to act

Reopen work when **either** lands upstream:

1. A `FastVideo/FastVideo-FastH3-8-Step-V2-MLX-INT*` repo appears under the FastVideo HF org, **or**
2. `scripts/checkpoint_conversion/convert_minimax_h3_mlx.py` upstream learns to emit the V2 sidecar and a Metal VSA (or dense-fallback) attention path lands.

## Work to do when it unblocks

1. **Plumb a checkpoint-defined DMD schedule.** V2 reads a `fastvideo_inference.json` sidecar: explicit rung ladder `[999, 874, 749, 624, 500, 375, 250, 125]` and **separate video and audio shifts** (upstream removed the unified `flow_shift` for H3). `scripts/generate_fastvideo.py` today passes a flat `--steps` (default 3) and has no shift concept at all — grep it for `shift`, there are no hits. Getting the ladder or the video/audio shift split wrong degrades output silently rather than erroring, so this is the part that needs care.
2. **Step count.** All four shipped `fasth3_*` rows in `data.reference/media-models.json` hardcode `steps: 4`. V2 is 8 forward passes.
3. **Catalog rows + migration.** Add the V2 row(s) following the `333`/`334` pattern, with `disclosure` (model card URL, MiniMax-H3 Community license, `estimatedDownloadGb`, `reviewedAt`) and a realistic `memoryGb` — the existing INT8 row already asks 48+ GB RAM against a ~144 GB download, so V2 at ~147.8 GB likely lands at or above that.
4. **Conversion cache.** `scripts/generate_fastvideo.py` already shells out to upstream's `scripts/checkpoint_conversion/convert_minimax_h3_mlx.py` and keys the cache on the snapshot commit sha (`~/.portos/fastvideo/mlx-checkpoints/-`), so the DiT conversion itself should need no PortOS change — but confirm the converter carries the sidecar into the converted checkpoint, otherwise the schedule is lost at conversion time.

## Labels

`model:medium` + `effort:medium` — medium capability because this is routine multi-file catalog + helper + migration work with a direct precedent in migrations 333/334/336, not novel design; medium effort because the DMD rung ladder and the video/audio shift split have real correctness stakes (a wrong shift produces plausible-looking but degraded video, with no error to catch it).

## Prior art

- #5351 — Track Apple Silicon support for FastH3 VSA preview (closed once the 2026-09-01 MLX weights landed)
- #5870 — Add the official FastVideo FastH3 MLX INT6/INT8 rows once upstream publishes their weights

Contributor guide

Open the contributing guide

Research direction

This is blocked until an Apple Silicon V2 path appears upstream. When unblocked, read scripts/generate_fastvideo.py, data.reference/media-models.json, and migrations 333/334/336; compare the converter and catalog patterns, verify the checkpoint sidecar and schedule handling, and finish with V2 rows and migration coverage matching the stated trigger and metadata requirements.

Written by the indexing model from the issue text.

Assessment

Tech stack
huggingface, javascript
Domain
ai, machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.