test(models): validate Inkling omni on real checkpoints
- Dominant language
- Rust
- Stars
- 467
- Forks
- 54
- Avg merge
- 4h 25m
- Merged PRs (30d)
- 310
Description
## Summary
Validate the merged Inkling omni implementation from epic #1313 on the published real checkpoints and record parity, quality, and throughput evidence that could not be produced on the Linux validation host.
## Context
The implementation is merged through PRs #1532, #1535, #1540, #1546, and #1548. Their deterministic CPU/reference tests, synthetic MLX tests, CI, lint, and contract checks pass. The available host did not have the approximately 153.5 GB affine Inkling-Small checkpoint, the approximately 170.7 GB native NVFP4 checkpoint, the 0.6B community checkpoint, or a compatible Apple GPU with sufficient unified memory.
## Validation matrix
- [ ] On Apple Silicon with at least 192 GB unified memory, load `mlx-community/Inkling-Small-mlx-4bit` and produce 64 greedy text tokens through the CLI and `/v1/chat/completions` using the checkpoint chat template.
- [ ] Compare the 0.6B checkpoint against the upstream reference for a RAW prompt, greedy decoding, 64 tokens, and `MLXCEL_FUSED_MOE=0`; require token-exact text parity.
- [ ] Run a grounded still-image request through both the CLI and server.
- [ ] Transcribe a clean five-second 16 kHz WAV through both the CLI and server; require placeholder count `ceil(samples / 800)` and word error rate below 20 percent.
- [ ] Describe motion in a synthetic left-to-right video through the adjacent-frame path; record sampled anchors, timestamps, and token count.
- [ ] Load the native MTP shard, verify 128-token greedy equality with plain decoding, record mean accepted length, and require a positive decode-throughput improvement.
- [ ] Validate the native NVFP4 checkpoint, including expert sidecar scales, on a reachable compatible host.
## Evidence to attach
- Exact model revisions and file hashes
- Hardware and OS details
- Commands and complete outputs
- Token sequences or parity diffs
- Audio transcript and word error rate
- Video sampling metadata and answer
- Plain versus MTP throughput and acceptance statistics
## Related
- Epic #1313
- Text #1318 / PR #1532
- Vision #1327 / PR #1535
- MTP #1315 / PR #1540
- Video #1323 / PR #1546
- Audio #1311 / PR #1548
Contributor guide
Research direction
Start by reading the merged validation work in PRs #1532, #1535, #1540, #1546, and #1548, then exercise the CLI and /v1/chat/completions entry points on the specified Apple Silicon host. Done means completing the validation matrix with exact model revisions, hashes, hardware details, full outputs, parity data, quality metrics, video metadata, and plain-versus-MTP throughput evidence.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- api, cli, machine-learning, testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 25/100