tracking: non-generative vision models present in the VLM reference but out of mlxcel's LLM/VLM-generation scope

## Summary

A model-coverage audit surfaced the model families below as not supported by mlxcel. They are task-specific vision models (object detection, instance segmentation, 3D body estimation, grounding, diffusion image generation) rather than the chat/completions LLMs and VLMs that mlxcel's text-generation runtime targets. Loading any of them today fails in `src/models/detection.rs::get_model_type`: a `config.json` whose `model_type` has no match arm errors with `Unsupported model type: `, and checkpoints without a top-level `model_type` (the diffusion image models) fail earlier with `model_type not found`.

This issue records the scope decision for each family so that coverage questions have a recorded answer. It is a tracking document, not a work item: closing it requires no code. One family on the original list, RT-DETRv2, has since been implemented and is noted below as resolved.

mlxcel is not strictly text-in/text-out today. Three non-chat paths already exist and serve as precedent for any future re-scoping:

- `mlxcel detect` (`src/commands/detect.rs`, `src/vision/detection/`): object detection, bounding-box output.
- Whisper (`src/models/whisper/`): encoder-decoder speech-to-text via `/v1/audio/transcriptions`.
- Kokoro (`src/models/kokoro/`): text-to-speech, audio output via `/v1/audio/speech`.

## Families

### `rt_detr_v2` (RT-DETRv2): resolved, already supported

Real-time detection transformer: ResNet-vd CNN backbone, hybrid encoder (AIFI plus FPN/PAN), and a 6-layer deformable-attention decoder with iterative box refinement. Outputs class logits and boxes for a fixed label set resolved through `id2label`. Task: object detection, including document layout detection (Docling layout checkpoints).

Status: supported. Implemented in `src/vision/detection/rt_detr_v2/` and exposed through the `mlxcel detect` subcommand (`src/commands/detect.rs`), which prints `l, t, r, b, label, confidence` rows or JSON. It sits outside the generate loop by design; detection models produce boxes, not a token stream.

### `sam3` (SAM 3)

Open-vocabulary detection, instance segmentation, and video tracking (~860M parameters). A ViT-L backbone with windowed plus global attention and 2D RoPE feeds an FPN neck, a DETR encoder/decoder stack (200 queries, box refinement), and a mask decoder; a CLIP text encoder embeds the text prompt for open-vocabulary scoring. A separate memory-based tracker propagates masklets across video frames. Checkpoints ship a hierarchy of component types under a top-level `model_type: "sam3_video"` with a `"sam3"` detector sub-config.

Task: detection plus segmentation plus tracking. Output: boxes, per-instance masks, confidence scores. It generates no text; the text encoder only embeds prompts. Supporting it would require a segmentation-mask output path (mask tensors or encoded mask images), a DETR/mask-decoder model family, and for video, streaming frame input with tracker memory state, none of which the generation runtime or the `detect` box path provides.

### `sam3_1` (SAM 3.1)

Extension of SAM 3 (~873M parameters) adding multi-object "Object Multiplex" tracking: a tri-branch FPN neck (detection, interactive, propagation), a multiplex mask decoder handling up to 16 objects per forward pass, and decoupled memory attention. Detection stack and outputs match SAM 3. Ships as `model_type: "sam3.1_video"` with a `"sam3.1"` detector sub-config.

Task: detection plus segmentation plus multi-object video tracking. Output: boxes, masks, tracked masklets. Same gap as SAM 3: mlxcel has no mask output path and no video/tracker-state loop; the two families would be one port effort since they share most components.

### `sam3d_body` (SAM 3D Body)

Single-image 3D human body mesh estimation (~720M parameters). A DINOv3-H+ ViT backbone encodes a cropped person image; a prompt encoder (keypoint and hand-box embeddings) plus a 6-layer cross-attention decoder produce pose, camera, and keypoint tokens; regression heads drive a parametric body model (forward kinematics, blend shapes, linear blend skinning). Ships as `model_type: "sam3d_body"` with a `"dinov3"` vision sub-config and no text component at all.

Task: 3D human pose/mesh regression. Output: mesh vertices, joint transforms, 3D keypoints, camera parameters. There is no text anywhere in the model, and the outputs are geometry buffers; supporting it would mean a new mesh/keypoint output path and a parametric body-model runtime with no overlap with mlxcel's existing code.

### `rfdetr` (RF-DETR)

Real-time detection transformer built on a DINOv2 ViT backbone with windowed attention, a multi-scale projector, two-stage query selection, and a deformable-attention decoder (300 queries, group-DETR), with an optional instance-segmentation head. Fixed COCO-style label set, no text components. Ships as `model_type: "rf-detr"`.

Task: object detection, optional instance segmentation. Output: boxes, scores, labels, optional masks. Its box path could in principle ride the existing `mlxcel detect` subcommand next to RT-DETRv2 (`src/vision/detection/`), so it is the cheapest family here to re-scope; it still needs a DINOv2 backbone, the deformable decoder, and (for masks) a mask output path.

### `locateanything` (LocateAnything)

A 3B generative grounding VLM: a MoonViT vision tower with an MLP connector into a Qwen2.5-style decoder-only text backbone. Its standard decode path is ordinary autoregressive text generation whose output interleaves coordinate control tokens (box/coord/ref markers) encoding located regions; it also ships a specialized parallel-box-decoding path with multi-token-prediction heads (`n_future_tokens: 6`) for fast box emission. Ships as `model_type: "locateanything"` with a `"qwen2"` text sub-config and `"moonvit"` vision sub-config.

Task: visual grounding, referring-expression localization, open-vocabulary localization. Output: text containing box coordinates. Architecturally it is close to mlxcel's VLM runtime (MoonViT already exists for Kimi-VL in `src/vision/encoders/kimi_vl.rs`, Qwen2 in `src/models/qwen2.rs`), which makes it the strongest VLM-shaped re-scoping candidate. It stays here because its value is the grounding contract: coordinate-token post-processing into boxes and the parallel-box-decoding path, neither of which the chat-completions API represents. A minimal port (plain autoregressive decode, raw coordinate tokens in the text) would work but deliver little without that contract.

### `florence2` (Florence-2)

A unified multi-task vision seq2seq model. A DaViT vision backbone produces image tokens that are concatenated with the text prompt and fed to a BART-style encoder-decoder transformer (6 encoder plus 6 decoder layers, learned absolute positions). One text-generation interface covers captioning, OCR, detection, dense region captioning, grounding, and segmentation: for spatial tasks, the decoder emits location tokens that post-process into boxes or regions. Ships as `model_type: "florence2"` with a `"davit"` vision sub-config.

Task: prompt-keyed multi-task vision (caption/OCR/detect/ground). Output: text tokens, some of which encode locations. It does generate text autoregressively, but as an encoder-decoder: mlxcel's generation engine is decoder-only, and the only encoder-decoder precedent is Whisper's dedicated ASR path. Supporting it would require either a seq2seq generation loop in the engine or a Whisper-style dedicated pipeline, plus task-prompt handling and location-token post-processing.

### `flux2` (FLUX.2 Klein)

Latent flow-matching text-to-image generation and image editing (4B and 9B variants). A dual-stream plus single-stream MMDiT transformer denoises latents over an iterative scheduler, conditioned on prompt embeddings from a bundled LLM-style text encoder, with a VAE for latent encode/decode. Checkpoints are multi-component directories (`transformer/`, `vae/`, text encoder) with no single top-level `model_type`; the family identifier is `flux2`. In mlxcel such a directory fails at load with `model_type not found`.

Task: image generation and editing. Output: images. Supporting it would require an entirely new inference stack: a diffusion sampling loop, a VAE, multi-component checkpoint loading, and an image output path (an images-generations style endpoint), none of which exist in mlxcel.

### `ideogram4` (Ideogram 4)

Diffusion text-to-image generation. A DiT transformer (34 layers, 4608 hidden, multimodal RoPE) denoises VAE latents, conditioned on features from a large external LLM text encoder; sampler presets trade steps for quality. Checkpoints are identified by a `model_index.json` pipeline class rather than a `config.json` `model_type`; the family identifier is `ideogram4`. The public checkpoint is FP8 and license-gated.

Task: image generation. Output: images. Same gap as FLUX.2: diffusion loop, VAE, multi-component loading, image output path. If mlxcel ever grows an image-generation stack, FLUX.2 and Ideogram 4 would share most of that infrastructure.

### `nemotron_labs_diffusion` (Nemotron Labs Diffusion 8B)

Despite the name, this is a text-only diffusion language model, not an image model: an 8B dense decoder-only transformer (GQA, SwiGLU, YaRN-scaled RoPE, 262k context) with an untied diffusion output head and a mask token. One checkpoint supports three decode modes: standard autoregressive, masked block-diffusion denoising (block size 32), and linear self-speculative decoding (diffusion draft, autoregressive verify, bundled LoRA adapter). Ships as `model_type: "nemotron_labs_diffusion"`.

Task: text generation. Output: text. This is the clearest re-scoping candidate in this issue: mlxcel already runs a block-diffusion text model (DiffusionGemma, `src/models/diffusion_gemma/`, accepted as `diffusion_gemma` in `src/models/detection.rs`), so the runtime shape exists. It is recorded here only because it arrived in the same audit batch; it should graduate to a dedicated port issue when prioritized rather than be treated as out of scope.

## Re-scoping criteria

A family moves from this issue to a dedicated port issue when all of the following hold:

1. A concrete user or product need names the family (a checkpoint someone wants to serve).
2. The output path exists or is being built: box outputs can ride `mlxcel detect` today; masks, meshes, and images need new output paths that should be scoped as their own infrastructure work first.
3. A specific public checkpoint is named for validation, with its `config.json` (or component layout) confirmed.
4. Weight layout is accounted for. Every family here carries convolution stacks (CNN backbones, ViT patch embeddings, VAEs), and checkpoints serialize 4-D convolution weights in `(out_channels, in_channels, kernel_h, kernel_w)` order, while mlxcel's MLX runtime runs channels-last convolutions expecting `(out_channels, kernel_h, kernel_w, in_channels)`. A port must permute these at load; `src/vision/detection/rt_detr_v2/sanitize.rs` is the in-tree precedent (it also guards against double-transposing checkpoints already stored in MLX layout).

Ranked by current proximity to mlxcel's runtime:

- `nemotron_labs_diffusion`: text-only, DiffusionGemma precedent covers the decode modes; needs only a port issue.
- `locateanything`: standard VLM architecture with in-tree components; needs a decision on the grounding output contract.
- `rfdetr`: fits the existing `detect` box path; needs a DINOv2 backbone and deformable decoder.
- `florence2`: needs a seq2seq generation loop or a Whisper-style dedicated pipeline.
- `sam3` / `sam3_1`: need a mask output path and (for video) tracker state streaming.
- `flux2` / `ideogram4`: need a full diffusion image stack.
- `sam3d_body`: needs a geometry output path and a parametric body model; furthest from any existing code.

When re-scoping, follow the port-issue format used by #533 through #547 (Summary with the `model_type` string, What it is, In-tree reuse, Scope, Effort) and link the new issue here.

## Non-goals

- Implementing any of these families as part of this issue; this is a decision record only.
- Building a general image-generation, segmentation-mask, or 3D-geometry output pipeline; each would be its own scoped infrastructure issue.
- Extending the OpenAI-compatible API surface with image/mask/mesh response types.
- Tracking generative VLM port candidates; those live in #548 and in the individual port issues #533 through #547.

Read the scope summary and family statuses in this issue, then review the referenced detection and model paths only when validating an entry. Done means the coverage decision and re-scoping criteria remain accurate; this issue explicitly requires no code and should move concrete ports into dedicated issues.

Written by the indexing model from the issue text.