Add DeepSeek V4 Flash Vision support
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 423
- Forks
- 53
- Avg merge
- 20h 26m
- Merged PRs (30d)
- 310
Description
Row: MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm
DeepSeek published deepseek-ai/DeepSeek-V4-Flash-Vision-Exp at revision 86f746b36186f0e567729a5c06a8c918caba82a9. The checkpoint reuses the DeepSeek-V4 text backbone and adds a 32-layer ViT, aligner, image sentinel embeddings, image-span attention visibility, interleaved image prompt encoding, and a DSpark tail.
vLLM at the repository parity pin has no vision path for DeepseekV4ForCausalLM; current vLLM, vLLM-Omni, SGLang, and Transformers also expose no complete implementation of this vision variant. The checkpoint repository's minimal PyTorch inference is the only complete executable reference and needs a model-specific oracle pin before it can gate a port.
Scope: add the model and oracle to the roadmap, commit the implementation spec before code, then implement the processor, vision tower and aligner, weight and quantized arms, DeepSeek image merge and visibility semantics, registered multimodal forward, OpenAI multi-image serving, and correctness/performance gates. The spec and implementation use one pull request, with the spec commit first.
CUDA, ROCm, and Vulkan are separate required device paths. Every accelerator build, correctness run, and benchmark must execute through resource-controller (rc) on a leased fleet device; no direct SSH run or CPU fallback counts. CUDA uses an eligible NVIDIA fleet device, ROCm uses strix:gpu0, and Vulkan uses the physical RADV GFX1151 device on strix:gpu0 rather than llvmpipe. A full-artifact memory blocker remains explicit per backend while reduced device-path correctness is still required.
The issue closes when the production entry point accepts the pinned checkpoint family through the documented quantized vehicle, ordinary autoregressive image generation is oracle-gated, text-only DeepSeek-V4 stays byte-identical, CUDA/ROCm/Vulkan device-path evidence is recorded from rc jobs, and the shipped checkpoint details are documented.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The payload names no repository files, tests, or production entry point. Start with the checkpoint repository's minimal PyTorch inference at revision 86f746b36186f0e567729a5c06a8c918caba82a9, then write the implementation spec before locating the corresponding entry points and gates. Done requires oracle-gated image generation, unchanged text-only behavior, documented rc evidence for CUDA, ROCm, and Vulkan, and shipped checkpoint documentation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- ai, backend, machine-learning, performance, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100