mudler / mudler/vllm.cpp

Add DeepSeek V4 Flash Vision support

Open
#2,411 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
423
Forks
53
Avg merge
20h 26m
Merged PRs (30d)
310

Description

Row: MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm

DeepSeek published deepseek-ai/DeepSeek-V4-Flash-Vision-Exp at revision 86f746b36186f0e567729a5c06a8c918caba82a9. The checkpoint reuses the DeepSeek-V4 text backbone and adds a 32-layer ViT, aligner, image sentinel embeddings, image-span attention visibility, interleaved image prompt encoding, and a DSpark tail.

vLLM at the repository parity pin has no vision path for DeepseekV4ForCausalLM; current vLLM, vLLM-Omni, SGLang, and Transformers also expose no complete implementation of this vision variant. The checkpoint repository's minimal PyTorch inference is the only complete executable reference and needs a model-specific oracle pin before it can gate a port.

Scope: add the model and oracle to the roadmap, commit the implementation spec before code, then implement the processor, vision tower and aligner, weight and quantized arms, DeepSeek image merge and visibility semantics, registered multimodal forward, OpenAI multi-image serving, and correctness/performance gates. The spec and implementation use one pull request, with the spec commit first.

CUDA, ROCm, and Vulkan are separate required device paths. Every accelerator build, correctness run, and benchmark must execute through resource-controller (rc) on a leased fleet device; no direct SSH run or CPU fallback counts. CUDA uses an eligible NVIDIA fleet device, ROCm uses strix:gpu0, and Vulkan uses the physical RADV GFX1151 device on strix:gpu0 rather than llvmpipe. A full-artifact memory blocker remains explicit per backend while reduced device-path correctness is still required.

The issue closes when the production entry point accepts the pinned checkpoint family through the documented quantized vehicle, ordinary autoregressive image generation is oracle-gated, text-only DeepSeek-V4 stays byte-identical, CUDA/ROCm/Vulkan device-path evidence is recorded from rc jobs, and the shipped checkpoint details are documented.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The payload names no repository files, tests, or production entry point. Start with the checkpoint repository's minimal PyTorch inference at revision 86f746b36186f0e567729a5c06a8c918caba82a9, then write the implementation spec before locating the corresponding entry points and gates. Done requires oracle-gated image generation, unchanged text-only behavior, documented rc evidence for CUDA, ROCm, and Vulkan, and shipped checkpoint documentation.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
ai, backend, machine-learning, performance, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.