mudler / mudler/vllm.cpp

vt::Conv2d has no CUDA provider, so the shared 2-D convolution seam cannot run on a GPU

Open
#2,709 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
423
Forks
53
Avg merge
20h 26m
Merged PRs (30d)
310

Description

Row: KERNEL-VT-CONV2D-CUDA

vt::Conv2d (include/vt/ops.h:4428) has exactly ONE provider in the tree:

$ grep -rn "OpId::kConv2d" src/ include/
src/vt/op_provider.cpp:566:    case OpId::kConv2d:
src/vt/ops.cpp:3282:  reinterpret_cast<Conv2dFn>(GetOp(OpId::kConv2d, q.device.type))(q, out, x, weight, bias, args);
src/vt/cpu/cpu_conv2d.cpp:111:    RegisterOp(OpId::kConv2d, DeviceType::kCPU,

src/vt/cpu/cpu_conv2d.cpp:111 is the CPU registration and there is no CUDA
one, so vt::Conv2d on a CUDA queue resolves no op. Any model that routes a
2-D convolution through the shared seam therefore ships a capability that
cannot run on the only device class that could serve it.

Why this is being filed now

dots3-note W7a (#2703) ports the dots speech encoder, whose stem is three
Conv2d(kernel=3, stride=2) layers (nvidia/audio_encoder.py:466-474 @
9035151d6). It composes them as host im2col + vt::MatmulBT and records ONE
EXACT TRACKED EXCEPTION to the kConv2d seam, with this issue as the reason.
The in-tree precedent is include/vllm/model_executor/models/whisper_audio.h:33,
which composes its Conv1d stem the same way and says "no new CUDA kernel".

The exception is the right call for that slice — a stem routed through
kConv2d would fault on the queue every real deployment uses — but the gap
outlives it. It is a vt gap, not a model gap.

What closing it looks like

A CUDA provider for OpId::kConv2d matching the CPU one's contract
(src/vt/cpu/cpu_conv2d.cpp), gated the way the other CUDA op ports are: a
red-first CPU/CUDA agreement test over the shapes the registered callers use,
plus the arch gate check-cuda-op-arch-gate already applies. Then each im2col
composition that exists only because of this gap can be reconsidered on a
measurement rather than on availability.

Not in scope here

vt::Conv3d (ops.h:4475) and vt::Conv1d (ops.h:4609) have their own
provider tables and are not measured by this issue.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with include/vt/ops.h:4428, src/vt/cpu/cpu_conv2d.cpp:111, and the dispatch in src/vt/ops.cpp:3282 to understand the Conv2d contract and provider registration. Run the existing check-cuda-op-arch-gate and inspect the red-first CPU/CUDA agreement coverage for registered caller shapes. Done means a CUDA OpId::kConv2d provider is registered, passes the agreement test, and satisfies the architecture gate.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
66/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.