mudler / mudler/vllm.cpp

Op provider: route runtime selection by full device

Open
#2,402 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
423
Forks
53
Avg merge
20h 26m
Merged PRs (30d)
310

Description

Row: BACKEND-ACCEL-PROVIDER

The backend registry is keyed by Device{type,index}, but the operation provider resolves runtime state by DeviceType. Every queue-bearing wrapper discards the index before GetOp.

This is memory-unsafe for the portable CPU reference tier. If index 0 is host-addressable, it can install and cache vt-cpu-ref for the whole type. A discrete index can then select that host function over non-host-addressable memory. Fallback draining also calls backend index 0 rather than the queue's backend.

The same runtime path has a publication race. MaybeInstallReferenceTier appends a provider after main() starts without synchronization. Two concurrent first misses can race slot.count and slot.providers.

Fresh review of #2377 found the device-index defect. Independent adversarial analysis confirmed that synchronizing every GPU does not fix host-addressability. A shared selected pointer with separate relaxed ref_selected state is also insufficient.

Acceptance:

  • Provider definitions remain type-keyed and immutable after static registration.
  • Runtime selection state, capability state, reference identity, and negative caches are keyed by the full Device.
  • Queue-bearing execution APIs carry Queue::device through normal and fallback resolution.
  • A CPU reference provider is a synthetic per-device candidate. Runtime code does not append it to the shared provider list.
  • Every reference invocation checks the exact backend's host-addressability and drains that exact backend.
  • A hardware-free two-index test covers both host-addressability polarities, resolution order, exact drain routing, and an already-warm reference selection.
  • A concurrent first-miss test passes under ThreadSanitizer and exposes one coherent selection.
  • Native provider ordering, disable behavior, statistics, and the cached native hot path retain their existing contracts.
  • Cached decline paths cannot bypass per-invocation reference eligibility or draining.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the BACKEND-ACCEL-PROVIDER path through GetOp, Queue::device, and MaybeInstallReferenceTier, then inspect how fallback draining and cached decline paths resolve providers. Use the acceptance list as the completion criteria, including the hardware-free two-index coverage and the concurrent first-miss test under ThreadSanitizer.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
api, backend, testing-qa
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.