mudler / mudler/vllm.cpp

fix(QUANT-GGUF-CPU-THREADPOOL): the CPU thread count reads hardware_concurrency() and ignores the cgroup quota

Open
#2,661 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
423
Forks
53
Avg merge
20h 26m
Merged PRs (30d)
310

Description

Row: QUANT-GGUF-CPU-THREADPOOL

Found by wave PORTQ-2 re-deriving PORT-NOW entry 66 of 5559679229..e126687a9a
(#2646). Upstream:
7635a9002b vllm#49919, "[Core] Explicitly manage torch CPU threads in workers".

Two of the commit's three halves are surface-absent; one is not

Surface-absent: there is no torch, so there is no intra-op thread pool to drop to
1, and there is no multiprocess executor — the executor is a single-process
pass-through (include/vllm/v1/executor/executor.h:41-47, "upstream
UniProcExecutor collapsed to a direct call"), so multiproc_executor.py and
gpu_worker.py have nothing to edit.

The half that does have a surface is the CPU-count resolution, and it is
quota-blind in exactly the way upstream's available_cpu_count() fixes.

src/vt/cpu/cpu_threadpool.cpp:111-120:

int ThreadsFromEnv() {
  int n = 0;
  if (const char* e = std::getenv("VLLM_CPP_CPU_THREADS")) {
    n = std::atoi(e);
  }
  if (n <= 0) {
    n = static_cast<int>(std::thread::hardware_concurrency());
  }
  return std::clamp(n, 1, kMaxThreads);
}

std::thread::hardware_concurrency() reports HOST cores inside a cpu-limited
container. Searches returning zero across src/ and include/: cgroup,
cpu.max, cpuset, sched_getaffinity, OMP_NUM_THREADS, set_num_threads.

This is not hypothetical for this project: AGENTS.md's own lease model runs jobs
in containers, and .agents/environment.md records that unconstrained
parallelism has OOM-rebooted the dgx box.

Scope

A cgroup- and affinity-aware usable-CPU count behind ThreadsFromEnv(): read
sched_getaffinity, then the tightest cpu.max / cpu.cfs_quota_us up the
hierarchy, take the minimum, floor at 1. The same treatment is owed to the two
other hardware_concurrency() sites,
src/vllm/model_executor/model_loader/nvfp4_dequant.cpp:144 and
src/vllm/entrypoints/openai/api_server.cpp:73.

Roughly 60-90 lines over 1-3 files, plus a hermetic test over synthetic cgroup
files. The explicit VLLM_CPP_CPU_THREADS override must keep priority.

Honest weight

Nothing was measured. The case is the code plus the container fact in
AGENTS.md, not an observed regression. Whether an over-subscribed pool has cost a
measurement on the leased containers is unknown, and this issue does not claim it
has.

Nothing was executed for this finding: no build, no test, no GPU.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with src/vt/cpu/cpu_threadpool.cpp:111-120 and inspect the two other hardware_concurrency() sites in src/vllm/model_executor/model_loader/nvfp4_dequant.cpp:144 and src/vllm/entrypoints/openai/api_server.cpp:73. Add a hermetic test using synthetic cgroup files and verify affinity and the tightest quota are respected while VLLM_CPP_CPU_THREADS remains the override. Done means all three sites use the usable CPU count without requiring a live container.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend, performance
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.