mudler / mudler/vllm.cpp

Derive the KV budget from real startup memory profiling

Open
#3,046 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
423
Forks
53
Avg merge
20h 26m
Merged PRs (30d)
310

Description

Row: KV-WARMUP-PROFILE

The existing engine inventory names this row for startup warmup and memory profiling. Its planned spec, .agents/specs/warmup-memory-profiling.md, is absent at main e1948ecebde085907171abce41a7b2e7541fd7d8. This issue gives that existing gap a queryable owner. It does not create a new profiling capability by declaration.

LoadedEngine::ResolveNumBlocks still falls back to 256 blocks when neither an explicit block count nor an absolute KV budget is supplied. The fraction path has no actual non-KV memory profile. Commit 31384c8b7 added a warning for an explicitly supplied fraction and deliberately retained that fallback. Its embedded legacy #83 link returns REMOTE_UNVERIFIED with the current contributor token. Searches for this row and memory profiling found no accessible implementation issue beyond #2773's characterization record.

The missing implementation now blocks gfx1100 campaign evidence. Native Qwen3.5-0.8B controls retain the 256-block fallback and reduce model length to 8192. Active vLLM e126687a9a828d513c01a07cd69f025f27d63280 executes the same short workload with its production 0.92 memory fraction and records an actual profiled cache allocation. Equal requested fractions therefore do not establish equal resolved controls. #2773 retains the state-comparison contract and the separate block-size, scheduling, format, and operation prerequisites. Profiling alone does not resolve all those differences.

Implement the existing row through a committed spec, fresh implementation, independent mutation review, and operator gates. Mirror the active pin's complete Worker::determine_available_memory, executing model-runner profile, and memory accounting chain. Run the oracle and retain its actual measurements before selecting local behavior. Preserve explicit block-count and absolute-budget precedence, supported backend behavior, real allocation limits, error paths, and production reachability through include/vllm.h and the ordinary loader. A free-memory multiplication without measured model/activation/non-framework costs is not the profile.

The spec must define the backend scope, actual ownership/lifetime of warmup allocations, CPU/resource limitations, red-before tests, production-call-site deletion mutations, and idle GPU verification. Resolve related inventory and roadmap records in the implementation flow. Keep unavailable hardware gates pending rather than making a broad unmeasured default change. Do not relax #2773's comparison checks or publish a throughput result to bypass this prerequisite.

This issue owns KV-WARMUP-PROFILE; #2773 owns the consuming gfx1100 numerical campaign. The landing PR must link this issue and the committed spec, with a closing keyword only when the declared implementation lands.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with LoadedEngine::ResolveNumBlocks and the active vLLM pin's Worker::determine_available_memory chain; read the missing .agents/specs/warmup-memory-profiling.md requirements from the issue first. Trace production reachability through include/vllm.h and the ordinary loader, then compare against #2773. Done means a committed spec, measured warmup allocation and accounting, mutation review, operator gates, preserved precedence and errors, and verified idle-GPU behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.