mudler / mudler/vllm.cpp

Wire production AsyncLLM requests into Prometheus /metrics

Open
#538 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
423
Forks
53
Avg merge
20h 26m
Merged PRs (30d)
310

Description

Problem

The shipped vllm-server --enable-metrics exposes the Prometheus catalog, but production requests run through AsyncLLM while the logger is attached only to the unused synchronous LLMEngine. Live scrapes therefore leave num_requests_running, token counters, request-success counters, and timing histograms at zero even while requests complete.

Reproduction

Run four concurrent /v1/chat/completions requests against a server started with --enable-metrics and scrape /metrics at 100 ms cadence. Requests complete correctly, but all nine aligned samples report num_requests_running=0, num_requests_waiting=0, with token and success counters unchanged.

Scope

Complete W6 in .agents/specs/prometheus-metrics.md:

  • attach the existing PrometheusStatLogger to production AsyncLLM;
  • build and record IterationStats in the output-handler path;
  • make concurrent Record() / Expose() safe;
  • preserve the null-logger no-stats path;
  • add RED-first async metric and concurrent-scrape tests;
  • gate affected async/sync/Prometheus/server targets.

No model kernels, sampling, scheduler policy, or Hermes client changes.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read .agents/specs/prometheus-metrics.md and trace the production AsyncLLM output-handler path alongside PrometheusStatLogger and IterationStats. Add the async metric recording and concurrency coverage described there, preserving the null-logger path, then run the affected async, sync, Prometheus, and server targets to verify RED metrics and concurrent scrapes.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, prometheus
Domain
backend, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.