vllm-project / vllm-project/vllm

[Doc]: metrics page is missing the spec-decode counters and cache_config_info

Open
#56,514 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
91.8k
Forks
22.2k
PR merge metrics
PR metrics pending

Description

### 📚 The doc issue

The metric tables on https://docs.vllm.ai/en/latest/usage/metrics/ come from `docs/mkdocs/gen_files/generate_metrics.py`. That script only picks up a metric when `name=` is a string literal. Two files don't do that:

- Since #45163, `vllm/v1/spec_decode/metrics.py` creates its counters in a loop over `counter_specs`, calling `self._counter_cls(name=name, ...)`.
- `vllm/v1/metrics/loggers.py` builds `vllm:cache_config_info` through a local variable.

As a result, these metrics are missing from the page:

- `vllm:spec_decode_num_drafts`, `vllm:spec_decode_num_draft_tokens`, `vllm:spec_decode_num_accepted_tokens`
- `vllm:diffusion_num_denoising_steps`, `vllm:diffusion_num_canvas_positions`, `vllm:diffusion_num_committed_tokens`
- `vllm:cache_config_info`

The Speculative Decoding table lists only `vllm:spec_decode_num_accepted_tokens_per_pos`. Meanwhile `features/speculative_decoding/acceptance_metrics.md` tells users to compute acceptance from `vllm:spec_decode_num_drafts_total` and `vllm:spec_decode_num_draft_tokens_total`, and neither of those is on the page.

### Suggest a potential alternative/fix

- Let the extractor resolve `name=` when it's a local variable, either assigned a string or unpacked from a literal table.
- Have the generator fail the docs build when a `vllm:*` name in a scanned file isn't extracted, so this can't regress silently again.

I have a fix ready and will open a PR.

### Before submitting a new issue...

- [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.