vllm-project / vllm-project/aibrix

trtllm metrics is undefined if we use absolute model path in the startup command

Open
#2,008 2 comments 0 reactions 0 assignees View on GitHub
kind/bug priority/critical-urgent
Dominant language
Go
Stars
5.1k
Forks
694
Avg merge
1d 20h
Merged PRs (30d)
98

Description

### 🚀 Feature Description and Motivation

there're some issues on the metrics fetching, If we use below config and then fetch the metrics
```
curl localhost:8000/prometheus/metrics
```

```
containers:
- args:
- |
cat <<'EOF' > /tmp/config.yaml
backend: pytorch
kv_cache_config:
free_gpu_memory_fraction: 0.85
max_num_tokens: 8192
max_batch_size: 16
trust_remote_code: true
return_perf_metrics: true
enable_iter_perf_stats: true
enable_iter_req_stats: true
perf_metrics_max_requests: 1000
EOF

trtllm-serve serve /models/Qwen3-8B \
--host 0.0.0.0 \
--port 8000 \
--extra_llm_api_options /tmp/config.yaml
```

the model_name is undefined which is unexpected. We also need to figure out the engine type usage

```
e2e_request_latency_seconds_bucket{engine_type="undefined",le="0.3",model_name="undefined"} 0.0
e2e_request_latency_seconds_bucket{engine_type="undefined",le="0.5",model_name="undefined"} 0.0
e2e_request_latency_seconds_bucket{engine_type="undefined",le="0.8",model_name="undefined"} 0.0
e2e_request_latency_seconds_bucket{engine_type="undefined",le="1.0",model_name="undefined"} 0.0
e2e_request_latency_seconds_bucket{engine_type="undefined",le="1.5",model_name="undefined"} 0.0
e2e_request_latency_seconds_bucket{engine_type="undefined",le="2.0",model_name="undefined"} 1.0
e2e_request_latency_seconds_bucket{engine_type="undefined",le="2.5",model_name="undefined"} 2.0
e2e_request_latency_seconds_bucket{engine_type="undefined",le="5.0",model_name="undefined"} 2.0
e2e_request_latency_seconds_bucket{engine_type="undefined",le="10.0",model_name="undefined"} 2.0
e2e_request_latency_seconds_bucket{engine_type="undefined",le="15.0",model_name="undefined"} 2.0
e2e_request_latency_seconds_bucket{engine_type="undefined",le="20.0",model_name="undefined"} 2.0
e2e_request_latency_seconds_bucket{engine_type="undefined",le="30.0",model_name="undefined"} 2.0
e2e_request_latency_seconds_bucket{engine_type="undefined",le="40.0",model_name="undefined"} 2.0
e2e_request_latency_seconds_bucket{engine_type="undefined",le="50.0",model_name="undefined"} 2.0
e2e_request_latency_seconds_bucket{engine_type="undefined",le="60.0",model_name="undefined"} 2.0
e2e_request_latency_seconds_bucket{engine_type="undefined",le="120.0",model_name="undefined"} 2.0
e2e_request_latency_seconds_bucket{engine_type="undefined",le="240.0",model_name="undefined"} 2.0
e2e_request_latency_seconds_bucket{engine_type="undefined",le="480.0",model_name="undefined"} 2.0
e2e_request_latency_seconds_bucket{engine_type="undefined",le="960.0",model_name="undefined"} 2.0
e2e_request_latency_seconds_bucket{engine_type="undefined",le="1920.0",model_name="undefined"} 2.0
e2e_request_latency_seconds_bucket{engine_type="undefined",le="7680.0",model_name="undefined"} 2.0
e2e_request_latency_seconds_bucket{engine_type="undefined",le="+Inf",model_name="undefined"} 2.0
e2e_request_latency_seconds_count{engine_type="undefined",model_name="undefined"} 2.0
# HELP time_to_first_token_seconds Histogram of time to first token in seconds.
# TYPE time_to_first_token_seconds histogram
time_to_first_token_seconds_sum{engine_type="undefined",model_name="undefined"} 0.13339699897915125
time_to_first_token_seconds_bucket{engine_type="undefined",le="0.001",model_name="undefined"} 0.0
time_to_first_token_seconds_bucket{engine_type="undefined",le="0.005",model_name="undefined"} 0.0
time_to_first_token_seconds_bucket{engine_type="undefined",le="0.01",model_name="undefined"} 0.0
time_to_first_token_seconds_bucket{engine_type="undefined",le="0.02",model_name="undefined"} 1.0
time_to_first_token_seconds_bucket{engine_type="undefined",le="0.04",model_name="undefined"} 1.0
time_to_first_token_seconds_bucket{engine_type="undefined",le="0.06",model_name="undefined"} 1.0
time_to_first_token_seconds_bucket{engine_type="undefined",le="0.08",model_name="undefined"} 1.0
time_to_first_token_seconds_bucket{engine_type="undefined",le="0.1",model_name="undefined"} 1.0
time_to_first_token_seconds_bucket{engine_type="undefined",le="0.25",model_name="undefined"} 2.0
time_to_first_token_seconds_bucket{engine_type="undefined",le="0.5",model_name="undefined"} 2.0
time_to_first_token_seconds_bucket{engine_type="undefined",le="0.75",model_name="undefined"} 2.0
time_to_first_token_seconds_bucket{engine_type="undefined",le="1.0",model_name="undefined"} 2.0
time_to_first_token_seconds_bucket{engine_type="undefined",le="2.5",model_name="undefined"} 2.0
time_to_first_token_seconds_bucket{engine_type="undefined",le="5.0",model_name="undefined"} 2.0
time_to_first_token_seconds_bucket{engine_type="undefined",le="7.5",model_name="undefined"} 2.0
time_to_first_token_seconds_bucket{engine_type="undefined",le="10.0",model_name="undefined"} 2.0
time_to_first_token_seconds_bucket{engine_type="undefined",le="20.0",model_name="undefined"} 2.0
time_to_first_token_seconds_bucket{engine_type="undefined",le="40.0",model_name="undefined"} 2.0
time_to_first_token_seconds_bucket{engine_type="undefined",le="80.0",model_name="undefined"} 2.0
time_to_first_token_seconds_bucket{engine_type="undefined",le="160.0",model_name="undefined"} 2.0
time_to_first_token_seconds_bucket{engine_type="undefined",le="640.0",model_name="undefined"} 2.0
time_to_first_token_seconds_bucket{engine_type="undefined",le="2560.0",model_name="undefined"} 2.0
time_to_first_token_seconds_bucket{engine_type="undefined",le="+Inf",model_name="undefined"} 2.0
time_to_first_token_seconds_count{engine_type="undefined",model_name="undefined"} 2.0
# HELP time_per_output_token_seconds Histogram of time per output token in seconds.
# TYPE time_per_output_token_seconds histogram
time_per_output_token_seconds_sum{engine_type="undefined",model_name="undefined"} 0.013218691888969944
time_per_output_token_seconds_bucket{engine_type="undefined",le="0.01",model_name="undefined"} 2.0
time_per_output_token_seconds_bucket{engine_type="undefined",le="0.025",model_name="undefined"} 2.0
time_per_output_token_seconds_bucket{engine_type="undefined",le="0.05",model_name="undefined"} 2.0
time_per_output_token_seconds_bucket{engine_type="undefined",le="0.075",model_name="undefined"} 2.0
time_per_output_token_seconds_bucket{engine_type="undefined",le="0.1",model_name="undefined"} 2.0
time_per_output_token_seconds_bucket{engine_type="undefined",le="0.15",model_name="undefined"} 2.0
time_per_output_token_seconds_bucket{engine_type="undefined",le="0.2",model_name="undefined"} 2.0
time_per_output_token_seconds_bucket{engine_type="undefined",le="0.3",model_name="undefined"} 2.0
time_per_output_token_seconds_bucket{engine_type="undefined",le="0.4",model_name="undefined"} 2.0
time_per_output_token_seconds_bucket{engine_type="undefined",le="0.5",model_name="undefined"} 2.0
time_per_output_token_seconds_bucket{engine_type="undefined",le="0.75",model_name="undefined"} 2.0
time_per_output_token_seconds_bucket{engine_type="undefined",le="1.0",model_name="undefined"} 2.0
time_per_output_token_seconds_bucket{engine_type="undefined",le="2.5",model_name="undefined"} 2.0
time_per_output_token_seconds_bucket{engine_type="undefined",le="5.0",model_name="undefined"} 2.0
time_per_output_token_seconds_bucket{engine_type="undefined",le="7.5",model_name="undefined"} 2.0
time_per_output_token_seconds_bucket{engine_type="undefined",le="10.0",model_name="undefined"} 2.0
time_per_output_token_seconds_bucket{engine_type="undefined",le="20.0",model_name="undefined"} 2.0
time_per_output_token_seconds_bucket{engine_type="undefined",le="40.0",model_name="undefined"} 2.0
time_per_output_token_seconds_bucket{engine_type="undefined",le="80.0",model_name="undefined"} 2.0
time_per_output_token_seconds_bucket{engine_type="undefined",le="+Inf",model_name="undefined"} 2.0
time_per_output_token_seconds_count{engine_type="undefined",model_name="undefined"} 2.0
# HELP request_queue_time_seconds Histogram of time spent in WAITING phase for request.
# TYPE request_queue_time_seconds histogram
request_queue_time_seconds_sum{engine_type="undefined",model_name="undefined"} 0.014286999590694904
request_queue_time_seconds_bucket{engine_type="undefined",le="0.3",model_name="undefined"} 2.0
request_queue_time_seconds_bucket{engine_type="undefined",le="0.5",model_name="undefined"} 2.0
request_queue_time_seconds_bucket{engine_type="undefined",le="0.8",model_name="undefined"} 2.0
request_queue_time_seconds_bucket{engine_type="undefined",le="1.0",model_name="undefined"} 2.0
request_queue_time_seconds_bucket{engine_type="undefined",le="1.5",model_name="undefined"} 2.0
request_queue_time_seconds_bucket{engine_type="undefined",le="2.0",model_name="undefined"} 2.0
request_queue_time_seconds_bucket{engine_type="undefined",le="2.5",model_name="undefined"} 2.0
request_queue_time_seconds_bucket{engine_type="undefined",le="5.0",model_name="undefined"} 2.0
request_queue_time_seconds_bucket{engine_type="undefined",le="10.0",model_name="undefined"} 2.0
request_queue_time_seconds_bucket{engine_type="undefined",le="15.0",model_name="undefined"} 2.0
```

### Use Case

fetch model names for grouping purpose.

### Proposed Solution

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.