Endpoint.live_stat returns null - inference metric collection not working
- Dominant language
- Python
- Stars
- 670
- Forks
- 183
- Avg merge
- 15h 13m
- Merged PRs (30d)
- 368
Description
## Bug Description
The Endpoint.live_stat GraphQL field returns null for all endpoints. Inference metrics collected by the appproxy worker are not being properly stored or retrieved.
## Analysis
The metric collection flow is as follows:
```
appproxy worker (periodic timer)
→ collect_inference_metric()
→ gather_inference_measures(circuit)
→ RuntimeVariant dispatch:
✅ VLLM → /metrics (Prometheus) collection
✅ HF_TGI → /metrics collection
✅ SGLANG → /metrics collection
✅ MODULAR_MAX → /metrics collection
❌ CUSTOM → return None (no collection)
❌ NIM → return None (no collection)
❌ CMD → return None (no collection)
→ store_inference_metrics() → Store to Valkey (TTL 120s)
```
Even for supported runtime variants (VLLM, HF_TGI, SGLANG, MODULAR_MAX), the live_stat field still returns null. The root cause needs further investigation — possible issues include:
- Metrics not being collected from /metrics endpoint correctly
- Metrics not being stored to Valkey properly
- GraphQL resolver not reading from Valkey correctly
- TTL expiration (120s) causing stale data loss before read
## Acceptance Criteria
- Identify root cause of null live_stat for supported runtime variants
- Endpoint.live_stat returns valid metric data for VLLM, HF_TGI, SGLANG, and MODULAR_MAX endpoints
- Verify the full pipeline: collect → store (Valkey) → resolve (GraphQL)
JIRA Issue: BA-4888
Contributor guide
Assessment
This issue has not been assessed yet.