lablup / lablup/backend.ai

Endpoint.live_stat returns null - inference metric collection not working

Open
#9,679 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
15h 13m
Merged PRs (30d)
368

Description

## Bug Description

The Endpoint.live_stat GraphQL field returns null for all endpoints. Inference metrics collected by the appproxy worker are not being properly stored or retrieved.

## Analysis

The metric collection flow is as follows:

```
appproxy worker (periodic timer)
→ collect_inference_metric()
→ gather_inference_measures(circuit)
→ RuntimeVariant dispatch:
✅ VLLM → /metrics (Prometheus) collection
✅ HF_TGI → /metrics collection
✅ SGLANG → /metrics collection
✅ MODULAR_MAX → /metrics collection
❌ CUSTOM → return None (no collection)
❌ NIM → return None (no collection)
❌ CMD → return None (no collection)
→ store_inference_metrics() → Store to Valkey (TTL 120s)
```

Even for supported runtime variants (VLLM, HF_TGI, SGLANG, MODULAR_MAX), the live_stat field still returns null. The root cause needs further investigation — possible issues include:

- Metrics not being collected from /metrics endpoint correctly
- Metrics not being stored to Valkey properly
- GraphQL resolver not reading from Valkey correctly
- TTL expiration (120s) causing stale data loss before read

## Acceptance Criteria

- Identify root cause of null live_stat for supported runtime variants
- Endpoint.live_stat returns valid metric data for VLLM, HF_TGI, SGLANG, and MODULAR_MAX endpoints
- Verify the full pipeline: collect → store (Valkey) → resolve (GraphQL)

JIRA Issue: BA-4888

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.