feature: latency metrics — time to first token, total response time, session percentiles
Open
@huang195 is already working on this.
Since Sep 10, 2026.
enhancement
high priority
- Dominant language
- Go
- Stars
- 13
- Forks
- 40
- Avg merge
- 12h 17m
- Merged PRs (30d)
- 156
Description
Feature Description
Response time is on the observability list and is not recorded today. It is also how we would
substantiate any claim about upstream (LiteLLM) performance rather than guessing.
Proposed Solution
- Time to first token and total response time per request
- p50 / p95 / max per session and per model
- Streaming and non-streaming responses both measured, and distinguishable
- Queue and proxy overhead separated from upstream time, so Cortex's own cost is visible
- Same field-name schema as per-session token and cost aggregation
Additional Context
Part of the Cortex v0.9.0 release.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.