Dashboard KPI for time to first token
- Dominant language
- Rust
- Stars
- 3
- Forks
- 0
- Avg merge
- 59m
- Merged PRs (30d)
- 5
Description
A cross-encoder rerank landed on 2026-05-29 and took p50 turn time from 2.8s to 21.5s. It went unnoticed until 2026-08-26. Nothing on the dashboards could have shown it.
Why it was invisible:
- `http_request_duration_seconds` is the only latency metric we have, and on the chat SSE routes (`POST /conversations` and `POST /conversations/{cid}/message`) it measures the whole stream duration. That is dominated by answer length, so "slow to start" and "long answer" are indistinguishable.
- `generation_ms` and `thinking_ms` are written per message into `messages` but never exported, so the regression was only findable by running SQL against prod.
- Completion tokens were flat across the step change, which is exactly the signal that separates an infra regression from longer answers, and that is not plotted either.
`rerank_seconds` (6babe9cd) now covers the largest pre-token step, but there is still no end to end student-facing number.
What I want:
1. `chat_ttft_seconds` histogram, measured from request received to the first `token` SSE event, labelled by strategy and `tool_use_enabled`. Tool use puts a research phase in front of the first visible token, so those have to be separable or the p50 is meaningless.
2. Export the per-phase timings that already exist in the strategy path: seed retrieval, embed, qdrant search, rerank, adversarial filter, extraction guard, research phase, writeup. A regression should point at a phase rather than at "chat got slower".
3. Export `generation_seconds` and completion tokens so tokens per second is plottable. Latency up with token count flat is the fingerprint we needed and did not have.
4. A student-facing KPI row at the top of `services-app`: TTFT p50/p95 plus the phase breakdown, and something that makes a step change obvious instead of requiring someone to eyeball a three month window.
Two related blind spots turned up in the same session, same theme of a panel existing but unable to show the truth:
- kube-state-metrics label clobbering left the OOMKilled table permanently empty, hiding a worker crashloop for 22 days (fixed in e5b024bb).
- The reranker benchmark scored 24 short sentences instead of the operational 40 chunks at 512 tokens, so it reported healthy throughput while the chat path paid seconds per turn (fixed in 6babe9cd).
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing the POST /conversations and POST /conversations/{cid}/message SSE routes through the strategy path, including the existing rerank_seconds instrumentation and per-message timings. Then inspect the services-app dashboard entry point and determine where the TTFT, phase, generation, and completion-token data should appear. Done means the requested metrics are exported and the student-facing KPI row shows TTFT percentiles, phase breakdowns, and visible step changes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- react, rust, typescript
- Domain
- backend, frontend, observability
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100