Edwinexd / Edwinexd/minerva

Dashboard KPI for time to first token

Open
#94 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
3
Forks
0
Avg merge
59m
Merged PRs (30d)
5

Description

A cross-encoder rerank landed on 2026-05-29 and took p50 turn time from 2.8s to 21.5s. It went unnoticed until 2026-08-26. Nothing on the dashboards could have shown it.

Why it was invisible:

- `http_request_duration_seconds` is the only latency metric we have, and on the chat SSE routes (`POST /conversations` and `POST /conversations/{cid}/message`) it measures the whole stream duration. That is dominated by answer length, so "slow to start" and "long answer" are indistinguishable.
- `generation_ms` and `thinking_ms` are written per message into `messages` but never exported, so the regression was only findable by running SQL against prod.
- Completion tokens were flat across the step change, which is exactly the signal that separates an infra regression from longer answers, and that is not plotted either.

`rerank_seconds` (6babe9cd) now covers the largest pre-token step, but there is still no end to end student-facing number.

What I want:

1. `chat_ttft_seconds` histogram, measured from request received to the first `token` SSE event, labelled by strategy and `tool_use_enabled`. Tool use puts a research phase in front of the first visible token, so those have to be separable or the p50 is meaningless.
2. Export the per-phase timings that already exist in the strategy path: seed retrieval, embed, qdrant search, rerank, adversarial filter, extraction guard, research phase, writeup. A regression should point at a phase rather than at "chat got slower".
3. Export `generation_seconds` and completion tokens so tokens per second is plottable. Latency up with token count flat is the fingerprint we needed and did not have.
4. A student-facing KPI row at the top of `services-app`: TTFT p50/p95 plus the phase breakdown, and something that makes a step change obvious instead of requiring someone to eyeball a three month window.

Two related blind spots turned up in the same session, same theme of a panel existing but unable to show the truth:

- kube-state-metrics label clobbering left the OOMKilled table permanently empty, hiding a worker crashloop for 22 days (fixed in e5b024bb).
- The reranker benchmark scored 24 short sentences instead of the operational 40 chunks at 512 tokens, so it reported healthy throughput while the chat path paid seconds per turn (fixed in 6babe9cd).

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the POST /conversations and POST /conversations/{cid}/message SSE routes through the strategy path, including the existing rerank_seconds instrumentation and per-message timings. Then inspect the services-app dashboard entry point and determine where the TTFT, phase, generation, and completion-token data should appear. Done means the requested metrics are exported and the student-facing KPI row shows TTFT percentiles, phase breakdowns, and visible step changes.

Written by the indexing model from the issue text.

Assessment

Tech stack
react, rust, typescript
Domain
backend, frontend, observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.