Live evaluation of demo.fanout.run: 33 findings, P0 correctness first

Findings from a live evaluation pass over https://demo.fanout.run (image 2026.9.3), driven through the browser as a user would, with every numeric claim verified against the API rather than read off the screen.

The agent itself came out well: from a single service-map click it root-caused a 504 (checkout's PlaceOrder hung ~103s while cart, payment, shipping and email each finished under 15ms; the proxy gave up at its 15s budget and returned 504 UT response_timeout on an order that had actually succeeded). Asked about zero ERROR logs it named the telemetry blind spot instead of answering "0". Dashboard-from-chat works end to end. The list below is everything that got in the way.

Ordered by priority. Each item is independent; P0 items should land before the demo is put in front of anyone else.

---

P0 — correctness

1. Stat tiles read the last, partial bucket while the card is labelled "Last 1h"

The frontend service card shows Operations 1,281, P95 48.2ms, coloured green. For the same service and window, /api/observability/topology?window=1h returns 15,283 spans and p95 13,691ms, and the card's own Compare tab prints 13.69s → 10.41s. One card contradicts itself by ~285x on latency and ~12x on volume, and paints a 13.7s-P95 service as healthy.

The tile appears to take the most recent 5-minute bucket, which is also incomplete at read time, so it under-reports on both axes.

Fix: aggregate tiles over the labelled window, or label them explicitly as the latest interval and exclude the in-flight bucket.

2. An empty namespace filter reports "Healthy"

Typing a namespace that does not exist (opentelemetry-demo) renders Health: Healthy · 0 services · Error rate 0.00% · 0 operations. No-data is being rendered as good news.

Fix: distinguish "no data" from "healthy" in the health card and every downstream widget. Add namespace autocomplete sourced from known values so the state is hard to reach in the first place.

3. Log severity is not normalised at ingest

Across a 400-entry sample over 6h: INFO 130, empty string 54, INFORMATION 16. System-wide there were zero ERROR or WARN logs in 6h while the platform reported a 1.77% error rate and 9 unhealthy services. Envoy access logs from frontend-proxy arrive with no severity at all; the .NET cart service emits INFORMATION.

Consequences: every severity filter is unreliable, and error-log views are structurally empty even for a system actively returning 504s.

Fix: map OTLP severity_number on ingest, fold INFORMATION into INFO, and give unlabelled records an explicit unspecified value rather than an empty string.

4. The trace picker ignores service scope

A dashboard scoped to checkout pinned a "Slow PlaceOrder Trace" widget that rendered a flagd.evaluation.v1.Service trace — 2 spans, recommendation ↔ flagd, 600.01s. Neither the service nor the operation matches the scope.

5. Health scoring is dominated by long-lived background spans

P95 sits at exactly 600.00s for flagd and fraud-detection, and 510.01s for product-reviews — consumer and poll spans, not request latency. As a result frontend is marked unhealthy despite a 0% error rate and a 3.9ms p50, and the demo permanently reads "Unhealthy · 9 of 20".

Fix: compute request-path percentiles separately from consumer/internal spans, or exclude long-running background span kinds from the health signal.

---

P1 — gaps that stall an evaluation

6. No ingest path in the UI. Signed in as admin, there is no ingest token, endpoint or copy-paste snippet anywhere. Someone who likes the demo cannot take the next step.
7. No route from insight to action. After the agent names a failing service there is nothing to click: no "create an alert from this", no pin-to-dashboard, no saved query. /api/alerts exists and returns data; nothing in the UI reaches it.
8. No logs explorer. 125,713 logs over 6h are reachable only through the agent or a severity-filtered widget that is empty because of item 3.
9. The dashboard generator pins widgets it has not checked. The generated checkout dashboard shipped "Checkout Error Logs" and "Checkout Warnings", both permanently empty. Probe for rows before pinning, or render an explicit empty state at creation time.
10. No drill-down from a Recent activity row. Rows highlight on hover like links and do nothing. Clicking a service should lead to its traces and logs.
11. Search only matches chat and dashboard titles. Searching checkout returns "No matching chats / No matching dashboards" while a service, its traces and an entire thread about it all exist.
12. State is not in the URL. Time window and namespace live outside it, so a shared dashboard link never reproduces what the sender was looking at. A bad dashboard id silently renders the default dashboard while keeping the bogus URL; an unknown route silently redirects to /chat. Both should be a not-found state.
13. Widget "Remove" deletes immediately with no confirmation and no undo.

---

P2 — number presentation

14. Duration formatting and mixed units. 600.00s should read 10m; 240.02s should read 4m 0s. One column currently prints 30.00s next to 25.0ms, and one axis prints 0.0ms at zero and 100.00s above it. The "Latency and error correlation" chart exposes a scaling hack in its legend as Error rate × 1000 instead of using a right-hand percentage axis.
15. Threshold colouring is noisy. A 0.00% cell renders red; a p50 move of 3.9ms → 4.0ms is flagged as a red 2.2% regression; the errors column flips grey↔red around 1% with no legend. Use explicit thresholds with a key, or a continuous scale.
16. P95 equals P99 on most rows because values land on histogram bucket bounds (2.5 / 5 / 10 / 25 / 50ms). Interpolate, or mark the values as approximate.
17. Downsample long series. The 24h performance chart draws 288 raw points and reads as a hairball.

---

P3 — design system

18. The type scale is not a scale. One screen renders 15 distinct size/weight pairs across 9, 10, 10.4, 11, 12, 14, 15, 16, 20 and 32px. Collapse to roughly six steps and two weights, drop everything below 11px (the 9px DEFAULT badge and 10px service-map labels are the ones users actually need to read), and add a step between 20px and 32px.
19. The dark-mode primary button fails contrast. "New chat", white on violet, measures 3.16:1 — below the 4.5:1 AA threshold for 14px text. The same button in light mode measures 5.54:1.
20. Dark-mode muted text is warm on a cool ground (rgb(191,189,182) on #0b0e14). Match the hue family of the background.
21. Status is encoded by hue alone in the service map, health bar, service pills and errors column. Add a redundant channel — shape, fill or glyph — for colour-vision deficiency.
22. The service map inverts salience. Normal edges are #8b8e99 at 3.5–4.0px width; error edges are #f26d78 at 1.5px, so the failing dependencies are the thinnest and palest lines on the canvas. Labels also collide with nodes (otelcol-contrib over load-generator, fraud-detection over flagd), and the circular layout routes every chord through the centre. A layered or directed layout would show the real request shape for 20 nodes and 23 edges.
23. Two status encodings disagree on one row. ad shows a pink "unhealthy" service pill next to a grey 0.69% errors value.
24. Card layout wastes vertical space. The health tile is half empty because it is height-matched to a sparkline tile; Recent activity scrolls internally at 6 rows while the page below it is blank; the generated Performance widget renders at roughly two-thirds width leaving a gutter.
25. The error-rate sparkline carries no information. No baseline, no min/max, no axis, plus a decorative hatch fill. A number with a delta would say more in less space.
26. The latency heatmap legend overlaps the x-axis tick labels and the colour scale has no unit.

---

P4 — hygiene

27. Ten icon buttons expose no accessible name in the accessibility tree.
28. The chat view jumps to the top of the thread after sending, so the answer arrives off-screen.
29. The theme-toggle tooltip stays visible after the click.
30. A dashboard load double-fetches dashboards, overview, topology and performance.
31. /robots.txt and every unknown path return SPA HTML with a 200. A public demo should serve a real robots file. The Cloudflare beacon also 503s on load.
32. Intermittent multi-second stalls on the default window: one pass measured 9.0s for performance and 12.7s for logs at 1h, where repeat calls run 440–550ms. Consistent with rollup/maintenance contention on the write mutex.
33. Timestamps carry no timezone and there is no unit or timezone preference.

Start by reproducing the P0 findings in the browser at demo.fanout.run, then compare the UI with /api/observability/topology?window=1h and the observed log data. Treat the five P0 findings as separate work items; done means the affected cards, filters, trace selection, and health state agree with the underlying data, with the listed empty and long-running cases handled explicitly.