[RUM] Control-plane load 3× and latency 2× in 24h — instances-list get_status fans out N polls/10s uncapped (one session: 5,271 requests)

## Summary

Control-plane (`fabric.harper.fast`) XHR volume **tripled** and median latency **doubled** in the last 24h, while session count rose only 1.3×. The volume is almost entirely one query: the per-instance `get_status` poll on the **instances list** page, which fans out one request per instance every 10s with no cap, no adaptive backoff, and no idle gating.

**96%** of that traffic came from a single view and **78%** from just **3 sessions** — one of which alone issued **5,271** operation requests.

Surfaced by the automated daily RUM review (app `f590deee-…`).

## Datadog findings

### Control plane overall

| Metric (`@resource.url_host:fabric.harper.fast`, xhr) | 48h→24h | 24h→now | Δ |
| --- | --- | --- | --- |
| Requests | 5,620 | **17,523** | **3.1×** |
| Median duration | 121ms | **259ms** | **2.1×** |
| Sessions (whole app) | 108 | 142 | 1.3× |

App-wide XHR median went **93ms → 253ms** and mean **323ms → 620ms**; `@view.loading_time` mean **502ms → 1,119ms**. So this is not confined to one endpoint's accounting — it is the dominant driver of a visible whole-app slowdown.

### By endpoint

| Endpoint | Requests 48h→24h | Requests 24h→now | Median 48h→24h | Median 24h→now |
| --- | --- | --- | --- | --- |
| `POST /HDBInstance/?/operation` | 1,538 | **10,514** (6.8×) | 126ms | **264ms** |
| `GET /Cluster/?` | 1,520 | 2,587 | 51ms | **471ms** (9.2×) |
| `GET /SystemStatus` | 1,495 | 1,645 | 253ms | 286ms |
| `POST /Login` | 34 | 35 | 126ms | **636ms** (5×) |

`/Cluster/{id}` is the striking one: **1.7× the traffic for 9× the latency**, on an endpoint whose volume barely moved. `/Login` at 636ms median is user-visible on every sign-in.

### Where the volume comes from

`POST /HDBInstance/?/operation` by view:

| View | 48h→24h | 24h→now |
| --- | --- | --- |
| `/$organizationId/$clusterId/instances/` | 1,370 | **10,078** |
| `/` | 130 | 371 |
| everything else | 38 | 65 |

Per-session concentration (top 10, last 24h): **5271, 1779, 1179**, 462, 445, 311, 190, 135, 102, 94 — across 37 sessions. The top 3 are 78% of the total. Prior 24h for comparison: 598, 271, 119, 101, 98, …

## Mechanism

[`InstanceStatusCell`](https://github.com/HarperFast/studio/blob/stage/src/features/cluster/InstanceStatusCell.tsx) renders once per row on the instances list and each instance runs its own `get_status` query via [`getStatusQueryOptions`](https://github.com/HarperFast/studio/blob/stage/src/integrations/api/instance/status/getStatus.ts):

```ts
refetchInterval: pollUnlessForbidden(10_000),
```

So steady-state cost is **N instances ÷ 10s**, for as long as the tab stays visible. A 12-instance cluster is 72 req/min → ~4,300/hour; 5,271 requests is roughly what one large cluster left open for an hour produces.

What is already handled, and what isn't:

- ✅ Initial burst is staggered (`index * 500`).
- ✅ Stopped/transitioning instances don't poll.
- ✅ 403 halts the timer (`pollUnlessForbidden`).
- ❌ **No cap or windowing** on how many rows poll concurrently — cost scales linearly with cluster size.
- ❌ **No adaptive interval.** 10s is fixed regardless of cluster size or how long the page has been open.
- ❌ **No off-screen gating.** Every row polls, including rows scrolled out of view in a long list.
- ⚠️ Background tabs *should* pause (React Query's `refetchIntervalInBackground` defaults to false, driven by `visibilitychange`), but a **visible-but-unfocused** window keeps polling — a second monitor left on the instances list polls indefinitely.

## Caveats — read before treating this as a pure regression

- Traffic **mix shifted** between the two windows (the prior window's top hosts included `compute.edge-stage`, absent from the current top 10), so some of the app-wide latency change is composition, not pure degradation.
- The clean apples-to-apples signals are the **same-endpoint** medians: `/Cluster/{id}` 51→471ms and `/HDBInstance/{id}/operation` 126→264ms.
- **Cause vs. effect is unresolved.** A 6.8× increase in polling load could be *causing* the control-plane latency, or a slower control plane could be inflating counts via retries. Worth checking against server-side metrics and any backend deploy in the window.
- Core Web Vitals are still dead per #1570, so LCP/FCP cannot corroborate the user-visible impact, and `pup` cannot compute percentiles — figures above are median/mean, **not p75-comparable** with #1405.

## Correlated signal: `AxiosError: timeout exceeded` is back

Quiet for two weeks, now returning:

| Window | Events |
| --- | --- |
| 24h → now | **6** (4 sessions) |
| 48h → 24h | 0 |
| 7d → 48h | 0 |
| 14d → 7d | 0 |
| 30d → 14d | 13 |

All 6 are on **Browse table** views (`/{org}/{cluster}/databases/data/`), routed through the global `queryClient` error handler. This is the same class as **#1371** (closed 2026-06-24), where the table view was likewise the top-affected view. Volume is low, but the two-week gap makes it a genuine early warning rather than ambient noise. Noting it here rather than reopening #1371.

## Suggested next steps

1. Correlate with server-side latency for `/Cluster/{id}` and `/HDBInstance/{id}/operation` to settle cause vs. effect, and check for a backend deploy in the window.
2. Bound the client cost on the instances list — the cheapest effective options being off-screen row gating (`IntersectionObserver`) and/or scaling the interval with row count.
3. Consider a single batched status endpoint for the list view instead of N per-row polls.
4. Watch the timeout count tomorrow; if it keeps climbing, reopen the #1371 thread.

Related: #1516 (per-tick analytics query cost), #1527 (500s on these same endpoints), #1569 / #1602 / #1601 (400s on them).

_Filed by the automated daily Datadog RUM review. No PII; customer org/cluster/instance identifiers and hostnames deliberately omitted._

Start with src/features/cluster/InstanceStatusCell.tsx and src/integrations/api/instance/status/getStatus.ts, then inspect the instances-list polling behavior and correlate the reported endpoints with server-side latency. Done means the list's status polling cost is bounded or appropriately gated, with the change verified against the request-volume and latency concerns described here.

Written by the indexing model from the issue text.