[RUM] Direct-to-instance :9925 connectivity failures at 3.4× baseline (19 sessions, 11 orgs) — and the whole class is invisible in Error Tracking by design

## Summary

Direct-to-instance operations calls (`POST https://:9925/`) are failing at the
**network level** (`status_code:0`, i.e. the request never completed) at **1.93% of all :9925
calls in the last 24h — 218 failures across 19 sessions and 16 distinct instance hosts in 11
organizations.** The trailing-7d rate is **0.57%**, so this is ~3.4× baseline.

**69% of those failures (151 of 218) burn the full 60s client timeout** before the UI learns
anything, on queries whose poll interval is 5–10s.

Separately, and independent of whether the rate above is a Studio problem at all: **none of
these failures produce an Error Tracking issue.** `shouldKeepEvent` drops connectivity-class
failures unconditionally, so a fleet-wide reachability degradation is invisible to Error
Tracking and to any alert built on it. The only trace is `@resource.status_code:0`, which
nothing currently monitors. That is why a 3.4× elevation across 11 orgs surfaced only via a
manual review.

Surfaced by the automated daily RUM review (app `f590deee-…`).

## Datadog findings

`@type:resource @resource.url:*:9925* @resource.status_code:0`

| Window | :9925 calls | status-0 | rate | sessions |
| --- | --- | --- | --- | --- |
| 24h → now | 11,284 | **218** | **1.93%** | **19** |
| 7d → 24h ago | 27,616 | 158 | 0.57% | — |
| 14d → 7d ago | 44,888 | 485 | 1.08% | — |

Breadth in the 24h window: **16 distinct instance hosts, 11 organizations, 19 of 118 sessions
(16%)**. 214 of the 218 are `POST /`; the other 4 are `GET /health`.

Duration distribution of the 218 failures — the spike at 60s is the axios client timeout
(`src/config/getInstanceClient.ts:68`, `timeout: 60000`):

| Duration | Count |
| --- | --- |
| ~60s (timeout) | **151** |
| 8–30s | 22 |
| <2s | 44 |
| other | 1 |

## What the per-session signature does and does not show

Worst single session: **77 failures over ~23 minutes against one host.** Timeline (host and
session identifiers withheld):

```
16:40:31 → 16:40:53 23 × POST / → 200, 25–533ms (p50 253ms)
16:41:03 host goes dark
16:41:35 … bursts of 6–7 concurrent POSTs, each hanging 60s,
16:49:42 repeating every ~61s
16:51:33 7 × fail fast (13.7s) — host now actively refusing
17:03:55 13 × 60s hang
```

**I want to be explicit that this session does not demonstrate a Studio retry bug**, because
that was my first read and the data refuted it:

- The 23 successes all *precede* 16:41:03; they are not interleaved with the hangs. So the
host genuinely became unreachable and stayed unreachable — the "instance is down,
restarting, or unreachable" case that `shouldKeepEvent` and `pollUnlessForbidden` both
explicitly treat as an expected state, not a bug.
- The observed cadence is **self-limiting**: because React Query will not refetch a key while
a request for it is in flight, and each request occupies the full 60s timeout, the 5–10s
poll timers degrade to ~1 request per key per 60s. ~3.3 requests/min across ~7 polling
queries is not runaway amplification.
- I also checked, and **disproved**, a connection-pool-starvation theory: successful requests
to the *same host in the same session* completed in 253ms p50 / 533ms max, so the stalled
requests were not blocking siblings.

So the elevated rate is most likely an **instance/infrastructure reachability** signal rather
than a Studio code defect. It is filed here because Studio's RUM is currently the only place
it is visible at all.

## The observability gap (this part is a Studio-side defect)

`src/integrations/datadog/shouldKeepEvent.ts:122-129`:

```ts
// A request timeout is a connectivity-class failure, never a Studio bug: the
// instance, cluster, or backend was too slow to answer in time. …
// Backend latency is tracked server-side, not from the browser.
if (/timeout of \d+ms exceeded/i.test(message)) {
return false;
}
```

Confirmed empirically: `@type:error @error.message:*timeout*` returns **0 events in 24h**
against 218 resource-level failures.

The per-event rationale is sound — these did flood Error Tracking (#1371). The gap is that
dropping them left **no aggregate signal in its place**. "Backend latency is tracked
server-side" does not cover *client-observed reachability*: whether a browser can reach a
customer instance on :9925 is only observable from the browser.

## Open questions for a human

1. **Is a 60s timeout right for a 5–10s poll?** `getInstanceClient` is shared by polls and by
genuinely slow operations (large queries, deploys), so a blanket reduction would risk
breaking the latter. A per-call or per-query-type timeout is the obvious shape, but this
touches core shared behavior and I did not want to change it unilaterally. 69% of failures
currently cost a user 60s of indeterminate loading.
2. **Should there be a monitor on `@resource.status_code:0` for first-party hosts?** That is
the signal that would have caught this without a manual review, and it does not reintroduce
the #1371 error flood because it is a resource-rate monitor, not an Error Tracking issue.
3. **Is the underlying reachability drop known to infra?** 16 hosts across 11 orgs in 24h
suggests something broader than individual customers stopping instances, but I cannot
distinguish those two from RUM alone — the operation name lives in the POST body, which RUM
does not capture.

## Related, not duplicate

Same endpoint, different failure modes: #1602 (400s on :9925), #1527 (500s on proxied
`/operation`), #1630 (404s on `Cluster/{id}/operation`), #1603 (poll fan-out volume). None of
those cover `status_code:0` / timeouts, and all of them are visible in Error Tracking, which
this class is not.

Start with src/integrations/datadog/shouldKeepEvent.ts:122-129 and src/config/getInstanceClient.ts:68 to understand the current filtering and shared timeout behavior. Review the related RUM resource signal and existing tests or monitoring entry points, then clarify whether the accepted outcome is aggregate monitoring for status_code:0, timeout changes, or both; done should include an agreed scope and validation against the reported failures.

Written by the indexing model from the issue text.