[RUM] New 400s on POST /HDBInstance/{id}/operation — ~41% of that endpoint's calls in one session, and a 400 never stops the poll

## Summary

Datadog RUM shows a **brand-new class of HTTP 400 on the instance operation endpoint**: `POST /HDBInstance/{id}/operation` returned **400 thirty times in the last 24h**, against **zero in each of the two preceding windows** (24–48h ago and 48h–7d ago). All 30 are on the current release **v2.154.3**.

This is a *different failure* from #1527 (500s on the same endpoints) — a 400 says the request itself was rejected as invalid, not that the instance failed to answer. Filing separately for that reason, but they may share a trigger.

Surfaced by the automated daily RUM review (app `f590deee-…`, last 24h).

## Datadog findings

**`@type:resource @resource.status_code:400 @resource.url_path_group:"/HDBInstance/?/operation"`**

| Window | 400s |
| --- | --- |
| 24h → now | **30** |
| 48h → 24h ago | 0 |
| 7d → 48h ago | 0 |

- **Reach:** 4 sessions, 4 users. Also 1 × 400 on `/Cluster/{id}/operation`.
- **First seen:** 2026-07-27 17:02 UTC. **Most recent:** 2026-07-28 11:08 UTC.
- **Server behavior:** fast, uniform rejections — 31–284ms, median **42ms** — with a **37-byte response body**. That's a validator/parse rejection, not work that failed partway.
- **Provider:** first party, `fabric.harper.fast`.
- **Not a sampling artifact.** `shouldKeepEvent` returns early for any `event.type !== 'error'`, so resource events are never filtered, and there has never been a 400 rule. The window-over-window comparison is apples to apples.
- **Traffic context:** 136 sessions in the last 24h vs 38 in the prior 24h (~3.6× more). The 400s are still new in absolute terms — the prior windows are *zero*, not merely lower.

### The shape that matters

The busiest session made **26 of the 30** — over ~16 minutes (01:15–01:31 UTC), all against a **single instance**. In that same session the same endpoint also returned **200 thirty-eight times**.

So roughly **41% of that session's operation calls failed while the rest succeeded.** A blanket auth or reachability problem would fail all of them. This points at **one specific operation in the polling mix being rejected** while its siblings are fine.

Other three sessions: 2, 1, and 1 occurrence.

## Why this is worth fixing

**A 400 never stops the poll.** `pollUnlessForbidden` (added in #1566, merged) is explicitly 403-only:

> `Only 403 stops the timer. 5xx, network failures, and timeouts are transient … and should keep polling so the UI self-heals.`

A 400 is *not* transient — a malformed or unsupported request will be rejected identically forever. But it currently gets:

- no poll stop — `refetchInterval` keeps firing on its timer, and
- React Query's **default `retry: 3`** (there are no `defaultOptions` in `src/react-query/queryClient.ts`), so each tick costs up to 4 doomed requests,
- plus a `console.error` + toast through the global error handler for each one.

That's the same amplifier #1566 was written to kill, one status code over.

## Candidate root causes

From reading the call sites — **not yet confirmed**, needs a backend/deploy correlation:

1. **`get_analytics` with a metric the instance's Harper build doesn't emit.** 22+ metric names are hard-coded in `status/analytics/pipeline/index.ts` (plus `table-size`, `mqtt-connections`, `ws-connections`), and the capability probe in `useAnalyticsCapability.ts` only proves that *one of four* metrics works — every other metric is sent unconditionally. The probe itself walks up to 4 metrics × `retry: 2` = up to 12 POSTs in a burst, and it treats a 4xx as "try the next metric". This is the highest-cardinality un-gated argument Studio sends, and it fits the partial-failure shape best.
2. **`get_status` from the cluster instances list.** `InstanceStatusCell.tsx` computes `statusPollEnabled` with **no Harper version check**, unlike `InstanceNavBar.tsx` which floors the same query at 4.6.0. It passes `forceFabricConnect: true`, so it always hits exactly this URL. On a pre-4.6 instance that 10s poll would 400 forever.
3. **Un-gated routes.** Nav links for deployments/secrets/certificates are version-gated, but the routes are not — a bookmark or back-nav mounts the component and starts its poll (`list_deployments` is an always-on 10s poll) against an instance whose Harper predates the feature.
4. **`bucket_ms`** is sent to every version on `get_analytics`; the code comment assumes older builds ignore unknown fields, which is untested against a strict validator.

RUM can't see the request body, so the operation name isn't directly observable — hence the list rather than a diagnosis.

## Suggested next steps

1. **Get the operation name.** Correlate server-side logs for `POST /HDBInstance/{id}/operation → 400` around 2026-07-28 01:15–01:31 UTC, or read the 37-byte body (likely `{"error":"Unknown operation…"}` or similar). That single fact collapses the candidate list.
2. **Generalize the poll stop to non-retryable 4xx.** Extend `pollUnlessForbidden`/`retryUnlessForbidden` from "403" to "400, 403, 404, 422" — a stable client error should stop the timer and skip retries exactly like a 403 does. Cheap, well-precedented by #1566, and it caps the damage regardless of which operation turns out to be at fault.
3. **Version-gate `get_status` in `InstanceStatusCell`** to match `InstanceNavBar`'s 4.6.0 floor.
4. **Consider gating each analytics metric** on the capability probe rather than assuming all 22+ exist on every build.
5. Optionally add a 400-on-operation-endpoint rule to `shouldKeepEvent` **only after** the cause is fixed — dropping it first would hide the signal.

## Related (do not dupe)

- **#1527** — 500s on the same instance/cluster `/operation` endpoints. Still active: 221 (`/HDBInstance`) + 137 (`/Cluster`) in the last 24h, ~1.6 per session, back to the rate seen 7 days ago after a quiet day.
- **#1546** — 403 step-up on these endpoints. Largely resolved by #1566: 403s across all resources fell **1049 → 65** window over window.
- **#1386** (closed) — 401 on the same polling endpoint.
- **#1566** (merged) — the 403 poll-stop this issue asks to generalize.

_Filed by the automated daily Datadog RUM review. No PII or customer identifiers included; instance/cluster IDs and user identities intentionally omitted._

Start by correlating the 400s with server logs or the 37-byte response body to identify the rejected operation. Then inspect pollUnlessForbidden/retryUnlessForbidden, status/analytics/pipeline/index.ts, useAnalyticsCapability.ts, InstanceStatusCell.tsx, InstanceNavBar.tsx, and src/react-query/queryClient.ts. Done means the offending request path is addressed and stable 4xx responses stop polling and retries without hiding the RUM signal.

Written by the indexing model from the issue text.