[RUM] Cluster /operation 404s 71 → 1,399 (31% of the endpoint) — the ChallengeCertificate 5s poll can never stop because its 404 is treated as success

## Summary

`POST /Cluster/{id}/operation/` is now **31% failures (1,443 of 4,627 calls in 7d)**, up from **10% (135 of 1,379)** over the preceding 23 days. Essentially all of the increase is **404s: 71 → 1,399**.

These 404s raise no error and produce no toast, because `getSearchByValue` deliberately treats 404 as an empty result. That is why none of the existing poll guards (#1527, #1546, #1569, #1602, #1603) can ever engage on them — and why this has been invisible in Error Tracking while still costing ~1,400 doomed control-plane round-trips a week.

## The signature

Worst single session in the window: **993 × 404 over 112 unbroken minutes**, inter-request gaps pinned at **5.1s**, all against **one** cluster, **100% 404 across 335+ consecutive calls** — while other sessions in the same window got clean `200`s on other clusters. So this is a stable per-cluster condition, not a transient.

```
window calls 404 403 500 400 200 fail%
168h..now 4,627 1,399 19 8 15 3,149 31%
720h..168h 1,379 71 25 27 1 1,237 10%
```

Breadth is **flat** — 26 sessions / 83 views in 7d vs 23 sessions / 69 views in the prior 23d. What changed is volume *per affected session*: **3 → 54**. This is a latent defect whose cost scales with how long a tab stays open, not a code regression. One long-lived tab produced ~71% of the week's total.

Note `POST /HDBInstance/{id}/operation` has **zero** 404s across the same 30 days (2% failures, all 500/400/403/401). The 404s are cluster-path-only, which is what pins the source below.

## Mechanism

`useChallengeCertificates(cluster.id)` → `getChallengeCertificatesQueryOptions` polls every **5s**, unconditionally, whenever the cluster Domains table is mounted:

- `src/features/cluster/domains/queries/getChallengeCertificates.ts` — `refetchInterval: pollUnlessForbidden(5000)`, `enabled: !!clusterId`
- it calls `getSearchByValue` with `forceFabricConnect: true` on a `clu-` id, so `getInstanceClient` routes it to `apiClient.baseURL + /Cluster/{id}/operation` (`src/config/getInstanceClient.ts:58-59`)
- `getSearchByValue` POSTs `search_by_value` for `data.ChallengeCertificate` to `/` → the observed `POST /Cluster/{id}/operation/`

The 404 is swallowed on purpose (`src/integrations/api/instance/database/getSearchByValue.ts`):

```ts
{ validateStatus: (status) => status >= 200 && status < 400 || status === 404 }
if (response.status === 404) { return { data: [] }; }
```

A cluster with no `data.ChallengeCertificate` table therefore answers 404 on **every** tick, the query **succeeds** with `[]`, `query.state.error` is never set, and `pollUnlessForbidden` cannot stop the timer. The poll re-fires every 5 seconds for as long as the tab is open.

The only consumer of the data is `isGeneratingCert` in `src/features/cluster/domains/constants/tableDefinition.tsx:25-29`:

```ts
const isGeneratingCert = !!challengeCertificates?.some((cert) => !cert.issueDate || cert.inProgress);
```

So a 5s cadence exists solely to watch an in-progress certificate — and it runs at full rate precisely when there is **nothing to watch**.

## Why this needs a design decision, not a one-liner

Nothing invalidates the `[clusterId, 'ChallengeCertificate']` query key anywhere in the codebase. The 5s poll is currently the *only* way a newly-started challenge is discovered, so simply stopping or slowing the idle poll would delay the certificate-progress UI in the one flow that matters (bind domain with `generateDomainCerts`, via `setDomainIdsOnCluster`).

Options, roughly in increasing order of correctness:

1. **Idle backoff** — keep 5s while `isGeneratingCert`, drop to 30–60s otherwise. ~6–12× fewer doomed requests, but adds up to 60s discovery latency after starting a challenge.
2. **Invalidate on mutation, then idle backoff** — have `setDomainIdsOnCluster` invalidate `[clusterId, 'ChallengeCertificate']` on success so discovery is immediate, which makes an aggressive idle interval (or `false`) safe. Preferred.
3. **Distinguish "table absent" from "no rows"** — a 404 here means the table has never existed on that cluster; that cannot change until a challenge is created, which option 2 already signals. Treating it as a terminal state would let the poll stop outright.

Option 2 is the smallest change that removes the waste without regressing the cert flow.

## Verification notes

- All figures from RUM app `f590deee-…`, `@type:resource` aggregates (not event samples, so no group-by truncation).
- Attribution to `getChallengeCertificates` is by elimination: it is the only cluster-scoped 5s `POST` to `/Cluster/{id}/operation/`, and `getSearchByValue` is the only caller that tolerates a 404 instead of throwing. The 5.1s observed cadence and the cluster-only 404 distribution both match. Not confirmed against a live reproduction — the Domains page needs a cluster with a missing `ChallengeCertificate` table to exercise.
- No customer, org, cluster, instance or session identifiers are included above.

*Filed by the daily automated Datadog RUM review (2026-08-17).*

Start with src/features/cluster/domains/queries/getChallengeCertificates.ts and trace useChallengeCertificates, then inspect setDomainIdsOnCluster and src/integrations/api/instance/database/getSearchByValue.ts. Compare the polling and query-key behavior with tableDefinition.tsx; done means missing ChallengeCertificate tables no longer generate unbounded 5-second requests while certificate-progress updates remain discoverable.

Written by the indexing model from the issue text.