HarperFast / HarperFast/studio

[RUM] PUT /Cluster 400s spiked to 39-of-40 in 24h (30d baseline 0–2/day) — one cluster, 3 sessions, bind and edit both blocked with no recovery

Open
#1,686 0 comments 0 reactions 0 assignees View on GitHub
rum
Dominant language
TypeScript
Stars
5
Forks
4
Avg merge
1d 8h
Merged PRs (30d)
40

Description

## What RUM shows

`PUT /Cluster/{id}` — the endpoint behind **Bind Domains**, **change region/plan**, and **change
version** — returned **400 on 39 of 40 calls in the last 24 hours**, against a 30-day baseline of
0–2 per day.

`@type:resource @resource.method:PUT @resource.url:*/Cluster/clu-*`, prod:

| window | 400 | 200 |
| --- | --- | --- |
| 24h → now | **39** | 1 |
| 48h → 24h | 0 | — |
| 72h → 48h | 0 | — |
| 96h → 72h | 0 | — |
| 120h → 96h | 0 | — |
| 144h → 120h | 1 | — |
| 168h → 144h | 2 | — |
| **30d total** | **42** | 27 (23 sessions) |

So **39 of the 42 PUT-400s in 30 days landed in one day**, and the 400s are concentrated on a
single cluster in a single organization, reached by 3 distinct sessions over ~6.5 hours. Every
400 also reaches Error Tracking as `AxiosError: Request failed with status code 400`,
fingerprint `v12.DFEB1F8CB54273FE5271CBBF98B28CFC` (40 events, `handling:handled`, via
`mutationCache.onError`).

## The users could not find a way around it

One session's attempts, by the view they were issued from (org/cluster ids omitted):

```
07:13:41 … 07:17:54 PUT /Cluster/{id} 400 × 11 from …/domains
07:23:23 POST /Cluster/{id}/container/stop 200
07:23:51 POST /HDBInstance/{id}/container/start 200
07:24:36 POST /Cluster/{id}/container/start 200
07:24:54 … 07:25:03 PUT /Cluster/{id} 400 × 3 from …/domains
07:38:34 PUT /Cluster/{id} 400 from …/domains
07:39:16 PUT /Cluster/{id} 400 from …/config/domains
07:44:12 … 07:44:16 PUT /Cluster/{id} 400 × 2 from …/edit
```

They restarted the cluster's containers mid-way and then tried the same operation from three
different pages. Nothing succeeded. An earlier session on the same cluster got a **200** at
01:23 and a **400** at 01:45, so the endpoint is not uniformly dead — something about the
request or the cluster's state changed in between.

## What is not yet known

RUM does not capture response bodies, so the *reason* for the 400 is not visible from the client
side. Candidates worth checking against central-manager logs for this cluster:

1. **A dangling domain id in `cluster.domainIds`.** `onBindClick` sends the union of the
cluster's existing ids and the newly selected ones —
[`Management.tsx:85-88`](https://github.com/HarperFast/studio/blob/stage/src/features/cluster/domains/Management.tsx#L85-L88):
```ts
setDomainIds({
clusterId: cluster.id,
domainIds: unique((cluster.domainIds?.slice() || []).concat(selectedDomainIds)),
generateDomainCerts: true,
});
```
If one id in `cluster.domainIds` no longer resolves server-side, every subsequent bind
re-sends it and can never succeed — the user is locked out of binding *anything* on that
cluster with no way to clear it from the UI. A `DELETE /Domain/{id}` → 204 did happen in this
org on the same day. This is the same whole-array-PUT fragility as #1626.
2. **Cluster state.** The 400s span a container stop/start, so a transient state may be part of
it — but the run continued 400ing for 20 minutes after the cluster came back up.
3. **A central-manager validation change** on the request body. `PUT /Cluster/{id}` carries three
different partial shapes from Studio — `{ domainIds, generateDomainCerts }`
([setDomainIdsOnCluster.ts:12](https://github.com/HarperFast/studio/blob/stage/src/features/clusters/mutations/setDomainIdsOnCluster.ts#L12)),
`{ regionPlans, skipGtmWait }` and `{ version, skipGtmWait }`
([useUpdateCluster.ts:14](https://github.com/HarperFast/studio/blob/stage/src/features/clusters/hooks/useUpdateCluster.ts#L14))
— and both call sites carry a `// TODO: API does not describe this endpoint permutation`
comment, so there is no schema pinning the contract either way.

Studio has not shipped in this window: prod is on **v2.169.3** for both the baseline and the
spike days (host-scoped `version` facet on `@view.url_host:fabric.harper.fast`), so this is not
a client rollout.

## Ask

1. Read the 400 bodies for this cluster in central-manager and say which validator rejects.
2. If it is (1) above, Studio needs to stop replaying ids it did not verify — send only the
delta, or reconcile `domainIds` against the org's live domain list before the PUT.
3. Either way, a 400 on this endpoint currently produces only a fading toast. Bind is a
multi-minute, DNS-dependent flow; the failure should persist on the page with the server's
reason, so a user is not left retrying across three pages.

## Related

- #1626 — unbinding one domain resets `generateDomainCerts` for the whole cluster (same
whole-array PUT).
- #1630 — the `/operation/` 404 poll from the same Domains view.

Contributor guide

Open the contributing guide

Research direction

Start with central-manager logs for the affected cluster to read the 400 response bodies and identify the rejecting validator. Then inspect Management.tsx:85-88, setDomainIdsOnCluster.ts:12, and useUpdateCluster.ts:14 to compare the PUT shapes and existing whole-array behavior. Done means the rejection cause is established, the confirmed client failure path is addressed, and the server reason remains visible on the page.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
api, frontend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.