HarperFast / HarperFast/studio
[RUM] New 400s on the direct-to-instance operations API (:9925) — a rejected poll re-fires every ~60s for 30 minutes
- Dominant language
- TypeScript
- Stars
- 5
- Forks
- 4
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 40
Description
## Summary
Datadog RUM shows a step-up in HTTP **400s on the direct-to-instance operations API** — `POST https://:9925/`. **26 in the last 24h against a ~1–3/day baseline.**
This is a **different code path from #1569**, which tracks 400s on the *proxied* `POST /HDBInstance/{id}/operation` on `fabric.harper.fast`. These go straight to the customer's instance host on port 9925, i.e. the non-`forceFabricConnect` client. Filing separately for that reason; they may well share a root cause.
Surfaced by the automated daily RUM review (app `f590deee-…`).
## Datadog findings
**`@type:resource @resource.status_code:400 @resource.url_path:"/" @resource.url_port:9925`**
| Window | 400s | Sessions |
| --- | --- | --- |
| 24h → now | **26** | 3 |
| 48h → 24h ago | 2 | 1 |
| 7d → 48h ago | 6 | 4 |
| 14d → 7d ago | 14 | 6 |
- **Reach:** 3 sessions across **3 distinct customer orgs / 4 instance hosts** (IDs and hostnames omitted — customer resources). So not one bad instance.
- **All on `v2.156.4`** (current prod).
- **Server behavior:** fast, uniform rejections — median ~**122ms**, `content-type: application/json`. A validator/parser saying no, not work that failed partway.
- The matching JS error stream is `AxiosError: Request failed with status code 400`, **26 events** — up from **2** in the prior 24h.
### The shape that matters
Three distinct clusters, and the repetition is the point:
1. **Applications page** — `/{org}/{cluster}/apps` — 8 errors, re-issuing the same rejected operation **every ~60s for ~31 minutes** (03:00 → 03:31 UTC). It never gave up.
2. **Instance overview** — `/{org}/{cluster}/instance/{id}` — **16 errors in 55 seconds** (10:59:23 → 11:00:19).
3. **Browse / data** — `/{org}/{cluster}/databases/data/` — 2 errors, 33s apart, against two different clusters in one session.
In every cluster the errors arrive in **near-simultaneous pairs** (~5–40ms apart), so two subscribers appear to be issuing the same rejected request.
RUM cannot see the request body, so the operation name is not directly observable.
## Why this is worth fixing
A 400 will be rejected identically forever, but nothing in the client treats it that way:
- `pollUnlessForbidden` ([src/react-query/pollUnlessForbidden.ts](https://github.com/HarperFast/studio/blob/stage/src/react-query/pollUnlessForbidden.ts)) is **403-only**, so the `refetchInterval` timer keeps firing.
- The retry predicate was also 403-only, so each doomed tick cost up to **4** requests on the callers using default exponential backoff.
- Each one is a `console.error` + a toast through the global handler in `queryClient.ts`.
**#1601 addresses the retry half** (stop retrying 400s) and deliberately leaves the poll timer running, because halting it would freeze a poll whose 400 came from state that is still settling. **The remaining question — should a *sustained* 400 stop the timer, and which call sites can safely do that — is still open**, and is the same question #1569 raises.
## Candidate root causes
Not confirmed. Because these are **direct-to-instance** calls (port 9925, not the fabric proxy), the most likely candidates are operations whose availability depends on the target's Harper build:
1. An operation the instance's Harper version doesn't implement, on a route that is nav-gated but not route-gated (a bookmark or back-nav mounts the component and starts its poll anyway).
2. `get_analytics`-style calls carrying metric names or `bucket_ms` that a stricter/older validator rejects.
3. The apps page's ~60s component/deployment poll sending an argument the instance rejects.
The ~60s cadence in cluster 1 is a useful fingerprint for whoever picks this up — it should identify the query directly.
## Suggested next steps
1. Identify the ~60s poll on the applications page and the burst source on the instance overview page; confirm the operation name server-side.
2. Version-gate the route (not just the nav link) if it turns out to be a capability mismatch.
3. Decide the poll-timer policy for a sustained 400 (see #1569 and #1601).
_Filed by the automated daily Datadog RUM review. No PII; customer org/cluster/instance identifiers and hostnames deliberately omitted._
Contributor guide
Research direction
Start by reading src/react-query/pollUnlessForbidden.ts and queryClient.ts, then trace the applications-page ~60s poll and the instance overview request source. Use the RUM timing patterns and server-side confirmation to identify the operations involved; done means the affected route or capability handling and sustained-400 poll policy are decided and reflected in the relevant call sites.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- frontend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100