HarperFast / HarperFast/studio

[RUM] New 400s on the direct-to-instance operations API (:9925) — a rejected poll re-fires every ~60s for 30 minutes

Open
#1,602 0 comments 0 reactions 0 assignees View on GitHub
rum
Dominant language
TypeScript
Stars
5
Forks
4
Avg merge
1d 8h
Merged PRs (30d)
40

Description

## Summary

Datadog RUM shows a step-up in HTTP **400s on the direct-to-instance operations API** — `POST https://:9925/`. **26 in the last 24h against a ~1–3/day baseline.**

This is a **different code path from #1569**, which tracks 400s on the *proxied* `POST /HDBInstance/{id}/operation` on `fabric.harper.fast`. These go straight to the customer's instance host on port 9925, i.e. the non-`forceFabricConnect` client. Filing separately for that reason; they may well share a root cause.

Surfaced by the automated daily RUM review (app `f590deee-…`).

## Datadog findings

**`@type:resource @resource.status_code:400 @resource.url_path:"/" @resource.url_port:9925`**

| Window | 400s | Sessions |
| --- | --- | --- |
| 24h → now | **26** | 3 |
| 48h → 24h ago | 2 | 1 |
| 7d → 48h ago | 6 | 4 |
| 14d → 7d ago | 14 | 6 |

- **Reach:** 3 sessions across **3 distinct customer orgs / 4 instance hosts** (IDs and hostnames omitted — customer resources). So not one bad instance.
- **All on `v2.156.4`** (current prod).
- **Server behavior:** fast, uniform rejections — median ~**122ms**, `content-type: application/json`. A validator/parser saying no, not work that failed partway.
- The matching JS error stream is `AxiosError: Request failed with status code 400`, **26 events** — up from **2** in the prior 24h.

### The shape that matters

Three distinct clusters, and the repetition is the point:

1. **Applications page** — `/{org}/{cluster}/apps` — 8 errors, re-issuing the same rejected operation **every ~60s for ~31 minutes** (03:00 → 03:31 UTC). It never gave up.
2. **Instance overview** — `/{org}/{cluster}/instance/{id}` — **16 errors in 55 seconds** (10:59:23 → 11:00:19).
3. **Browse / data** — `/{org}/{cluster}/databases/data/` — 2 errors, 33s apart, against two different clusters in one session.

In every cluster the errors arrive in **near-simultaneous pairs** (~5–40ms apart), so two subscribers appear to be issuing the same rejected request.

RUM cannot see the request body, so the operation name is not directly observable.

## Why this is worth fixing

A 400 will be rejected identically forever, but nothing in the client treats it that way:

- `pollUnlessForbidden` ([src/react-query/pollUnlessForbidden.ts](https://github.com/HarperFast/studio/blob/stage/src/react-query/pollUnlessForbidden.ts)) is **403-only**, so the `refetchInterval` timer keeps firing.
- The retry predicate was also 403-only, so each doomed tick cost up to **4** requests on the callers using default exponential backoff.
- Each one is a `console.error` + a toast through the global handler in `queryClient.ts`.

**#1601 addresses the retry half** (stop retrying 400s) and deliberately leaves the poll timer running, because halting it would freeze a poll whose 400 came from state that is still settling. **The remaining question — should a *sustained* 400 stop the timer, and which call sites can safely do that — is still open**, and is the same question #1569 raises.

## Candidate root causes

Not confirmed. Because these are **direct-to-instance** calls (port 9925, not the fabric proxy), the most likely candidates are operations whose availability depends on the target's Harper build:

1. An operation the instance's Harper version doesn't implement, on a route that is nav-gated but not route-gated (a bookmark or back-nav mounts the component and starts its poll anyway).
2. `get_analytics`-style calls carrying metric names or `bucket_ms` that a stricter/older validator rejects.
3. The apps page's ~60s component/deployment poll sending an argument the instance rejects.

The ~60s cadence in cluster 1 is a useful fingerprint for whoever picks this up — it should identify the query directly.

## Suggested next steps

1. Identify the ~60s poll on the applications page and the burst source on the instance overview page; confirm the operation name server-side.
2. Version-gate the route (not just the nav link) if it turns out to be a capability mismatch.
3. Decide the poll-timer policy for a sustained 400 (see #1569 and #1601).

_Filed by the automated daily Datadog RUM review. No PII; customer org/cluster/instance identifiers and hostnames deliberately omitted._

Contributor guide

Open the contributing guide

Research direction

Start by reading src/react-query/pollUnlessForbidden.ts and queryClient.ts, then trace the applications-page ~60s poll and the instance overview request source. Use the RUM timing patterns and server-side confirmation to identify the operations involved; done means the affected route or capability handling and sustained-400 poll policy are decided and reflected in the relevant call sites.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
frontend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.