HarperFast / HarperFast/studio

[RUM] 500s on instance/cluster /operation polling endpoints — ~15× step-up since ~2026-07-14

Open
#1,527 1 comment 0 reactions 0 assignees View on GitHub
rum
Dominant language
TypeScript
Stars
5
Forks
4
Avg merge
1d 8h
Merged PRs (30d)
40

Description

## Summary

Datadog RUM shows a **~15× step-up in first-party HTTP 500 responses on the instance/cluster operation endpoints**, beginning around **2026-07-14** and sustained since. These are `POST` requests Studio makes while polling instance/cluster state.

Surfaced by the daily automated RUM review (RUM app `f590deee-…`, last 24h).

## Datadog findings

**Resource 500s, POST, `provider: first party (harper.fast)`** — daily trend:

| Window (rel. to 2026-07-16) | 500s on `*/operation` |
| --- | --- |
| 72h → 48h ago | **8** |
| 48h → 24h ago | **119** |
| 24h → now | **116** |

- **Endpoints:** `POST /HDBInstance/{id}/operation` and `POST /Cluster/{id}/operation` (IDs redacted — customer resources).
- **Spread:** across ~8 distinct instances/clusters, so not a single bad resource.
- **Reach (last 24h):** 10 sessions, 6 users.
- **Server behavior:** fast failures (~84–240ms), i.e. the backend rejects quickly rather than timing out.
- **Client pattern:** bursty — a handful of sessions each produce a run of 500s, consistent with Studio's polling loop continuing to hammer the endpoint after a 500 (no visible backoff/circuit-break).

A correlated, smaller signal appears in the JS error stream over the same window: handled `AxiosError: Request failed with status code 500` from `ensureUserIsLoaded → loadUser` on the `/` view, trending **7 → 23 → 25** across the same three days. Both are first-party 500s that stepped up together ~2 days ago, which points at a **backend change around 2026-07-14** as the likely trigger.

## Why this is worth tracking

- The overall RUM error volume actually **dropped sharply** in the last 24h (1627 → 467 → 109 error events/day) as the Monaco worker-OOM fixes (#1499/#1407/#1406) landed — so this 500 regression stands out as the one thing trending the wrong way in an otherwise-improving window.
- Distinct from #1386 (that was a **401** on the same polling endpoint, now closed); this is a **500**.

## Suggested next steps

1. Confirm on the backend whether `/HDBInstance/{id}/operation` and `/Cluster/{id}/operation` began returning 500 around 2026-07-14 (deploy correlation).
2. Determine whether the 500s are expected for a resource state (e.g. stopped/errored instance) that Studio should handle gracefully rather than retry against.
3. If retries are the amplifier, add backoff / stop polling on repeated 5xx so a single unhealthy resource can't generate a burst of 500s per session.

_Filed by the automated daily Datadog RUM review. No PII or customer identifiers included; specific instance/cluster IDs intentionally omitted._

Contributor guide

Open the contributing guide

Research direction

Start by checking the POST /HDBInstance/{id}/operation and /Cluster/{id}/operation handlers and correlating their 500s with the backend deploy around 2026-07-14. Then inspect Studio's operation polling loop and the ensureUserIsLoaded → loadUser path. Done means identifying the regression and either fixing the server rejection or making repeated 5xx responses stop or back off, with coverage for the affected behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
api, frontend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.