HarperFast / HarperFast/harper
App-dev tooling for throttling / backpressure self-service (umbrella)
- Dominant language
- JavaScript
- Stars
- 89
- Forks
- 10
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 200
Description
**Investigative / design umbrella.** Customer (PXG team) wants better self-service tooling around Harper's throttling and event-loop backpressure so app developers can react to it earlier and customize behavior, rather than only seeing it as warnings or errors in logs.
## Heart of the request (from Dawson)
- Capability to prioritize requests and relieve backpressure *when* throttling is a problem (e.g. deprio reads vs. writes).
- Earlier mechanisms to detect and deal with throttling.
- Better surfacing of problems in user code that cause throttling — long-running functions that aren't yielding.
- Warnings about read/write transaction queues *before* they become errors.
## Mechanisms to consider
- Easier profiling workflows on top of the 4.7 profiling work.
- Expose queues and warnings as programmatic events / observables.
- A `backpressure` hook/function the app dev can call to make app-side decisions (stop accepting new work, shut down a background loop, etc.).
- Expose a curated set of internal utilities (e.g. [`server/throttle.ts`](https://github.com/HarperDB/harperdb/blob/main/server/throttle.ts) — the throttle higher-order function) so app devs can use the same heuristics Harper itself uses, without re-implementing them incorrectly.
## Why this is Not Ready
The ask is broad enough that it needs decomposition before scheduling:
- Which surface? A `utils` global? Plugin-API additions on `scope`? Operations-API hooks?
- Which signals are stable enough to expose? (read txn queue depth, write queue depth, event-loop utilization, throttle counter…)
- How does the app dev's `backpressure` hook compose with Harper's internal throttling — does it pre-empt or coexist?
Decomposition into 2-4 concrete sub-issues is the right next step; this umbrella stays open as the design context.
## Related
- bari notes profiling improvements have already landed in 4.7.
- The `throttle` higher-order function referenced is in [`server/throttle.ts`](server/throttle.ts).
---
Tracked in Jira: [CORE-3021](https://harperdb.atlassian.net/browse/CORE-3021)
**Status:** Not Ready — needs decomposition
🤖 Filed by Claude on behalf of Kris.
Contributor guide
Assessment
This issue has not been assessed yet.