HarperFast / HarperFast/harper
Write-rejection guard: bulk deletes are exempt from the backpressure they create, and the rejecting thread does not recover
- Dominant language
- JavaScript
- Stars
- 89
- Forks
- 10
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 205
Description
## What happened
On a production cluster (harper-pro 5.2.x), an application component ran a bulk cleanup issuing parallel `Table.delete` batches (~50 deletes × ~6 cascading writes in flight). The outstanding-commit guard tripped:
> a commit has been outstanding for 47984ms (exceeds the 45000ms limit) ... started from Target.delete ... (deletes, and writes applied from a canonical source ... bypass this check)
Two behaviors compounded into an incident:
1. **The offending write class is exempt from the guard it trips.** Deletes (and canonical-source writes) bypass the check — so the bulk delete never slowed down, while every *other* application write on that thread was rejected with 503. The victims were unrelated hot-path writes: ~2,000 `POST` result submissions were discarded in 30 minutes. The guard punished everyone except the workload that created the pressure.
2. **The rejecting state never cleared on its own.** The thread kept 503ing application writes after the bulk delete finished; only a container restart recovered it. If there is a latch, it did not release when the outstanding commit resolved.
## Asks
- Either bulk deletes participate in the backpressure they create, or the guard targets the offending source rather than rejecting bystander writes on the thread.
- The per-thread rejection state should clear once the outstanding commit resolves, without a restart.
## Notes
The application side has since serialized its deletes (self-limiting), which avoids tripping the guard — this issue is about the guard's fairness and recovery, which bite any bulk-writing component. Timestamps, node logs, and the exact message are available on request.
Contributor guide
Research direction
Start by tracing the outstanding-commit guard from the Table.delete path and the rejecting state described in the issue. Reproduce the parallel bulk-delete scenario and verify that deletes are accounted for or rejection is scoped to the offending source, and that rejection clears after the outstanding commit resolves without a restart.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, nodejs
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100