HarperFast / HarperFast/harper

Write-rejection guard: bulk deletes are exempt from the backpressure they create, and the rejecting thread does not recover

Open
#2,321 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 2h
Merged PRs (30d)
205

Description

## What happened

On a production cluster (harper-pro 5.2.x), an application component ran a bulk cleanup issuing parallel `Table.delete` batches (~50 deletes × ~6 cascading writes in flight). The outstanding-commit guard tripped:

> a commit has been outstanding for 47984ms (exceeds the 45000ms limit) ... started from Target.delete ... (deletes, and writes applied from a canonical source ... bypass this check)

Two behaviors compounded into an incident:

1. **The offending write class is exempt from the guard it trips.** Deletes (and canonical-source writes) bypass the check — so the bulk delete never slowed down, while every *other* application write on that thread was rejected with 503. The victims were unrelated hot-path writes: ~2,000 `POST` result submissions were discarded in 30 minutes. The guard punished everyone except the workload that created the pressure.
2. **The rejecting state never cleared on its own.** The thread kept 503ing application writes after the bulk delete finished; only a container restart recovered it. If there is a latch, it did not release when the outstanding commit resolved.

## Asks

- Either bulk deletes participate in the backpressure they create, or the guard targets the offending source rather than rejecting bystander writes on the thread.
- The per-thread rejection state should clear once the outstanding commit resolves, without a restart.

## Notes

The application side has since serialized its deletes (self-limiting), which avoids tripping the guard — this issue is about the guard's fairness and recovery, which bite any bulk-writing component. Timestamps, node logs, and the exact message are available on request.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the outstanding-commit guard from the Table.delete path and the rejecting state described in the issue. Reproduce the parallel bulk-delete scenario and verify that deletes are accounted for or rejection is scoped to the offending source, and that rejection clears after the outstanding commit resolves without a restart.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, nodejs
Domain
databases
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.