HarperFast / HarperFast/harper
main is red: Integration Tests 3/6 fails on drop_table with 'Invalid column family specified in write batch' since #2492
- Dominant language
- JavaScript
- Stars
- 89
- Forks
- 10
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 200
Description
## Summary
`Integration Tests 3/6 (Node.js v24)` is failing on `main` with a RocksDB error during `drop_table`. It has failed on every `main` run since `70387f7a9` ("Stop the RocksDB WriteBufferManager from stalling every writer by default", #2492) and passed on the commit before it.
## Evidence
Bisected by CI run, on `main`:
| Run | Head | Result |
|---|---|---|
| 33810995509 | `3adefa7a4` — Return each record once from indexed array-element scans | **success** |
| 33811401862 | `70387f7a9` — Stop the RocksDB WriteBufferManager from stalling every writer (#2492) | **failure** |
| 33811425108 | `bba79a810` — test(mqtt): promote QA-649 MQTT connect wedge regression anchor (#2488) | **failure** |
Same two tests, same order, in `Blob lifecycle`:
```
✖ drop_table BlobCache succeeds
Error: expected 200 "OK", got 500 "Internal Server Error"
✖ restart HTTP workers and create another blob for drop_schema test
Error: expected 200 "OK", got 500 "Internal Server Error"
[main/0] [error]: Error: Remove failed: Invalid argument: Invalid column family specified in write batch
```
The second failure looks like a cascade of the first: the table drop leaves the store in a state the following blob write cannot use.
## Why #2492 is the suspect
The failing operation is a RocksDB write batch against a dropped column family, and #2492 changes RocksDB write-buffer behaviour (`writeBufferManagerAllowStall`). A stall that previously serialized writers would have masked a race between a `drop_table` and an in-flight cross-worker write; removing the stall lets both proceed concurrently.
That is a hypothesis from the diff's subject and the error, not a verified root cause — I have not read #2492's diff or reproduced locally. It is possible #2492 is correct and merely *unmasked* an existing defect. #1381 ("Cross-worker write can race RocksDB table drop and poison catalog cleanup") describes almost exactly this shape and may be the same bug, now reachable by default.
## Impact
`main` is red, so every PR branched from or merged with it inherits the failure. Beyond CI: if the race is real rather than test-only, a `drop_table` concurrent with a write can leave a write batch referencing a dropped column family, which is a data-path error surfacing as a 500 rather than a handled conflict.
## Suggested next step
Confirm by re-running the failing suite at `3adefa7a4` and `70387f7a9`. If it reproduces, decide between reverting #2492 while the race is fixed properly, and fixing the drop/write ordering it exposes — the latter being what #1381 already asks for.
## Provenance
Found while investigating a CI failure on https://github.com/HarperFast/harper/pull/2476, which inherits it: shard 3/6 was green on that PR's four earlier heads and failed only once its base picked up `70387f7a9`. Nothing in that PR touches RocksDB or the blob path.
Contributor guide
Research direction
Re-run Integration Tests 3/6 at commits 3adefa7a4 and 70387f7a9, focusing on the Blob lifecycle failures in drop_table. Read #2492 and the related #1381 issue, then trace the drop_table and concurrent write path to confirm whether the failure is a race. Done means the suite passes on main and the drop/write behavior is fixed or the regression is clearly isolated for a separate change.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, nodejs
- Domain
- backend, databases, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100