HarperFast / HarperFast/harper

main is red: Integration Tests 3/6 fails on drop_table with 'Invalid column family specified in write batch' since #2492

Open
#2,496 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

## Summary

`Integration Tests 3/6 (Node.js v24)` is failing on `main` with a RocksDB error during `drop_table`. It has failed on every `main` run since `70387f7a9` ("Stop the RocksDB WriteBufferManager from stalling every writer by default", #2492) and passed on the commit before it.

## Evidence

Bisected by CI run, on `main`:

| Run | Head | Result |
|---|---|---|
| 33810995509 | `3adefa7a4` — Return each record once from indexed array-element scans | **success** |
| 33811401862 | `70387f7a9` — Stop the RocksDB WriteBufferManager from stalling every writer (#2492) | **failure** |
| 33811425108 | `bba79a810` — test(mqtt): promote QA-649 MQTT connect wedge regression anchor (#2488) | **failure** |

Same two tests, same order, in `Blob lifecycle`:

```
✖ drop_table BlobCache succeeds
Error: expected 200 "OK", got 500 "Internal Server Error"
✖ restart HTTP workers and create another blob for drop_schema test
Error: expected 200 "OK", got 500 "Internal Server Error"

[main/0] [error]: Error: Remove failed: Invalid argument: Invalid column family specified in write batch
```

The second failure looks like a cascade of the first: the table drop leaves the store in a state the following blob write cannot use.

## Why #2492 is the suspect

The failing operation is a RocksDB write batch against a dropped column family, and #2492 changes RocksDB write-buffer behaviour (`writeBufferManagerAllowStall`). A stall that previously serialized writers would have masked a race between a `drop_table` and an in-flight cross-worker write; removing the stall lets both proceed concurrently.

That is a hypothesis from the diff's subject and the error, not a verified root cause — I have not read #2492's diff or reproduced locally. It is possible #2492 is correct and merely *unmasked* an existing defect. #1381 ("Cross-worker write can race RocksDB table drop and poison catalog cleanup") describes almost exactly this shape and may be the same bug, now reachable by default.

## Impact

`main` is red, so every PR branched from or merged with it inherits the failure. Beyond CI: if the race is real rather than test-only, a `drop_table` concurrent with a write can leave a write batch referencing a dropped column family, which is a data-path error surfacing as a 500 rather than a handled conflict.

## Suggested next step

Confirm by re-running the failing suite at `3adefa7a4` and `70387f7a9`. If it reproduces, decide between reverting #2492 while the race is fixed properly, and fixing the drop/write ordering it exposes — the latter being what #1381 already asks for.

## Provenance

Found while investigating a CI failure on https://github.com/HarperFast/harper/pull/2476, which inherits it: shard 3/6 was green on that PR's four earlier heads and failed only once its base picked up `70387f7a9`. Nothing in that PR touches RocksDB or the blob path.

Contributor guide

Open the contributing guide

Research direction

Re-run Integration Tests 3/6 at commits 3adefa7a4 and 70387f7a9, focusing on the Blob lifecycle failures in drop_table. Read #2492 and the related #1381 issue, then trace the drop_table and concurrent write path to confirm whether the failure is a race. Done means the suite passes on main and the drop/write behavior is fixed or the regression is clearly isolated for a separate change.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, nodejs
Domain
backend, databases, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.