HarperFast / HarperFast/harper
Dropped table can be resurrected after cluster restart — drop is not a durable, replayable txn-log event
- Dominant language
- JavaScript
- Stars
- 89
- Forks
- 10
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 200
Description
## Summary
Dropping a table via the Ops API `drop_table` with `replicated: true` removes the table across the cluster, but after a cluster restart the table can come back **repopulated with pre-drop data**. The drop is not recorded as a durable, timestamped, replicated event, so nothing in the recovery/replication path knows the table was *intentionally* dropped at a point in time. Pre-drop records that still exist (in the database-level audit/transaction log locally, and/or on peers) get re-materialized once the table exists again.
Reported behavior (operator):
> Used Ops API `drop_table` with `replicated: true`, verified the table was dropped across all nodes, then restarted the cluster to recreate the table. Pre-v5, that order of operations produced a clean (empty) table cluster-wide. This time the table was recreated and **repopulated with whatever was locally in the logs for that table on each node.**
The operator explicitly does **not** want to drop the whole database — they want a cluster-wide drop + recreate that yields a clean slate for that one table.
## Why this happens (grounded in code; v5 core)
1. **`drop_table` replicates as an operation RPC, not a logged event.** `dropTable()` calls `harperBridge.dropTable()` + `dropTableMeta()`, then `server.replication.replicateOperation(dropTableObject)` — it broadcasts the *operation* to peers, who each run the same handler. See `dataLayer/schema.ts:208-214`. Nothing writes a "table dropped at version/timestamp T" entry anywhere durable.
2. **There is no drop/tombstone entry type in the audit (txn) log.** The audit entry types are PUT / PATCH / DELETE / MESSAGE / RELOCATE / INVALIDATE / STRUCTURES (`resources/auditStore.ts`). There is no `DROP_TABLE` marker.
3. **The audit/txn log is database-level and is not pruned of the table's history on drop.** `replayLogs()` reads a single `rootStore.auditStore` for the whole database, not a per-table log (`resources/replayLogs.ts:52-53`). Historical entries for the dropped table's `tableId` remain in that shared log.
4. **Recovery replay re-applies any entries whose `tableId` matches a currently-existing table.** `replayLogs()` builds `tableById` from the tables that exist *now* and replays every audit entry matching one of them (`resources/replayLogs.ts:42-72`, replay switch at `123-177`). It has no notion of "this table was dropped, suppress its pre-drop history."
5. **No drop timestamp = the drop loses every ordering race.** Because the drop isn't an ordered, replicated, replayable event, any path that still holds pre-drop data — local audit replay on boot, or replication catch-up/full-copy from a peer that dropped later or hadn't pruned — can resurrect the table's old contents once the table name/id exists again.
## Proposed direction
Make table drop a **first-class, durable, replicated, replayable event** rather than a fire-and-forget operation broadcast:
- Add a drop marker / tombstone (e.g. a new audit entry type such as `DROP_TABLE`) written to the transaction log **with a version/timestamp**, before the physical removal.
- Replicate the drop **as that logged event** (so it participates in the same ordering/catch-up guarantees as data), not only as an operation RPC.
- Honor the marker in both recovery (`replayLogs`) and replication catch-up: any record for that table older than the drop's timestamp must be suppressed/not resurrected. A subsequent legitimate recreate (newer timestamp) starts clean.
This gives the "clean slate for one table, cluster-wide" behavior the operator expects, without dropping the whole database.
## Open questions / to confirm with a repro
- **Exact resurrection vector.** `tableId` is assigned from a monotonic `NEXT_TABLE_ID` counter that is **not** rolled back on drop (`resources/databases.ts:571-579, 1043-1046`), so a recreated table likely gets a *new* id — which means local `replayLogs` (matched by `tableId`) would *not* match the old entries. That points to **replication catch-up from a peer** as the more likely vector here, rather than (or in addition to) local replay. Needs a controlled repro to confirm which path(s) fire.
- Whether the physical drop fully purges primary-store data on the RocksDB (v5) bridge, or leaves recoverable remnants.
- Behavior difference vs. pre-v5 that changed the observed outcome (operator notes pre-v5 produced a clean table).
## Repro sketch (to validate)
1. Multi-node cluster, table `dev.foo` with data, fully replicated.
2. `drop_table { schema: dev, table: foo }` with `replicated: true`; verify gone on all nodes.
3. Restart the cluster; recreate `dev.foo` (schema/component definition or API).
4. Observe whether `foo` is empty (expected) or repopulated with pre-drop rows (bug).
---
_Filed from an operator report; root-cause analysis is grounded in v5 core code but the precise resurrection path is not yet confirmed by a repro — see open questions._
Contributor guide
Assessment
This issue has not been assessed yet.