commonwarexyz / commonwarexyz/constantinople
QMDB indexer cannot reconnect once a namespace exceeds u32::MAX operation locations
- Dominant language
- Rust
- Stars
- 28
- Forks
- 8
- PR merge metrics
- No merged PRs in 30d
Description
### Summary
Two related defects prevented the chain indexer's QMDB publisher from reconnecting after a restart, permanently halting ingestion. Both are fixed on branch `michael/qmdb-suffix-recovery` (commits `2139fa0`, `812146e`).
**1. u32 recovery ceiling.** On reconnect, `recover_writer_state` rebuilt writer state by fetching an operation-range checkpoint over the *entire* history (from location 0 to the watermark) and converting the location count to `u32`. Once a namespace exceeds `u32::MAX` (4,294,967,295) operations, the conversion fails on every attempt and the publisher can never reconnect — it spins in a 1s retry loop and ingestion stops. At ~100k txs/block the state namespace advances ~196k locations/block, crossing the ceiling around height ~21,800.
**2. Producer redelivery wedge.** A crash between a store commit and the durable-queue ack causes an already-uploaded block to be redelivered on restart. The producer validated writer cursors with no tolerance for the "already uploaded" case and retried forever on `WriterOutOfSync`.
### Environment
- `deploy.sh` config: c8a.4xlarge validators, external chain store, `--spammer-accounts 49152`, ~280k TPS, ~100k txs/block.
### Reproduction
- Ceiling: run a network under load until a QMDB namespace crosses ~4.29B operations, then restart the indexer. Observed error: `cannot initialize QMDB writer from 4572755410 operation locations`, repeating indefinitely.
- Wedge: kill the indexer in the window after a store commit but before the queue ack; on restart the redelivered block loops on `WriterOutOfSync ... writer is at operation N, but finalized block starts at N-k`.
### Fix
- **Suffix-window recovery**: fetch a single-operation checkpoint ending at the watermark instead of the full prefix; reconstruct writer peaks from the pinned nodes. Recovery cost drops from O(history) to O(log n) and the u32 ceiling is removed. Requires a pin-aware `reconstruct_peaks` in exoware-qmdb (carried as the `[patch]` exoware git-rev bump in `2139fa0`); no commonware API change needed.
- **Redelivery skip**: when both writer cursors are at or past the block's operation ranges, treat it as already-uploaded and skip capture instead of erroring.
- Validated: read-only recovery against a live store rebuilt a 4.57B-location watermark in seconds; reconnect covered by unit tests.
Contributor guide
Research direction
Start with recover_writer_state and the reconnect unit tests described in the issue, then inspect the pin-aware reconstruct_peaks work in exoware-qmdb and the producer’s cursor validation path. Compare the behavior with commits 2139fa0 and 812146e; done means recovery works beyond u32::MAX locations and redelivered blocks no longer loop on WriterOutOfSync.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 20/100