commonwarexyz / commonwarexyz/constantinople

QMDB indexer cannot reconnect once a namespace exceeds u32::MAX operation locations

Open
#40 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
28
Forks
8
PR merge metrics
No merged PRs in 30d

Description

### Summary

Two related defects prevented the chain indexer's QMDB publisher from reconnecting after a restart, permanently halting ingestion. Both are fixed on branch `michael/qmdb-suffix-recovery` (commits `2139fa0`, `812146e`).

**1. u32 recovery ceiling.** On reconnect, `recover_writer_state` rebuilt writer state by fetching an operation-range checkpoint over the *entire* history (from location 0 to the watermark) and converting the location count to `u32`. Once a namespace exceeds `u32::MAX` (4,294,967,295) operations, the conversion fails on every attempt and the publisher can never reconnect — it spins in a 1s retry loop and ingestion stops. At ~100k txs/block the state namespace advances ~196k locations/block, crossing the ceiling around height ~21,800.

**2. Producer redelivery wedge.** A crash between a store commit and the durable-queue ack causes an already-uploaded block to be redelivered on restart. The producer validated writer cursors with no tolerance for the "already uploaded" case and retried forever on `WriterOutOfSync`.

### Environment

- `deploy.sh` config: c8a.4xlarge validators, external chain store, `--spammer-accounts 49152`, ~280k TPS, ~100k txs/block.

### Reproduction

- Ceiling: run a network under load until a QMDB namespace crosses ~4.29B operations, then restart the indexer. Observed error: `cannot initialize QMDB writer from 4572755410 operation locations`, repeating indefinitely.
- Wedge: kill the indexer in the window after a store commit but before the queue ack; on restart the redelivered block loops on `WriterOutOfSync ... writer is at operation N, but finalized block starts at N-k`.

### Fix

- **Suffix-window recovery**: fetch a single-operation checkpoint ending at the watermark instead of the full prefix; reconstruct writer peaks from the pinned nodes. Recovery cost drops from O(history) to O(log n) and the u32 ceiling is removed. Requires a pin-aware `reconstruct_peaks` in exoware-qmdb (carried as the `[patch]` exoware git-rev bump in `2139fa0`); no commonware API change needed.
- **Redelivery skip**: when both writer cursors are at or past the block's operation ranges, treat it as already-uploaded and skip capture instead of erroring.
- Validated: read-only recovery against a live store rebuilt a 4.57B-location watermark in seconds; reconnect covered by unit tests.

Contributor guide

Open the contributing guide

Research direction

Start with recover_writer_state and the reconnect unit tests described in the issue, then inspect the pin-aware reconstruct_peaks work in exoware-qmdb and the producer’s cursor validation path. Compare the behavior with commits 2139fa0 and 812146e; done means recovery works beyond u32::MAX locations and redelivered blocks no longer loop on WriterOutOfSync.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, databases
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.