RocksDB: dead-workflow backfill materializes unbounded range before batch cutoff

Summary

In Rivet engine 2.3.17, gasoline_dead_wf_backfill requests the entire remaining workflow-data range without a row limit. The RocksDB adapter materializes that range before yielding the first entry. Consequently, the backfill's MAX_BACKFILLS = 1000 cutoff does not bound the underlying read allocation.

This report is based on public source inspection. It does not include a completed minimal runtime reproduction or claim a measured OOM threshold.

Affected code path

1. Backfill workflow declares MAX_BACKFILLS = 1000, runs a chunk, and persists its continuation after the activity returns.
2. backfill_dead_workflows scans the entire remaining workflow-data subspace using StreamingMode::WantAll, with no explicit range limit.
3. The default range limit is None.
4. The RocksDB stream adapter awaits the complete get_range result and collects a vector before yielding entries.
5. The RocksDB transaction task uses limit.unwrap_or(usize::MAX) and copies matching keys and values into a vector.
6. The backfill's workflow-count cutoff executes only after the underlying range has been materialized.

Lowering MAX_BACKFILLS alone therefore does not bound the initial allocation. Subsequent chunks can also re-materialize the remaining range. A sufficiently large workflow-data subspace could cause high memory consumption or OOM during this migration.

Introduction

The backfill and its startup dispatch were introduced by #5656, commit b92d85a820e00c41054c2804587d673fcde6ea1f (chore: sync engine changes from enterprise), listed in the 2.3.15 release notes. The backfill file is absent in 2.3.13 and this code path remains in 2.3.17.

Expected behavior

Memory consumption should be bounded by a page/batch budget rather than the size of the entire remaining workflow-data range. Progress should survive restart, and each workflow should only be classified after all required fields have been read.

Proposed reproduction / regression test

This is a suggested synthetic test, not a completed reproduction:

1. Populate a filesystem/RocksDB-backed test store with many workflow-data records and substantial synthetic payloads.
2. Leave the gasoline_dead_wf completion marker unset, as on an upgrade from 2.3.13.
3. Run the backfill while measuring peak memory and the number/bytes of key-value pairs materialized by each range fetch.
4. Check that the first fetch currently materializes the remaining range despite the 1,000-workflow processing cutoff.
5. After a fix, assert bounded fetches, correct dead-workflow indexing, and continuation correctness when one workflow spans multiple pages or execution restarts between pages.

Possible fix

Either paginate this backfill explicitly or implement genuinely bounded pagination in the RocksDB range-stream adapter. Preserve partial workflow aggregation across page boundaries (or an equivalent safe continuation scheme) so a row limit does not misclassify workflows or repeatedly rescan a single oversized workflow without making progress. A broader adapter change would need transaction/snapshot-semantics coverage.

Start with gasoline/src/db/kv/debug.rs, especially backfill_dead_workflows and its MAX_BACKFILLS cutoff, then trace range handling through universaldb/src/range_option.rs and the RocksDB transaction and transaction_task.rs files. Use the proposed filesystem/RocksDB test scenario to check materialized fetch size, partial workflow aggregation, dead-workflow indexing, and continuation across pages or restart. Done means memory-bounded fetching without misclassification or stalled progress.