Preempted Rewrite strands its written data+index files as unverified orphans for 7 days — recurring conflicts grew table dir 26G→71G/day

Summary

When a Rewrite (compaction via table.optimize()) is preempted by a concurrent transaction and fails with the documented "Retryable commit conflict", the transaction has already written its full compacted data files and a complete new index generation to disk. Since these files belong to no committed version, the conservative cleanup path (delete_unverified=False) will not reclaim them for 7 days. Under a workload where such conflicts recur, the table directory grows essentially unboundedly — we measured 26G → 71G in a single day on a table whose live data is ~3.5G.

I understand the conflict itself is by design (the transaction spec classifies Rewrite-vs-Delete as retryable, and application-level retry is the intended remedy — #1597 tracks built-in retry, and automatic resolution was closed as not planned in #3068). This issue is not about the conflict; it is about the failed transaction's disk footprint.

Environment

• lancedb (Python) 0.30.1, bundled Rust core lance 3.0.1
• Local ext4 filesystem on NVMe, Linux
• Single service process, multiple async tasks: a continuous ingest pipeline (appends + periodic optimize()) plus webhook-triggered re-indexing that issues deletes against the same table
• Table: ~316k rows, ~3.5G live data, IVF_PQ index (~3.1G per generation)

What happened

Over one day, 8 optimize(cleanup_older_than=timedelta(minutes=5), delete_unverified=False) passes were preempted by concurrent Delete (and in one case Rewrite) transactions:

WARNING: vector store optimize failed: lance error: Retryable commit conflict for version 55770:
This Rewrite transaction was preempted by concurrent transaction Delete at version 55770.
Please retry., .../lance-3.0.1/src/io/commit/conflict_resolver.rs:670:34

Each failed pass left behind ~6.6G that no committed version references:

• a full compacted data file (~3.5G) in data/
• a complete new index generation (~3.1G auxiliary.idx + friends) in _indices/

End state after one day: table dir 71G (data/ 37G, _indices/ 35G), 207 version manifests, 173 index generation dirs — against ~3.5G of live data. Disk pressure aside, the recovery path is awkward: delete_unverified=False cleanup won't touch the orphans for 7 days, and delete_unverified=True is only safe with all writers quiesced (and there is an open race in that area, #3718), which is hard to arrange on a live service.

Why this feels worse than a "lost pass"

A preempted Rewrite is supposed to cost only the wasted compaction work. In practice it also costs (table size + index size) of disk per occurrence for 7 days. With compaction running after ingest passes (dozens of times/day) and any other writer issuing deletes, a busy day multiplies this: 8 conflicts ≈ +44G for us. The growth is invisible until you monitor the table directory size explicitly.

Proposal

A Rewrite transaction that aborts on a commit conflict knows exactly which files it has written. Options, roughly in order of preference:

1. Self-cleanup on abort: delete (or schedule verified deletion of) the transaction's own written files when the commit fails terminally. Since the files were never part of any committed version and the writer owns them, this seems safe without the 7-day quarantine.
2. Expose a targeted reclaim API: something like "clean up files written by my failed transaction(s)" that doesn't carry the general delete_unverified=True risk.
3. At minimum, document the disk-cost-per-failed-Rewrite so operators size their monitoring/retention accordingly.

Workaround we applied (as a datapoint)

• Application-level retry on retryable conflicts (checkout_latest + backoff, as in the error message's suggestion) — prevents most occurrences.
• Serializing delete/optimize transactions within our process behind a lock — removes the conflict source for in-process writers entirely.
• Raised cleanup_older_than from 5 to 15 minutes per the guidance around #3718.
• The stranded orphans themselves we are simply waiting out (7-day floor), since an aggressive delete_unverified=True sweep on a live service seemed riskier than the disk usage.

Happy to provide fuller logs or directory listings if useful.

Start by reading the conflict handling around src/io/commit/conflict_resolver.rs:670:34 and the cleanup behavior discussed with #3718. Trace how a preempted Rewrite records its written data and index files, then determine whether the proposed self-cleanup or a targeted reclaim path can be made safe. Done means failed Rewrite files no longer wait for broad unverified cleanup, or the disk-cost limitation is clearly documented.