HarperFast / HarperFast/harper
RocksDB transaction conflict (40 retries exhausted) under concurrent writes to HNSW-indexed table
- Dominant language
- JavaScript
- Stars
- 89
- Forks
- 10
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 200
Description
## Problem
During a high-throughput ingestion job that writes ~2000 records to a table with an HNSW vector index on a Float[] column, ~1% of writes fail with:
```
After 40 retries, unable to commit transaction, transaction is in conflict with ongoing writes
```
The errors are non-deterministic — re-running the ingest succeeds for most of the previously-failed records — so this looks like a contention / retry-budget issue rather than a logical conflict.
## Repro
Verified on `harperfast/harper-pro-gpu:5.1.100` running on `ada-4000-medium.harper-gpu-testing.harperfabric.com` (Harper 5.1.0).
1. Define a table:
```graphql
type DocChunk @table {
id: ID @primaryKey
sourceUrl: String @indexed
content: String
embedding: [Float] @indexed(type: "HNSW", distance: "cosine")
# ... a few more scalar columns
}
```
2. Run an ingestion job that fans out ~8 concurrent `put()` calls, each writing a row whose `embedding` is a freshly-computed ~768-dim Float32 vector from `scope.models.embed()`.
3. Total ~2000 records. Expect a small but non-zero number to fail with the transaction-conflict error above.
In the run that produced the data on `gpu-c`: 2003 chunks attempted, 18 failed (~0.9%). Errors were spread across pages — not concentrated on any particular row — and remaining ~1985 chunks were successfully indexed.
## Storage engine
This database is using **RocksDB** (confirmed by the on-disk `.sst` / `MANIFEST` / `CURRENT` / `OPTIONS` files in `database/data/`). The retry-budget message language ("After N retries, unable to commit transaction") suggests this comes from Harper's optimistic-concurrency wrapper around RocksDB writes, presumably contention on the HNSW index's internal graph structure rather than on the row itself.
## Why this matters
For ingestion-style workloads (RAG index builds, bulk vector imports, document indexing) the user has to either:
- Serialize their writes (loses parallelism)
- Add their own retry loop on top of the one Harper already failed at
- Accept ~1% data loss on every batch
For applications doing live writes (per-document indexing as documents arrive) this is mostly fine — natural concurrency rarely hits 8-way contention on the same index. But the failure mode is silent enough that a batch script could lose records without noticing.
## Suggested directions
- Raise the retry budget for HNSW-indexed columns specifically (40 may be too low under sustained fan-out on the index lock).
- Add exponential backoff / jitter between retries — if all 8 writers retry at the same wall-clock cadence they keep colliding.
- Surface a clearer error / consider auto-batching: if the user is consistently hitting the retry ceiling, the engine could spool writes into a smaller-concurrency batch automatically.
- Document the concurrency ceiling for indexed-column writes so library authors building ingestion loops can pick a safe `concurrency` from the start.
## Workaround
Drop the ingestion concurrency from 8 → 2. Hasn't been tested but I'll confirm if I get a chance. Re-ingestion is idempotent in the demo app so the alternative ("rerun until errorCount is 0") also works in practice but isn't great UX.
## Context
- Surfaced while building a docs.harperdb.io RAG demo on `agent-example-harper` (branch `kris/docs-rag`).
- Cross-references the model-analytics emission PR #779 — the new `model-embed` count metric will eventually let us correlate ingestion throughput against the retry-conflict rate from monitoring.
Contributor guide
Assessment
This issue has not been assessed yet.