chroma-core / chroma-core/chroma
[Bug]: No public API to force HNSW persistence: index stays WAL-only until sync_threshold, causing unbounded replay on reopen
- Dominant language
- Rust
- Stars
- 29.3k
- Forks
- 2.5k
- Avg merge
- 1d 4h
- Merged PRs (30d)
- 38
Description
### What happened?
## Summary
Follow-up to the discussion on PR #7047
(https://github.com/chroma-core/chroma/pull/7047#issuecomment-4998004637), which
targets the clean-shutdown case of #6975. As discussed there with @Raditya0902,
this is a distinct mechanism — steady-state behavior, no shutdown involved — so
filing it separately.
To be clear up front: **this is not a data-loss report.** We initially suspected
data loss and built a hard-kill repro to verify; the repro disproved it. What
remains is an operational cost issue and a missing primitive.
## Environment
chromadb 1.5.9, PersistentClient, macOS ARM64, Python 3.12,
hnsw:space=cosine, default hnsw:sync_threshold=1000.
(Production context: single collection, ~160k embeddings, 768-dim.)
## Verified behavior
1. `upsert()` is immediately durable: the record is committed to
`embeddings_queue` (SQLite) before returning. SIGKILL below sync_threshold
loses nothing — after reopen, both `get()` and semantic `query()` return the
record.
2. However, below sync_threshold the HNSW index itself is never persisted: the
VECTOR segment has no `max_seq_id` row and `data_level0.bin` stays empty.
Queries after reopen work only because Chroma **replays the WAL from the last
sync point on open**. We verified this directly: emptying `embeddings_queue`
on a copy before reopening yields `get()` → hit, `query()` → empty (the
on-disk index is empty).
3. `_persist()` fires only when cumulative write ops cross sync_threshold.
Queries, reads, and elapsed time never trigger it. A low-write instance
(e.g. ~300 upserts/day) crosses it every ~3 days; in production we measured
a 335-op window that persisted only when an unrelated nightly batch pushed
cumulative ops past the threshold.
4. Clean shutdown does not sync either (the #6975 family): in a long-lived
server the client lives in module globals and `System.stop()` is never
invoked, so "restart to flush" is not available. PR #7047 addresses the
clean-shutdown path; it does not cover steady-state control.
## Why it matters operationally
- **Unbounded replay on open**: every reopen (or first access after a crash)
replays all ops since the last sync. With a large backlog this cost lands on
the first `get_collection()`/`query()`, unpredictably.
- **No control over the sync point**: applications that want a known-good
on-disk index (before backups, snapshots, or maintenance windows) have no
supported way to obtain one.
- **The only workaround is a hack**: issuing idempotent re-upserts of existing
records purely to advance the write counter past sync_threshold. It works,
but it rewrites data solely as a side effect — clearly a stand-in for a
missing primitive.
## Proposal
A public `collection.flush()` (or client-level `persist()`) that:
- applies pending WAL ops to the HNSW segment and persists it (advancing the
VECTOR `max_seq_id`),
- is a no-op when there is nothing pending,
- is safe to call from the owning process at any time.
This complements #7047 (shutdown path) by giving applications steady-state
control over persistence and reopen cost.
Happy to share the repro scripts and more measurements — we run this in
production in [TAILOR](https://github.com/tailormemory/tailor), a self-hosted
personal memory system, with instrumentation around exactly this path.
### Versions
1.5.9
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at the PersistentClient and collection persistence path, then trace how embeddings_queue updates the VECTOR segment and its max_seq_id; data_level0.bin and the reopen replay behavior are the concrete checks described here. Validate the behavior with the reported reopen or SIGKILL reproduction. Done means a supported flush or persist operation applies pending WAL work, advances the persisted index state, and is a no-op when there is nothing pending.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust, sqlite
- Domain
- backend-api-design, databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100