chroma-core / chroma-core/chroma
[Bug]: SIGSEGV in chromadb_rust_bindings HNSW walk (1.5.8) — deterministic under caller blocking-syscall timing
- Dominant language
- Rust
- Stars
- 29.3k
- Forks
- 2.5k
- Avg merge
- 1d 4h
- Merged PRs (30d)
- 38
Description
## Summary
`collection.count()` segfaults the Rust binding via a null neighbour pointer in the HNSW graph walk on 1.5.8, but only when the caller's main thread is blocked in a syscall. Same call from a bare Python script returns cleanly. Segment-rebuild repair doesn't fix it; re-adding every record from `chroma.sqlite3` to a fresh collection does.
## Cross-references
- **#6852**: same binary, same platform, same null-deref class. Filed 2026-04-14, no maintainer reply. This report adds 1.5.8 offsets, a deterministic GIL-timing repro, and the rebuild-doesn't-cure / SQL-replay-does-cure narrowing.
- **#6963**: same null-deref class on Linux x86_64.
- **#6895**: sibling crash one layer down in `hnswlib::repairConnectionsForUpdate`.
- **#6949**: same "Rust null-derefs on input Python should reject" pattern.
- **#1584** (closed): historical concurrent-write corruption thread cited by #6852 / #6895.
## Environment
- `chromadb==1.5.8` (current PyPI)
- Python 3.12.13
- macOS 15.x (Darwin 25.4.0), Apple Silicon (arm64)
- Persistent local backend; ~514k vectors at 384 dims, cosine
## Crash signature
```
Exception Type: EXC_BAD_ACCESS (SIGSEGV)
Exception Codes: KERN_INVALID_ADDRESS at 0x0000000000000000
```
Faulting thread is a background worker; main thread sits in `_pthread_cond_wait` waiting on the compactor.
Two crashes a day apart hit `+0x18c2fe8` and `+0x18c3008` inside `chromadb_rust_bindings.abi3.so`: same function, 32 bytes apart. Five frames repeat 3× before the deref:
```
+0x9b0fcc → +0x9eac80 → +0x9bbdec → +0x1bc6630 → +0x9b0de8
... (×3) ...
+0x18c2fe8 <-- null neighbour pointer dereference
```
Graph-traversal recursion. Bug class is "this slot's `neighbour_list` is null on entry to the walker," not a single bad branch.
## GIL / syscall timing ladder
Same `count()`, same on-disk palace, four outcomes:
| Calling context | Outcome | Time |
|---|---|---|
| `chromadb.PersistentClient(path=…).get_collection(…).count()` from a bare script | Returns | ~6 s |
| Same call from imported module code; main thread shares GIL with compactor | Hangs in `_pthread_cond_wait` | 90 s+ |
| **JSON-RPC server whose main loop blocks in `sys.stdin.readline()`** (syscall releases GIL) | **SIGSEGV** | **5-8 s, deterministic** |
| Same server with a sibling sqlite-WAL init removed (perturbs scheduling) | Hangs (no longer crashes) | indefinite |
The more uninterrupted runtime the compactor's worker gets, the further it walks the graph, the more likely it lands on the null slot. A blocking syscall on the main thread surrenders the GIL without sharing runtime. Scheduling perturbation hides the crash behind a hang.
## Faulthandler-captured Python frame
Faulthandler pinpoints `chromadb/api/rust.py:397` (`_count`) inside the user's `count()` call when the Rust binding crashes.
## What didn't fix it
A repair path that reads each record's vector from sqlite, rebuilds a fresh HNSW segment in-place via hnswlib, verifies via sampled top-1 self-query, and swaps the segment dir under the live palace: runs clean, and the rebuilt segment still has at least one slot with a null `neighbour_list`. Either the in-place rebuild has the same defect as the original write path, or some inputs produce null neighbours during the build itself.
## What did fix it
Reading every row out of `chroma.sqlite3` directly (raw SQL over `embeddings` / `embedding_metadata`) and re-adding to a freshly-created collection via `add()` produces a palace that does **not** crash under the same blocking-syscall timing. Same data, same dim, same metric. The difference is the segment is built by `collection.add(documents=…)` from scratch rather than by segment-level rebuild.
Points to (a) the segment file format / serialiser used by in-place rebuild, or (b) state the original segment retained across migrations that the SQL replay strips out.
## Ruled out
Storage backend wrappers, stdio fd-redirects, sqlite `seq_id` migration touches, `get_collection` vs `get_or_create_collection(metadata=…)`, and import chain. Bare `count()` on the same on-disk segment is sufficient under the right calling-thread timing; none of the above changes that.
## Asks
1. Does this match a known fix class in the segment-rebuild path or in the HNSW walker's neighbour-list invariant? If `rc/2026-04-24` lands a relevant null check, a pointer to the change would let downstream callers stop running the SQL replay.
2. A diagnostic that enumerates HNSW slots with null `neighbour_list` would let downstream users self-test segments before they hit the crash.
3. Rough window on the next PyPI release with relevant fixes, so downstream callers know whether to run the SQL replay or wait.
cc @tazarov @rescrv
---
`.ips` dumps, faulthandler tracebacks, bisection probes, and a synthesised repro available on request.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at chromadb/api/rust.py:397 and trace the Rust binding's HNSW walk and segment-rebuild path; the payload does not name the relevant Rust source file. Reproduce with the JSON-RPC server whose main loop blocks in sys.stdin.readline(), then compare it with SQL replay into a fresh collection. Done means identifying the null neighbour-list invariant failure and documenting a diagnostic or verified fix.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100