chroma-core / chroma-core/chroma
HNSW Rust loader segfaults (exit 139) when index_metadata.pickle contains dimensionality=None instead of raising Python exception
- Dominant language
- Rust
- Stars
- 29.3k
- Forks
- 2.5k
- Avg merge
- 1d 4h
- Merged PRs (30d)
- 38
Description
## Summary
When `index_metadata.pickle` for a persisted HNSW segment contains `dimensionality: None` (instead of an integer), `chromadb_rust_bindings.abi3.so` crashes with a null pointer dereference (KERN_INVALID_ADDRESS) on the first `collection.query()` call, rather than raising a Python exception. This makes the root cause impossible to diagnose without a crash reporter.
## Environment
- ChromaDB: 1.5.8
- Python: 3.13.12
- OS: macOS 15.x ARM64 (Apple Silicon, Darwin 25.4.0)
- Embedding function: `DefaultEmbeddingFunction` (all-MiniLM-L6-v2, 384 dims)
## Steps to Reproduce
1. Create or obtain a persisted ChromaDB collection with a large HNSW segment (e.g. 267K+ vectors)
2. Corrupt the segment's `index_metadata.pickle` to have `dimensionality: None`:
```python
import pickle
path = "~/.mempalace/palace//index_metadata.pickle"
with open(path, "rb") as f:
meta = pickle.load(f)
meta["dimensionality"] = None
with open(path, "wb") as f:
pickle.dump(meta, f)
```
3. Open the collection and run a query:
```python
import chromadb
client = chromadb.PersistentClient(path="...")
col = client.get_collection("my_collection")
col.query(query_texts=["test"], n_results=1)
```
4. Process exits with code 139
## Expected Behaviour
ChromaDB should validate the pickle contents before passing them to the Rust HNSW loader and raise a descriptive `ValueError` or `RuntimeError` such as:
```
ValueError: HNSW segment : index_metadata.pickle has dimensionality=None, expected int.
Delete the pickle file to allow ChromaDB to reload from binary, or reinitialise the segment.
```
## Actual Behaviour
Process crashes with exit code 139. macOS crash report shows:
```
Thread N crashed with ARM Thread State:
Exception Type: EXC_BAD_ACCESS (SIGSEGV)
Exception Subtype: KERN_INVALID_ADDRESS at 0x0000000000000000
Crashed Thread: chromadb_rust_bindings.abi3.so
```
## Workaround
Deleting the pickle entirely lets ChromaDB load the HNSW binary directly without crashing — the binary files (`data_level0.bin`, `header.bin`, etc.) are read correctly when no pickle is present. No data is lost.
## Additional Notes
This was triggered by a third-party tool (MemPalace) writing `None` into the pickle during a large mining run. The ChromaDB crash makes it very hard for users to diagnose the problem — a Python-level validation check before the Rust boundary would make this class of issue immediately self-diagnosing.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at the collection.query() path and the chromadb_rust_bindings.abi3.so HNSW loader boundary, then trace how index_metadata.pickle is read before the Rust call. Reproduce with dimensionality=None and verify that a descriptive Python exception is raised instead of process exit 139; also cover the malformed metadata case with a regression test if the existing test location is found.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100