chroma-core / chroma-core/chroma

DefaultEmbeddingFunction.__call__ constructs a new ONNXMiniLM_L6_V2 on every call (10× slowdown on repeated embeds)

Open
#6,941 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
29.3k
Forks
2.5k
Avg merge
1d 4h
Merged PRs (30d)
38

Description

## What happens

`chromadb/api/types.py` ships (chromadb 1.5.8):

```python
class DefaultEmbeddingFunction(EmbeddingFunction[Documents]):
def __call__(self, input: Documents) -> Embeddings:
from chromadb.utils.embedding_functions.onnx_mini_lm_l6_v2 import (
ONNXMiniLM_L6_V2,
)
return ONNXMiniLM_L6_V2()(input)
```

`DefaultEmbeddingFunction.__call__` constructs a fresh `ONNXMiniLM_L6_V2` every time it runs, triggering cold lazy-init of the tokenizer (~5ms) and the ONNX `InferenceSession` (~180ms) per invocation. Users whose workload hits embedding on a hot path (per-request retrieval, streaming ingest, RAG loops) pay ~200ms of avoidable tokenizer+model setup per call.

## Secondary issue — thread contention under concurrency

Even with a cached instance, the default `intra_op_num_threads=0` ("use all cores") causes severe context-switch thrashing when multiple concurrent queries hit the same session: each embed call fans out across all CPUs, producing worse-than-serial scaling.

## Measurement

AMD EPYC-Rome VPS, 16 cores, 300-drawer palace, chromadb 1.5.8:

| Scenario | Pre-fix | Post-fix (singleton + intra_op=1) | Speedup |
|---|---|---|---|
| `semantic_search` p95 (single-user; per `/search` call, one embed inside) | 412 ms | 95 ms | 4.3× |
| Composite 4-layer wake-up p95 | 768 ms | 106 ms | 7.2× |
| 4-concurrent scaling ratio vs single-call | 4.36× | 1.35× | 3.2× better |
| Ingest per drawer | 299 ms | 104 ms | 2.9× |

The 4-concurrent ratio is the one worth dwelling on: 4.36× means four parallel embed calls take **longer than running them in series** would — the default thread fan-out produces negative scaling.

## Proposed fix

Two parts:

1. Cache a single `ONNXMiniLM_L6_V2` instance at the class level (or via module-level singleton) so `DefaultEmbeddingFunction.__call__` routes every invocation through one instance.
2. Construct that instance with `intra_op_num_threads=1, inter_op_num_threads=1` so concurrent embeds parallelize across separate cores rather than thrashing contention on shared cores.

`InferenceSession.run()` is documented as thread-safe; the Rust-backed `tokenizers.Tokenizer` is thread-safe for `encode`. Vectors are byte-identical pre- and post-patch for identical input — no index rebuild required by users applying the change.

## Workaround downstream

A monkey-patch of `DefaultEmbeddingFunction.__call__` landing the singleton + `intra_op=1` settings produces the numbers above. Version-pinned to chromadb 1.5.8 because the patch depends on the exact call site. Discovered during MemPalace-sidecar integration work (xelauvas/xelasphere, April 2026); happy to share the patch module and full benchmark artifacts if useful.

## Version

chromadb 1.5.8 (reproducible back to at least 1.4.x based on the identical `__call__` body). Python 3.12.3, onnxruntime 1.24.4.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in chromadb/api/types.py at DefaultEmbeddingFunction.__call__, then inspect ONNXMiniLM_L6_V2 in chromadb/utils/embedding_functions/onnx_mini_lm_l6_v2.py, including its session options. Confirm the singleton and thread settings preserve byte-identical vectors, then verify repeated and concurrent embedding performance against the reported measurements.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning, performance
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
64/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.