[Feature Request]: Add Chroma vector database ingestion connector for RAG pipelines
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
### What would you like to happen?
## Motivation
Beam's Python ml/rag ingestion module currently supports Milvus and Qdrant (`sdks/python/apache_beam/ml/rag/ingestion/`). Chroma is one of the most widely-adopted open-source vector databases, especially for local and embedded RAG workloads, and is the missing piece in Beam's OSS vector-DB coverage.
## Proposal
Add `sdks/python/apache_beam/ml/rag/ingestion/chroma.py` implementing a `WriteToChroma` PTransform matching the shape of the existing Milvus and Qdrant connectors:
- Accepts `Chunk` records from the existing RAG chunking transforms
- Supports both `chromadb.HttpClient` (remote) and `chromadb.PersistentClient`
(embedded / local) client modes
- Configurable collection name, embedding + metadata mapping, batch size
- Standard retry/backoff
## Benefits
- Completes Beam's coverage of the OSS local-first vector-DB tier
- Chroma is Apache-2.0 licensed — no dependency-license concerns
- **`PersistentClient` runs in-process**, so integration tests do not need
Docker or a remote service — a meaningful advantage over Milvus/Qdrant
which require running servers in CI
## Testing plan
- Unit tests mirroring `qdrant_test.py`
- Integration test using `chromadb.PersistentClient` against a tmpdir
(no external service required)
- Docs + example consistent with existing Milvus / Qdrant docs
### Issue Priority
Priority: 2 (default / most feature requests should be filed as P2)
### Issue Components
- [x] Component: Python SDK
- [ ] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [x] Component: IO connector
- [ ] Component: Beam YAML
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Infrastructure
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Prism Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
Contributor guide
Assessment
This issue has not been assessed yet.