apache / apache/beam

[Feature Request]: Add Chroma vector database ingestion connector for RAG pipelines

Open
#40,047 1 comment 0 reactions 1 assignee Claimed by @nitinware View on GitHub
io new feature P2 python
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

### What would you like to happen?

## Motivation

Beam's Python ml/rag ingestion module currently supports Milvus and Qdrant (`sdks/python/apache_beam/ml/rag/ingestion/`). Chroma is one of the most widely-adopted open-source vector databases, especially for local and embedded RAG workloads, and is the missing piece in Beam's OSS vector-DB coverage.

## Proposal

Add `sdks/python/apache_beam/ml/rag/ingestion/chroma.py` implementing a `WriteToChroma` PTransform matching the shape of the existing Milvus and Qdrant connectors:

- Accepts `Chunk` records from the existing RAG chunking transforms
- Supports both `chromadb.HttpClient` (remote) and `chromadb.PersistentClient`
(embedded / local) client modes
- Configurable collection name, embedding + metadata mapping, batch size
- Standard retry/backoff

## Benefits

- Completes Beam's coverage of the OSS local-first vector-DB tier
- Chroma is Apache-2.0 licensed — no dependency-license concerns
- **`PersistentClient` runs in-process**, so integration tests do not need
Docker or a remote service — a meaningful advantage over Milvus/Qdrant
which require running servers in CI

## Testing plan

- Unit tests mirroring `qdrant_test.py`
- Integration test using `chromadb.PersistentClient` against a tmpdir
(no external service required)
- Docs + example consistent with existing Milvus / Qdrant docs

### Issue Priority

Priority: 2 (default / most feature requests should be filed as P2)

### Issue Components

- [x] Component: Python SDK
- [ ] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [x] Component: IO connector
- [ ] Component: Beam YAML
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Infrastructure
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Prism Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.