redpanda-data / redpanda-data/connect
Add a `weaviate` output for writing vectors/objects to Weaviate
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 8.8k
- Forks
- 969
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 64
Description
Summary
Add a native output component for Weaviate, so pipelines
can write objects (with vectors and properties) directly into a Weaviate
collection. This would sit alongside the existing vector-database outputs
(pinecone, qdrant, cyborgdb).
Use case / motivation
Weaviate is one of the more widely used open-source vector databases, and it's a
common sink for RAG and semantic-search pipelines. Redpanda Connect is already a
natural fit for these workloads — openai_embeddings / other embedding
processors upstream, then a vector store downstream — but Weaviate users
currently have to fall back to a generic http output and hand-roll batching,
auth, UUID handling, and error semantics.
A first-class weaviate output would let a full ingest → embed → upsert pipeline
be expressed in a single config file, the same way it already can be for Pinecone
and Qdrant.
Prior art / feasibility
- Redpanda Connect already ships
pinecone,qdrant, andcyborgdboutputs, so
the component pattern and interface are well established. - Weaviate maintains an official Kafka Connect sink
(weaviate/kafka-connect-weaviate) supporting INSERT/UPSERT/DELETE,
bring-your-own-vector, and gRPC batching — so the semantics and batching model
are already proven against Weaviate's API. - Weaviate provides an official Go client
(github.com/weaviate/weaviate-go-client), which should make a native Go
implementation straightforward and avoid any cgo/external-library concerns.
Proposed configuration (illustrative)
Roughly following the shape of the existing qdrant output:
output:
weaviate:
rest_url: "https://my-cluster.weaviate.network"
grpc_url: "grpc-my-cluster.weaviate.network:443" # optional, for gRPC batching
api_key: "${WEAVIATE_API_KEY}"
collection: "Documents"
id: 'root = uuid_v5(this.doc_id, "namespace")' # Bloblang -> deterministic UUID for upsert
vector_mapping: 'root = this.embedding' # bring-your-own-vector
properties_mapping: |
root.text = this.text
root.source = this.source
operation: "upsert" # insert | upsert | delete
batching:
count: 100
period: "1s"
max_in_flight: 64
Additional considerations
- Upsert semantics: support deterministic UUIDs (e.g.
uuid_v5) so records
are idempotent — important for at-least-once delivery and re-processing. - Bring-your-own-vector vs. server-side vectorization: allow an explicit
vector_mapping, and optionally defer to a collection's configured vectorizer
when no vector is supplied. - Delete support: a
deleteoperation keyed by UUID would round out CDC-style
sync use cases.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the existing pinecone, qdrant, and cyborgdb outputs to understand the component pattern and configuration conventions. Review the official Weaviate Go client and its batching, authentication, UUID, vector, property, and delete APIs. Done means a native weaviate output supports the proposed operations and mappings with documented configuration and coverage for success and error behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, data-engineering, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100