redpanda-data / redpanda-data/connect

Add a `weaviate` output for writing vectors/objects to Weaviate

Open
#4,630 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
8.8k
Forks
969
Avg merge
1d 13h
Merged PRs (30d)
64

Description

Summary

Add a native output component for Weaviate, so pipelines
can write objects (with vectors and properties) directly into a Weaviate
collection. This would sit alongside the existing vector-database outputs
(pinecone, qdrant, cyborgdb).

Use case / motivation

Weaviate is one of the more widely used open-source vector databases, and it's a
common sink for RAG and semantic-search pipelines. Redpanda Connect is already a
natural fit for these workloads — openai_embeddings / other embedding
processors upstream, then a vector store downstream — but Weaviate users
currently have to fall back to a generic http output and hand-roll batching,
auth, UUID handling, and error semantics.

A first-class weaviate output would let a full ingest → embed → upsert pipeline
be expressed in a single config file, the same way it already can be for Pinecone
and Qdrant.

Prior art / feasibility
  • Redpanda Connect already ships pinecone, qdrant, and cyborgdb outputs, so
    the component pattern and interface are well established.
  • Weaviate maintains an official Kafka Connect sink
    (weaviate/kafka-connect-weaviate) supporting INSERT/UPSERT/DELETE,
    bring-your-own-vector, and gRPC batching — so the semantics and batching model
    are already proven against Weaviate's API.
  • Weaviate provides an official Go client
    (github.com/weaviate/weaviate-go-client), which should make a native Go
    implementation straightforward and avoid any cgo/external-library concerns.
Proposed configuration (illustrative)

Roughly following the shape of the existing qdrant output:

output:
  weaviate:
    rest_url: "https://my-cluster.weaviate.network"
    grpc_url: "grpc-my-cluster.weaviate.network:443"    # optional, for gRPC batching
    api_key: "${WEAVIATE_API_KEY}"
    collection: "Documents"
    id: 'root = uuid_v5(this.doc_id, "namespace")'      # Bloblang -> deterministic UUID for upsert
    vector_mapping: 'root = this.embedding'             # bring-your-own-vector
    properties_mapping: |
      root.text = this.text
      root.source = this.source
    operation: "upsert"                                 # insert | upsert | delete
    batching:
      count: 100
      period: "1s"
    max_in_flight: 64
Additional considerations
  • Upsert semantics: support deterministic UUIDs (e.g. uuid_v5) so records
    are idempotent — important for at-least-once delivery and re-processing.
  • Bring-your-own-vector vs. server-side vectorization: allow an explicit
    vector_mapping, and optionally defer to a collection's configured vectorizer
    when no vector is supplied.
  • Delete support: a delete operation keyed by UUID would round out CDC-style
    sync use cases.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the existing pinecone, qdrant, and cyborgdb outputs to understand the component pattern and configuration conventions. Review the official Weaviate Go client and its batching, authentication, UUID, vector, property, and delete APIs. Done means a native weaviate output supports the proposed operations and mappings with documented configuration and coverage for success and error behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, data-engineering, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.