debezium / debezium/dbz

PyDebeziumAI - Real-Time CDC Integration for LangChain & LangGraph

Open
#2,066 0 comments 0 reactions 1 assignee Claimed by @KMohnishM View on GitHub
component/ai-python type/task
Dominant language
HTML
Stars
6
Forks
8
Avg merge
2d 19h
Merged PRs (30d)
1

Description

## Overview

This issue tracks the GSoC 2026 activity for building **PyDebeziumAI** - a first-class Python library that integrates directly with Debezium (via embedded mode or JSON-over-broker inputs) to enable real-time Retrieval-Augmented Generation (RAG) synchronization.

PyDebeziumAI listens to database WAL changes in real time and automatically updates vector database embeddings with sub-second latency, keeping vector stores and LLM application context automatically in sync with the relational database source of truth.

## Goals

### Ingestion & Logical Type Conversion Layer
- Implement canonical event model contracts (`DebeziumEventModel`, `DebeziumPayloadModel`, `DebeziumSchemaModel`).
- Build JPype type conversion engine to map Java/JVM `SourceRecord` schema types to native Python types.
- Implement `IngestionHandler` base structure and adapters for JSON mode (`ConnectMessageExtractor`) and Connect mode (`SourceRecordExtractor`).

### Transformation Layer
- Implement `DocumentBuilder` to map database event payloads into LangChain-compatible `Document` objects.
- Implement configurable projection policies (mapping select columns to page content vs metadata attributes).
- Implement deterministic ID generation strategies (e.g., `TablePkIdStrategy`) to guarantee correct upsert and delete semantics on vector search indexes.
- Implement metadata sanitization to coerce non-standard database data types (e.g., Decimals, binary bytes, dates) into vector-compatible primitives.

### Synchronization & Vector Adapters Layer
- Implement `SyncManager` to route database events to vector database endpoints and coordinate transactional updates/deletes.
- Support atomic update execution via sequential delete-then-upsert pipeline on vector databases.
- Support soft-delete handling (flagging deleted rows via `_is_deleted` in metadata).
- Implement reliable delivery mechanics: exponential backoff with randomized jitter retry configurations and Dead-Letter Queue (DLQ).
- Provide out-of-the-box, pluggable `VectorStoreAdapter` implementations for **Chroma**, **PGVector**, and **Milvus**.

### Retrievers & LLM Integrations
- Implement LangChain-compatible custom retrievers to query synchronized vector collections.
- Implement LangGraph-compatible node integrations allowing LLM agents to interact with reactive, real-time CDC updates.
- Package end-to-end RAG application examples (e.g., PostgreSQL -> Debezium -> PyDebeziumAI -> Chroma -> Chatbot).

## References

- Repository: https://github.com/debezium/debezium-ai-python
- Design Document: https://github.com/debezium/debezium-design-documents/pull/50

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.