PyDebeziumAI - Real-Time CDC Integration for LangChain & LangGraph
- Dominant language
- HTML
- Stars
- 6
- Forks
- 8
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 1
Description
## Overview
This issue tracks the GSoC 2026 activity for building **PyDebeziumAI** - a first-class Python library that integrates directly with Debezium (via embedded mode or JSON-over-broker inputs) to enable real-time Retrieval-Augmented Generation (RAG) synchronization.
PyDebeziumAI listens to database WAL changes in real time and automatically updates vector database embeddings with sub-second latency, keeping vector stores and LLM application context automatically in sync with the relational database source of truth.
## Goals
### Ingestion & Logical Type Conversion Layer
- Implement canonical event model contracts (`DebeziumEventModel`, `DebeziumPayloadModel`, `DebeziumSchemaModel`).
- Build JPype type conversion engine to map Java/JVM `SourceRecord` schema types to native Python types.
- Implement `IngestionHandler` base structure and adapters for JSON mode (`ConnectMessageExtractor`) and Connect mode (`SourceRecordExtractor`).
### Transformation Layer
- Implement `DocumentBuilder` to map database event payloads into LangChain-compatible `Document` objects.
- Implement configurable projection policies (mapping select columns to page content vs metadata attributes).
- Implement deterministic ID generation strategies (e.g., `TablePkIdStrategy`) to guarantee correct upsert and delete semantics on vector search indexes.
- Implement metadata sanitization to coerce non-standard database data types (e.g., Decimals, binary bytes, dates) into vector-compatible primitives.
### Synchronization & Vector Adapters Layer
- Implement `SyncManager` to route database events to vector database endpoints and coordinate transactional updates/deletes.
- Support atomic update execution via sequential delete-then-upsert pipeline on vector databases.
- Support soft-delete handling (flagging deleted rows via `_is_deleted` in metadata).
- Implement reliable delivery mechanics: exponential backoff with randomized jitter retry configurations and Dead-Letter Queue (DLQ).
- Provide out-of-the-box, pluggable `VectorStoreAdapter` implementations for **Chroma**, **PGVector**, and **Milvus**.
### Retrievers & LLM Integrations
- Implement LangChain-compatible custom retrievers to query synchronized vector collections.
- Implement LangGraph-compatible node integrations allowing LLM agents to interact with reactive, real-time CDC updates.
- Package end-to-end RAG application examples (e.g., PostgreSQL -> Debezium -> PyDebeziumAI -> Chroma -> Chatbot).
## References
- Repository: https://github.com/debezium/debezium-ai-python
- Design Document: https://github.com/debezium/debezium-design-documents/pull/50
Contributor guide
Assessment
This issue has not been assessed yet.