apache / apache/camel-quarkus

Add a sync ledger SPI for keeping RAG vector stores in sync with source documents

Open
#9,003 0 comments 0 reactions 1 assignee Claimed by @JiriOndrusek View on GitHub
Dominant language
Java
Stars
302
Forks
232
Avg merge
1d 22h
Merged PRs (30d)
108

Description

Vector stores cannot be enumerated for "what did I already ingest", and LangChain4j has no equivalent of LangChain-Python's indexing API (`RecordManager`/`cleanup`), so today any ingestion pipeline is either append-only (restarts re-embed the whole corpus, edits join their previous vectors instead of replacing them) or wipe-and-reload (loses everything between passes).

Upstream context showing this gap is real and repeatedly asked for, never fully closed:

- langchain4j/langchain4j#1085 — "Does langchain4j have indexing API?" (closed after only the removal-support half shipped; the record-manager/indexing API itself never followed)
- langchain4j/langchain4j#2931 — "Incremental update of the knowledge base" (open; maintainer confirms "there is currently no such functionality in LangChain4j, you need to track that yourself")
- quarkiverse/quarkus-langchain4j#473 — "Easy RAG: strategies to control whether data should be ingested or not" (closed; a checksum/diff re-ingestion idea was floated and called "awesome if we could do" but never built)

A `SyncLedger` SPI + `JdbcSyncLedger` JDBC implementation now exist on the `feature/sql-ladger` branch (`extensions-support/langchain4j`), with a behavioural contract test (`SyncLedgerContract`) verified against both H2 and real PostgreSQL (`integration-tests/langchain4j-sync-ledger`, JVM + native). It's currently experimental and internal (not yet wired into a user-facing extension) — this issue tracks turning it into a real PR and, longer term, wiring it into a `langchain4j-ingest` extension.

Note: this is deliberately a stopgap. If langchain4j ever ships its own record-manager/indexing API upstream (per langchain4j/langchain4j#1085 / langchain4j/langchain4j#2931 above), our `SyncLedger`/`JdbcSyncLedger` are candidates for removal in favor of an adapter over the upstream mechanism — `SyncLedger` is designed as a swappable SPI for exactly this reason, so this feature carries no long-term compatibility promise.

Co-Authored-By: Claude Sonnet 5

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.