Add a sync ledger SPI for keeping RAG vector stores in sync with source documents
- Dominant language
- Java
- Stars
- 302
- Forks
- 232
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 108
Description
Vector stores cannot be enumerated for "what did I already ingest", and LangChain4j has no equivalent of LangChain-Python's indexing API (`RecordManager`/`cleanup`), so today any ingestion pipeline is either append-only (restarts re-embed the whole corpus, edits join their previous vectors instead of replacing them) or wipe-and-reload (loses everything between passes).
Upstream context showing this gap is real and repeatedly asked for, never fully closed:
- langchain4j/langchain4j#1085 — "Does langchain4j have indexing API?" (closed after only the removal-support half shipped; the record-manager/indexing API itself never followed)
- langchain4j/langchain4j#2931 — "Incremental update of the knowledge base" (open; maintainer confirms "there is currently no such functionality in LangChain4j, you need to track that yourself")
- quarkiverse/quarkus-langchain4j#473 — "Easy RAG: strategies to control whether data should be ingested or not" (closed; a checksum/diff re-ingestion idea was floated and called "awesome if we could do" but never built)
A `SyncLedger` SPI + `JdbcSyncLedger` JDBC implementation now exist on the `feature/sql-ladger` branch (`extensions-support/langchain4j`), with a behavioural contract test (`SyncLedgerContract`) verified against both H2 and real PostgreSQL (`integration-tests/langchain4j-sync-ledger`, JVM + native). It's currently experimental and internal (not yet wired into a user-facing extension) — this issue tracks turning it into a real PR and, longer term, wiring it into a `langchain4j-ingest` extension.
Note: this is deliberately a stopgap. If langchain4j ever ships its own record-manager/indexing API upstream (per langchain4j/langchain4j#1085 / langchain4j/langchain4j#2931 above), our `SyncLedger`/`JdbcSyncLedger` are candidates for removal in favor of an adapter over the upstream mechanism — `SyncLedger` is designed as a swappable SPI for exactly this reason, so this feature carries no long-term compatibility promise.
Co-Authored-By: Claude Sonnet 5
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.