Introduce a ChronicleQueue-based ChangeEventQueue implementation
- Dominant language
- HTML
- Stars
- 6
- Forks
- 8
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 1
Description
## Feature request or enhancement
_For feature requests or enhancements, please provide the following information:_
**Which use case/requirement will be addressed by the proposed feature?**
As a follow-up to DDD-36, we determined that the vast majority of the issues with Oracle occur during large commits, where there can easily be back pressure due to the overflooding of the `ChangeEventQueue` when we buffer entire transactions and emit them all at once.
Currently, users can optimize the default `ChangeEventQueue` implementation by raising the `max.queue.size`, but if they are already using off-heap stores like Ehcache or Infinipsan, they begin to trade off-heap storage for heap storage, which is still finite. So there should be solutions that can bridge this gap.
We should implement a new `debezium-storage-chronicle-queue` module that provides custom `ChangeEventQueue` implementations that can be wired in as a replacement for the default.
---
**Implementation ideas (optional)**
There should ideally be two implementations in this module, both that support ChronicleQueue.
1. An implementation that relies solely on `ChronicleQueue`, flushing to disk and reading from disk with no JVM heap store.
2. A hybrid implementation that uses heap up to `max.queue.size` and spills to `ChronicleQueue`.
Contributor guide
Assessment
This issue has not been assessed yet.