debezium / debezium/dbz

Introduce a ChronicleQueue-based ChangeEventQueue implementation

Open
#1,938 0 comments 0 reactions 0 assignees View on GitHub
component/core-library component/storage type/enhancement
Dominant language
HTML
Stars
6
Forks
8
Avg merge
2d 19h
Merged PRs (30d)
1

Description

## Feature request or enhancement

_For feature requests or enhancements, please provide the following information:_

**Which use case/requirement will be addressed by the proposed feature?**

As a follow-up to DDD-36, we determined that the vast majority of the issues with Oracle occur during large commits, where there can easily be back pressure due to the overflooding of the `ChangeEventQueue` when we buffer entire transactions and emit them all at once.

Currently, users can optimize the default `ChangeEventQueue` implementation by raising the `max.queue.size`, but if they are already using off-heap stores like Ehcache or Infinipsan, they begin to trade off-heap storage for heap storage, which is still finite. So there should be solutions that can bridge this gap.

We should implement a new `debezium-storage-chronicle-queue` module that provides custom `ChangeEventQueue` implementations that can be wired in as a replacement for the default.

---

**Implementation ideas (optional)**

There should ideally be two implementations in this module, both that support ChronicleQueue.

1. An implementation that relies solely on `ChronicleQueue`, flushing to disk and reading from disk with no JVM heap store.
2. A hybrid implementation that uses heap up to `max.queue.size` and spills to `ChronicleQueue`.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.