History compaction tool [DBZ-747]
Nobody has claimed this yet.
- Dominant language
- HTML
- Stars
- 6
- Forks
- 9
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 1
Description
Migrated from [DBZ-747](https://issues.redhat.com/browse/DBZ-747)
Restarting a Debezium connector with a schema history topic can take a long time, depending on the number of schema history records to be recovered. E.g. there have been reports of this taking about 40 min for 100K schema history records. This can drastically sped up by discarding all old records, which are not needed any longer for restarting the connector from it's latest persisted offset.
To address this concern, there should be a tool which copies the existing DB history topic to a topic with a given new name, only copying those entries which are needed as per the connector's current offset. Here is an example:
{code:java}
offset | DDL | notes
------------------------------------------------------
001 | CREATE TABLE test ... |
002 | ALTER TABLE test ADD COLUMN foo ... |
003 | ALTER TABLE test ADD COLUMN bar ... | connector gets stopped at this offset
004 | ALTER TABLE test ADD COLUMN baz ... |
{code}
When the connector gets stopped at offset 003, upon restarting it currently will have to parse the statements 001, 002, 003, so to build up the full table schema valid at 003. The idea is to "compact" this information, persisting the table state valid as of 003, i.e. including the "foo" and "bar" columns, which then can be read back from this single event. After such compaction, the history topic would look like this (note the JSON representation should be used instead of DDL statements, which are used here just for illustration purposes):
{code:java}
offset | DDL | notes
------------------------------------------------------
003 | CREATE TABLE test (foo, bar) ... |
004 | ALTER TABLE test ADD COLUMN baz ... |
{code}
The tool would be used like this:
* Stop the connector
* Start the tool for copying the new, compacted history topic
** There should be an option which recreates the JSON-based representation ("tableChanges") by re-parsing the DDL field
* Point connector to the new DB history topic
* (Delete the old topic or keep it for analysis purposes)
* Re-start connector
Connector start-up times will be improved, as many old entries won't be processed again.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue names no files, tests, or code entry points. Start by tracing how connector offsets and schema-history topic records are read during restart, then determine how the current offset and tableChanges representation would be handled. Done means a tool copies only the required history into a new topic, with the specified JSON reconstruction option, and the connector can restart from it.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- json
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100