Implement ability to select columns from the join tables in snapshot custom statements [DBZ-4995]
- Dominant language
- HTML
- Stars
- 6
- Forks
- 9
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 1
Description
Migrated from [DBZ-4995](https://issues.redhat.com/browse/DBZ-4995)
The Debezium engine doesn't allow to select columns from the join tables into '{_}snapshot.select.statement.overrides{_}'. If we add a 'JOIN' operation to a custom snapshot statement and try to select columns from the join tables, we get an _IllegalArgumentException_ "{_}Column xxx not found in result set xxxx for table xxx. This might be caused by DBZ-4350.{_}"
Our use case is a postgresql database with an EAV (entity-attribute-value, open shema) data model. Our data tables: {_}objects{_}, {_}params{_}, {_}references{_}. We stream them using Debezium and also convert them into flat tables according to the _objects.object_type_ field, so the objects and their params/references must go into one flat table according to the value of the _objects.object_type_ field.
During a 0-day-snapshot or full refill we got the situation that we get the entire _objects_ table from debezium, and then the entire _parameters_ table. Therefore, we must keep the complete '{_}objects{_}' table in memory in order to be able to route records from the {_}params{_}/{_}references{_} tables to the correct flat table. If we could use a snapshot statement like
{quote}select {*}o.object_type{*}, p.*
from params p
left join objects o on o.object_id=p.object_id
{quote}
then we wouldn't have to worry about the amount of memory consumed.
I suggest adding the following setting to the connector config:
{quote}{color:#00875a}snapshot.select.statement.overrides...join.tables:"inventory.table1,customers.table2"{color}
{quote}
as addition to the existing settings:
{quote}{color:#403294}snapshot.select.statement.overrides{color}
{color:#403294}snapshot.select.statement.overrides..{color}
{quote}
If you agree with such an improvement, I can make and suggest an implementation to speed up the availability of this feature.
Contributor guide
Research direction
Start by reviewing the existing snapshot.select.statement.overrides configuration and the Debezium engine's custom snapshot statement handling. Compare the proposed join.tables setting with the PostgreSQL objects, params, and references example; done means snapshot statements can select columns from joined tables without the reported IllegalArgumentException.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgresql
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100