debezium / debezium/dbz

Implement ability to select columns from the join tables in snapshot custom statements [DBZ-4995]

Open
#616 0 comments 0 reactions 0 assignees View on GitHub
component/core-library migrated-from-jira type/enhancement
Dominant language
HTML
Stars
6
Forks
9
Avg merge
2d 19h
Merged PRs (30d)
1

Description

Migrated from [DBZ-4995](https://issues.redhat.com/browse/DBZ-4995)

The Debezium engine doesn't allow to select columns from the join tables into '{_}snapshot.select.statement.overrides{_}'. If we add a 'JOIN' operation to a custom snapshot statement and try to select columns from the join tables, we get an _IllegalArgumentException_ "{_}Column xxx not found in result set xxxx for table xxx. This might be caused by DBZ-4350.{_}"

Our use case is a postgresql database with an EAV (entity-attribute-value, open shema) data model. Our data tables: {_}objects{_}, {_}params{_}, {_}references{_}. We stream them using Debezium and also convert them into flat tables according to the _objects.object_type_ field, so the objects and their params/references must go into one flat table according to the value of the  _objects.object_type_ field.

During a 0-day-snapshot or full refill we got the situation that we get the entire _objects_ table from debezium, and then the entire _parameters_ table. Therefore, we must keep the complete '{_}objects{_}' table in memory in order to be able to route records from the {_}params{_}/{_}references{_} tables to the correct flat table. If we could use a snapshot statement like
{quote}select {*}o.object_type{*}, p.*

from params p

         left join objects o on o.object_id=p.object_id
{quote}
then we wouldn't have to worry about the amount of memory consumed.

I suggest adding the following setting to the connector config:
{quote}{color:#00875a}snapshot.select.statement.overrides...join.tables:"inventory.table1,customers.table2"{color}
{quote}
as addition to the existing settings:
{quote}{color:#403294}snapshot.select.statement.overrides{color}
{color:#403294}snapshot.select.statement.overrides..{color}
{quote}
 

If you agree with such an improvement, I can make and suggest an implementation to speed up the availability of this feature.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the existing snapshot.select.statement.overrides configuration and the Debezium engine's custom snapshot statement handling. Compare the proposed join.tables setting with the PostgreSQL objects, params, and references example; done means snapshot statements can select columns from joined tables without the reported IllegalArgumentException.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.