Aiven-Open / Aiven-Open/bigquery-connector-for-apache-kafka

Config enableBatchLoad does not work correctly when original topic name was mutated in SMT

Open
#72 9 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
37
Forks
45
Avg merge
19h 50m
Merged PRs (30d)
5

Description

Following [my previous PR](https://github.com/Aiven-Open/bigquery-connector-for-apache-kafka/pull/70), when `enableBatchLoad` config is set, it doesn't work correctly when the topic name was mutated in `SinkRecord`.

I will create a separate PR if I have time to work on this. If you have any thoughts on implementation (for example, should we make the same handle for `SinkRecord.originalTopic` or approach it differently), please let me know. Ref to the problematic code is [here](https://github.com/Aiven-Open/bigquery-connector-for-apache-kafka/blob/0142bdac1bddbac74188715b0a5d984b8348fb7e/kcbq-connector/src/main/java/com/wepay/kafka/connect/bigquery/BigQuerySinkTask.java#L300).

By the way, what is the use case for having `enableBatchLoad` as list? As a user, I would expect this to be a boolean value, applied to all topics handled by the connector, not a subset - that would be much more convenient.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.