Aiven-Open / Aiven-Open/bigquery-connector-for-apache-kafka

Revisit threading model

オープン
#12 コメント 0 件 リアクション 0 件 担当者 1 名 @Claudenw が担当を希望しています GitHub で見る
主要言語
Java
スター
37
フォーク
45
平均マージ
19時間 50分
マージ済み PR(30日)
5

説明

Right now, for every batch of records delivered to a task in its `poll` method, and for every table that the connector will write to from that batch, a new thread is spun up. The `queueSize` and `threadPoolSize` properties provide some control over this behavior, and users can also tune the connector's consumer properties by setting, e.g., `consumer.override.max.poll.records`. However, tuning these properties in practice can be difficult (especially if batches are small), inefficient (especially when table creation/updates take place and there is a period where no writes to the table will succeed), and the overall threading model can lead to out-of-order delivery (which required special handling to overcome when implementing upsert/delete support on top of the insertAll API, and likely will again when implementing upsert/delete on top of the Storage Write API).

Instead, we may consider creating at most one thread at a time per table, or if that degrades performance too much, per upstream topic partition. In `poll`, tasks can terminate stale threads (e.g., ones that haven't received new records for some duration or number of consecutive polls), instantiate new threads for any tables/topic partitions that do not currently have one, and dispatch records to the appropriate corresponding threads.

For batch resizing logic (which is necessary when a group of records sent to BigQuery exceeds the maximum size), we would have to tweak some of the existing logic to allow batch sizes to go back up after being reduced (otherwise, a single abnormally-large record could get the batch size down to some really low number and then every subsequent write would have to be limited to that small batch size).

コントリビューションガイド

このリポジトリのコントリビューションガイドは索引されていません

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。