Aiven-Open / Aiven-Open/bigquery-connector-for-apache-kafka

Revisit threading model

未關閉
#12 0 則留言 0 個 reaction 已指派 1 人 已被 @Claudenw 認領 在 GitHub 檢視
主要語言
Java
星號
37
分支
45
平均合併
19 小時 50 分鐘
30 天內合併 PR
5

描述

Right now, for every batch of records delivered to a task in its `poll` method, and for every table that the connector will write to from that batch, a new thread is spun up. The `queueSize` and `threadPoolSize` properties provide some control over this behavior, and users can also tune the connector's consumer properties by setting, e.g., `consumer.override.max.poll.records`. However, tuning these properties in practice can be difficult (especially if batches are small), inefficient (especially when table creation/updates take place and there is a period where no writes to the table will succeed), and the overall threading model can lead to out-of-order delivery (which required special handling to overcome when implementing upsert/delete support on top of the insertAll API, and likely will again when implementing upsert/delete on top of the Storage Write API).

Instead, we may consider creating at most one thread at a time per table, or if that degrades performance too much, per upstream topic partition. In `poll`, tasks can terminate stale threads (e.g., ones that haven't received new records for some duration or number of consecutive polls), instantiate new threads for any tables/topic partitions that do not currently have one, and dispatch records to the appropriate corresponding threads.

For batch resizing logic (which is necessary when a group of records sent to BigQuery exceeds the maximum size), we would have to tweak some of the existing logic to allow batch sizes to go back up after being reduced (otherwise, a single abnormally-large record could get the batch size down to some really low number and then every subsequent write would have to be limited to that small batch size).

貢獻指南

這個儲存庫沒有索引到貢獻指南

評估

這個 Issue 還沒有評估資料。

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。