Aiven-Open / Aiven-Open/cloud-storage-connectors-for-apache-kafka

[File size flush inteverval] Best practices for reducing network costs

未關閉
#255 3 則留言 1 個 reaction 已指派 0 人 在 GitHub 檢視
GCS
主要語言
Java
星號
58
分支
39
平均合併
2 天 10 小時
30 天內合併 PR
5

描述

Hi all, hope everithing is fine.
Currently we are working on a feature that uses this connector to dump debezium CDC messages. Our deployed connector is currently working based on default values, and as far as I can get, the only ways to control flush intervals are based on time (offset.flush.interval.ms, which defaults to 60 secs) and num of records per flushed files (file.max.records, didn't see a default value, so I'm assuming it will write as many lines as possible to the file during the flush interval window).

Our current cost for writing these files is pretty large, so I was wondering if you'd have any bet practices to improve performance of the connectors in order to reduce network costs during the write to gcs.

Thanks for any help!
Cheers

貢獻指南

開啟貢獻指南

評估

這個 Issue 還沒有評估資料。

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。