Aiven-Open / Aiven-Open/cloud-storage-connectors-for-apache-kafka

[File size flush inteverval] Best practices for reducing network costs

Aberta
#255 3 comentários 1 reação 0 responsáveis Ver no GitHub
GCS
Linguagem predominante
Java
Estrelas
58
Forks
39
Merge médio
2d 10h
PRs com merge (30d)
5

Descrição

Hi all, hope everithing is fine.
Currently we are working on a feature that uses this connector to dump debezium CDC messages. Our deployed connector is currently working based on default values, and as far as I can get, the only ways to control flush intervals are based on time (offset.flush.interval.ms, which defaults to 60 secs) and num of records per flushed files (file.max.records, didn't see a default value, so I'm assuming it will write as many lines as possible to the file during the flush interval window).

Our current cost for writing these files is pretty large, so I was wondering if you'd have any bet practices to improve performance of the connectors in order to reduce network costs during the write to gcs.

Thanks for any help!
Cheers

Guia de contribuição

Abrir o guia de contribuição

Avaliação

Esta issue ainda não foi avaliada.

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.