ClickHouse / ClickHouse/clickhouse-rs
Support overriding buffer size(BUFFER_SIZE) in Insert
- Dominant language
- Rust
- Stars
- 559
- Forks
- 172
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 3
Description
### Use case
InsertFormatted supports buffered_with_capacity() for user provided buffer capacity, however this is not supported in Insert. Insert by default use 256 KB as the BUFFER_SIZE.
Use case is to support reading from Iggy stream, batch messages and insert into ClickHouse however if the batch size >= MIN_CHUNK_SIZE(depends on BUFFER_SIZE) the batch is flushed to server in the background. If there is an error then retry becomes complicated especially if the background batch was a success and failure on commit/force-commit. Support to commit/force_commit (only) on batch size simplifies the retry logic.
### Describe the solution you'd like
Similar to InsertFormatted support buffered_with_capacity() for Insert which can be used to override the default BUFFER_SIZE(256KB). This will enable overriding the default for using RowBinary/RowBinaryWithNamesAndTypes which cannot be used with InsertFormatted.
### Describe the alternatives you've considered
InsertFormatted does not support RowBinary/RowBinaryWithNamesAndTypes, it is mainly for self-describing input formats.
### Additional context
This will help with Refer# [Iggy ClickHouse Connector](https://github.com/apache/iggy/issues/2539) which I am implementing.
I can work on the PR to implement this as well.
Contributor guide
Research direction
Start by locating Insert and InsertFormatted and compare how buffered_with_capacity() is exposed. Check the behavior with RowBinary and RowBinaryWithNamesAndTypes, and confirm that the buffer capacity can be overridden so batching does not flush before commit or force_commit; done means the requested API works for these formats.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- clickhouse, rust
- Domain
- api, database
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100