ClickHouse / ClickHouse/clickhouse-rs

Support overriding buffer size(BUFFER_SIZE) in Insert

Open
#389 2 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
559
Forks
172
Avg merge
1d 19h
Merged PRs (30d)
3

Description

### Use case
InsertFormatted supports buffered_with_capacity() for user provided buffer capacity, however this is not supported in Insert. Insert by default use 256 KB as the BUFFER_SIZE.
Use case is to support reading from Iggy stream, batch messages and insert into ClickHouse however if the batch size >= MIN_CHUNK_SIZE(depends on BUFFER_SIZE) the batch is flushed to server in the background. If there is an error then retry becomes complicated especially if the background batch was a success and failure on commit/force-commit. Support to commit/force_commit (only) on batch size simplifies the retry logic.

### Describe the solution you'd like
Similar to InsertFormatted support buffered_with_capacity() for Insert which can be used to override the default BUFFER_SIZE(256KB). This will enable overriding the default for using RowBinary/RowBinaryWithNamesAndTypes which cannot be used with InsertFormatted.

### Describe the alternatives you've considered
InsertFormatted does not support RowBinary/RowBinaryWithNamesAndTypes, it is mainly for self-describing input formats.

### Additional context
This will help with Refer# [Iggy ClickHouse Connector](https://github.com/apache/iggy/issues/2539) which I am implementing.
I can work on the PR to implement this as well.

Contributor guide

Open the contributing guide

Research direction

Start by locating Insert and InsertFormatted and compare how buffered_with_capacity() is exposed. Check the behavior with RowBinary and RowBinaryWithNamesAndTypes, and confirm that the buffer capacity can be overridden so batching does not flush before commit or force_commit; done means the requested API works for these formats.

Written by the indexing model from the issue text.

Assessment

Tech stack
clickhouse, rust
Domain
api, database
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.