ClickHouse / ClickHouse/clickhouse-rs

Expose api to allow freeze buffer in `Insert`, but not send

Open
#233 5 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
559
Forks
172
Avg merge
1d 19h
Merged PRs (30d)
3

Description

### Use case
In our case, we will have many stateless writer nodes:
- Fetch logs from Kafka firstly
- Buffering logs in such nodes for some seconds
- Finally flush the logs into remote Clickhouse

The problem I encounter is:
- I want to call `take_and_prepare_chunk` to compress the logs data frequently to save memory
(log data is usually large, and such simple writer node usually only have small memory).

- However, I need to limit the actual `send` call, we can't write the remote Clickhouse too aggressively—we need to keep it healthy.

So I think, maybe expose such a api will be useful

### Describe the solution you'd like
It is just a small change, I can help it.

### Describe the alternatives you've considered

### Additional context

Contributor guide

Open the contributing guide

Research direction

Start by reading the existing Insert flow and the take_and_prepare_chunk entry point. Define an API that permits preparing or freezing buffered data without sending it, while leaving the actual send under caller control; done means these operations can be performed independently.

Written by the indexing model from the issue text.

Assessment

Tech stack
clickhouse, rust
Domain
databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.