ClickHouse / ClickHouse/clickhouse-rs
Expose api to allow freeze buffer in `Insert`, but not send
- Dominant language
- Rust
- Stars
- 559
- Forks
- 172
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 3
Description
### Use case
In our case, we will have many stateless writer nodes:
- Fetch logs from Kafka firstly
- Buffering logs in such nodes for some seconds
- Finally flush the logs into remote Clickhouse
The problem I encounter is:
- I want to call `take_and_prepare_chunk` to compress the logs data frequently to save memory
(log data is usually large, and such simple writer node usually only have small memory).
- However, I need to limit the actual `send` call, we can't write the remote Clickhouse too aggressively—we need to keep it healthy.
So I think, maybe expose such a api will be useful
### Describe the solution you'd like
It is just a small change, I can help it.
### Describe the alternatives you've considered
### Additional context
Contributor guide
Research direction
Start by reading the existing Insert flow and the take_and_prepare_chunk entry point. Define an API that permits preparing or freezing buffered data without sending it, while leaving the actual send under caller control; done means these operations can be performed independently.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- clickhouse, rust
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100