ClickHouse / ClickHouse/clickhouse-go

Retriable/reusable batches

Open
#728 2 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Go
Stars
3.3k
Forks
680
Avg merge
2d 3h
Merged PRs (30d)
14

Description

Somewhat related to #529

**Suggestion.**
Make driver.Batch retriable and/or reusable after Send() - both for the same, or for another connection.
As far as I understood from the code, driver.Batch is tied to the specific connection (guess it is fetching the column types etc from CH), however, it would be tremendously useful to be able to reuse the batch after it was sent once.

**Use case**
* Retrying the same batch on the same server in case of some errors
* Retrying the same batch on the *different* server (to the same replicated table)
* Implementing majority writes for better write durability

Right now to implement those features, client has to keep the data for the batch somewhere else, which is not only inconvenient (especially in streaming-style use-cases, where client itself does not have access to all of the data needed to be inserted to CH), but essentially doubles the memory footprint.

Contributor guide

Open the contributing guide

Research direction

Start by reading the driver.Batch lifecycle and the Send() behavior described in the issue, then review related issue #529. Determine how batch state and connection-specific metadata are handled. Done means defining and implementing a clear way to retry or reuse a sent batch on the same or another ClickHouse connection without requiring callers to retain all inserted data.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.