ClickHouse / ClickHouse/clickhouse-go
Retriable/reusable batches
- Dominant language
- Go
- Stars
- 3.3k
- Forks
- 680
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 14
Description
Somewhat related to #529
**Suggestion.**
Make driver.Batch retriable and/or reusable after Send() - both for the same, or for another connection.
As far as I understood from the code, driver.Batch is tied to the specific connection (guess it is fetching the column types etc from CH), however, it would be tremendously useful to be able to reuse the batch after it was sent once.
**Use case**
* Retrying the same batch on the same server in case of some errors
* Retrying the same batch on the *different* server (to the same replicated table)
* Implementing majority writes for better write durability
Right now to implement those features, client has to keep the data for the batch somewhere else, which is not only inconvenient (especially in streaming-style use-cases, where client itself does not have access to all of the data needed to be inserted to CH), but essentially doubles the memory footprint.
Contributor guide
Research direction
Start by reading the driver.Batch lifecycle and the Send() behavior described in the issue, then review related issue #529. Determine how batch state and connection-specific metadata are handled. Done means defining and implementing a clear way to retry or reuse a sent batch on the same or another ClickHouse connection without requiring callers to retain all inserted data.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100