ClickHouse / ClickHouse/clickhouse-go
Improving `Tuple` write performance (open for discussion)
- Dominant language
- Go
- Stars
- 3.3k
- Forks
- 680
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 14
Description
## Observed
1. high number of slices being allocated for inserting `Tuple` type objects
## Details
Split off from https://github.com/ClickHouse/clickhouse-go/pull/1426 , to have a further discussion on the direction and its usefulness.
---
When using the `Tuple` type for inserting values, the values need an underlying `Array` / `Slice` type for ClickHouse Go to process them correctly. But depending on how data is internally presented before inserting, it might mean new slices need to be allocated. If these tuples then form the values of a map, the overhead of those allocations can quickly become significant.
Take for example the following struct:
```
type ValueWithTypeTuple struct {
V string
T int8
}
```
Currently to insert these, you need to do something like:
```
func (avt ValueWithTypeTuple) Value() (driver.Value, error) {
return []any{avt.V, avt.T}, nil
}
```
So I'm wondering if a specific Tuple type makes sense, with specific types for the most common lengths (like Tuple2, Tuple3 and Tuple4).
The method to insert these would then look something like this:
```
// Get implements the column.Tuple2 interface from ClickHouse. It returns two values that can be inserted as Tuple.
func (avt ValueWithTypeTuple) Get() (any, any) {
return &avt.V, int8(avt.T)
}
```
It then no longer requires the slice allocation, instead directly referencing the values.
I'd like to know any thoughts or other ideas to see if, and what kind of implementation might make sense.
### Environment
* [x] `clickhouse-go` version: `v2.30.0`
* [x] Interface: ClickHouse API
* [x] Go version: 1.23.4
* [x] Operating system: Linux
* [ ] ~ClickHouse version:~
* [ ] ~Is it a ClickHouse Cloud? No~
* [ ] ~ClickHouse Server non-default settings, if any:~
* [ ] ~`CREATE TABLE` statements for tables involved:~
* [ ] ~Sample data for all these tables, use [clickhouse-obfuscator](https://github.com/ClickHouse/ClickHouse/blob/master/programs/obfuscator/Obfuscator.cpp#L42-L80) if necessary~
Contributor guide
Research direction
Start by reviewing the discussion and the implementation context from pull request #1426, then trace how Tuple values are currently converted for insertion. Compare the proposed Tuple2/Tuple3/Tuple4 approach with other ways to avoid slice allocations. Done means an agreed implementation direction and scope, rather than a patch based on an unresolved design discussion.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100