ClickHouse / ClickHouse/clickhouse-go

Improving `Tuple` write performance (open for discussion)

Open
#1,473 1 comment 0 reactions 0 assignees View on GitHub
enhancement performance
Dominant language
Go
Stars
3.3k
Forks
680
Avg merge
2d 3h
Merged PRs (30d)
14

Description

## Observed

1. high number of slices being allocated for inserting `Tuple` type objects

## Details

Split off from https://github.com/ClickHouse/clickhouse-go/pull/1426 , to have a further discussion on the direction and its usefulness.

---

When using the `Tuple` type for inserting values, the values need an underlying `Array` / `Slice` type for ClickHouse Go to process them correctly. But depending on how data is internally presented before inserting, it might mean new slices need to be allocated. If these tuples then form the values of a map, the overhead of those allocations can quickly become significant.

Take for example the following struct:

```
type ValueWithTypeTuple struct {
V string
T int8
}
```

Currently to insert these, you need to do something like:

```
func (avt ValueWithTypeTuple) Value() (driver.Value, error) {
return []any{avt.V, avt.T}, nil
}
```

So I'm wondering if a specific Tuple type makes sense, with specific types for the most common lengths (like Tuple2, Tuple3 and Tuple4).

The method to insert these would then look something like this:

```
// Get implements the column.Tuple2 interface from ClickHouse. It returns two values that can be inserted as Tuple.
func (avt ValueWithTypeTuple) Get() (any, any) {
return &avt.V, int8(avt.T)
}
```

It then no longer requires the slice allocation, instead directly referencing the values.

I'd like to know any thoughts or other ideas to see if, and what kind of implementation might make sense.

### Environment
* [x] `clickhouse-go` version: `v2.30.0`
* [x] Interface: ClickHouse API
* [x] Go version: 1.23.4
* [x] Operating system: Linux
* [ ] ~ClickHouse version:~
* [ ] ~Is it a ClickHouse Cloud? No~
* [ ] ~ClickHouse Server non-default settings, if any:~
* [ ] ~`CREATE TABLE` statements for tables involved:~
* [ ] ~Sample data for all these tables, use [clickhouse-obfuscator](https://github.com/ClickHouse/ClickHouse/blob/master/programs/obfuscator/Obfuscator.cpp#L42-L80) if necessary~

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the discussion and the implementation context from pull request #1426, then trace how Tuple values are currently converted for insertion. Compare the proposed Tuple2/Tuple3/Tuple4 approach with other ways to avoid slice allocations. Done means an agreed implementation direction and scope, rather than a patch based on an unresolved design discussion.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.