ClickHouse / ClickHouse/clickhouse-go

Expose struct column extraction helper for AppendStruct inserts

Open Beginner friendly
#1,874 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
3.3k
Forks
680
Avg merge
2d 3h
Merged PRs (30d)
14

Description

## Observed

1. Existing issue #587 documents that `AppendStruct` fails when a table contains columns that are absent from the Go struct, even when those columns have server-side `DEFAULT` expressions.
2. The current workaround is to explicitly list the inserted columns in `PrepareBatch`, for example `INSERT INTO events (id, name, timestamp)`, so ClickHouse can apply defaults to omitted columns.
3. That workaround is effective, but it duplicates the struct's `ch` tags in raw SQL. In wide or frequently evolving ingestion structs, the column list can drift from the struct definition.

## Expected behaviour

The driver should expose a small helper that extracts the ordered ClickHouse column names represented by a struct, so callers can build explicit insert column lists from the same source of truth used by `AppendStruct`.

This does not need to change `AppendStruct` semantics or add an implicit `DESCRIBE` round trip. It would make the documented workaround for #587 easier to use correctly.

## Code example

```go
package main

import (
"context"
"fmt"
"strings"

"github.com/ClickHouse/clickhouse-go/v2"
)

type Event struct {
ID uint64 `ch:"id"`
Name string `ch:"name"`
}

func insert(ctx context.Context, conn clickhouse.Conn, events []Event) error {
columns, err := clickhouse.StructColumns(Event{})
if err != nil {
return err
}

batch, err := conn.PrepareBatch(ctx, fmt.Sprintf(
"INSERT INTO events (%s)",
strings.Join(columns, ", "),
))
if err != nil {
return err
}
defer batch.Close()

for i := range events {
if err := batch.AppendStruct(&events[i]); err != nil {
return err
}
}
return batch.Send()
}
```

With a table like this, ClickHouse can apply `source` server-side because the generated insert statement only names `id` and `name`:

```sql
CREATE TABLE events (
id UInt64,
name String,
source String DEFAULT 'server default'
) ENGINE = MergeTree()
ORDER BY id;
```

## Error log

Current open-ended insert behaviour still fails as described in #587:

```log
clickhouse [AppendStruct]: missing destination name "source" in *main.Event
```

## Details

### Environment
* [x] `clickhouse-go` version: current `main` and released v2 versions with `AppendStruct`
* [x] Interface: ClickHouse API
* [x] Go version: not version-specific
* [x] Operating system: not OS-specific
* [x] ClickHouse version: any version where the table has `DEFAULT` columns
* [ ] Is it a ClickHouse Cloud?
* [ ] ClickHouse Server non-default settings, if any:
* [x] `CREATE TABLE` statements for tables involved: see code example above
* [ ] Sample data for all these tables, use [clickhouse-obfuscator](https://github.com/ClickHouse/ClickHouse/blob/master/programs/obfuscator/Obfuscator.cpp#L42-L80) if necessary

Related: #587

Contributor guide

Open the contributing guide

Research direction

Start with the existing AppendStruct entry point and the struct tag extraction it already uses, then review related issue #587 for the documented failure mode. Add a public helper that returns the ordered ClickHouse column names represented by a struct, and verify that callers can use it to build explicit insert columns without changing AppendStruct semantics.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, databases
Issue type
Feature
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
74/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.