ClickHouse / ClickHouse/clickhouse-go

Support Avro as an insertion format

Open
#769 23 comments 0 reactions 0 assignees View on GitHub
discuss enhancement
Dominant language
Go
Stars
3.3k
Forks
680
Avg merge
2d 3h
Merged PRs (30d)
14

Description

Hi,

We have an Avro file which is directly consumed from Kafka. We don't want to deserialize Avro to Google struct, and then give it to Clickhouse client to serialize it in Clickhouse proto format.. Because it's extra work on the API.

What we imagine is, supporting Avro as an insertion format, so just passing Avro file content as []byte, and then Clickhouse client will send it. And we will define our insert query as "INSERT INTO table FORMAT AVRO".

Do you think this is something easy to implement?

PS: We are also working on this driver to provide a PR but at the moment we are learning how it works..

Contributor guide

Open the contributing guide

Research direction

Start by tracing how clickhouse-go handles insertion formats and []byte payloads; the issue identifies INSERT INTO table FORMAT AVRO as the intended entry point. Done means Avro file content can be passed directly without conversion through a Google struct and is accepted by ClickHouse using that format query.

Written by the indexing model from the issue text.

Assessment

Tech stack
clickhouse, go
Domain
databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
39/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.