ClickHouse / ClickHouse/ClickHouse

Allow specifying offset policy via SETTINGS for kafka engine

Open
#113,056 0 comments 0 reactions 0 assignees View on GitHub
comp-message-queues external feature
Dominant language
C++
Stars
49.9k
Forks
9k
Avg merge
21h 32m
Merged PRs (30d)
515

Description

### Company or project name

n/a

### Use case

When subscribing to a kafka topic, currently it seems the only way to set a policy for offset handling is via an XML config file for Clickhouse.

Choosing the policy feels like something that should be available as part of table creation. As it stands, it requires the policy be set before a table is created.

### Describe the solution you'd like

I'd like to be able to pass the `rdkafka` supported offset options when creating a consumer, this would also have the benefit of being visible in a `SHOW CREATE TABLE` too.

I'm more than happy to do the work to implement such a feature if the team are prepared to consider a submission.

### Describe alternatives you've considered

Alternatives are to just stick with the existing behaviour.

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by tracing Kafka engine table creation and the existing XML configuration path for rdkafka offset options. Check how CREATE TABLE settings are represented by SHOW CREATE TABLE, then define completion as accepting the offset policy through table SETTINGS, persisting it, displaying it, and applying it when the consumer is created.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, kafka, sql
Domain
data-engineering, databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.