ClickHouse / ClickHouse/ClickHouse
Allow specifying offset policy via SETTINGS for kafka engine
- Dominant language
- C++
- Stars
- 49.9k
- Forks
- 9k
- Avg merge
- 21h 32m
- Merged PRs (30d)
- 515
Description
### Company or project name
n/a
### Use case
When subscribing to a kafka topic, currently it seems the only way to set a policy for offset handling is via an XML config file for Clickhouse.
Choosing the policy feels like something that should be available as part of table creation. As it stands, it requires the policy be set before a table is created.
### Describe the solution you'd like
I'd like to be able to pass the `rdkafka` supported offset options when creating a consumer, this would also have the benefit of being visible in a `SHOW CREATE TABLE` too.
I'm more than happy to do the work to implement such a feature if the team are prepared to consider a submission.
### Describe alternatives you've considered
Alternatives are to just stick with the existing behaviour.
### Additional context
_No response_
Contributor guide
Research direction
Start by tracing Kafka engine table creation and the existing XML configuration path for rdkafka offset options. Check how CREATE TABLE settings are represented by SHOW CREATE TABLE, then define completion as accepting the offset policy through table SETTINGS, persisting it, displaying it, and applying it when the consumer is created.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, kafka, sql
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100