apache / apache/pinot

Handle change in default groupId behavior in kafka consumer

Open
#9,183 4 comments 0 reactions 0 assignees View on GitHub
good first issue
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

Starting kafka 2.2.0 [onwards](https://issues.apache.org/jira/browse/KAFKA-6774), when `group.id` is not specified in the kafka consumer config, it is set as `null`, instead of an empty string. Due to this change, even though auto-commit is enabled, it doesn't show up in the consumer offsets topics. This breaks existing kafka monitoring tools.

One proposed solution is for Pinot to set a default group id when it was not explicitly configured by user (say, table name as the default group id).

Contributor guide

Open the contributing guide

Research direction

Start by tracing Pinot's Kafka consumer configuration and how an omitted group.id reaches Kafka 2.2.0. Reproduce the case with auto-commit enabled and inspect whether the consumer appears in the offsets topic. Done means an explicitly or implicitly assigned group id preserves the expected monitoring behavior without overriding a user-provided value.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kafka
Domain
stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.