Handle change in default groupId behavior in kafka consumer
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 189
Description
Starting kafka 2.2.0 [onwards](https://issues.apache.org/jira/browse/KAFKA-6774), when `group.id` is not specified in the kafka consumer config, it is set as `null`, instead of an empty string. Due to this change, even though auto-commit is enabled, it doesn't show up in the consumer offsets topics. This breaks existing kafka monitoring tools.
One proposed solution is for Pinot to set a default group id when it was not explicitly configured by user (say, table name as the default group id).
Contributor guide
Research direction
Start by tracing Pinot's Kafka consumer configuration and how an omitted group.id reaches Kafka 2.2.0. Reproduce the case with auto-commit enabled and inspect whether the consumer appears in the offsets topic. Done means an explicitly or implicitly assigned group id preserves the expected monitoring behavior without overriding a user-provided value.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, kafka
- Domain
- stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100