confluentinc / confluentinc/parallel-consumer
Missing topic and offset infos when login error in ConsumerOffsetCommitter
- Dominant language
- Java
- Stars
- 299
- Forks
- 172
- PR merge metrics
- No merged PRs in 30d
Description
First thanks for this library,
I'm trying to use it at its best to get the maximum throughput in my application.
I got some issue when scaling up and down my app, due to "infinite rebalance".
I feel that something may be going wrong between these two configurations : PERIODIC_CONSUMER_ASYNCHRONOUS and "partition.assignment.strategy" = org.apache.kafka.clients.consumer.CooperativeStickyAssignor
I would be very helpful to add topic/partition/offset that are tried to be commited in this log (Ideally the ones that failed)
https://github.com/confluentinc/parallel-consumer/blob/master/parallel-consumer-core/src/main/java/io/confluent/parallelconsumer/internal/ConsumerOffsetCommitter.java#L100
I don't see these informations in the current message
```
{extendedStackTrace=org.apache.kafka.common.errors.RebalanceInProgressException: Offset commit cannot be completed since the consumer is undergoing a rebalance for auto partition assignment. You can try completing the rebalance by calling poll() and then retry the operation.
, localizedMessage=Offset commit cannot be completed since the consumer is undergoing a rebalance for auto partition assignment. You can try completing the rebalance by calling poll() and then retry the operation., name=org.apache.kafka.common.errors.RebalanceInProgressException, commonElementCount=0, message=Offset commit cannot be completed since the consumer is undergoing a rebalance for auto partition assignment. You can try completing the rebalance by calling poll() and then retry the operation.}
```
Regards,
Benoit
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in parallel-consumer-core/src/main/java/io/confluent/parallelconsumer/internal/ConsumerOffsetCommitter.java around line 100, where the login error is logged. Trace the offset commit failure path and inspect the existing tests for this component. Done means the relevant log includes the topic, partition, and offset being committed, ideally identifying the failed entries.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, distributed-systems
- Issue type
- Feature
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100