confluentinc / confluentinc/parallel-consumer

Missing topic and offset infos when login error in ConsumerOffsetCommitter

Open Beginner friendly
#629 3 comments 0 reactions 0 assignees View on GitHub
good first issue
Dominant language
Java
Stars
299
Forks
172
PR merge metrics
No merged PRs in 30d

Description

First thanks for this library,

I'm trying to use it at its best to get the maximum throughput in my application.

I got some issue when scaling up and down my app, due to "infinite rebalance".

I feel that something may be going wrong between these two configurations : PERIODIC_CONSUMER_ASYNCHRONOUS and "partition.assignment.strategy" = org.apache.kafka.clients.consumer.CooperativeStickyAssignor

I would be very helpful to add topic/partition/offset that are tried to be commited in this log (Ideally the ones that failed)

https://github.com/confluentinc/parallel-consumer/blob/master/parallel-consumer-core/src/main/java/io/confluent/parallelconsumer/internal/ConsumerOffsetCommitter.java#L100

I don't see these informations in the current message
```
{extendedStackTrace=org.apache.kafka.common.errors.RebalanceInProgressException: Offset commit cannot be completed since the consumer is undergoing a rebalance for auto partition assignment. You can try completing the rebalance by calling poll() and then retry the operation.
, localizedMessage=Offset commit cannot be completed since the consumer is undergoing a rebalance for auto partition assignment. You can try completing the rebalance by calling poll() and then retry the operation., name=org.apache.kafka.common.errors.RebalanceInProgressException, commonElementCount=0, message=Offset commit cannot be completed since the consumer is undergoing a rebalance for auto partition assignment. You can try completing the rebalance by calling poll() and then retry the operation.}
```

Regards,
Benoit

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in parallel-consumer-core/src/main/java/io/confluent/parallelconsumer/internal/ConsumerOffsetCommitter.java around line 100, where the login error is logged. Trace the offset commit failure path and inspect the existing tests for this component. Done means the relevant log includes the topic, partition, and offset being committed, ideally identifying the failed entries.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, distributed-systems
Issue type
Feature
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.