confluentinc / confluentinc/parallel-consumer
Transactional Producer instance gets timeout getting commit lock while second instance starts
- Dominant language
- Java
- Stars
- 299
- Forks
- 172
- PR merge metrics
- No merged PRs in 30d
Description
Hi,
While instance A is consuming/producing records from topic full of records, instance B is started, the following sequence takes place:
1. time 0:
- Instance B requests joining group, at this time several timeout clock start.
- Instance A stops consuming and produces message _Request joining group due to: group is already rebalancing_ every 3sec.
2. time 0 + max.poll.interval.ms:
- Instance B produces message _JoinGroup failed: The coordinator is not aware of this member_
- Instance A produces message: _consumer poll timeout has expired. This means the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms_
- Instance B continues requesting joining group and then produces this sequence of messages:
> Successfully joined group with generation Generation{...}
> Successfully synced group in generation Generation{...}
> Notifying assignor about the new Assignment(partitions=[...])
> Assigned 4 total (4 new) partition(s) [...]
> Setting offset for partition [...]
- Instance B starts consuming
3. time 0 + request.timeout.ms:
- Instance A outputs message _Resetting the last seen epoch of partition_
4. time 0 + commitLockAcquisitionTimeout:
- Instance A finally produces stack trace with following messages and is definitely stuck
```
User provided listener io.confluent.parallelconsumer.ParallelEoSStreamProcessor failed on invocation of onPartitionsRevoked for partitions
Caused by: java.util.concurrent.TimeoutException: Timeout getting commit lock (which was set to PT5M). Slow processing or too many records being ack'd? Try increasing the commit lock timeout (commitLockAcquisitionTimeout), or reduce your record processing time.
Error from poll control thread, will attempt controlled shutdown, then rethrow. Error: There is a newer producer with the same transactionalId which fences the current one
```
The used configuration is:
- PC version : 0.5.2.8
- number of partitions : 4
- max.poll.records: 150
- max.poll.interval.ms: 180000
- request.timeout.ms: 210000
- linger.ms = 65
- delivery.timeout.ms: 240000
- allowEagerProcessingDuringTransactionCommit=false,
- commitLockAcquisitionTimeout=PT5M,
- produceLockAcquisitionTimeout=PT1M,
- commitInterval=PT0.1S,
- ordering=UNORDERED,
- commitMode=PERIODIC_TRANSACTIONAL_PRODUCER,
- maxConcurrency=64,
- invalidOffsetMetadataPolicy=FAIL,
- defaultMessageRetryDelay=PT1S,
- retryDelayProvider=null,
- sendTimeout=PT10S,
- offsetCommitTimeout=PT10S,
- batchSize=1,
- thresholdForTimeSpendInQueueWarning=PT10S,
- maxFailureHistory=10,
- shutdownTimeout=PT10S,
- drainTimeout=PT30S,
- messageBufferSize=50,
- initialLoadFactor=2,
- maximumLoadFactor=100
Contributor guide
No contributing guide indexed for this repository
Research direction
Reproduce the two-instance startup sequence with the reported transactional producer settings, especially max.poll.interval.ms and commitLockAcquisitionTimeout. Start by tracing the onPartitionsRevoked failure and the poll control thread around the commit-lock timeout and producer fencing messages. Done means the described rebalance does not leave instance A stuck or trigger the reported controlled shutdown.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, kafka
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 42/100