confluentinc / confluentinc/parallel-consumer

Transactional Producer instance gets timeout getting commit lock while second instance starts

Open
#803 4 comments 0 reactions 0 assignees View on GitHub
verified bug
Dominant language
Java
Stars
299
Forks
172
PR merge metrics
No merged PRs in 30d

Description

Hi,

While instance A is consuming/producing records from topic full of records, instance B is started, the following sequence takes place:

1. time 0:
- Instance B requests joining group, at this time several timeout clock start.
- Instance A stops consuming and produces message _Request joining group due to: group is already rebalancing_ every 3sec.
2. time 0 + max.poll.interval.ms:
- Instance B produces message _JoinGroup failed: The coordinator is not aware of this member_
- Instance A produces message: _consumer poll timeout has expired. This means the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms_
- Instance B continues requesting joining group and then produces this sequence of messages:
> Successfully joined group with generation Generation{...}
> Successfully synced group in generation Generation{...}
> Notifying assignor about the new Assignment(partitions=[...])
> Assigned 4 total (4 new) partition(s) [...]
> Setting offset for partition [...]
- Instance B starts consuming
3. time 0 + request.timeout.ms:
- Instance A outputs message _Resetting the last seen epoch of partition_
4. time 0 + commitLockAcquisitionTimeout:
- Instance A finally produces stack trace with following messages and is definitely stuck

```
User provided listener io.confluent.parallelconsumer.ParallelEoSStreamProcessor failed on invocation of onPartitionsRevoked for partitions
Caused by: java.util.concurrent.TimeoutException: Timeout getting commit lock (which was set to PT5M). Slow processing or too many records being ack'd? Try increasing the commit lock timeout (commitLockAcquisitionTimeout), or reduce your record processing time.
Error from poll control thread, will attempt controlled shutdown, then rethrow. Error: There is a newer producer with the same transactionalId which fences the current one
```

The used configuration is:

- PC version : 0.5.2.8
- number of partitions : 4
- max.poll.records: 150
- max.poll.interval.ms: 180000
- request.timeout.ms: 210000
- linger.ms = 65
- delivery.timeout.ms: 240000
- allowEagerProcessingDuringTransactionCommit=false,
- commitLockAcquisitionTimeout=PT5M,
- produceLockAcquisitionTimeout=PT1M,
- commitInterval=PT0.1S,
- ordering=UNORDERED,
- commitMode=PERIODIC_TRANSACTIONAL_PRODUCER,
- maxConcurrency=64,
- invalidOffsetMetadataPolicy=FAIL,
- defaultMessageRetryDelay=PT1S,
- retryDelayProvider=null,
- sendTimeout=PT10S,
- offsetCommitTimeout=PT10S,
- batchSize=1,
- thresholdForTimeSpendInQueueWarning=PT10S,
- maxFailureHistory=10,
- shutdownTimeout=PT10S,
- drainTimeout=PT30S,
- messageBufferSize=50,
- initialLoadFactor=2,
- maximumLoadFactor=100

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the two-instance startup sequence with the reported transactional producer settings, especially max.poll.interval.ms and commitLockAcquisitionTimeout. Start by tracing the onPartitionsRevoked failure and the poll control thread around the commit-lock timeout and producer fencing messages. Done means the described rebalance does not leave instance A stuck or trigger the reported controlled shutdown.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kafka
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.