confluentinc / confluentinc/parallel-consumer

ParallelConsumer would run for a while and then exit due to InternalRuntimeException(Timeout)

Open
#833 9 comments 1 reaction 0 assignees View on GitHub
Dominant language
Java
Stars
299
Forks
172
PR merge metrics
No merged PRs in 30d

Description

Hi team,
version 0.5.3.1

**InternalRuntimeException:**
My test scenario is a scenario where 50% of records fail, and there's 1000 keys in total, parallelConsumer would run for a while and then exit due to InternalRuntimeException at 24/09/13 21:33:37.130
io.confluent.parallelconsumer.internal.InternalRuntimeException: Timeout waiting for commit response PT30S to request ConsumerOffsetCommitter.CommitRequest(id=79c3ac04-b8c7-4dc2-9b09-c77d6ad6bee4, requestedAtMs=1726234432425)
Screenshot 2024-09-16 at 15 24 18

And we can see the code from ConsumerOffsetCommitter.commitAndWait()

` CommitResponse take = commitResponseQueue.poll(commitTimeout.toMillis(), TimeUnit.MILLISECONDS); // blocks, drain until we find our response
`
cause take is null then throw InternalRuntimeException.

**metrics from pc_processed_records_total**
During this time, there's no successful records:
Screenshot 2024-09-18 at 10 39 16

**Adding commit response to queue:**
And below logs from ConsumerOffsetCommitter.maybeDoCommit() show the last time add commit response to queue is 24/09/13 21:16:54.105

Screenshot 2024-09-16 at 15 25 49

**Waiting on a commit response:**

And we can see the code from ConsumerOffsetCommitter.commitAndWait() show the last time wait commit response from queue is 24/09/13 21:33:52.426

Screenshot 2024-09-16 at 15 31 17

Here's my concerns:
First time adding commit response to queue time is 24/09/13 21:16:54.105, and waiting on a commit response time is 24/09/13 21:16:54.084, within 30s
second time there's no adding commit response to queue but waiting on a commit response time is 24/09/13 21:33:39.194.
so it lead to InternalRuntimeException.

please help to check

BRS,
Dumont

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with ConsumerOffsetCommitter.commitAndWait() and maybeDoCommit(), using the reported 0.5.3.1 scenario with 50% failing records and 1,000 keys. Trace why a commit response is not added to the queue before the 30-second poll timeout. Done means the failure cause is identified and the consumer no longer exits unexpectedly in this scenario.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems, performance, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.