confluentinc / confluentinc/parallel-consumer
Truncating state
- Dominant language
- Java
- Stars
- 299
- Forks
- 172
- PR merge metrics
- No merged PRs in 30d
Description
Hi @astubbs ,
We started using the parallel consumer recently for some of our services. Thanks for creating this great new consumer.
We sometimes see the following warnings in our logs and wonder what could be causing them.
**Kafka Configuration**
- no compact topics
- retention.ms=604800000 (7 days)
- auto.offset.reset=latest
- broker version=2.8.1
- we do not use Kafka transactions
- we use parallel-consumer version 0.5.2.4
**PC Configuration**
```
ParallelConsumerOptions.builder()
.ordering(ProcessingOrder.KEY)
.maxConcurrency(20)
.commitInterval(Duration.ofSeconds(2))
.consumer(consumer)
.commitMode(CommitMode.PERIODIC_CONSUMER_ASYNCHRONOUS)
.build();
```
**Warning message**
```
Truncating state - removing records lower than 208437. Offsets have been removed from the partition by the broker or committed offset has been raised. Bootstrap polled 208437 but expected 1 from loaded commit data. Could be caused by record retention or compaction and offset reset policy LATEST.
```
The expected value is also sometimes 0 instead of 1
**We also see the following warning, which might be related**
```
Bootstrap polled offset has been reset to an earlier offset (1081798) - truncating state - all records above (including this) will be replayed. Was expecting 1081799 but bootstrap poll was 1081798. Could be caused by record retention or compaction and offset reset policy EARLIEST.
```
I have also created a [PR](https://github.com/confluentinc/parallel-consumer/pull/545) to add the partition and topic to the truncate logging.
Do you know how we could explain these warnings?
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.