confluentinc / confluentinc/parallel-consumer

Truncating state

Open
#546 23 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
299
Forks
172
PR merge metrics
No merged PRs in 30d

Description

Hi @astubbs ,

We started using the parallel consumer recently for some of our services. Thanks for creating this great new consumer.
We sometimes see the following warnings in our logs and wonder what could be causing them.

**Kafka Configuration**
- no compact topics
- retention.ms=604800000 (7 days)
- auto.offset.reset=latest
- broker version=2.8.1
- we do not use Kafka transactions
- we use parallel-consumer version 0.5.2.4

**PC Configuration**
```
ParallelConsumerOptions.builder()
.ordering(ProcessingOrder.KEY)
.maxConcurrency(20)
.commitInterval(Duration.ofSeconds(2))
.consumer(consumer)
.commitMode(CommitMode.PERIODIC_CONSUMER_ASYNCHRONOUS)
.build();
```

**Warning message**
```
Truncating state - removing records lower than 208437. Offsets have been removed from the partition by the broker or committed offset has been raised. Bootstrap polled 208437 but expected 1 from loaded commit data. Could be caused by record retention or compaction and offset reset policy LATEST.
```
The expected value is also sometimes 0 instead of 1

**We also see the following warning, which might be related**
```
Bootstrap polled offset has been reset to an earlier offset (1081798) - truncating state - all records above (including this) will be replayed. Was expecting 1081799 but bootstrap poll was 1081798. Could be caused by record retention or compaction and offset reset policy EARLIEST.
```

I have also created a [PR](https://github.com/confluentinc/parallel-consumer/pull/545) to add the partition and topic to the truncate logging.

Do you know how we could explain these warnings?

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.