confluentinc / confluentinc/confluent-kafka-python

[Version 2.2.0] list_topics() messes up offsets

Open
#2,152 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
509
Forks
964
Avg merge
2d 2h
Merged PRs (30d)
14

Description

I know, we're using an old version, but can't upgrade due to dropped CentOS support.
Really weird behaviour....

What I do is roughly:

```python
consumer = make_consumer(enable_auto_commit=False)
consumer.subscribe(my_topic)

while True:
msgs = consumer.consume(timeout=timeout_sec, num_messages=max_records)
consumer.list_topics() # This is the crucial line - if present, next consume will be from committed offset, otherwise, will continue from last polled
consumer.commit(message=msgs[0])
```

Somehow `list_topics()` ruins the expected behaviour of polling the next batch every time - in the example it polls overlapping batches.

Interesting note: if I do `assign()` instead of `subscribe()` it works as expected even with `list_topics()` calls.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.