aio-libs / aio-libs/aiokafka

how to handle UnknownTopicOrPartitionError?

Open
#793 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.4k
Forks
269
Avg merge
1d 1h
Merged PRs (30d)
6

Description

What I want to do is subscribing topics that have name with specific pattern like `email.service.*` dynamically.
that dynamically means that while it is consuming I should be able to create or delete topics and it should handle it.

Creating a topic and subscribing it dynamically was fine..
However if I delete a topic that already being subscribed, consumer spew out a lot of error messages like follows:

```
OffsetCommit failed for group email.service on partition TopicPartition(topic='email.service.order', partition=0) with offset OffsetAndMetadata(offset=1, metadata=''): UnknownTopicOrPartitionError
OffsetCommit failed for group email.service on partition TopicPartition(topic='email.service.order', partition=0) with offset OffsetAndMetadata(offset=1, metadata=''): UnknownTopicOrPartitionError
OffsetCommit failed for group email.service on partition TopicPartition(topic='email.service.order', partition=0) with offset OffsetAndMetadata(offset=1, metadata=''): UnknownTopicOrPartitionError
...
```

But I couldn't handle this, because it didn't raise any exceptions. So I checked the code.

https://github.com/aio-libs/aiokafka/blob/724e8105ce0163b2b2df176af65605aafd3ed060/aiokafka/consumer/group_coordinator.py#L1146-L1149

for UnknownTopicOrPartitionError, it just warn it not raise it. How can I handle this exception?
if I handled this exception, I would go like this:

```python
...
try:
result = await consumer.getmany(timeout_ms=500)
except UnknownTopicOrPartitionError as e:
consumer.unsubscribe()
consumer.subscribe(pattern='email.service.*')
...
```

maybe this is a xy problem, if are there any better way to subscribe topics dynamically, please let me know!
thanks!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.