confluentinc / confluentinc/confluent-kafka-python
How to correctly handle a message.error() for a corrupted message vs retryable error ?
- Dominant language
- Python
- Stars
- 509
- Forks
- 964
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 14
Description
This is not a bug, just a question.
I'm not clear on what I should do if I encounter a `message.error()`. My goal is to process 100% of the input messages, never discarding any message, except for those which are corrupted and can never be processed even after retrying.
I am basically doing this kind of loop:
```
while True:
message = consumer.poll(1)
if message:
if message.error():
continue
print(message.value())
```
My assumption in this loop is that there can never be a case where a message gets discarded. For example, there is no such thing as a message that is "temporarily" corrupted which would be fixed if I implemented a retry. Also there is not a case where the message gets popped off the internal queue but for some reason a error was returned at the same time.
Is that a correct assumption?
Is there any situation in which it would make sense to implement some sort of retry mechanism instead, such as breaking out of the loop and then reconnecting to the last committed offset? This way requires some method of identifying messages that are permanently corrupted in order to skip these and not get stuck in the loop.
Thank you.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.