aio-libs / aio-libs/aiokafka

Document how to handle long message processing with max_poll_interval_ms

Aberta
#848 3 comentários 0 reações 0 responsáveis Ver no GitHub
enhancement
Linguagem predominante
Python
Estrelas
1.4k
Forks
269
Merge médio
1d 1h
PRs com merge (30d)
6

Descrição

**Describe the solution you'd like**
I think it'd be good to give a clear warning in the docs about processing long messages and focus people's attention on max_poll_interval_ms. Maybe another section - "long processing" - in [here](https://aiokafka.readthedocs.io/en/stable/consumer.html#consumer-usage) would be good.

**The problem that I had**
- A message was taking a long time to process.
- After 5 minutes, the consumer doing the processing left the group.
- The group rebalanced
- Another consumer got the same message to process.
- The original consumer finished processing (after 7 minutes), tried to commit the offset, failed with UnknownMemberID, which also causes a rebalance of the whole group, preventing the second consumer from commiting the message.
- That looped forever.

**My understanding of max_poll_interval_ms**
The number isn't sent to Kafka.
It's only used internally in aiokafka to leave the consumer group if the consumer didn't poll for messages in a while, suggesting it's stuck.

Is that correct?

Guia de contribuição

Abrir o guia de contribuição

Avaliação

Esta issue ainda não foi avaliada.

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.