aio-libs / aio-libs/aiokafka

[QUESTION] How do I use a topic like a database table and query it as fast as I possibly can?

Aberta
#608 4 comentários 0 reações 0 responsáveis Ver no GitHub
question
Linguagem predominante
Python
Estrelas
1.4k
Forks
269
Merge médio
1d 1h
PRs com merge (30d)
6

Descrição

Using `offsets_for_times`, I'd like to be able to query all message between a certain time range, as fast as I can.

Using the `confluent-kafka-python` client, I was able to do this at about 1415.25348 messages/second, with a message size of about 1.8 MB.

I would like to run my kafka consumer like a short-lived database query.

I understand I can run multiple instances of my consumer as a consumer group to increase parallelism, but I would like to run a single instance in a multi-threaded fashion. If my topic has 3 partitions, I would like to consume messages using three consumers. I'm not quite sure how to do this from a single instance. I'm trying to create a library that another engineer can use to query a topic between a time range, as already mentioned above.

Guia de contribuição

Abrir o guia de contribuição

Avaliação

Esta issue ainda não foi avaliada.

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.