[QUESTION] How do I use a topic like a database table and query it as fast as I possibly can?
- Linguagem predominante
- Python
- Estrelas
- 1.4k
- Forks
- 269
- Merge médio
- 1d 1h
- PRs com merge (30d)
- 6
Descrição
Using `offsets_for_times`, I'd like to be able to query all message between a certain time range, as fast as I can.
Using the `confluent-kafka-python` client, I was able to do this at about 1415.25348 messages/second, with a message size of about 1.8 MB.
I would like to run my kafka consumer like a short-lived database query.
I understand I can run multiple instances of my consumer as a consumer group to increase parallelism, but I would like to run a single instance in a multi-threaded fashion. If my topic has 3 partitions, I would like to consume messages using three consumers. I'm not quite sure how to do this from a single instance. I'm trying to create a library that another engineer can use to query a topic between a time range, as already mentioned above.
Guia de contribuição
Avaliação
Esta issue ainda não foi avaliada.