confluentinc / confluentinc/confluent-kafka-python
Consumer not consuming from all assigned partitions
- Dominant language
- Python
- Stars
- 509
- Forks
- 964
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 14
Description
Description
===========
We had 100 partitions and 100 consumers and things worked as expected. But last night I increased number of partitions to 1000 and number of consumers to 200.
As expected each of the 200 consumers got assigned 5 partitions each. I confirmed this from consumer logs and `./kafka-consumer-groups.sh --describe`
Partitions numbered 0-99 were consumed fully as expected. But only about a fifth of partitions numbered 100-999 were consumed.
Either the partitions were fully consumed, or not a single message was consumed. It is as if the consumer didn't even "connect" or bother to consume from those partitions. It is really bizarre! E.g. see below:
```
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID
ETLJob etl_runs 745 - 124 - ETLJob-b9fed09c-6b77-4098-8bdf-42859c8f57de
ETLJob etl_runs 749 121 121 0 ETLJob-b9fed09c-6b77-4098-8bdf-42859c8f57de
ETLJob etl_runs 746 - 136 - ETLJob-b9fed09c-6b77-4098-8bdf-42859c8f57de
ETLJob etl_runs 747 - 138 - ETLJob-b9fed09c-6b77-4098-8bdf-42859c8f57de
ETLJob etl_runs 748 - 144 - ETLJob-b9fed09c-6b77-4098-8bdf-42859c8f57de
ETLJob etl_runs 93 24312 24312 0 ETLJob-2190c0c3-d562-4e50-89bc-ec3a54c94e89
ETLJob etl_runs 90 24034 24034 0 ETLJob-2190c0c3-d562-4e50-89bc-ec3a54c94e89
ETLJob etl_runs 94 25189 25189 0 ETLJob-2190c0c3-d562-4e50-89bc-ec3a54c94e89
ETLJob etl_runs 91 24027 24027 0 ETLJob-2190c0c3-d562-4e50-89bc-ec3a54c94e89
ETLJob etl_runs 92 24887 24887 0 ETLJob-2190c0c3-d562-4e50-89bc-ec3a54c94e89
ETLJob etl_runs 763 - 134 - ETLJob-beb213ff-4bcb-43d3-b5b7-c3853f5b0e56
ETLJob etl_runs 760 - 142 - ETLJob-beb213ff-4bcb-43d3-b5b7-c3853f5b0e56
ETLJob etl_runs 764 - 133 - ETLJob-beb213ff-4bcb-43d3-b5b7-c3853f5b0e56
ETLJob etl_runs 761 - 148 - ETLJob-beb213ff-4bcb-43d3-b5b7-c3853f5b0e56
ETLJob etl_runs 762 138 138 0 ETLJob-beb213ff-4bcb-43d3-b5b7-c3853f5b0e56
```
How to reproduce
================
Don't know
Checklist
=========
Please provide the following information:
- [x] confluent-kafka-python and librdkafka version (`confluent_kafka.version()` and `confluent_kafka.libversion()`): `('1.8.2', 17302016)` and `('1.8.2', 17302271)` respectively
- [x] Apache Kafka broker version: `2.8.1 (Commit:839b886f9b732b15)`
- [x] Client configuration: `{...}`
```
self.conf = {
"group.id": group_id,
"client.id": client_id or group_id,
"bootstrap.servers": get_server_set(brokers or get_env_variable(KAFKA_BROKERS_VARIABLE)),
"allow.auto.create.topics": "false",
"enable.auto.commit": "false",
"isolation.level": "read_committed",
"max.poll.interval.ms": (max_batch_processing_seconds + 60) * 1000, # convert s to ms and add 1 minute
}
```
- [x] Operating system: `Linux 5.4.219-126.411.amzn2.x86_64`
- [ ] Provide client logs (with `'debug': '..'` as necessary)
- [ ] Provide broker log excerpts
- [x] Critical issue
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.