confluentinc / confluentinc/confluent-kafka-python
AdminClient caching all topics in metadata request calls
- Dominant language
- Python
- Stars
- 509
- Forks
- 964
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 14
Description
Description
===========
I was doing some testing with the Admin client where I delete and recreate topics over and over again with new names over a period of three days and I see that the admin client is caching every topic name and will cause timeouts. I have gotten to a point where only 5 topics exist on the broker but its looking for 8k topics
How to reproduce
================
normal admin client can be 1.9.2 or 2.3, but list current topics, then delete them, then create new topics again over and over for about a week and you will see them.
Checklist
=========
Please provide the following information:
- [x] confluent-kafka-python and librdkafka version (`confluent_kafka.version()` and `confluent_kafka.libversion()`):
- [1.9.2]
- [x] Apache Kafka broker version: Microsoft Event Hubs
- [x] Client configuration: `{...}`

conf = {
'bootstrap.servers': self.stressconfigs.get_appr_broker(self.mode),
'security.protocol': 'SASL_SSL',
'ssl.ca.location': '',
'sasl.mechanism': 'PLAIN',
'sasl.username': '$ConnectionString',
'sasl.password': self.stressconfigs.get_appr_Connectiontring(self.mode),
'client.id': 'python-admin-client' + self.clientId,
'allow.auto.create.topics': 'false',
'connections.max.idle.ms': 180000,
'socket.keepalive.enable' : True,
'request.timeout.ms': 60000,
'socket.timeout.ms': 60000,
'message.timeout.ms': 60000,
}
- [x] Operating system: linux Azure kubernetes
- [x] Provide client logs (with `'debug': '..'` as necessary)
- [ ] Provide broker log excerpts
- [x] Critical issue Yes
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.