confluentinc / confluentinc/confluent-kafka-python

AdminClient caching all topics in metadata request calls

Open
#1,737 13 comments 0 reactions 0 assignees View on GitHub
component:admin investigate further priority:high status:planned
Dominant language
Python
Stars
509
Forks
964
Avg merge
2d 2h
Merged PRs (30d)
14

Description

Description
===========
I was doing some testing with the Admin client where I delete and recreate topics over and over again with new names over a period of three days and I see that the admin client is caching every topic name and will cause timeouts. I have gotten to a point where only 5 topics exist on the broker but its looking for 8k topics

How to reproduce
================
normal admin client can be 1.9.2 or 2.3, but list current topics, then delete them, then create new topics again over and over for about a week and you will see them.

Checklist
=========
Please provide the following information:

- [x] confluent-kafka-python and librdkafka version (`confluent_kafka.version()` and `confluent_kafka.libversion()`):
- [1.9.2]
- [x] Apache Kafka broker version: Microsoft Event Hubs
- [x] Client configuration: `{...}`
![image](https://github.com/confluentinc/confluent-kafka-python/assets/10121257/3dfe1f82-2018-4ee3-be01-01775c7c5619)
conf = {
'bootstrap.servers': self.stressconfigs.get_appr_broker(self.mode),
'security.protocol': 'SASL_SSL',
'ssl.ca.location': '',
'sasl.mechanism': 'PLAIN',
'sasl.username': '$ConnectionString',
'sasl.password': self.stressconfigs.get_appr_Connectiontring(self.mode),
'client.id': 'python-admin-client' + self.clientId,
'allow.auto.create.topics': 'false',
'connections.max.idle.ms': 180000,
'socket.keepalive.enable' : True,
'request.timeout.ms': 60000,
'socket.timeout.ms': 60000,
'message.timeout.ms': 60000,
}
- [x] Operating system: linux Azure kubernetes
- [x] Provide client logs (with `'debug': '..'` as necessary)
- [ ] Provide broker log excerpts
- [x] Critical issue Yes

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.