confluentinc / confluentinc/confluent-kafka-python

describe_acls() is very slow and crashes with segmentation fault for large ACL sets (>55k entries)

Open
#2,031 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
509
Forks
964
Avg merge
2d 2h
Merged PRs (30d)
14

Description

We have an internal Python API that fetches all ACLs from a Kafka cluster.

Previously, we used a ZooKeeper-based API for this purpose. As ZooKeeper is being deprecated in favor of KRaft mode, we are migrating to the Confluent Kafka Python API.

However, when using `confluent_kafka.admin.AdminClient.describe_acls(acl_filter)`, we observe that:

- **The call is significantly slower** than both the Java AdminClient API and our earlier ZooKeeper-based Python API.

- On the same cluster and ACL dataset, Java AdminClient or ZooKeeper API returns results in under 1 second, whereas the Confluent API takes over 5 seconds.

- When the ACL count exceeds ~55,000 entries, t**he Confluent API call frequently terminates with a segmentation fault**.

Reproducer code (**Note**: My Kafka cluster has around 2.5 Lakh acls. Nothing suspicious found in broker logs.)
```
from confluent_kafka.admin import AdminClient, AclBindingFilter, ResourcePatternType, ResourceType, AclOperation, AclPermissionType
import time

# Replace with your Kafka bootstrap servers
BOOTSTRAP_SERVERS = "localhost:9092"

client = AdminClient({
'bootstrap.servers': BOOTSTRAP_SERVERS,
'security.protocol': 'SASL_PLAINTEXT',
'sasl.mechanism': 'GSSAPI',
'sasl.kerberos.service.name': 'kafka',
})

acl_filter = AclBindingFilter(
restype=ResourceType.ANY,
name=None,
resource_pattern_type=ResourcePatternType.ANY,
principal=None,
host=None,
operation=AclOperation.ANY,
permission_type=AclPermissionType.ANY
)

start = time.time()
try:
acls = client.describe_acls(acl_filter).result()
print(f"Fetched {len(acls)} ACLs")
except Exception as e:
print(f"Error fetching ACLs: {e}")
end = time.time()

print(f"Time taken: {end - start:.2f} seconds")

```

- Confluent Kafka Version: 2.8.0
- Librdkafka Version: 2.2.0
- Kafka Cluster Version: 3.9.0

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.