confluentinc / confluentinc/confluent-kafka-python
Frequent Broker Disconnects Caused by "non-nullable field metrics was serialized as null" with Telemetry Subscriptions
- Dominant language
- Python
- Stars
- 509
- Forks
- 964
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 14
Description
### Summary
When using **confluent_kafka v2.8.0** (with librdkafka v2.8.0) and enabling client metrics telemetry subscriptions, we observe consistent broker disconnects for all Python client types (AdminClient, Producer, and Consumer). The broker throws an InvalidRequestException related to a non-nullable "metrics" field, resulting in Disconnected errors on the client, even though authentication is successful.
Note: This is observed only with the Python client; Java clients do not experience these issues under the same broker and metric subscription configuration.
**Environment**
confluent_kafka Version: 2.8.0
Kafka : 3.9
### Broker-Side Error
> ERROR Exception while processing request from 10.218.247.36:9192-10.219.33.168:58922-9 (kafka.network.Processor)
> org.apache.kafka.common.errors.InvalidRequestException: Error getting request for apiKey: PUSH_TELEMETRY, apiVersion: 0, connectionId: 10.218.247.36:9192-10.219.33.168:58922-9, listenerName: ListenerName(SASL_PLAINTEXT), principal: User:guptrith
> Caused by: java.lang.RuntimeException: non-nullable field metrics was serialized as null
> at org.apache.kafka.common.message.PushTelemetryRequestData.read(PushTelemetryRequestData.java:106)
> at org.apache.kafka.common.message.PushTelemetryRequestData.(PushTelemetryRequestData.java:70)
> at org.apache.kafka.common.requests.PushTelemetryRequest.parse(PushTelemetryRequest.java:95)
> at org.apache.kafka.common.requests.AbstractRequest.doParseRequest(AbstractRequest.java:322)
> at org.apache.kafka.common.requests.AbstractRequest.parseRequest(AbstractRequest.java:172)
> at org.apache.kafka.common.requests.RequestContext.parseRequest(RequestContext.java:120)
> at kafka.network.RequestChannel$Request.(RequestChannel.scala:102)
> at kafka.network.Processor.$anonfun$processCompletedReceives$1(SocketServer.scala:1151)
> at java.base/java.util.LinkedHashMap$LinkedValues.forEach(LinkedHashMap.java:647)
> at kafka.network.Processor.processCompletedReceives(SocketServer.scala:1129)
> at kafka.network.Processor.run(SocketServer.scala:1015)
> at java.base/java.lang.Thread.run(Thread.java:840)
**To Reproduce**
Configure a client with the following settings:
> config = {
> 'security.protocol': 'SASL_PLAINTEXT',
> 'sasl.kerberos.service.name': 'kafka',
> 'sasl.kerberos.kinit.cmd': '/bin/true',
> 'sasl.mechanism': 'GSSAPI',
> 'bootstrap.servers': 'qtekafka-alpha-01.nyc.com:9192,qtekafka-alpha-02.nyc.com:9192,...'
> }
> from confluent_kafka.admin import AdminClient
> ac = AdminClient(config)
Subscribe to client metrics topics using configurations such as:
> Client metrics configs for dzlUw3JaR3WzL1coeEoAAQ are:
> interval.ms=60000
> metrics=org.apache.kafka.producer.,org.apache.kafka.consumer.
> Client metrics configs for HlGcemxAQn2EMbDqWFcwtw are:
> interval.ms=3000
> metrics=org.apache.kafka.consumer.
> Client metrics configs for _yTegRipQjCCW8zvJ2X2Pg are:
> interval.ms=5000
> metrics=org.apache.kafka.producer.
Observe that the broker logs show repeated disconnects and exceptions.
### Client Log Example
> In [8]: %6|1766719941.098|GETSUBSCRIPTIONS|rdkafka#producer-1| [thrd:main]: Telemetry client instance id changed from AAAAAAAAAAAAAAAAAAAAAA to c6inLoBGRd6Wk4WYwdy/2Q
> %6|1766719944.050|FAIL|rdkafka#producer-1| [thrd:sasl_plaintext://qtekafka-alpha-01.dr.com:9192/bootstrap]: sasl_plaintext://qtekafka-alpha-01.dr.com:9192/3: Disconnected (after 2958ms in state UP)
> %6|1766719950.144|FAIL|rdkafka#producer-1| [thrd:sasl_plaintext://qtekafka-alpha-01.dr.com:9192/bootstrap]: sasl_plaintext://qtekafka-alpha-01.dr.com:9192/3: Disconnected (after 6072ms in state UP, 1 identical error(s) suppressed)
> %6|1766719956.095|FAIL|rdkafka#producer-1| [thrd:sasl_plaintext://qtekafka-alpha-02.tbd.com:9192/bootstra]: sasl_plaintext://qtekafka-alpha-02.tbd.com:9192/6: Disconnected (after 12013ms in state UP)
...
### Analysis
The issue appears only with certain metric subscription values.
**Produces Errors :**
`Client metrics configs for trial1 are:
interval.ms=1000
metrics=org.apache.kafka.producer.connection.creation.rate,org.apache.kafka.producer.connection.creation.total,org.apache.kafka.producer.node.request.latency.avg,org.apache.kafka.consumer.poll.idle.ratio.avg,org.apache.kafka.consumer.coordinator.rebalance.latency.avg,org.apache.kafka.consumer.coordinator.rebalance.latency.max
Client metrics configs for trial are:
interval.ms=1000
metrics=org.apache.kafka `
**Dont produce Errors :**
`Client metrics configs for trial1 are:
interval.ms=1000
metrics=org.apache.kafka.producer.connection.creation.rate,org.apache.kafka.producer.connection.creation.total,org.apache.kafka.producer.node.request.latency.avg,org.apache.kafka.consumer.poll.idle.ratio.avg,org.apache.kafka.consumer.coordinator.rebalance.latency.avg,org.apache.kafka.consumer.coordinator.rebalance.latency.max
Client metrics configs for trial are:
interval.ms=1000
metrics=org.apache `
## Question / Request
- By when can it be fixed ?
- Is this a known serialization bug in confluent_kafka/librdkafka?
- Is there a supported way to tell, from the client side, before connecting, whether a given metrics subscription is likely to fail?
- Additional Context
- This affects a lot , as Kafka brokers log repeated exceptions and drop connections for Python clients.
- Can you suggest some workaround till the time its not fixed
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.