confluentinc / confluentinc/confluent-kafka-python

Frequent Broker Disconnects Caused by "non-nullable field metrics was serialized as null" with Telemetry Subscriptions

Open
#2,185 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
509
Forks
964
Avg merge
2d 2h
Merged PRs (30d)
14

Description

### Summary
When using **confluent_kafka v2.8.0** (with librdkafka v2.8.0) and enabling client metrics telemetry subscriptions, we observe consistent broker disconnects for all Python client types (AdminClient, Producer, and Consumer). The broker throws an InvalidRequestException related to a non-nullable "metrics" field, resulting in Disconnected errors on the client, even though authentication is successful.

Note: This is observed only with the Python client; Java clients do not experience these issues under the same broker and metric subscription configuration.

**Environment**
confluent_kafka Version: 2.8.0
Kafka : 3.9

### Broker-Side Error

> ERROR Exception while processing request from 10.218.247.36:9192-10.219.33.168:58922-9 (kafka.network.Processor)
> org.apache.kafka.common.errors.InvalidRequestException: Error getting request for apiKey: PUSH_TELEMETRY, apiVersion: 0, connectionId: 10.218.247.36:9192-10.219.33.168:58922-9, listenerName: ListenerName(SASL_PLAINTEXT), principal: User:guptrith
> Caused by: java.lang.RuntimeException: non-nullable field metrics was serialized as null
> at org.apache.kafka.common.message.PushTelemetryRequestData.read(PushTelemetryRequestData.java:106)
> at org.apache.kafka.common.message.PushTelemetryRequestData.(PushTelemetryRequestData.java:70)
> at org.apache.kafka.common.requests.PushTelemetryRequest.parse(PushTelemetryRequest.java:95)
> at org.apache.kafka.common.requests.AbstractRequest.doParseRequest(AbstractRequest.java:322)
> at org.apache.kafka.common.requests.AbstractRequest.parseRequest(AbstractRequest.java:172)
> at org.apache.kafka.common.requests.RequestContext.parseRequest(RequestContext.java:120)
> at kafka.network.RequestChannel$Request.(RequestChannel.scala:102)
> at kafka.network.Processor.$anonfun$processCompletedReceives$1(SocketServer.scala:1151)
> at java.base/java.util.LinkedHashMap$LinkedValues.forEach(LinkedHashMap.java:647)
> at kafka.network.Processor.processCompletedReceives(SocketServer.scala:1129)
> at kafka.network.Processor.run(SocketServer.scala:1015)
> at java.base/java.lang.Thread.run(Thread.java:840)

**To Reproduce**
Configure a client with the following settings:

> config = {
> 'security.protocol': 'SASL_PLAINTEXT',
> 'sasl.kerberos.service.name': 'kafka',
> 'sasl.kerberos.kinit.cmd': '/bin/true',
> 'sasl.mechanism': 'GSSAPI',
> 'bootstrap.servers': 'qtekafka-alpha-01.nyc.com:9192,qtekafka-alpha-02.nyc.com:9192,...'
> }
> from confluent_kafka.admin import AdminClient
> ac = AdminClient(config)

Subscribe to client metrics topics using configurations such as:

> Client metrics configs for dzlUw3JaR3WzL1coeEoAAQ are:
> interval.ms=60000
> metrics=org.apache.kafka.producer.,org.apache.kafka.consumer.
> Client metrics configs for HlGcemxAQn2EMbDqWFcwtw are:
> interval.ms=3000
> metrics=org.apache.kafka.consumer.
> Client metrics configs for _yTegRipQjCCW8zvJ2X2Pg are:
> interval.ms=5000
> metrics=org.apache.kafka.producer.

Observe that the broker logs show repeated disconnects and exceptions.

### Client Log Example

> In [8]: %6|1766719941.098|GETSUBSCRIPTIONS|rdkafka#producer-1| [thrd:main]: Telemetry client instance id changed from AAAAAAAAAAAAAAAAAAAAAA to c6inLoBGRd6Wk4WYwdy/2Q
> %6|1766719944.050|FAIL|rdkafka#producer-1| [thrd:sasl_plaintext://qtekafka-alpha-01.dr.com:9192/bootstrap]: sasl_plaintext://qtekafka-alpha-01.dr.com:9192/3: Disconnected (after 2958ms in state UP)
> %6|1766719950.144|FAIL|rdkafka#producer-1| [thrd:sasl_plaintext://qtekafka-alpha-01.dr.com:9192/bootstrap]: sasl_plaintext://qtekafka-alpha-01.dr.com:9192/3: Disconnected (after 6072ms in state UP, 1 identical error(s) suppressed)
> %6|1766719956.095|FAIL|rdkafka#producer-1| [thrd:sasl_plaintext://qtekafka-alpha-02.tbd.com:9192/bootstra]: sasl_plaintext://qtekafka-alpha-02.tbd.com:9192/6: Disconnected (after 12013ms in state UP)

...
### Analysis
The issue appears only with certain metric subscription values.

**Produces Errors :**
`Client metrics configs for trial1 are:
interval.ms=1000
metrics=org.apache.kafka.producer.connection.creation.rate,org.apache.kafka.producer.connection.creation.total,org.apache.kafka.producer.node.request.latency.avg,org.apache.kafka.consumer.poll.idle.ratio.avg,org.apache.kafka.consumer.coordinator.rebalance.latency.avg,org.apache.kafka.consumer.coordinator.rebalance.latency.max
Client metrics configs for trial are:
interval.ms=1000
metrics=org.apache.kafka `

**Dont produce Errors :**

`Client metrics configs for trial1 are:
interval.ms=1000
metrics=org.apache.kafka.producer.connection.creation.rate,org.apache.kafka.producer.connection.creation.total,org.apache.kafka.producer.node.request.latency.avg,org.apache.kafka.consumer.poll.idle.ratio.avg,org.apache.kafka.consumer.coordinator.rebalance.latency.avg,org.apache.kafka.consumer.coordinator.rebalance.latency.max
Client metrics configs for trial are:
interval.ms=1000
metrics=org.apache `

## Question / Request

- By when can it be fixed ?
- Is this a known serialization bug in confluent_kafka/librdkafka?
- Is there a supported way to tell, from the client side, before connecting, whether a given metrics subscription is likely to fail?
- Additional Context
- This affects a lot , as Kafka brokers log repeated exceptions and drop connections for Python clients.
- Can you suggest some workaround till the time its not fixed

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.