confluentinc / confluentinc/confluent-kafka-python

Producer hide librdkafka failures and silently return None as message on poll

Open
#1,187 2 comments 0 reactions 0 assignees View on GitHub
bug code:python priority:high
Dominant language
Python
Stars
509
Forks
964
Avg merge
2d 2h
Merged PRs (30d)
14

Description

Description
===========
In case an error happened on Kerberos request authentification `librdkafka` will provide some logs which also are not returned into python space and for just `None` returned from message poll without an error indication.

How to reproduce
================
```python
consumer_config = {
'bootstrap.servers': 'kafka-int.epm-eco.projects.example.com:9095}',
'schema.registry.url': 'http://schema-registry-int.epm-eco.projects.example.com:8081',
'group.id': 'epm-ddo.consumer.2',
'auto.offset.reset': 'earliest',
'security.protocol': 'SASL_SSL',
'ssl.ca.location': 'conf/integration/epm-eco-int.ca.crt',
'sasl.mechanisms': 'GSSAPI',
'sasl.kerberos.principal': 'kafkaclient',
'sasl.kerberos.kinit.cmd': 'bash -c "cat conf/integration/paswd | /usr/bin/kinit Pavel_Alexeev@PETERSBURG.EXAMPLE.COM"',
'sasl.kerberos.service.name': 'kafka',
# 'log_level': 7, 'debug': 'all' ## Logs will be ONLY in container output
}

topic = '....cdm.LocationChartNode'

consumer = AvroConsumer(consumer_config)
consumer.subscribe([topic])

message = consumer.poll(5)
print(message)

consumer.close()
```
Printed result `None`, if look into the container, `librdkafka` produce errors like:
```
%3|1628784442.369|SASLREFRESH|rdkafka#consumer-34| [thrd:main]: Kerberos ticket refresh failed: bash -c "cat conf/integration/paswd | /usr/bin/kinit Pavel_Alexeev@PETERSBURG.EXAMPLE.COM": exited with code 1
```

Checklist
=========
Please provide the following information:

- [✔] confluent-kafka-python and librdkafka version (`confluent_kafka.version()` and `confluent_kafka.libversion()`):
('1.7.0', 17235968), ('1.7.0', 17236223)
- [✔] Apache Kafka broker version: 5.3.2 (uncertain)
- [✔] Client configuration: Kerberros (SASL_SSL), see full config above in reproduce section
- [✔] Operating system: Linux. Fedora and Ubuntu checked
- [✔] Provide client logs (with `'debug': '..'` as necessary)
- [✔] Provide broker log excerpts: sems irrelevant andI have no access
- [✔] Critical issue: yes. Silently skip errors behavior do not allow rely on behaviour and understand what is there something go wrong or just end of topic reached

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.