confluentinc / confluentinc/confluent-kafka-python

Missing log messages when closing the program

Open
#1,719 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
509
Forks
964
Avg merge
2d 2h
Merged PRs (30d)
14

Description

Description
===========

The `Producer` and `Consumer` can use Python logging via the `logger` configuration option. This works well as long as the program is running. Unfortunately, the last log messages get lost when the program exits. Those messages can only be seen if the logger feature is not used.

How to reproduce
================

**logger.py:**
```python
import logging
import sys
from confluent_kafka import Producer, Consumer
import confluent_kafka

logger = logging.getLogger("kafka")
logger.setLevel(logging.DEBUG)
logger.addHandler(logging.StreamHandler())

c = Consumer({
"bootstrap.servers": "not-available-broker",
"group.id": "my-group",
"debug": "msg",
"logger": logger if sys.argv[-1] == "logger" else None,
})
c.poll(0)
c.close()
print("version:", confluent_kafka.version(), "lib-version:", confluent_kafka.libversion())
```

**output with logger:**
```
$ ./logger.py logger
INIT [rdkafka#consumer-1] [thrd:app]: librdkafka v2.3.0 (0x20300ff) rdkafka#consumer-1 initialized (builtin.features gzip,snappy,ssl,sasl,regex,lz4,sasl_plain,sasl_scram,plugins,zstd,sasl_oauthbearer,http,oidc, STRIP STATIC_LINKING GCC GXX PKGCONFIG INSTALL GNULD LIBDL PLUGINS ZLIB SSL ZSTD CURL HDRHISTOGRAM SYSLOG SNAPPY SOCKEM SASL_SCRAM SASL_OAUTHBEARER OAUTHBEARER_OIDC CRC32C_HW, debug 0x40)
version: ('2.3.0', 33751040) lib-version: ('2.3.0', 33751295)
```

**output without logger:**
```
$ ./logger.py
%7|1709646389.068|INIT|rdkafka#consumer-1| [thrd:app]: librdkafka v2.3.0 (0x20300ff) rdkafka#consumer-1 initialized (builtin.features gzip,snappy,ssl,sasl,regex,lz4,sasl_plain,sasl_scram,plugins,zstd,sasl_oauthbearer,http,oidc, STRIP STATIC_LINKING GCC GXX PKGCONFIG INSTALL GNULD LIBDL PLUGINS ZLIB SSL ZSTD CURL HDRHISTOGRAM SYSLOG SNAPPY SOCKEM SASL_SCRAM SASL_OAUTHBEARER OAUTHBEARER_OIDC CRC32C_HW, debug 0x40)
%7|1709646389.068|DESTROY|rdkafka#consumer-1| [thrd:app]: Terminating instance (destroy flags none (0x0))
%7|1709646389.068|DESTROY|rdkafka#consumer-1| [thrd:main]: Destroy internal
%7|1709646389.068|DESTROY|rdkafka#consumer-1| [thrd:main]: Removing all topics
%3|1709646389.114|FAIL|rdkafka#consumer-1| [thrd:not-available-broker:9092/bootstrap]: not-available-broker:9092/bootstrap: Failed to resolve 'not-available-broker:9092': Temporary failure in name resolution (after 46ms in state CONNECT)
version: ('2.3.0', 33751040) lib-version: ('2.3.0', 33751295)
```

Expected behaviour
=================

Both outputs should be similar.

`Consumer.close()` should really terminate the consumer. The `Producer` should provide a `close()` method to terminate it cleanly.

Checklist
=========
Please provide the following information:

- [x] confluent-kafka-python and librdkafka version (`confluent_kafka.version()` and `confluent_kafka.libversion()`):
- [x] Apache Kafka broker version: not relevant
- [x] Client configuration: `{...}`
- [x] Operating system: Ubuntu 22.04
- [x] Provide client logs (with `'debug': '..'` as necessary)
- [ ] Provide broker log excerpts
- [ ] Critical issue

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.