confluentinc / confluentinc/librdkafka

Increased number of `rd_kafka_cgrp_terminated` with 2.5.0 and shutdown stability degradation

Open
#4,792 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
1k
Forks
3.3k
Avg merge
1d 8h
Merged PRs (30d)
5

Description

Description
===========

I do not know yet the reason, but after the upgrade of librdkafka to 2.5.0 (without any more changes) the Karafka ecosystem CI crashes more often with:

```
*** rdkafka_cgrp.c:3312:rd_kafka_cgrp_terminated: assert: !rd_kafka_assignment_in_progress(rkcg->rkcg_rk) ***
Aborted (core dumped)
```

on shutdown and the consumer destroy also hangs, which has not happened with 2.4.0.

https://github.com/confluentinc/librdkafka/blob/6eaf89fb124c421b66b43b195879d458a3a31f86/src/rdkafka_cgrp.c#L3312

I recall a different issue a while ago having a similar problem where the solution was to gracefully unsubscribe the consumer prior to the shutdown. I wonder if this would mitigate this as well :thinking:

update: I can now trigger segfaults. Not sure yet exactly why but at least I can crash it on my machine.

How to reproduce
================

I was unable to reproduce it so far in an isolated environment, and my stress tests on my test setup do not show this behavior. However, all of my previous reports (including fixed) would always have some specs failing on valid issues. I will keep investigating and will provide more details when available.

Checklist
=========

**IMPORTANT**: We will close issues where the checklist has not been completed.

Please provide the following information:

- [x] librdkafka version (release number or git tag): `2.5.0` (not happening with `2.4.0`)
- [x] Apache Kafka version: `confluentinc/cp-kafka:7.6.1`
- [x] librdkafka client configuration: absolute defaults
- [x] Operating system: ubuntu-latest from Github Actions CI shared runner
- [ ] Provide logs (with `debug=..` as necessary) from librdkafka (will be provided, trying to repro with logs) - I cannot at this stage because only crashes in CI which runs without log collection (will work on this)
- [ ] Provide broker log excerpts - same as above. When CI crashes, VM is shutdown. I will try introducing a mode with heavy debug tracing.
- [ ] Critical issue

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.