confluentinc / confluentinc/confluent-kafka-python

Support for connect_cb

Open
#1,097 8 comments 0 reactions 0 assignees View on GitHub
code:C enhancement investigate further status:waiting-for-interest
Dominant language
Python
Stars
509
Forks
964
Avg merge
2d 2h
Merged PRs (30d)
14

Description

We have an experimental patch which adds support for setting `connect_cb` as a configuration, however looking back we found [this issue from 2017
](https://github.com/confluentinc/confluent-kafka-python/issues/219#issuecomment-315983155) which says that adding it would be hard.

And it is! By doing so we discovered a deadlock that would occur since the `connect_cb` is called in a different thread than the other callback configs due to in part the GIL being held during calls to `rd_kafka_assign`. Trying to solve this, we released the GIL for the duration of `rd_kafka_assign` and things seem to be working well now.

That context is here because I'm curious to know:
1. Is supporting `connect_cb` hard for other specific reasons?
2. Does the assign call need to hold the GIL for other safety reasons?

Basically I'm wondering if there are some invariants I may be stepping on with these changes. Thanks.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.