apache / apache/pulsar

When using the Python pulsar client, crashes occasionally happen on connect/disconnect/close due to pthread_mutex_lock assertion failures

Open
#16,009 1 comment 0 reactions 0 assignees View on GitHub
Stale type/bug
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

**Describe the bug**
Occasionally, in flakey CI tests that run Python code in a heavily threaded environment, we have segfaults when calling `connect`, `create_producer`, or `producer.close()` on Pulsar client objects.

I wish I had more debugging info or a full C system call trace, but the failures occur only in CI where I can't use `gdb`, and Python's `faulthandler` doesn't seem to provide a stacktrace unfortunately.

The errors always have these characteristics:
- They exist in an environment with many threads.
- Some of the threads have previously used a Pulsar client, and still exist, but are not using that client.
- The current thread attempting to use the Pulsar client is doing a `connect`, `create_producer`, or `producer.close()` operation.
- The error line is either an unhandled SIGSEGV, or an abort with the description `[__pthread_mutex_lock: Assertion `mutex->__data.__owner == 0' failed](https://stackoverflow.com/questions/9239999/pthread-mutex-lock-c62-pthread-mutex-lock-assertion-mutex-data-owner)`.

I'm sorry I don't have more specific debugging or reproduction information.

This only occurs on client 2.10.0; we're using that client with Python 3.7.13 on Linux, in Docker, x86_64 arch.

Contributor guide

Open the contributing guide

Research direction

Start by investigating the Python client paths exercised by connect, create_producer, and producer.close() in heavily threaded Linux CI environments. The report provides no file, test, or reproducible case; done would require a reproducible failure or sufficient diagnostics to identify and verify a fix for the pthread_mutex_lock assertion or segmentation fault.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, linux, python
Domain
distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.