When using the Python pulsar client, crashes occasionally happen on connect/disconnect/close due to pthread_mutex_lock assertion failures
- Dominant language
- Java
- Stars
- 15.3k
- Forks
- 3.8k
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 160
Description
**Describe the bug**
Occasionally, in flakey CI tests that run Python code in a heavily threaded environment, we have segfaults when calling `connect`, `create_producer`, or `producer.close()` on Pulsar client objects.
I wish I had more debugging info or a full C system call trace, but the failures occur only in CI where I can't use `gdb`, and Python's `faulthandler` doesn't seem to provide a stacktrace unfortunately.
The errors always have these characteristics:
- They exist in an environment with many threads.
- Some of the threads have previously used a Pulsar client, and still exist, but are not using that client.
- The current thread attempting to use the Pulsar client is doing a `connect`, `create_producer`, or `producer.close()` operation.
- The error line is either an unhandled SIGSEGV, or an abort with the description `[__pthread_mutex_lock: Assertion `mutex->__data.__owner == 0' failed](https://stackoverflow.com/questions/9239999/pthread-mutex-lock-c62-pthread-mutex-lock-assertion-mutex-data-owner)`.
I'm sorry I don't have more specific debugging or reproduction information.
This only occurs on client 2.10.0; we're using that client with Python 3.7.13 on Linux, in Docker, x86_64 arch.
Contributor guide
Research direction
Start by investigating the Python client paths exercised by connect, create_producer, and producer.close() in heavily threaded Linux CI environments. The report provides no file, test, or reproducible case; done would require a reproducible failure or sufficient diagnostics to identify and verify a fix for the pthread_mutex_lock assertion or segmentation fault.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, linux, python
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100