[enhancement][broker] Introduce thread block monitoring mechnism.
- Dominant language
- Java
- Stars
- 15.3k
- Forks
- 3.8k
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 160
Description
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
### Motivation
Although it was an old story that worker threads maybe blocked by blocking operations.
But for earlier version we still found blocking occurs in our env.
I think we can just `ping` those worker thread to check if the thread is blocking by send a task in the thread task queue.
If thread is blocking for long time, `ping` task won't be executed. so we can add an alart to notify us potential thread blocking and restart the process.
This complete the current mechnism,
current we only have thread task queue size, complete task number by enable task tracing.
but they are diffcult to set an alart to justify the blocking condition and only support `OrderedExecutor`.
### Solution
_No response_
### Alternatives
_No response_
### Anything else?
_No response_
### Are you willing to submit a PR?
- [X] I'm willing to submit a PR!
Contributor guide
Research direction
Start by reviewing the existing thread task queue, task tracing, and OrderedExecutor mechanisms mentioned in the issue. Define how a ping detects a blocked worker, what alert and restart behavior is required, and which tests would demonstrate the completed monitoring mechanism.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100