apache / apache/pulsar

[enhancement][broker] Introduce thread block monitoring mechnism.

Open
#18,959 1 comment 0 reactions 0 assignees View on GitHub
Stale type/enhancement
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.

### Motivation

Although it was an old story that worker threads maybe blocked by blocking operations.
But for earlier version we still found blocking occurs in our env.

I think we can just `ping` those worker thread to check if the thread is blocking by send a task in the thread task queue.
If thread is blocking for long time, `ping` task won't be executed. so we can add an alart to notify us potential thread blocking and restart the process.

This complete the current mechnism,
current we only have thread task queue size, complete task number by enable task tracing.
but they are diffcult to set an alart to justify the blocking condition and only support `OrderedExecutor`.

### Solution

_No response_

### Alternatives

_No response_

### Anything else?

_No response_

### Are you willing to submit a PR?

- [X] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the existing thread task queue, task tracing, and OrderedExecutor mechanisms mentioned in the issue. Define how a ping detects a blocked worker, what alert and restart behavior is required, and which tests would demonstrate the completed monitoring mechanism.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.