stackabletech / stackabletech/hbase-operator
Enable TaskMonitor warnings of slow tasks by default
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 21
- Forks
- 6
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 9
Description
HBase has a TaskMonitor that can warn on any tasks that take too long.
https://github.com/apache/hbase/blob/47996d6c2128815e45bb8bdb6e3a470bfddd6106/hbase-server/src/main/java/org/apache/hadoop/hbase/monitoring/TaskMonitor.java#L49
By default it does not warn as the setting hbase.taskmonitor.rpc.warn.time defaults to 0 which disables the check. But it's definitely useful to have a warning like this.
We saw tasks stuck for a week at a customer and this went unnoticed. As I'm not 100% sure on which tasks (i.e. also Procedures?) are monitored I suggest a rather high threshold.
Maybe 1 hour? (Even though most tasks should finish within seconds)
This is a warning that we then - later - should extract from the logs to convert into an alert and/or at least have a dashboard (in Grafana or Opensearch Dashboard) to show these.
The log message will start with: "Task may be stuck"
Also note that older versions have a bug that can wrong warnings: https://issues.apache.org/jira/browse/HBASE-22935
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the linked hbase-server/src/main/java/org/apache/hadoop/hbase/monitoring/TaskMonitor.java and inspect how hbase.taskmonitor.rpc.warn.time disables the check. Confirm which tasks are monitored and settle the default threshold, such as one hour. Done means slow-task warnings are enabled by default without the older false-warning behavior described in HBASE-22935.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100