stackabletech / stackabletech/hbase-operator

Enable TaskMonitor warnings of slow tasks by default

Open
#278 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

type/feature-improvement
Dominant language
Rust
Stars
21
Forks
6
Avg merge
1d 19h
Merged PRs (30d)
9

Description

HBase has a TaskMonitor that can warn on any tasks that take too long.
https://github.com/apache/hbase/blob/47996d6c2128815e45bb8bdb6e3a470bfddd6106/hbase-server/src/main/java/org/apache/hadoop/hbase/monitoring/TaskMonitor.java#L49

By default it does not warn as the setting hbase.taskmonitor.rpc.warn.time defaults to 0 which disables the check. But it's definitely useful to have a warning like this.

We saw tasks stuck for a week at a customer and this went unnoticed. As I'm not 100% sure on which tasks (i.e. also Procedures?) are monitored I suggest a rather high threshold.
Maybe 1 hour? (Even though most tasks should finish within seconds)

This is a warning that we then - later - should extract from the logs to convert into an alert and/or at least have a dashboard (in Grafana or Opensearch Dashboard) to show these.
The log message will start with: "Task may be stuck"

Also note that older versions have a bug that can wrong warnings: https://issues.apache.org/jira/browse/HBASE-22935

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the linked hbase-server/src/main/java/org/apache/hadoop/hbase/monitoring/TaskMonitor.java and inspect how hbase.taskmonitor.rpc.warn.time disables the check. Confirm which tasks are monitored and settle the default threshold, such as one hour. Done means slow-task warnings are enabled by default without the older false-warning behavior described in HBASE-22935.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.