apache / apache/accumulo

Add shell command for marking tservers as "dead"

Open
#3,138 7 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
1.2k
Forks
487
Avg merge
4d 5h
Merged PRs (30d)
13

Description

**Is your feature request related to a problem? Please describe.**

***Problem***
There is an operational situation where a tserver cannot be terminated locally (either ssh or physical access) and is causing issues in accumulo.
The tserver needs to marked as "Dead" and accumulo will not automatically change the tserver state due to operational health checks still passing.

***Current workaround***
In situations where an admin cannot ssh into or remotely terminate a failed node running tservers, an admin needs to be able to mark the tservers on that node as "dead" so accumulo starts reassigning tablets to other tservers.

Currently this has been handled by getting the active zlock for that specific tserver from zookeeper and then running a delete zlock command via the zookeeper shell.

This seems risky and complicated for what should be a simple operation.

**Describe the solution you'd like**
As an Admin, I would like the ability to mark tservers as "dead" via the accumulo shell with requiring ssh access to the specific tservers.

I should be able to do this via a manual accumulo shell command on either a per-tserver basis or via a tserver regex to match all tservers against a specific node.
This command should prompt the Admin with a confirmation action and a preview list of tservers to mark 'dead'"

**Describe alternatives you've considered**
Alternative solution would be increasing the ability for accumulo to detect issues with the tservers.
However this doesn't help with node maintenance actions that may require taking tservers offline without ssh access.

**Additional context**
Add any other context or screenshots about the feature request here.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing existing Accumulo shell commands and the tserver state handling behind the current ZooKeeper active-zlock workaround. Trace how a tserver is identified and marked dead, then define the preview and confirmation flow for a single tserver or node-matching regex. Done means the command safely supports both selection modes and triggers tablet reassignment without requiring SSH access.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
cli, distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.