apache / apache/accumulo

Tablets processed by multiple tservers

Open
#1,994 2 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
1.2k
Forks
487
Avg merge
4d 5h
Merged PRs (30d)
13

Description

**Describe the bug**
On a very active 1.9.3 cluster, Zookeeper reached a certain state where it wasn't functioning 100%, causing tservers to see warnings about Transient connections. The tservers did not lose the ZK lock during this time but were spamming Transient connection warnings. In this time, the Master marked the tserver as dead and began reassigning its tablets to other tservers. ZK was able to recovery and allowed most of the tservers to resume functioning normally. The half alive tservers eventually lost their locks and reported rfiles missing because they were still trying to process files that had been assigned to and compacted by other tservers.

**Versions (OS, Maven, Java, and others, as appropriate):**
- Affected version(s) of this project: 1.9.3, 1.10

Timeline:
- Master: tablet 4 was loaded on Tserver1
- Tserver1: MinC of tablet 4 -> Fabcd.rf
- Tserver1: WARN: Saw (possibly) transient exception communicating with ZooKeeper (DistributedWorkQueue:217, ZooReader:161)
- Tserver1: Many WARN: Saw (possibly) transient exception communicating with ZooKeeper
- Master: tablet 4 was loaded on Tserver2
- Tserver2: MajC of tablet 4: Fabcd.rf -> Cefgh.rf
- Tserver1: Lost tablet server lock, exiting.
- Tserver1: Starting MajC of tablet 4: Fabcd.rf, ...
- Tserver1: WARN: Some problem opening map file Fabcd.rf does not exist
- Tserver1 finishes dying

Contributor guide

Open the contributing guide

Research direction

Start by tracing the Master tablet reassignment flow and the tserver lock, ZooKeeper warning, and compaction paths described in the timeline; DistributedWorkQueue and ZooReader are named in the warnings. Reproduce or reason through the sequence where a tserver is marked dead before losing its lock, and define done as preventing both tservers from processing the same tablet and the resulting missing-file errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.