Tablets processed by multiple tservers
- Dominant language
- Java
- Stars
- 1.2k
- Forks
- 487
- Avg merge
- 4d 5h
- Merged PRs (30d)
- 13
Description
**Describe the bug**
On a very active 1.9.3 cluster, Zookeeper reached a certain state where it wasn't functioning 100%, causing tservers to see warnings about Transient connections. The tservers did not lose the ZK lock during this time but were spamming Transient connection warnings. In this time, the Master marked the tserver as dead and began reassigning its tablets to other tservers. ZK was able to recovery and allowed most of the tservers to resume functioning normally. The half alive tservers eventually lost their locks and reported rfiles missing because they were still trying to process files that had been assigned to and compacted by other tservers.
**Versions (OS, Maven, Java, and others, as appropriate):**
- Affected version(s) of this project: 1.9.3, 1.10
Timeline:
- Master: tablet 4 was loaded on Tserver1
- Tserver1: MinC of tablet 4 -> Fabcd.rf
- Tserver1: WARN: Saw (possibly) transient exception communicating with ZooKeeper (DistributedWorkQueue:217, ZooReader:161)
- Tserver1: Many WARN: Saw (possibly) transient exception communicating with ZooKeeper
- Master: tablet 4 was loaded on Tserver2
- Tserver2: MajC of tablet 4: Fabcd.rf -> Cefgh.rf
- Tserver1: Lost tablet server lock, exiting.
- Tserver1: Starting MajC of tablet 4: Fabcd.rf, ...
- Tserver1: WARN: Some problem opening map file Fabcd.rf does not exist
- Tserver1 finishes dying
Contributor guide
Research direction
Start by tracing the Master tablet reassignment flow and the tserver lock, ZooKeeper warning, and compaction paths described in the timeline; DistributedWorkQueue and ZooReader are named in the warnings. Reproduce or reason through the sequence where a tserver is marked dead before losing its lock, and define done as preventing both tservers from processing the same tablet and the resulting missing-file errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100