linkedin / linkedin/dynamometer
Hadoop 3.0 NN drops replicas silently
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 134
- Forks
- 34
- PR merge metrics
- No merged PRs in 30d
Description
In Hadoop 3.0/CDH5.7 and above,
HDFS-9260 (Improve the performance and GC friendliness of NameNode startup and full block reports) changed the internal representation of block replicas, as well as the block report processing logic in NameNode.
After HDFS-9260, NN expects block replicas to be reported in ascending order of block id. If a block id is not in order, NN discards it silently. Because simulated DataNode in Dynamometer uses hash map to store block replicas, the replicas are not reported in order. The Dynamometer cluster would then see missing blocks gradually increase several minutes after NN starts.
Suggest to change SimulatedBPStorage.blockMap to a TreeMap sorted by block id. Will supply a patch for the proposed change.
Credit: @fangyurao for identifying the issue, and help verifying the fix.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Locate SimulatedBPStorage.blockMap and review how simulated DataNode block replicas are stored and reported. Verify the behavior against the NameNode block-report ordering requirement described in the issue, then run the relevant Dynamometer or HDFS simulation tests if available. Done means replicas are reported in ascending block-id order without blocks being silently dropped.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- hadoop, java
- Domain
- distributed-systems, testing
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100