stackabletech / stackabletech/hdfs-operator

Duplicate datanodes after a disk failure

Open
#285 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

area/documentation customer-request
Dominant language
Rust
Stars
53
Forks
9
Avg merge
1d 13h
Merged PRs (30d)
10

Description

Affected version

0.6.0

Current and expected behavior

In an HDFS cluster with 2 namenodes and 3 datanodes using local disks, one of the datanodes had a disk failure.
After fixing the disk issue and restarting the datanode, this node showed up multiple times in the datanode list.

The node did register and become an active datanode, but had two rogue identities remaining listed as dead in the datanode list.

4.4$ /stackable/hadoop/bin/hdfs dfsadmin -report
Configured Capacity: 643930521600 (599.71 GB)
Present Capacity: 637323175133 (593.55 GB)
DFS Remaining: 637294691549 (593.53 GB)
DFS Used: 28483584 (27.16 MB)
DFS Used%: 0.00%
Replicated Blocks:
        Under replicated blocks: 0
        Blocks with corrupt replicas: 0
        Missing blocks: 0
        Missing blocks (with replication factor 1): 0
        Low redundancy blocks with highest priority to recover: 0
        Pending deletion blocks: 0
Erasure Coded Block Groups:
        Low redundancy block groups: 0
        Block groups with corrupt internal blocks: 0
        Missing block groups: 0
        Low redundancy blocks with highest priority to recover: 0
        Pending deletion blocks: 0
-------------------------------------------------
Live datanodes (3):
Name: 100.64.13.117:9866 (simple-hdfs-datanode-default-1.simple-hdfs-datanode-default.stack-dev.svc.cluster.local)
Hostname: simple-hdfs-datanode-default-1.simple-hdfs-datanode-default.stack-dev.svc.cluster.local
Decommission Status : Normal
Configured Capacity: 214643507200 (199.90 GB)
DFS Used: 9478144 (9.04 MB)
Non DFS Used: 1531338752 (1.43 GB)
DFS Remaining: 212431602079 (197.84 GB)
DFS Used%: 0.00%
DFS Remaining%: 98.97%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 10
Last contact: Mon Dec 12 10:14:04 UTC 2022
Last Block Report: Mon Dec 12 08:07:50 UTC 2022
Num of Blocks: 861
Name: 100.64.43.71:9866 (100-64-43-71.simple-hdfs-datanode-default-2.stack-dev.svc.cluster.local)
Hostname: simple-hdfs-datanode-default-2.simple-hdfs-datanode-default.stack-dev.svc.cluster.local
Decommission Status : Normal
Configured Capacity: 214643507200 (199.90 GB)
DFS Used: 9502720 (9.06 MB)
Non DFS Used: 1531371520 (1.43 GB)
DFS Remaining: 212431544735 (197.84 GB)
DFS Used%: 0.00%
DFS Remaining%: 98.97%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 10
Last contact: Mon Dec 12 10:14:04 UTC 2022
Last Block Report: Mon Dec 12 05:26:22 UTC 2022
Num of Blocks: 861
Name: 100.64.45.118:9866 (100-64-45-118.simple-hdfs-datanode-default-0.stack-dev.svc.cluster.local)
Hostname: simple-hdfs-datanode-default-0.simple-hdfs-datanode-default.stack-dev.svc.cluster.local
Decommission Status : Normal
Configured Capacity: 214643507200 (199.90 GB)
DFS Used: 9502720 (9.06 MB)
Non DFS Used: 1531371520 (1.43 GB)
DFS Remaining: 212431544735 (197.84 GB)
DFS Used%: 0.00%
DFS Remaining%: 98.97%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 10
Last contact: Mon Dec 12 10:14:04 UTC 2022
Last Block Report: Mon Dec 12 06:52:28 UTC 2022
Num of Blocks: 861
Dead datanodes (2):
Name: 100.64.13.116:9866 (100.64.13.116)
Hostname: simple-hdfs-datanode-default-1.simple-hdfs-datanode-default.stack-dev.svc.cluster.local
Decommission Status : Normal
Configured Capacity: 214643507200 (199.90 GB)
DFS Used: 53775 (52.51 KB)
Non DFS Used: 1533402609 (1.43 GB)
DFS Remaining: 213110050816 (198.47 GB)
DFS Used%: 0.00%
DFS Remaining%: 99.29%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 0
Last contact: Mon Dec 12 08:07:09 UTC 2022
Last Block Report: Mon Dec 12 08:05:21 UTC 2022
Num of Blocks: 0
Name: 100.64.48.102:9866 (100.64.48.102)
Hostname: simple-hdfs-datanode-default-1.simple-hdfs-datanode-default.stack-dev.svc.cluster.local
Decommission Status : Normal
Configured Capacity: 214643507200 (199.90 GB)
DFS Used: 7147520 (6.82 MB)
Non DFS Used: 1531076608 (1.43 GB)
DFS Remaining: 212434194847 (197.84 GB)
DFS Used%: 0.00%
DFS Remaining%: 98.97%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 0
Last contact: Fri Dec 09 10:28:20 UTC 2022
Last Block Report: Fri Dec 09 09:13:23 UTC 2022
Num of Blocks: 0

To a certain extent this is expected behavior, as the datanode generates a new datanode uid when it is started up with an empty data directory, as can easily be shown by shutting down a datanode, removing the persistent volume it stores its data on and then restarting it:

# Datanode ID before restart with empty PV
bash-4.4$ cat /data/data/current/VERSION
#Tue Dec 13 11:59:14 UTC 2022
datanodeUuid=f6ee6a85-1ef1-427c-8a59-b1025ce7560f
storageType=DATA_NODE
cTime=0
clusterID=CID-d3fe3677-072d-4699-b2cf-d4d24414c62c
layoutVersion=-57
storageID=DS-976c3639-f61b-4acb-bbd0-6e4d7af2da92
bash-4.4$

# Datanode ID after restart with empty PV
bash-4.4$ cat /data/data/current/VERSION
#Tue Dec 13 12:24:05 UTC 2022
datanodeUuid=0ea49c48-09c3-4443-9d91-c767ed5e0426
storageType=DATA_NODE
cTime=0
clusterID=CID-d3fe3677-072d-4699-b2cf-d4d24414c62c
layoutVersion=-57
storageID=DS-9c219b84-bceb-4e72-900f-c569442978f2
bash-4.4$

Here you can see that the datanodeUuid changes over the restart, and since this uid is used by the namenode to look up if this datanode is known to the cluster, the node registers as a new node after the restart.

This can be mitigated by using multiple data directories, as the uuid is stored in all of the directories and recovered to an empty one in the case of disk failure.

Possible solution

Document this behavior, especially the effect of having multiple data directories and workarounds for removing the dead datanodes from the nodelist (restart namenodes).

Additional context

No response

Environment

No response

Would you like to work on fixing this bug?

None

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the HDFS datanode identity example in /data/data/current/VERSION and the hdfs dfsadmin -report output. Document the effect of multiple data directories and the workaround of restarting namenodes to remove dead datanodes from the node list; done means these disk-failure behaviors and mitigations are covered.

Written by the indexing model from the issue text.

Assessment

Domain
distributed-systems, documentation
Issue type
Documentation
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.