Duplicate datanodes after a disk failure

Affected version

0.6.0

Current and expected behavior

In an HDFS cluster with 2 namenodes and 3 datanodes using local disks, one of the datanodes had a disk failure.
After fixing the disk issue and restarting the datanode, this node showed up multiple times in the datanode list.

The node did register and become an active datanode, but had two rogue identities remaining listed as dead in the datanode list.

4.4$ /stackable/hadoop/bin/hdfs dfsadmin -report
Configured Capacity: 643930521600 (599.71 GB)
Present Capacity: 637323175133 (593.55 GB)
DFS Remaining: 637294691549 (593.53 GB)
DFS Used: 28483584 (27.16 MB)
DFS Used%: 0.00%
Replicated Blocks:
        Under replicated blocks: 0
        Blocks with corrupt replicas: 0
        Missing blocks: 0
        Missing blocks (with replication factor 1): 0
        Low redundancy blocks with highest priority to recover: 0
        Pending deletion blocks: 0
Erasure Coded Block Groups:
        Low redundancy block groups: 0
        Block groups with corrupt internal blocks: 0
        Missing block groups: 0
        Low redundancy blocks with highest priority to recover: 0
        Pending deletion blocks: 0
-------------------------------------------------
Live datanodes (3):
Name: 100.64.13.117:9866 (simple-hdfs-datanode-default-1.simple-hdfs-datanode-default.stack-dev.svc.cluster.local)
Hostname: simple-hdfs-datanode-default-1.simple-hdfs-datanode-default.stack-dev.svc.cluster.local
Decommission Status : Normal
Configured Capacity: 214643507200 (199.90 GB)
DFS Used: 9478144 (9.04 MB)
Non DFS Used: 1531338752 (1.43 GB)
DFS Remaining: 212431602079 (197.84 GB)
DFS Used%: 0.00%
DFS Remaining%: 98.97%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 10
Last contact: Mon Dec 12 10:14:04 UTC 2022
Last Block Report: Mon Dec 12 08:07:50 UTC 2022
Num of Blocks: 861
Name: 100.64.43.71:9866 (100-64-43-71.simple-hdfs-datanode-default-2.stack-dev.svc.cluster.local)
Hostname: simple-hdfs-datanode-default-2.simple-hdfs-datanode-default.stack-dev.svc.cluster.local
Decommission Status : Normal
Configured Capacity: 214643507200 (199.90 GB)
DFS Used: 9502720 (9.06 MB)
Non DFS Used: 1531371520 (1.43 GB)
DFS Remaining: 212431544735 (197.84 GB)
DFS Used%: 0.00%
DFS Remaining%: 98.97%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 10
Last contact: Mon Dec 12 10:14:04 UTC 2022
Last Block Report: Mon Dec 12 05:26:22 UTC 2022
Num of Blocks: 861
Name: 100.64.45.118:9866 (100-64-45-118.simple-hdfs-datanode-default-0.stack-dev.svc.cluster.local)
Hostname: simple-hdfs-datanode-default-0.simple-hdfs-datanode-default.stack-dev.svc.cluster.local
Decommission Status : Normal
Configured Capacity: 214643507200 (199.90 GB)
DFS Used: 9502720 (9.06 MB)
Non DFS Used: 1531371520 (1.43 GB)
DFS Remaining: 212431544735 (197.84 GB)
DFS Used%: 0.00%
DFS Remaining%: 98.97%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 10
Last contact: Mon Dec 12 10:14:04 UTC 2022
Last Block Report: Mon Dec 12 06:52:28 UTC 2022
Num of Blocks: 861
Dead datanodes (2):
Name: 100.64.13.116:9866 (100.64.13.116)
Hostname: simple-hdfs-datanode-default-1.simple-hdfs-datanode-default.stack-dev.svc.cluster.local
Decommission Status : Normal
Configured Capacity: 214643507200 (199.90 GB)
DFS Used: 53775 (52.51 KB)
Non DFS Used: 1533402609 (1.43 GB)
DFS Remaining: 213110050816 (198.47 GB)
DFS Used%: 0.00%
DFS Remaining%: 99.29%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 0
Last contact: Mon Dec 12 08:07:09 UTC 2022
Last Block Report: Mon Dec 12 08:05:21 UTC 2022
Num of Blocks: 0
Name: 100.64.48.102:9866 (100.64.48.102)
Hostname: simple-hdfs-datanode-default-1.simple-hdfs-datanode-default.stack-dev.svc.cluster.local
Decommission Status : Normal
Configured Capacity: 214643507200 (199.90 GB)
DFS Used: 7147520 (6.82 MB)
Non DFS Used: 1531076608 (1.43 GB)
DFS Remaining: 212434194847 (197.84 GB)
DFS Used%: 0.00%
DFS Remaining%: 98.97%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 0
Last contact: Fri Dec 09 10:28:20 UTC 2022
Last Block Report: Fri Dec 09 09:13:23 UTC 2022
Num of Blocks: 0

To a certain extent this is expected behavior, as the datanode generates a new datanode uid when it is started up with an empty data directory, as can easily be shown by shutting down a datanode, removing the persistent volume it stores its data on and then restarting it:

# Datanode ID before restart with empty PV
bash-4.4$ cat /data/data/current/VERSION
#Tue Dec 13 11:59:14 UTC 2022
datanodeUuid=f6ee6a85-1ef1-427c-8a59-b1025ce7560f
storageType=DATA_NODE
cTime=0
clusterID=CID-d3fe3677-072d-4699-b2cf-d4d24414c62c
layoutVersion=-57
storageID=DS-976c3639-f61b-4acb-bbd0-6e4d7af2da92
bash-4.4$

# Datanode ID after restart with empty PV
bash-4.4$ cat /data/data/current/VERSION
#Tue Dec 13 12:24:05 UTC 2022
datanodeUuid=0ea49c48-09c3-4443-9d91-c767ed5e0426
storageType=DATA_NODE
cTime=0
clusterID=CID-d3fe3677-072d-4699-b2cf-d4d24414c62c
layoutVersion=-57
storageID=DS-9c219b84-bceb-4e72-900f-c569442978f2
bash-4.4$

Here you can see that the datanodeUuid changes over the restart, and since this uid is used by the namenode to look up if this datanode is known to the cluster, the node registers as a new node after the restart.

This can be mitigated by using multiple data directories, as the uuid is stored in all of the directories and recovered to an empty one in the case of disk failure.

Possible solution

Document this behavior, especially the effect of having multiple data directories and workarounds for removing the dead datanodes from the nodelist (restart namenodes).

Additional context

No response

Environment

No response

Would you like to work on fixing this bug?

None

Inizia con l'esempio dell'identità del datanode HDFS in /data/data/current/VERSION e con l'output di hdfs dfsadmin -report. Documenta l'effetto di più directory di dati e la soluzione alternativa di riavviare i namenode per rimuovere i datanode inattivi dall'elenco dei nodi; il lavoro è completo quando questi comportamenti e queste mitigazioni in caso di guasto del disco sono coperti.