deepseek-ai / deepseek-ai/3FS

storage failover的场景,个别target处于OFFLINE状态,无法自愈

Open
#345 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
10.2k
Forks
1.1k
PR merge metrics
No merged PRs in 30d

Description

出现问题的场景如下:
三副本同步NodeA -> NodeB -> NodeC,在写数据时,最后一个节点NodeC完成了commit操作,返回response给前驱节点前进程crash了,当前在处理的chunk的状态如下:
updateVer == commitVer == $seqNum
chunkState == COMMIT
而此时,前驱节点engine的状态如下
updateVer == $seqNum
commitVer == $seqNum - 1
chunkState == CLEAN

此时NodeC重启恢复数据,在跟NodeB进行resync期间,在如下阶段报错返回
`

} else if (meta.checksum() != remoteMeta.checksum) {

if (meta.chainVer != vChainId.chainVer) {
XLOGF(DFATAL, "chain {} checksum not equal, local {}, remote {}", vChainId, meta, remoteMeta);
++currentSyncingRemoteFullSyncLightCount;
hasFatalEvents = true;
break;
} else {

`
这个场景算是个常规case,长期卡在OFFLINE的状态,应该是不符合预期的。此外,代码中判断meta.chainVer != vChainId.chainVer也有点匪夷所思,有哪些特殊考虑吗? 为什么要拿chain最新的版本去跟engine存量数据去比较?NodeC停服期间,chainVer还是有可能发生变化的。

修复:
是不是可以考虑在meta.chainVer != vChainId.chainVer判断之前,跳过remoteMeta.chunkState == ChunkState::COMMIT的case?

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.