apple / apple/foundationdb

Injected disk error in AsyncFileChaos is not tracked and causing StorageServerDurabilityError

Open
#9,747 1 comment 0 reactions 0 assignees View on GitHub
bug testing
Dominant language
C++
Stars
16.7k
Forks
1.6k
Avg merge
1d 20h
Merged PRs (30d)
126

Description

The fault injection is here
https://github.com/apple/foundationdb/blob/27e1b76b68990422418ab7195f35494286117e71/fdbrpc/include/fdbrpc/AsyncFileChaos.h#L99-L105

This error can later cause storage server `SevError` of `StorageServerDurabilityError` here https://github.com/apple/foundationdb/blob/27e1b76b68990422418ab7195f35494286117e71/fdbserver/storageserver.actor.cpp#L10408

In 7.2 cherrypicks https://github.com/apple/foundationdb/pull/9732, commit 79d0687e5, seed
` -f ./tests/slow/DiskFailureCycle.toml -s 282036857 -b off`. I found that the corruption happened for one of the disk queue file, but the page was copied to another file and was discarded after the reboot, which makes it hard to track the dirty page.

```
120.205235 CorruptedByteInjection ID=0000000000000000 Filename=/root/simfdb/fa11e779fcbb57e0bfa2a1dcb38615c4/storage-fb3b1d127a41c6afa254b6d5dc693cfb-1.fdq Position=293707 BK,CC,CP,SS

126.428277 DQRecInvalidPage ID=fb3b1d127a41c6af NextReadLocation=1093668 HashCheck=0 Seq=1093632 Expect=1093632 File0Name=/root/simfdb/fa11e779fcbb57e0bfa2a1dcb38615c4/storage-fb3b1d127a41c6afa254b6d5dc693cfb-0.fdq SS
126.428277 DQTruncateFile ID=fb3b1d127a41c6af File=1 Pos=290816 File0Name=/root/simfdb/fa11e779fcbb57e0bfa2a1dcb38615c4/storage-fb3b1d127a41c6afa254b6d5dc693cfb-0.fdq SS
126.428477 FindPhysicalLocation ID=fb3b1d127a41c6af Page0Valid=1 Page0Seq=425984 Page1Valid=1 Page1Seq=802816 Location=946986 Context=lastPoppedSeq File0Name=/root/simfdb/fa11e779fcbb57e0bfa2a1dcb38615c4/storage-fb3b1d127a41c6afa254b6d5dc693cfb-0.fdq SS
126.428477 FoundPhysicalLocation ID=fb3b1d127a41c6af PageIndex=1 PageLocation=35 SizeofPage=4096 PageSequence=802816 Location=946986 Context=lastPoppedSeq File0Name=/root/simfdb/fa11e779fcbb57e0bfa2a1dcb38615c4/storage-fb3b1d127a41c6afa254b6d5dc693cfb-0.fdq SS
126.428477 DQTruncateFile ID=fb3b1d127a41c6af File=0 Pos=0 File0Name=/root/simfdb/fa11e779fcbb57e0bfa2a1dcb38615c4/storage-fb3b1d127a41c6afa254b6d5dc693cfb-0.fdq SS

126.428877 StorageServerDurabilityError ID=fb3b1d127a41c6af RestoredVersion=302130779 Checking=min MinVersion=303134785 MaxVersion=303722114 Backtrace=addr2line -e fdbserver.debug -p -C -f -i 0x47a9867 0x47a9b15 0x47a44a4 0x46c42f5 0x46c455a 0x2c9876c 0x2c97e19 0x2c97750 0x2c16c2e 0x2cc4fa6 0x2cc3cd7 0x22727c8 0x21872f8 0x21871a6 0x2187079 0x2187aae 0x2176c38 0x46b0888 0x46b0594 0x288ce38 0x17278a8 0x4696e03 0x469691a 0x2b7e6bc 0x7f0b6fe49555 SS
```

Contributor guide

Open the contributing guide

Research direction

Read fdbrpc/include/fdbrpc/AsyncFileChaos.h at the fault-injection lines and fdbserver/storageserver.actor.cpp around StorageServerDurabilityError. Reproduce with tests/slow/DiskFailureCycle.toml using seed 282036857 and inspect the injected-error and disk-queue traces; done means the injected failure is tracked through the relevant recovery path without the reported unaccounted durability error.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.