"Corrupt" range request into last fragment
- Dominant language
- C++
- Stars
- 2k
- Forks
- 874
- Avg merge
- 6d 15h
- Merged PRs (30d)
- 46
Description
We have an interesting production case where a specific fragment of an object (on 1 machine) appears to be corrupted, however, the full file is not:
- the md5 of the full file from $bad machine is equal to the md5 of the full file from $good machine. eg,
```bash
$ curl -s "$url" | md5
a5ad1e4cbbdd080a6f5aff822f891bb5
$ curl -s "$url" --resolve $domain:443:$bad | md5
a5ad1e4cbbdd080a6f5aff822f891bb5
```
- the md5 of every range request past a specific byte number from $bad machine is different than the same range request to $good machine. eg
```bash
$ curl -s -H 'Range: bytes=23068672-23091472’ “$url" | md5
634e9686f4812f9faa929995bc9fa854
$ curl -s -H 'Range: bytes=23068672-23091472' "$url" --resolve $domain:443:$bad | md5
f5badfebd9d9410292ae9ff3d0af905d
```
- if you do a 1 byte range request past that specific byte number on $bad machine, each has the same md5. eg,
```bash
$ for begin in $(jot 25 151); do end=$((b+1)); curl -s --resolve $domain:443:$bad "$url" -r 23067$begin-23067$end | md5; done
791c2566d1af1ec27718457b25be9c7d
791c2566d1af1ec27718457b25be9c7d
791c2566d1af1ec27718457b25be9c7d
791c2566d1af1ec27718457b25be9c7d
791c2566d1af1ec27718457b25be9c7d
791c2566d1af1ec27718457b25be9c7d
...
```
- if you have another machine pull the object from $bad machine, the new object does not see the same issue.
- Restarting ATS does *not* make the issue go away.
- The issue appears both in RAM cache as well as Disk Cache.
- This is the second report we've seen of this. In both cases, the "magic" byte was in the last fragment. In this case, the magic byte was 23067089; the full content length was 23091472 -- 22*2^20 == 23068672 (1583 bytes off).
- PURGEing the object makes that url/machine problem go away.
- So far, gdb and debug log sessions have not yielded more useful insights.
Contributor guide
Assessment
This issue has not been assessed yet.