apache / apache/trafficserver

"Corrupt" range request into last fragment

Open
#6,674 14 comments 0 reactions 0 assignees View on GitHub
Bug Cache Stale
Dominant language
C++
Stars
2k
Forks
874
Avg merge
6d 15h
Merged PRs (30d)
46

Description

We have an interesting production case where a specific fragment of an object (on 1 machine) appears to be corrupted, however, the full file is not:
- the md5 of the full file from $bad machine is equal to the md5 of the full file from $good machine. eg,
```bash
$ curl -s "$url" | md5
a5ad1e4cbbdd080a6f5aff822f891bb5
$ curl -s "$url" --resolve $domain:443:$bad | md5
a5ad1e4cbbdd080a6f5aff822f891bb5
```
- the md5 of every range request past a specific byte number from $bad machine is different than the same range request to $good machine. eg
```bash
$ curl -s -H 'Range: bytes=23068672-23091472’ “$url" | md5
634e9686f4812f9faa929995bc9fa854
$ curl -s -H 'Range: bytes=23068672-23091472' "$url" --resolve $domain:443:$bad | md5
f5badfebd9d9410292ae9ff3d0af905d
```
- if you do a 1 byte range request past that specific byte number on $bad machine, each has the same md5. eg,
```bash
$ for begin in $(jot 25 151); do end=$((b+1)); curl -s --resolve $domain:443:$bad "$url" -r 23067$begin-23067$end | md5; done
791c2566d1af1ec27718457b25be9c7d
791c2566d1af1ec27718457b25be9c7d
791c2566d1af1ec27718457b25be9c7d
791c2566d1af1ec27718457b25be9c7d
791c2566d1af1ec27718457b25be9c7d
791c2566d1af1ec27718457b25be9c7d
...
```
- if you have another machine pull the object from $bad machine, the new object does not see the same issue.
- Restarting ATS does *not* make the issue go away.
- The issue appears both in RAM cache as well as Disk Cache.
- This is the second report we've seen of this. In both cases, the "magic" byte was in the last fragment. In this case, the magic byte was 23067089; the full content length was 23091472 -- 22*2^20 == 23068672 (1583 bytes off).
- PURGEing the object makes that url/machine problem go away.
- So far, gdb and debug log sessions have not yielded more useful insights.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.