The new SeparateBodyFileCache design makes updates/reads non-atomic

The problem

Scenario 1

Program crashes after writing metadata but before writing body. Now the cache thinks the body is empty.

Scenario 2

Multiple processes using the same cache directory. Process A writes metadata, but not yet body. Process B gets metadata, then gets body, and now it thinks the body is empty.

Workarounds

For pip I added logic that basically pretends a cache entry is missing if it doesn't have both body and metadata files.

Solutions

Probably need both a new API that involves writing and reading both at once, and an implementation that makes sure that's atomic. See also the somewhat related #325.

Start by examining the SeparateBodyFileCache design described in the issue and the related discussion in #325. Define how metadata and body reads and writes should behave when a process crashes or multiple processes share a cache directory; done means incomplete entries are not observed as valid cache entries and both operations are atomic.