ctengel / ctengel/simpler-objects
find_object returns false 404 for objects whose only copy is on a sleeping disk
- Dominant language
- Python
- Stars
- 0
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
Split out from the #72 timeout work (the bucket-listing path was fixed there; this single-object path was intentionally left as-is).
## Problem
`find_object` (`GET`/`HEAD /{bucket}/{key}` on the locator, `locator_api.py:68`) HEADs each object server with `timeout=1`:
```python
result = await client.head(server + object_path, timeout=1)
```
By design, a transport error / timeout here is treated as "this server doesn't have it" — the loop just `continue`s, and if no server returns 200 the locator returns **404** (only an explicit upstream 503 escalates to 503; see the `find_object` comment block and the locator-busy-vs-missing reasoning).
That design is fine when the server genuinely lacks the object. But a sleeping HDD takes ~15s to spin up (per #72, "even 15 seconds is normal"). So if the **only** copy of an object lives on a server whose disk is asleep, the HEAD times out at 1s, the server is skipped, and the client gets a **404 for an object that actually exists**. Unlike a 503, a 404 gives the client no reason to retry — the object effectively "disappears" until the disk happens to be awake.
This is the "1/3 single-object errors went unnoticed" case @ctengel described in #72: on the current 1/3-SSD / 2/3-HDD layout with 2x replication, most objects have a fast SSD copy, masking it. On an all-HDD cluster it would be common.
Note this is a **correctness** symptom (a present object reported absent), not just an availability/latency one like the #72 bucket-listing 503.
## Options (not mutually exclusive)
- Lengthen the `find_object` HEAD timeout to tolerate spin-up (~16s), accepting slower GETs when a disk is cold. Trade-off: a genuinely-down server now stalls the per-server leg for the full timeout.
- Distinguish "timed out / unreachable" from "confirmed 404" and, when at least one server timed out (but none confirmed the object and none returned 503), respond **503 + Retry-After** rather than 404, so the client retries instead of treating it as missing.
- Metadata caching / a "disk asleep" health signal so the locator can answer without waking the disk and/or give the slow server a few seconds (see #72 discussion and #1, #3).
## Related
- #72 — bucket-listing 503 from the same sleeping-disk root cause (fixed for `GET /bucket/`)
- #63 — partial results
- #1, #3 — metadata caching
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.