ctengel / ctengel/simpler-objects

find_object returns false 404 for objects whose only copy is on a sleeping disk

Open
#75 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
0
Forks
0
PR merge metrics
No merged PRs in 30d

Description

Split out from the #72 timeout work (the bucket-listing path was fixed there; this single-object path was intentionally left as-is).

## Problem

`find_object` (`GET`/`HEAD /{bucket}/{key}` on the locator, `locator_api.py:68`) HEADs each object server with `timeout=1`:

```python
result = await client.head(server + object_path, timeout=1)
```

By design, a transport error / timeout here is treated as "this server doesn't have it" — the loop just `continue`s, and if no server returns 200 the locator returns **404** (only an explicit upstream 503 escalates to 503; see the `find_object` comment block and the locator-busy-vs-missing reasoning).

That design is fine when the server genuinely lacks the object. But a sleeping HDD takes ~15s to spin up (per #72, "even 15 seconds is normal"). So if the **only** copy of an object lives on a server whose disk is asleep, the HEAD times out at 1s, the server is skipped, and the client gets a **404 for an object that actually exists**. Unlike a 503, a 404 gives the client no reason to retry — the object effectively "disappears" until the disk happens to be awake.

This is the "1/3 single-object errors went unnoticed" case @ctengel described in #72: on the current 1/3-SSD / 2/3-HDD layout with 2x replication, most objects have a fast SSD copy, masking it. On an all-HDD cluster it would be common.

Note this is a **correctness** symptom (a present object reported absent), not just an availability/latency one like the #72 bucket-listing 503.

## Options (not mutually exclusive)

- Lengthen the `find_object` HEAD timeout to tolerate spin-up (~16s), accepting slower GETs when a disk is cold. Trade-off: a genuinely-down server now stalls the per-server leg for the full timeout.
- Distinguish "timed out / unreachable" from "confirmed 404" and, when at least one server timed out (but none confirmed the object and none returned 503), respond **503 + Retry-After** rather than 404, so the client retries instead of treating it as missing.
- Metadata caching / a "disk asleep" health signal so the locator can answer without waking the disk and/or give the slow server a few seconds (see #72 discussion and #1, #3).

## Related

- #72 — bucket-listing 503 from the same sleeping-disk root cause (fixed for `GET /bucket/`)
- #63 — partial results
- #1, #3 — metadata caching

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.