ctengel / ctengel/simpler-objects

add_object can write a duplicate copy when the existing copy is on a sleeping disk

Open
#76 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
0
Forks
0
PR merge metrics
No merged PRs in 30d

Description

Split out from the #72 sleeping-disk timeout work. This is the **write-path** correctness analogue of #75 (#72 = read 503, #75 = read false-404, this = write false-"doesn't-exist").

## Problem

Before choosing a write target, `add_object` (`PUT /{bucket}/{key}` on the locator) fans out a HEAD to every server to confirm the key doesn't already exist (`locator_api.py:97`):

```python
async def check_exists(server):
try:
result = await client.head(server + object_path, timeout=1)
return server, result.status_code
except httpx.HTTPError:
return server, None
...
for server, status in exist_results:
if status is None:
candidates.pop(server, None)
elif status != 404:
raise HTTPException(status_code=409)
```

A sleeping HDD takes ~15s to spin up (per #72, "even 15 seconds is normal"), so the `timeout=1` HEAD against it raises and returns `status is None`. The code treats that as "no answer, drop this server from the write candidates" and proceeds — it never learns the object already exists there.

If the **only** existing copy of that key is on the sleeping server, the locator picks a *different* server and writes the object there. The result is **two copies of a key the system treats as unique and immutable** — objects are `O_CREAT | O_EXCL`, never overwritten, and the whole design assumes a key resolves to one logical object. The duplicate is silent (no 409), and the two copies can later diverge in how readers/replication see them.

This is a **correctness** symptom (a present object reported absent, leading to a duplicate write), distinct from the availability symptoms in #72.

## Notes / options

- The intended 409-on-conflict path only fires for an *explicit* non-404 status. A timeout currently can't yield 409 because we can't tell "absent" from "asleep" in 1s.
- Lengthening the HEAD timeout to tolerate spin-up (~16s) would let the existence check actually see the sleeping copy, at the cost of slower PUTs when a disk is cold.
- Alternatively, distinguish "timed out / unreachable" from "confirmed 404": if any server's existence check is merely unreachable (not a confirmed 404), refuse to write (e.g. 503 + Retry-After) rather than risk a duplicate — fail safe instead of duplicating.
- Metadata caching / a "disk asleep" health signal would let the locator answer without waking the disk (see #72 discussion, #1, #3).

## Related

- #72 — bucket-listing 503, same sleeping-disk root cause
- #75 — read-path false 404 (sole copy on a sleeping disk)
- #1, #3 — metadata caching

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.