ctengel / ctengel/simpler-objects

Complete the sleeping-disk timeout sweep on the write & replication paths

Open
#77 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
0
Forks
0
PR merge metrics
No merged PRs in 30d

Description

Follow-up to #72. The fix there raised the timeouts on the **bucket-listing read path** (`list_bucket` / `head_bucket` fan-out and the replicator's `auto_replica` / `get_bucket_contents` client legs). But the same sleeping-disk root cause — an HDD takes ~15s to spin up, "even 15 seconds is normal" — still bites two other places that were left at `timeout=1`/`timeout=2`. Both run a real risk because the replicator runs off-hours, exactly when drives are spun down.

(The two single-object correctness analogues are tracked separately: #75 read false-404, #76 write duplicate.)

## 1. `add_object` bucket-existence check → spurious 507

`locator_api.py:120`, the final write-candidate stage:

```python
async def check_bucket(server):
try:
result = await client.head(server + bucket + "/", timeout=1)
result.raise_for_status()
return server, True
except httpx.HTTPError:
return server, False
```

A sleepy candidate's bucket HEAD times out at 1s → `ok=False` → the candidate is pruned. If every candidate is pruned this way the locator returns **507 (no space)** even though space exists — a write fails purely because a disk was asleep.

## 2. Replicator direct object-server HEADs → false "no space" / aborted run

The replicator talks straight to object servers (no locator in front), so it hits cold disks directly:

- `async_replicate.py:21` (`find_space`), `httpx.head(server + bucket + "/", timeout=1)` — a sleepy replication target times out, gets popped, and you get a false `No space to replicate object` warning + non-zero exit.
- `async_replicate.py:33` (`get_object_size`), `httpx.head(obj, timeout=2)` — a sleepy source or destination times out and raises, aborting the replication of that object (and the run).

Because replication runs during off-hours when drives are spun down, this is the **most likely** real-world trigger of the whole sleeping-disk class.

## Suggested fix

Raise these to a spin-up-tolerant timeout (~16s, matching what #72 used for the locator fan-out; keep the ladder client ≥ locator-internal ≥ ~15s), or adopt whichever distinguish-asleep-from-down strategy comes out of #72/#75/#76. Out of scope here: `get_object_server_health` (`locator_api.py:39`, `timeout=1`) is left short on purpose so genuinely-down nodes fail fast for candidate selection, and `statvfs` doesn't spin a disk anyway.

## Related

- #72 — read-path fix this completes
- #75 — read-path false 404 (single object)
- #76 — write-path duplicate (single object)
- #49 — data-path streaming timeout
- #63 — partial results; #1, #3 — metadata caching

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.