Complete the sleeping-disk timeout sweep on the write & replication paths #77

Follow-up to #72. The fix there raised the timeouts on the bucket-listing read path (list_bucket / head_bucket fan-out and the replicator's auto_replica / get_bucket_contents client legs). But the same sleeping-disk root cause — an HDD takes ~15s to spin up, "even 15 seconds is normal" — still bites two other places that were left at timeout=1/timeout=2. Both run a real risk because the replicator runs off-hours, exactly when drives are spun down.

(The two single-object correctness analogues are tracked separately: #75 read false-404, #76 write duplicate.)

1. add_object bucket-existence check → spurious 507

locator_api.py:120, the final write-candidate stage:

async def check_bucket(server):
    try:
        result = await client.head(server + bucket + "/", timeout=1)
        result.raise_for_status()
        return server, True
    except httpx.HTTPError:
        return server, False

A sleepy candidate's bucket HEAD times out at 1s → ok=False → the candidate is pruned. If every candidate is pruned this way the locator returns 507 (no space) even though space exists — a write fails purely because a disk was asleep.

2. Replicator direct object-server HEADs → false "no space" / aborted run

The replicator talks straight to object servers (no locator in front), so it hits cold disks directly:

• async_replicate.py:21 (find_space), httpx.head(server + bucket + "/", timeout=1) — a sleepy replication target times out, gets popped, and you get a false No space to replicate object warning + non-zero exit.
• async_replicate.py:33 (get_object_size), httpx.head(obj, timeout=2) — a sleepy source or destination times out and raises, aborting the replication of that object (and the run).

Because replication runs during off-hours when drives are spun down, this is the most likely real-world trigger of the whole sleeping-disk class.

Suggested fix

Raise these to a spin-up-tolerant timeout (~16s, matching what #72 used for the locator fan-out; keep the ladder client ≥ locator-internal ≥ ~15s), or adopt whichever distinguish-asleep-from-down strategy comes out of #72/#75/#76. Out of scope here: get_object_server_health (locator_api.py:39, timeout=1) is left short on purpose so genuinely-down nodes fail fast for candidate selection, and statvfs doesn't spin a disk anyway.

Related

• locator API gives 503 if even one obj server takes 2 seconds #72 — read-path fix this completes
• find_object returns false 404 for objects whose only copy is on a sleeping disk #75 — read-path false 404 (single object)
• add_object can write a duplicate copy when the existing copy is on a sleeping disk #76 — write-path duplicate (single object)
• is 34 minutes long enough to replicate huge files? #49 — data-path streaming timeout
• GET /{bucket}/: optional override to return partial results when a server is down #63 — partial results; Locator performance enhancement on GET #1, Locator periodically read whole list of files #3 — metadata caching