find_object returns false 404 for objects whose only copy is on a sleeping disk #75

Split out from the #72 timeout work (the bucket-listing path was fixed there; this single-object path was intentionally left as-is).

Problem

find_object (GET/HEAD /{bucket}/{key} on the locator, locator_api.py:68) HEADs each object server with timeout=1:

result = await client.head(server + object_path, timeout=1)

By design, a transport error / timeout here is treated as "this server doesn't have it" — the loop just continues, and if no server returns 200 the locator returns 404 (only an explicit upstream 503 escalates to 503; see the find_object comment block and the locator-busy-vs-missing reasoning).

That design is fine when the server genuinely lacks the object. But a sleeping HDD takes ~15s to spin up (per #72, "even 15 seconds is normal"). So if the only copy of an object lives on a server whose disk is asleep, the HEAD times out at 1s, the server is skipped, and the client gets a 404 for an object that actually exists. Unlike a 503, a 404 gives the client no reason to retry — the object effectively "disappears" until the disk happens to be awake.

This is the "1/3 single-object errors went unnoticed" case @ctengel described in #72: on the current 1/3-SSD / 2/3-HDD layout with 2x replication, most objects have a fast SSD copy, masking it. On an all-HDD cluster it would be common.

Note this is a correctness symptom (a present object reported absent), not just an availability/latency one like the #72 bucket-listing 503.

Options (not mutually exclusive)

• Lengthen the find_object HEAD timeout to tolerate spin-up (~16s), accepting slower GETs when a disk is cold. Trade-off: a genuinely-down server now stalls the per-server leg for the full timeout.
• Distinguish "timed out / unreachable" from "confirmed 404" and, when at least one server timed out (but none confirmed the object and none returned 503), respond 503 + Retry-After rather than 404, so the client retries instead of treating it as missing.
• Metadata caching / a "disk asleep" health signal so the locator can answer without waking the disk and/or give the slow server a few seconds (see locator API gives 503 if even one obj server takes 2 seconds #72 discussion and Locator performance enhancement on GET #1, Locator periodically read whole list of files #3).

Related

• locator API gives 503 if even one obj server takes 2 seconds #72 — bucket-listing 503 from the same sleeping-disk root cause (fixed for GET /bucket/)
• GET /{bucket}/: optional override to return partial results when a server is down #63 — partial results
• Locator performance enhancement on GET #1, Locator periodically read whole list of files #3 — metadata caching