Repository navigation
locator API gives 503 if even one obj server takes 2 seconds #72
Description
maybe related
$ simpler-objects-async-replicate --replicas 2 http://localhost:29164/ bucket Traceback (most recent call last): File "simpler-objects-async-replicate", line 8, in <module> sys.exit(cli()) ~~~^^ File "simpler_objects/async_replicate.py", line 123, in cli results = [auto_replica(args.locator, b, args.replicas) for b in buckets] ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "simpler_objects/async_replicate.py", line 83, in auto_replica res.raise_for_status() ~~~~~~~~~~~~~~~~~~~~^^ File "httpx/_models.py", line 829, in raise_for_status raise HTTPStatusError(message, request=request, response=self) httpx.HTTPStatusError: Server error '503 Service Unavailable' for url 'http://localhost:29164/bkup-20250818/' For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/503There is an 8 second timeout for the locator, but the locator API only waits 2 seconds for the object servers. If ANY timeout it will 503 back. 2 seconds is usually enough unless an HDD is in powersave mode and has to spin up.
Two non-mutually-exclusive options
- increase that timeout
- explore giving partial results vs 503
Activity
ctengel commentedon May 28, 2026
to be clear this is
GET /bucket/in the Locator APIctengel commentedon May 28, 2026
manifest during replicate because that's the main system that does that call, and it's likely to run during off hours when drives have spun down
ctengel commentedon May 28, 2026
See also #63 - that should cover the optional partial results. The short/quick fix here is to increase the timeout. We DO want the drives to sleep when idle to save power. Apparently even 15 seconds is normal. We may need to look at a bunch of our timeouts - it would be nice if some way to skip down hard servers but give sleepy ones a few seconds to wake up. (they would need to somehow indicate "I'm alive")
Alternate/future concept would be for locator or obj server to cache metadata so it doesn't need to wake disk up (see #1 #3 )
ctengel commentedon May 28, 2026
Two thoughts:
So fundamental question here - long timeouts or short timeouts and respect the 503 retry?
ctengel commentedon May 28, 2026
Alternatively, cache all metadata in memory on the object servers. This would allow all requests other than GET/PUT object (these tend to have long timeouts anyway due to large object sizes. we could still do HEAD object quickly) to go quick. Maybe also do some kind of /health indication that drive is asleep.
ctengel commentedon May 28, 2026
If locator caches hard down systems then that would allow decent performance generally there. Short timeout it system is suspected hard down, longish otherwise to allow for drive spin up
Various bug fixes