Object server

Possibly rewrite or replace altogether - our semantics are similar to S3 or WebDAV

Requirements

Support Range request for get (useful for Web browser video playback)
Don't load the whole file on GET or PUT (i.e. don't max out server memory)
List out contents
Disk free (so "locator" can query to see if there is enough space before redirecting a PUT here)
Status/Health (can be future, allows read only mode, etc)

Basic bucket listing and replication

#8 included as well; especially checksum

time to beat 8-16 MB/s download

Loading whole file into memory looks like it may be a deal breaker for the pi hardware

Basically there are 3 choices:

Improve the super basic code we have
Switch to FastAPI or some other framework
Use a basic S3ish storage instead like gofakes3 or rclone or versitygw

Spend a pom or two on each in order

Issue averted via f66b88e and fc4a37c

Currently about 16 MB/s or 128 Mbps replication - read+write - pretty good!!!

We may actually be essentially maxing out 100 Mbps Ethernet on Pi 3! So software performance is really good :-) need to build out Pi 4 object servers to get much better

Added /health endpoint to object server

https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/Range_requests

down to as low as 3 Mbps - maybe related to SMR buffers filling up or parallel - regardless next step is to try pi 4

Move to pi 4
Implement range request

416

https://werkzeug.palletsprojects.com/en/stable/utils/#werkzeug.utils.send_from_directory

https://smoores.dev/post/http_range_requests/

https://www.zeng.dev/post/2023-http-range-and-play-mp4-in-browser/

https://starlette.dev/responses/

Looks like we may get this in Python 3.15: python/cpython#118949

Or in the meantime: https://github.com/danvk/RangeHTTPServer

see also https://pkg.go.dev/net/http which currently implements range requests via FileServer, ServeFile ServeContent

which if any existing option makes it easy to handle puts and digests?

3.15
fastapi
flask
go

416 or 200 on multirange

Rewrite object server as FastAPI

regression
test range
perf comp
gradual deploy

async

problem - pre checksumming we were getting ~230 Mbps / 27 MB/s upload but with it (either fastapi or python) but with it effectively 100 Mbps / 12 MB/s - sha256 on pi 4 is slow... back to pi 3 speeds...

Still quick enough for 1x 4k stream or 10x HD stream on a single node...

I think just go with it and look to

scale to 3 nodes
implement Optional synchronous checksum #31
also do Async replication #5 and Proxy with sync replication #10 ... Data integrity is key, moreso than speed
clever client concurrency - i.e. can start another download after upload is complete just waiting for 201

server code is done, just need to scale and deploy

Navigation Menu

Repository navigation

Object server #2

2 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Object server #2

Description

Activity

ctengel commented on Aug 18, 2025

ctengel commented on Aug 19, 2025

ctengel commented on Aug 19, 2025

ctengel commented on Aug 19, 2025

ctengel commented on Aug 19, 2025

ctengel commented on Aug 19, 2025

ctengel commented on Aug 21, 2025

ctengel commented on Aug 25, 2025

ctengel commented on Aug 25, 2025

ctengel commented on Sep 28, 2025

2 remaining items

ctengel commented on Jan 2, 2026

ctengel commented on Jan 3, 2026

ctengel commented on Jan 3, 2026

ctengel commented on Jan 3, 2026

ctengel commented on Jan 3, 2026

ctengel commented on Jan 3, 2026

ctengel commented on Jan 3, 2026

ctengel commented on Jan 4, 2026

ctengel commented on Jan 5, 2026

ctengel commented on Jan 5, 2026

ctengel commented on Jan 5, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions