Stream uploads straight to S3 instead of staging them on disk #250

Summary

Stream uploads from the client straight into an S3 multipart upload, for both upload paths (tus and form-data), instead of writing the whole file to local disk and uploading it again. Account quotas must stay enforced, and the design must leave clear places to add per-upload rules later (#197).

Supersedes #163. Should also remove the causes of #202. Includes the "share post-processing between tus and form-data uploads" half of #160. Moving processing to background jobs stays in #160.

Today

• tus (apps/api/src/tus/tus-server.ts): @tus/file-store writes every chunk to tmp/tus-uploads. On finish, finalize.ts chooses the media ID, copies the file into a working folder, processes it, then putObjects it to S3.
• form-data (apps/api/src/media/upload-guards.ts): express-fileupload with useTempFiles: true writes the file to disk, then media/service.ts repeats the same processing and putObjects it.

Every byte is written to disk at least once, read back, and sent to S3 in a single request. That costs disk space and I/O, needs sticky sessions for tus on more than one instance, and fails at the last step when S3 closes the connection (#202). The two paths also duplicate the processing code.

Decisions

1. The media ID is chosen when the upload starts. The file streams straight to its final key, {mediaId}/main.{ext}, in the private bucket, so no copy within S3 is needed. tus stores the media ID on its tus_uploads row at create, and form-data generates it at the start of the request.
2. WebP replaces the original. When WebP is on and the file is an image, finalize reads main.{ext} back, converts it, writes main.webp, and deletes main.{ext}. The stored media points at main.webp, as it does today.
3. size is the size of the stored file. That's the WebP size after conversion. It's what the user is consuming, so it's what counts toward storage. The checks before and during the upload use the uploaded size, because that's all that is known then. This also fixes the bug where size is currently recorded as the uploaded size.
4. tus uploads without a declared size (Upload-Defer-Length) are rejected in onUploadCreate. Every upload must declare its size up front so it can be checked before any bytes arrive.
5. tus state:
   • Postgres (tus_uploads) keeps what MediaLit itself uses: user, app, signature, metadata, declared size, media ID, completion.
   • The bucket, through @tus/s3-store's default .info objects, keeps tus's own resume bookkeeping (offsets, part numbers). MediaLit doesn't need to manage that.
   • Drop tempFilePath. Drop uploadOffset from Postgres too, unless something outside tus reads it.
6. One shared post-upload function for both paths (from Async jobs for file transformations #160). It takes { mediaId, key, size, metadata, user, app, signature } and runs the final checks, WebP, thumbnails, creating the media record and cleaning up the signature. tus onUploadFinish and the form-data handler both call it.
7. Object tags follow the existing DISABLE_TAGGING flag (services/s3.ts). Pass Tagging when the multipart upload is created unless the flag is set, so providers that reject tags keep working as they do today.
8. Memory and disk defaults:
   • form-data (@aws-sdk/lib-storage Upload): partSize 8 MiB and queueSize 4, so at most 32 MiB of memory per upload.
   • tus (@tus/s3-store): partSize 8 MiB and maxConcurrentPartUploads 4, so at most about 32 MiB of buffered parts on disk per upload.
   • Make both configurable through environment variables, documented in .env.example.

Proposal

tus

• Replace @tus/file-store with @tus/s3-store, configured so the object key is {mediaId}/main.{ext}. Use namingFunction and the upload ID mapping, or s3-store's key option. If tus can't produce that key, fall back to a staging key plus a copy within S3, and note that in the PR.
• Any API instance can resume an upload, because the tus state is in the bucket and the MediaLit state is in Postgres.

form-data

• Replace express-fileupload temp files with busboy streaming into @aws-sdk/lib-storage Upload. That library uploads in parts, retries each part, and aborts on failure.

Processing

• Video thumbnails don't download the video again. The shared function passes ffmpeg a short-lived presigned URL for the object, and ffmpeg fetches only the bytes it needs for the first frame. @medialit/thumbnail already passes the source straight to -i, so the package doesn't change. The URL must always be one MediaLit generated, never a value from the client.
• Image thumbnails and WebP read the file back from S3. Images are small, and the GIF check in @medialit/thumbnail uses the file extension, which would break on a URL with a query string.
• Keep the current API response unchanged: the media header for tus and the response body for form-data must still include the thumbnail. CourseLit, SendLit and FrontLit depend on it.

Enforcement points

Every rule runs through one shared function (extend validateUploadConstraints), so the two paths can't drift apart.

Rejecting at any point calls AbortMultipartUpload, so a rejected upload never becomes an object in the bucket. Storage is checked one last time against the stored size, after WebP conversion, before the media record is created.

Reliability

• Each part is retried on its own (network errors and 5xx), so one dropped connection no longer fails a whole upload.
• Add a lifecycle rule to the bucket that aborts incomplete multipart uploads after 1 day, and document it in self-hosting. Parts from uploads that are left unfinished stop costing storage. Also remove leftover tus .info objects, either with s3-store's expiry or in the existing cleanup job.
• The temp-file cleanup job no longer has local files to delete, which removes the ENOENT noise and leftover .json files from Retry S3 uploads on network errors and fix the tus cleanup ENOENT noise and leftover .json files #202.

Spike (first task)

• @tus/s3-store works with AWS S3, Cloudflare R2, Backblaze B2 and MinIO. R2 requires every part except the last to be the same size, so partSize must stay fixed.
• tus can write to the key {mediaId}/main.{ext} (decision 1). If it can't, use the fallback described under Proposal.
• Multipart uploads with Tagging work on each provider, or DISABLE_TAGGING covers the ones where they don't.
• The 8 MiB part size doesn't break small or slow uploads.
• Reading an image back from S3 for thumbnails and WebP doesn't noticeably slow down the upload response.
• Video thumbnails from a presigned URL are fast on a large MP4 whose index is at the end of the file (no faststart), on S3, R2 and MinIO.
• A rollout plan for tus uploads that are in progress on disk when the new version deploys. For example, keep the file store read-only until they expire.

Acceptance criteria

• Neither upload path writes the whole file to local disk.
• tus and form-data uploads go through one shared post-upload function.
• An upload over the plan's max file size is rejected before any bytes are stored (tus and form-data).
• A tus upload with Upload-Defer-Length is rejected at create.
• An upload that lies about its size is cut off at the limit and nothing is stored in the bucket.
• An upload that would exceed account storage is rejected at create, and again at completion if storage ran out in the meantime.
• With WebP on, only main.webp remains in the bucket, and size is the WebP file's size.
• After a rejection or a client abort, no object and no unfinished multipart upload is left in the bucket (or the lifecycle rule removes it).
• An S3 connection reset during an upload is retried and the upload succeeds.
• Upload responses (media header and REST body) have the same shape as today, including thumbnails.
• Making a video thumbnail doesn't download the whole video again.
• A tus upload can resume on a different API instance.
• Memory and disk use per upload stay within the configured part size × concurrency.
• The REST, MCP and CLI integration suites pass against QA, with S3 and with MinIO locally.