Retry S3 uploads on network errors and fix the tus cleanup ENOENT noise and leftover .json files #202

Problem 1: uploads sometimes fail at the last step with socket hang up

When an upload finishes, the API writes the file to S3 with putObject, using createReadStream(path) as the body. If S3, or a proxy in between, has closed an idle keep-alive connection, the request fails with socket hang up or ECONNRESET. The AWS SDK can't retry it, because the stream body has already been read.

In tus uploads, onUploadFinish (apps/api/src/tus/tus-server.ts) catches the error and returns 403 with the message socket hang up. tus clients don't retry 4xx responses, so the upload fails even though every byte arrived. The user has to upload the whole file again.

• Seen in the CLI integration suite against QA (multi-file medialit upload) and previously in production logs (Error finalizing tus upload).
• Affects every product that uses MediaLit: CourseLit, SendLit, FrontLit.

Fix

• In apps/api/src/services/s3.ts, make putObject accept the file path and open a fresh read stream on each attempt.
• Retry up to 3 times with a short backoff on network errors (ECONNRESET, socket hang up, EPIPE, ETIMEDOUT) and S3 5xx responses. Throw any other error straight away.
• Update the four call sites:
  • tus/finalize.ts: main file and thumbnail.
  • media/service.ts: main file and thumbnail on the multipart path.
• Don't change status codes, response bodies or storage keys.

Problem 2: tus cleanup logs ENOENT errors and can leave .json files behind

When an upload succeeds, finalize deletes its temporary files but keeps the tus_uploads row until it expires. The cleanup job (tus/cleanup.ts) then calls removeTusFiles again, gets ENOENT, and logs Error removing tus files for every successful upload.

removeTusFiles (tus/utils.ts) deletes the data file and then the .json file in one try block. If the data file is already gone, the .json file is never deleted. After a crash or restart in the middle of an upload, those files pile up in /tmp/tus-uploads.

Fix

• Delete both files with fs.rmSync(path, { force: true }), each independently.
• Only log errors other than "file not found".

Acceptance criteria

• A unit test where the first putObject attempt fails with ECONNRESET and the second succeeds. The upload succeeds, and the second attempt uses a fresh stream.
• A unit test where putObject fails with a non-network error (for example AccessDenied). It isn't retried.
• A unit test where removeTusFiles runs with the data file missing. The .json file is still deleted, and nothing is logged.
• Running the cleanup job after successful uploads logs no Error removing tus files.
• The REST, MCP and CLI integration suites pass against QA.