Fresh Linux deployments crash-loop rustfs and the API because ./rustfs/data is created root-owned #1752

Activity

1. joshunrau

   added

   BugType: existing behavior is wrongType: existing behavior is wrong

   Priority: HighSignificant or frequently hit user impact; target the next release or twoSignificant or frequently hit user impact; target the next release or two

   Difficulty: LowIsolated change, about 2 hours or lessIsolated change, about 2 hours or less

   Area: InfrastructureDeployment, Docker, CI, logging, CLIs, build and test toolingDeployment, Docker, CI, logging, CLIs, build and test tooling

   on Oct 7, 2026
2. joshunrau commented on Oct 26, 2026

   

   joshunrau

   on Oct 26, 2026

   CollaboratorAuthor

   More actions

   Still reproduces on main (d86740e). The rustfs service (docker-compose.yaml:130-154) and .env.template:27 have not changed since this was filed.

   This is not limited to fresh clones. Every Linux deployment that followed the 2.0.0 upgrade steps (CHANGELOG.md:164-171) got the same root-owned ./rustfs/data. Those that kept storage off (CHANGELOG.md:126) still start, with rustfs restarting in the background. mongo bind-mounts its directories the same way (docker-compose.yaml:123-125) without breaking, because its entrypoint starts as root, runs find -L /data/configdb /data/db \! -user mongodb -exec chown mongodb '{}' +, and only then drops to mongodb.

   Changes to the suggested fix

   • Run the init step with the rustfs image instead of busybox: the same image: as rustfs, user: '0:0', entrypoint: ['chown', '-R', 'rustfs:rustfs', '/data']. The runtime stage is Alpine with coreutils, and rustfs:rustfs resolves from the image's own /etc/passwd, so the owner stays right if upstream renumbers the user again (it moved from 1000 to 10001 in November 2025). Make it recursive, as mongo's is, so a ./rustfs/data that root restored from a backup is fixed too.
   • Give it exactly the profiles of rustfs (docker-compose.yaml:151-154). A service with no profiles is always enabled, so it would run on a gateway-only host and create ./rustfs/data there. With fewer profiles, rustfs would depend on a service Compose has not enabled. Leave restart unset: unless-stopped restarts a container whatever its exit code, so the one-shot would loop.
   • condition: service_healthy works, because the image installs curl for the healthcheck at docker-compose.yaml:145-150. The cost: an unhealthy rustfs then fails docker compose up -d even with STORAGE_ENABLED=false, unless the admin applies the override in docs/en/5-reference/5.6-docker.md:63-71.
   • Pinning .env.template:27 and the :-latest fallback at docker-compose.yaml:131 reaches new installs only. Every .env generated since a988993 sets RUSTFS_VERSION=latest explicitly. A note in the commit body does not reach those admins either: scripts/changelog.ts copies only the subject and BREAKING CHANGE notes into the release notes. To pin existing deployments too, write the tag into docker-compose.yaml and drop RUSTFS_VERSION from .env.template. An admin who wants another tag sets image: in docker-compose.override.yaml, as 5.6-docker.md:40 already recommends for local changes. Since this was filed, latest has moved to 1.0.2 (2026-10-16), 1.0.3 (10-17) and 1.0.4 (10-25). Pin 1.0.4 or later, so that no deployment that already pulled latest goes backwards.
   • Keep the bind mount. A named volume needs no init step, but it moves the data out of ./rustfs/data, which the 2.0.0 notes tell admins to back up (CHANGELOG.md:166, :177).

   Also in this change

   • docs/en/5-reference/5.6-docker.md: add the init service to the services table (:12-20) and the profiles table (:30-33, including "all seven services"). Also move it into the storage profile in the override at :63-71, or it keeps running on stacks without storage and creates ./rustfs/data.
   • Optional, as a separate fix(api) commit that also helps Development and deployment tutorials leave file storage enabled but unconfigured, so pnpm dev crashes and deployed file uploads target localhost #1787: the bare catch at apps/api/src/storage/storage.service.ts:100 turns every HeadBucket failure (DNS, connection refused, bad keys) into a CreateBucket call, and the error that then stops Nest does not mention storage. Call CreateBucket only on NotFound from @aws-sdk/client-s3, and rethrow anything else as an error naming STORAGE_ENDPOINT, with the original as cause. The existing test at apps/api/src/storage/__tests__/storage.service.test.ts:159 rejects with a plain new Error('NotFound'), so it has to switch to the SDK class.

   Tests

   No unit or e2e test can reach the compose change, and root AGENTS.md:49-51 exempts only scripts/. The API change has the same e2e gap: the api's dev:test sets NODE_ENV=test, which onModuleInit skips (storage.service.ts:94). Both need an explicit exception, as #1792 and #1794 do, unless CI gains a compose job. Today it never starts the stack (release.yaml:28 only reads its config). A job on ubuntu-latest that runs generate-env.sh and docker compose up -d, then polls /api/v1/setup, would fail on main now.

   In addition to the manual check in the issue, run up:

   • again on an existing ./rustfs/data that already holds the bucket;
   • with --profile nogateway;
   • with the storage-off override, and check that no ./rustfs/data appears;
   • on a 2.0.0-upgraded host whose rustfs is already restart-looping;
   • on Docker Desktop for macOS, where the stack works today. rustfs now waits for the chown to succeed, so file sharing that refuses chown would take storage down.

   Land this with #1792 if the gateway takes the compose route. The same pattern works there with the gateway image and chown -R node:node /app/sqlite. It needs -R because the gateway has run as root since 7c40422, so an existing gateway.db is root-owned.

   _{Triage note written with Claude Code. Code references point at main @ d86740e.}
3. caoyadong888-wq commented on Oct 26, 2026

   

   caoyadong888-wq

   on Oct 26, 2026

   More actions

   This matches the classic “Docker creates the bind-mount parent as root:root before the container starts” failure mode on Linux.

   docker-compose.yaml bind-mounts ./rustfs/data → /data, and the rustfs/rustfs image runs as non-root (uid 10001). On a fresh clone that directory does not exist (gitignored), so the daemon creates it as root:root 0755. The process then cannot write RUSTFS_VOLUMES=/data, exits with Permission denied, and restart: unless-stopped crash-loops it. With STORAGE_ENABLED=true (default in .env.template), the API’s StorageService.onModuleInit then fails talking to an unhealthy rustfs, which is why every API route shows up as 502 with little signal in the app logs.

   Immediate recovery (no image rebuild)

   On the host, before (or after stopping) the stack:

   # from the compose project root
   docker compose stop rustfs api || true
   mkdir -p ./rustfs/data
   # rustfs image user is typically uid/gid 10001 — confirm once:
   docker run --rm --entrypoint id rustfs/rustfs:${RUSTFS_VERSION:-latest}
   sudo chown -R 10001:10001 ./rustfs/data
   # optional: keep group-writable for host tooling
   sudo chmod 775 ./rustfs/data
   docker compose up -d rustfs
   docker compose ps rustfs
   docker compose logs --tail=50 rustfs

   Then bring the API back once rustfs is healthy:

   docker compose up -d api
   docker compose logs --tail=80 api | grep -Ei 'storage|rustfs|HeadBucket|CreateBucket|Error' || true

   Sanity checks worth pasting back if it still loops:

   # host side
   namei -l ./rustfs/data
   stat -c '%U:%G %a %n' ./rustfs/data
   
   # inside the container (must succeed as the rustfs user)
   docker compose exec -u 10001 rustfs sh -c 'touch /data/.write-test && rm /data/.write-test && ls -la /data'

   Compose-side options (pick one)

   1. Named volume (avoids host-uid surprises): replace ./rustfs/data:/data with rustfs_data:/data and declare volumes: { rustfs_data: {} }.
   2. Init/fixperms sidecar or entrypoint: a one-shot user: "0:0" service that chown -R 10001:10001 /data before rustfs starts (same pattern as many linuxserver images).
   3. Document in the deploy tutorial: mkdir -p rustfs/data && sudo chown 10001:10001 rustfs/data before the first docker compose up -d, plus a note that SELinux hosts may also need :Z on the bind.

   Until (1) or (2) lands, (3) + the recovery commands above unblock fresh Linux installs without disabling storage.

   Paid help + https://github.com/caoyadong888-wq/linux-docker-troubleshoot-minilab