Temp cleanup half-deletes root-owned trees in `_github_home` under Docker ≥ 29.6 (exec umask 0000), corrupting Bazel's install base on persistent runners

[nekronos]

Describe the bug

On persistent self-hosted runners, the worker's temp-directory cleanup partially deletes tool state that container steps wrote into $HOME (/github/home). Under Docker ≥ 29.6 this leaves Bazel's install base half-deleted, and every subsequent containerized Bazel job on that runner fails at startup with FATAL: corrupt installation. The job that plants the damage succeeds; a later, unrelated job fails; no log mentions the deletion. The runner stays broken until the directory is removed manually.

Root cause — three long-standing behaviors and one recent Docker change:

1. /github/home is a bind mount of _work/_temp/_github_home (ContainerOperationProvider.cs), so container steps write their $HOME state into the per-job temp directory.
2. The worker deletes all of _temp at job start and end (TempDirectoryManager.cs) with continueOnContentDeleteError: true — undeletable entries are skipped silently (IOUtil.cs).
3. The worker runs as the non-root service user, but container-created files are root-owned. Unlinking requires write permission on the parent directory, so the root-owned tree's directory modes decide what the cleanup can delete.
4. Docker 29.6 clears the daemon umask (daemon: Set umask to 0000  moby/moby#52892); docker exec sessions — every container job step — inherit umask 0000, so directories created with mkdir(0777) are now world-writable instead of 0755.

Bazel's install base mixes both modes: outer directories from mkdir(0777) & ~umask, inner content explicitly chmod'ed read-only (umask-independent).

• Docker ≤ 29.5: outer dirs are root-owned 0755, the cleanup can delete nothing, the install base survives intact.
• Docker ≥ 29.6: outer dirs are 0777, the cleanup deletes the top-level files (A-server.jar, …) but not the contents of the read-only inner dirs — leaving a husk Bazel refuses to start on rather than repair.

Full protection (old) and full deletion (a root cleanup) would both be harmless; only the new partial outcome is fatal, and it recurs on every job.

To Reproduce

Steps to reproduce the behavior (no Bazel needed), on a persistent self-hosted runner with Docker ≥ 29.6:

1. Run this container job:

jobs:
  demo:
    runs-on: [self-hosted, <persistent-label>]
    container:
      image: ubuntu:24.04
    steps:
      - run: |
          if [ -d "$HOME/.cache/demo" ]; then
            echo "=== state left by previous job + cleanup:"
            ls -laR "$HOME/.cache/demo"
          fi
          mkdir -p "$HOME/.cache/demo/inner"
          touch "$HOME/.cache/demo/top.file" "$HOME/.cache/demo/inner/inner.file"
          chmod 555 "$HOME/.cache/demo/inner"

2. Run the same job again on the same runner.
3. The second run's listing shows top.file deleted while inner/inner.file survives — the half-delete. On Docker ≤ 29.5 the tree is fully intact instead.

The umask difference is directly observable on the runner host:

$ docker run --rm ubuntu:24.04 bash -c umask   # container init
0022
$ docker exec <container> bash -c umask        # job-step path, Docker 29.6
0000

Replacing the demo tree with bazelisk build //... reproduces the real failure: job 1 succeeds (fresh extraction of Bazel's install base), job 2 fails with FATAL: corrupt installation.

Expected behavior

Tool state under /github/home is either removed completely between jobs or left intact — and when the cleanup cannot delete entries, that is surfaced (a warning listing skipped paths) instead of silently leaving a partially-deleted tree that breaks later jobs.

Runner Version and Platform

2.335.1

Linux x86_64, self-hosted, persistent (non-ephemeral), runner service running as a non-root user. Docker Engine 29.6.2 (29.3.1 unaffected). container: jobs running as root (default, no userns remapping).

What's not working?

Every containerized Bazel job on a poisoned runner fails immediately:

FATAL: corrupt installation: file '/github/home/.cache/bazel/_bazel_root/install/<md5>/A-server.jar' is missing or modified.
Please remove '/github/home/.cache/bazel/_bazel_root/install/<md5>' and try again.
##[error]Process completed with exit code 36.

On-disk state of the half-deleted install base (host view; all top-level files deleted, read-only subdirs stranded):

/…/_work/_temp/_github_home/.cache/bazel/_bazel_root/install/<md5>/
├── embedded_tools/   (r-xr-xr-x, contents intact)
└── platforms/        (r-xr-xr-x, contents intact)
    # A-server.jar and all other top-level files: gone

Job Log Output

Included above. The job that causes the damage completes with result: Succeeded and logs nothing unusual.

Runner and Worker's Diagnostic Logs

The _diag logs contain no trace of the skipped deletions — continueOnContentDeleteError: true swallows the failures, which is part of the problem. Happy to provide full Runner_/Worker_ logs for an affected job pair on request.

[blairlearn]

This sounds a lot like the issue I ran into with privileged containers (#4597). The workaround I came up with is to add a "cleanup" step at the end of your job.

In my case, the job was running inside a container and the files causing the problem were in the _actions tree. Presumably you could use a similar technique to fix the permissions in your affected directory. Here's the step I used; hopefully it will be of some help. You'll probably want to start off with the fourth line action_dir=...

      - name: Restore ownership of action and working directories
        if: ${{ always() }}
        run: |
          action_dir="/__w/_actions/"
          runner_owner="$(stat -c '%u:%g' "${GITHUB_ENV}")"

          printf "Changing ownerships to %s\n" "${runner_owner}"
          chown -R "${runner_owner}" "${action_dir}"
          chown -R "${runner_owner}" "${GITHUB_WORKSPACE}"

          printf "Remaining root-owned entries:\n"
          root_owned="$(
            find /__w/_actions "${GITHUB_WORKSPACE}" \
              -user 0 -print
          )"

          if [ -n "${root_owned}" ]; then
            printf '%s\n' "${root_owned}"
          else
            printf "None\n"
          fi

[nekronos]

@blairlearn I just pinned docker to the latest version before the umask change.