BuildKit client hangs after successful image push (v0.23.2)

[wujunwei]

Contributing guidelines and issue reporting guide

• I've read the contributing guidelines and wholeheartedly agree. I've also read the issue reporting guide.

Well-formed report checklist

• I have found a bug that the documentation does not mention anything about my problem
• I have found a bug that there are no open or closed issues that are related to my problem
• I have provided version/information about my environment and done my best to provide a reproducer

Description of bug

BuildKit client hangs after successful image push (v0.23.2)

Summary

BuildKit client (buildctl) hangs indefinitely after successfully pushing an image to a registry, even though the push operation completes successfully (receives HTTP 201 Created response). The client process never exits and must be forcefully terminated.

Environment

• BuildKit Version: v0.23.2 (official Docker Hub image)
• BuildKit Image: moby/buildkit:v0.23.2 from Docker Hub
• Platform: Linux (Kubernetes environment)
• Container Runtime: containerd
• Registry: Harbor (private registry)
• Worker: containerd worker
• Network: Host networking mode
• Deployment: Kubernetes using official Docker Hub image

Steps to Reproduce

1. Execute a build with push to registry:

buildctl --addr tcp://buildkitd-service:12346 build \
    --frontend dockerfile.v0 \
    --no-cache \
    --output type=image,name=registry.example.com/repo/image:tag,push=true \
    --opt platform=linux/amd64 \
    --opt force-network-mode=host \
    --allow network.host \
    --local context=. \
    --local dockerfile=. \
    --progress plain

3. Observe that:
   • Build completes successfully
   • Image layers are pushed successfully
   • Manifest is pushed successfully (HTTP 201 Created)
   • Image appears in containerd: ctr -n buildkit images ls
   • Image appears in registry
   • But buildctl process never exits

Expected Behavior

The buildctl process should exit normally after the push operation completes successfully.

Actual Behavior

The buildctl process hangs indefinitely and must be killed with SIGKILL. The process appears to be waiting for some internal operation to complete.

Debugging Information

Client Process Analysis

Using strace on the hanging client process shows:

[pid 20119] futex(0x198e420, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>

Multiple threads are stuck in FUTEX_WAIT_PRIVATE, suggesting the main thread is waiting in errgroup.Wait().

Daemon Process Analysis

The BuildKit daemon (official v0.23.2 Docker Hub image) shows:

[pid 29510] futex(0x2f4d920, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>

Network connections remain established:

tcp6  0  0  daemon:12346  client:57532  ESTABLISHED

But no data is being transmitted (Recv-Q and Send-Q are both 0).

Log Analysis - Key Finding

Client logs show the push completing successfully:

11:48:51  #17 pushing manifest for harbor.XXX.io/test/XXXXX:main-271313406-18@sha256:1444138e6cb07041a256d102a9ec7f27628f2e4afd691e24340c94a03d68ff6e
11:48:52  #17 pushing manifest for harbor.XXX.io/test/XXXXX:main-271313406-18@sha256:1444138e6cb07041a256d102a9ec7f27628f2e4afd691e24340c94a03d68ff6e 0.4s done
11:48:52  #17 DONE 1.3s

Daemon logs (from official Docker Hub image) stop abruptly after the successful push:

time="2025-08-11T11:16:43Z" level=debug msg="fetch response received" 
response.status="201 Created" span="exporting to image"

🔍 Critical Observation: The buildkitd logs show no corresponding "session finished" message after the push completes. In normal operations, we would expect to see:

time="2025-08-11T11:16:43Z" level=debug msg="session finished: <nil>" spanID=xxx traceID=xxx

This missing "session finished" log entry strongly suggests that the session cleanup process is stuck or never initiated, which explains why the client remains hanging.

Session State Analysis

The absence of the "session finished" log indicates:

1. Session cleanup never started - The session termination process was never triggered
2. Session cleanup stuck - The cleanup process started but got stuck somewhere
3. Session reference leak - The session object is still being referenced, preventing cleanup

This correlates with the resource state showing Usage count: 0 but Reclaimable: false.

Resource State

Using buildctl du shows:

ID:             1jjc57vswpm46fex2yvsj97jt
Created at:     2025-08-11 11:16:22.613823175 +0000 UTC
Mutable:        false
Reclaimable:    false
Shared:         false
Size:           0B
Description:    local source for context
Usage count:    0

The resource has Usage count: 0 but Reclaimable: false, indicating a resource leak that correlates with the missing session cleanup.

Additional Context

• Using the official open-source Docker Hub image moby/buildkit:v0.23.2
• Key diagnostic: Missing "session finished" log entries in daemon logs
• Issue occurs in Kubernetes environment with official image
• The problem appears to be in the session cleanup mechanism after successful operations
• Session state becomes inconsistent (usage count 0 but not reclaimable)
• The problem is reproducible but intermittent

---

Note: Happy to provide additional debugging information if needed.

[MalteMagnussen]

Thank you for creating this issue. We are experiencing the same.

We also use moby/buildkit:v0.23.2. We also have it running in Kubernetes.

We are experiencing the jobs timing out in GitLab CI after saying "DONE".

Example logs from our GitLab CI, where the job just times out after 60 minutes. Blocking a lot of CI across the company.

% buildctl --addr tcp://buildkit.redacted.com:1234 build --frontend=dockerfile.v0 --local context=. --local dockerfile=. -opt filename=Dockerfile --import-cache type=registry,ref=registry:5000/redacted:buildcache --export-cache type=registry,mode=max,oci-mediatypes=true,ref=registry:5000/redacted:buildcache --output type=image,name=harbor.redacted.com/redacted:33f5019c,push=true
...
image building and pushing just fine
...
#11 preparing build cache for export 4.1s done
#11 writing cache image manifest sha256:8ab59059b541c3dff5d318f70c4bb3e1e73537c34e5cbd793249d0abb4f34df5 0.6s done
#11 DONE 4.1s

We also observe all of this:

    Build completes successfully
    Image layers are pushed successfully
    Manifest is pushed successfully (HTTP 201 Created)
    Image appears in registry
    But buildctl process never exits

[MalteMagnussen]

Does anyone know if it helps to roll our company instance of buildkit back to an earlier version, until this is fixed?

EDIT1: Rolling back to 0.23.1 seems to have fixed the problem for now for us. I'll update this, if the problem returns on 0.23.1 as well.

EDIT2: The problem is still there on 0.23.1. Just happened again for one of the teams. 🤞 for a fix soon.

[wujunwei]

Does anyone know if it helps to roll our company instance of buildkit back to an earlier version, until this is fixed?

EDIT: Rolling back to 0.23.1 seems to have fixed the problem for now for us. I'll update this, if the problem returns on 0.23.1 as well.

I added more logs to try to find out where it is stuck, if nothing works I will also downgrade to 0.23.1 like you.

[tonistiigi]

Please follow https://github.com/moby/buildkit/blob/master/.github/issue_reporting_guide.md#reporting-deadlock and report back. If it seems to suggest waiting on the client side then you can get the stacktrace from client side as well. If it shows runc open then add ps aux as well.

[wujunwei]

Please follow https://github.com/moby/buildkit/blob/master/.github/issue_reporting_guide.md#reporting-deadlock and report back. If it seems to suggest waiting on the client side then you can get the stacktrace from client side as well. If it shows runc open then add ps aux as well.

Maybe I have to wait until the next time the problem occurs. It will appear once or twice in hundreds of builds a day.

[JosiahWhite]

I have somehow found myself in a situation where I can reproduce the issue repeatedly. The source of the deadlock seems to be boltcachestorage returning "ErrNotFound" but the record not being nil here:

buildkit/solver/exporter.go

Line 115 in ae2b7ea

if v == nil {

Based on the commit that added this code and the fact that another build completed while the build that locked up was still running, I think a snapshot that was needed by the original build got removed during the GC process.

Here is a stacktrace for the relevant goroutine:

goroutine 79330 [runnable]:
github.com/pkg/errors.callers()
        /src/vendor/github.com/pkg/errors/stack.go:167 +0x4a
github.com/pkg/errors.WithStack(...)
        /src/vendor/github.com/pkg/errors/errors.go:151
github.com/moby/buildkit/solver/bboltcachestorage.(*Store).Load.func1(0xc00019a488?)
        /src/solver/bboltcachestorage/storage.go:136 +0x28a
go.etcd.io/bbolt.(*DB).View(0xdb96d0?, 0xc0178b7c80)
        /src/vendor/go.etcd.io/bbolt/db.go:939 +0x6c
github.com/moby/buildkit/solver/bboltcachestorage.(*Store).Load(0xc00005a950, {0xc0048c4500, 0x47}, {0xc00a7d50c0, 0x34})
        /src/solver/bboltcachestorage/storage.go:124 +0xd3
github.com/moby/buildkit/solver.(*exporter).ExportTo(0xc008eaeb00, {0x20f0300, 0xc00d4503c0}, {0x20edf00, 0xc001cf6250}, {0x1e810d0, 0x2, {0x0, 0x0}, 0x0, ...})
        /src/solver/exporter.go:125 +0x58f
github.com/moby/buildkit/solver.(*mergedExporter).ExportTo(0xc000a26ec0?, {0x20f0300, 0xc00d4503c0}, {0x20edf00, 0xc001cf6250}, {0x1e810d0, 0x2, {0x0, 0x0}, 0x0, ...})
        /src/solver/exporter.go:275 +0x131
github.com/moby/buildkit/solver.(*exporter).ExportTo(0xc009e2a280, {0x20f0300, 0xc00d4503c0}, {0x20edf00, 0xc001cf6250}, {0x1e810d0, 0x2, {0x0, 0x0}, 0x0, ...})
        /src/solver/exporter.go:187 +0xef7
github.com/moby/buildkit/solver.(*exporter).ExportTo(0xc002d0c500, {0x20f0300, 0xc00d4503c0}, {0x20edf00, 0xc001cf6250}, {0x1e810d0, 0x2, {0x0, 0x0}, 0x0, ...})
        /src/solver/exporter.go:199 +0xc78
github.com/moby/buildkit/solver.(*exporter).ExportTo(0xc0014f4380, {0x20f0300, 0xc00d4503c0}, {0x20edf00, 0xc001cf6250}, {0x1e810d0, 0x2, {0x0, 0x0}, 0x0, ...})
        /src/solver/exporter.go:187 +0xef7
github.com/moby/buildkit/solver.(*mergedExporter).ExportTo(0xee03abded?, {0x20f0300, 0xc00d4503c0}, {0x20edf00, 0xc001cf6250}, {0x1e810d0, 0x2, {0x0, 0x0}, 0x0, ...})
        /src/solver/exporter.go:275 +0x131
github.com/moby/buildkit/solver.(*exporter).ExportTo(0xc004be29c0, {0x20f0300, 0xc00d4503c0}, {0x20edf00, 0xc001cf6250}, {0x1e810d0, 0x2, {0x0, 0x0}, 0x0, ...})
        /src/solver/exporter.go:187 +0xef7
github.com/moby/buildkit/solver.(*mergedExporter).ExportTo(0xee03abe1c?, {0x20f0300, 0xc00d4503c0}, {0x20edf00, 0xc001cf6250}, {0x1e810d0, 0x2, {0x0, 0x0}, 0x0, ...})
        /src/solver/exporter.go:275 +0x131
github.com/moby/buildkit/solver.(*exporter).ExportTo(0xc0066eee00, {0x20f0300, 0xc00d4503c0}, {0x20edf00, 0xc001cf6250}, {0x1e810d0, 0x2, {0x0, 0x0}, 0x0, ...})
        /src/solver/exporter.go:187 +0xef7
github.com/moby/buildkit/solver.(*mergedExporter).ExportTo(0xee03abe8a?, {0x20f0300, 0xc00d4503c0}, {0x20edf00, 0xc001cf6250}, {0x1e810d0, 0x2, {0x0, 0x0}, 0x0, ...})
        /src/solver/exporter.go:275 +0x131
github.com/moby/buildkit/solver.(*exporter).ExportTo(0xc012f69b00, {0x20f0300, 0xc00d4503c0}, {0x20edf00, 0xc001cf6250}, {0x1e810d0, 0x2, {0x0, 0x0}, 0x0, ...})
        /src/solver/exporter.go:187 +0xef7
github.com/moby/buildkit/solver.(*mergedExporter).ExportTo(0xc01650f040?, {0x20f0300, 0xc00d4503c0}, {0x20edf00, 0xc001cf6250}, {0x1e810d0, 0x2, {0x0, 0x0}, 0x0, ...})
        /src/solver/exporter.go:275 +0x131
github.com/moby/buildkit/solver.(*exporter).ExportTo(0xc014bf3980, {0x20f0300, 0xc00d4503c0}, {0x20edf00, 0xc001cf6250}, {0x1e810d0, 0x2, {0x0, 0x0}, 0x0, ...})
        /src/solver/exporter.go:187 +0xef7
github.com/moby/buildkit/solver.(*mergedExporter).ExportTo(0xee03abe8e?, {0x20f0300, 0xc00d4503c0}, {0x20edf00, 0xc001cf6250}, {0x1e810d0, 0x2, {0x0, 0x0}, 0x0, ...})
        /src/solver/exporter.go:275 +0x131
github.com/moby/buildkit/solver.(*exporter).ExportTo(0xc01715a9c0, {0x20f0300, 0xc00d4503c0}, {0x20edf00, 0xc001cf6250}, {0x1e810d0, 0x2, {0x0, 0x0}, 0x0, ...})
        /src/solver/exporter.go:187 +0xef7
github.com/moby/buildkit/solver.(*mergedExporter).ExportTo(0xee03abe8e?, {0x20f0300, 0xc00d4503c0}, {0x20edf00, 0xc001cf6250}, {0x1e810d0, 0x2, {0x0, 0x0}, 0x0, ...})
        /src/solver/exporter.go:275 +0x131
github.com/moby/buildkit/solver.(*exporter).ExportTo(0xc01594b9c0, {0x20f0300, 0xc00d4503c0}, {0x20edf00, 0xc001cf6250}, {0x1e810d0, 0x2, {0x0, 0x0}, 0x0, ...})
        /src/solver/exporter.go:187 +0xef7
github.com/moby/buildkit/solver.(*mergedExporter).ExportTo(0xee03abe9b?, {0x20f0300, 0xc00d4503c0}, {0x20edf00, 0xc001cf6250}, {0x1e810d0, 0x2, {0x0, 0x0}, 0x0, ...})
        /src/solver/exporter.go:275 +0x131
github.com/moby/buildkit/solver.(*exporter).ExportTo(0xc01706ab40, {0x20f0300, 0xc00d450300}, {0x20edf00, 0xc001cf6250}, {0x1e810d0, 0x2, {0x0, 0x0}, 0x0, ...})
        /src/solver/exporter.go:187 +0xef7
github.com/moby/buildkit/solver.(*mergedExporter).ExportTo(0x20f5600?, {0x20f0300, 0xc00d450300}, {0x20edf00, 0xc001cf6250}, {0x1e810d0, 0x2, {0x0, 0x0}, 0x0, ...})
        /src/solver/exporter.go:275 +0x131
github.com/moby/buildkit/solver/llbsolver.NewProvenanceCreator.func1({0x20f0300, 0xc00fee1920})
        /src/solver/llbsolver/provenance.go:421 +0x1a4
github.com/moby/buildkit/solver/llbsolver.(*ProvenanceCreator).Predicate(0xc00d450270, {0x20f0300, 0xc00fee1920})
        /src/solver/llbsolver/provenance.go:478 +0xb5
github.com/moby/buildkit/solver/llbsolver.(*Solver).recordBuildHistory.func1.2({0x1de998a?, 0x411a37?}, {0x20f5600, 0xc0000b5e80}, 0xc000d1e0c0)
        /src/solver/llbsolver/solver.go:248 +0x20b
github.com/moby/buildkit/solver/llbsolver.(*Solver).recordBuildHistory.func1.3()
        /src/solver/llbsolver/solver.go:286 +0x5d
golang.org/x/sync/errgroup.(*Group).add.func1()
        /src/vendor/golang.org/x/sync/errgroup/errgroup.go:130 +0x7e
created by golang.org/x/sync/errgroup.(*Group).add in goroutine 372
        /src/vendor/golang.org/x/sync/errgroup/errgroup.go:98 +0x76

[tonistiigi]

@JosiahWhite Can you check #6129 . As there is no deadlock in your trace are you saying it goes to loop in the condition you are describing?

[mtharp]

I'm also seeing this with v0.23.1 with an identical stack to what JosiahWhite posted - in some cases multiple builds on the same daemon are stuck in the same manner. And yes, it looked more like an infinite loop with 200% CPU usage rather than a deadlock.

I was previously using v0.20.0 without issue. Like the original poster, I am not exporting the cache to a registry.

[JosiahWhite]

@JosiahWhite Can you check #6129 . As there is no deadlock in your trace are you saying it goes to loop in the condition you are describing?

You are correct in that I am not seeing a deadlock but it was an infinite loop that burns CPU. I have since modified my usage such that I am not hitting this bug anymore but I will try and reproduce it when I have some time. It seemed to be occurring when I had a smaller cache volume that was getting cleaned up due to being close to the storage limits. I was never exporting to a remote cache, just letting the default local buildkit cache be used.

[tonistiigi]

I was never exporting to a remote cache, just letting the default local buildkit cache be used.

The same cache exporting functions get called when provenance attestation is created, where it tries to log all the remote descriptors(layer blob pointers) that can be associated with some location of your build.