Add support for pushing separately from building (`docker buildx push`)

[solidDoWant]

Description

Request: buildx should support pushing built manifests separately from the build process. There should be a buildx equivalent of docker push, probably docker buildx push.

Use cases:

• Retry on container registry errors without attempting to build again (Can you split up the build and push steps? #2718 (comment)
• Building a manifest/images before the targeted container registry is available (Can you split up the build and push steps? #2718 (comment))
• Avoiding changes to the image when created images are not bit per bit reproducible (Can you split up the build and push steps? #2718 (comment), Can you split up the build and push steps? #2718 (comment))
• Testing locally prior to pushing remotely (Can you split up the build and push steps? #2718 (comment), Sequential build of targets #1976 (comment), Can you split up the build and push steps? #2718 (comment))

Workarounds (all painful):

• Avoid buildx and use docker build + docker push + docker manifest create + docker manifest push (the hard way)
• Use a local "intermediary" registry (Can you split up the build and push steps? #2718 (comment))

The previous issue (#1152) got converted to a discussion (#2718) because the initial prompt was a question. To be clear, I'm not asking for alternatives with this issue - I'm asking for built-in buildx support.

[tonistiigi]

Buildx is a CLI for BuildKit builders and BuildKit does not provide APIs for image store (create/delete/list images). You can't have buildx push without having buildx images ls, buildx images rm etc. at least as well and unlike build cache, image storage needs to be actively maintained by the user as it can't be garbage collected. Many other tools exist for storing images, like Docker daemon, Containerd, Container registry etc. and Buildx integrates with all of them. Additionally buildx allows exporting images to OCI layout where you can manage it as immutable files.

[solidDoWant]

To be frank, as a consumer of the tool, the underlying implementation of this feature (whether it be via something that the CLI tool handles, buildkit handles, or the tools that buildkit integrates with) isn't really important to me. What is important to me (again, purely as a user) is that I am able to use the tool for the aforementioned use cases.

Myself and at least 40 other people see the need for this, making it the third most requested issue currently open, and 6th of all time.

For what it's worth, I'm not opposed to having buildx images {ls,rm} etc. commands, and I'm not opposed to managing cleanup of old images. The above uses cases would also be satisfied if docker push could push manifests + referenced images. Any of these options work, so long as manifests/images created via docker buildx can be pushed after being built, separately from being built.

[tonistiigi]

The above uses cases would also be satisfied if docker push could push manifests + referenced images.

It can, with containerd image store https://docs.docker.com/desktop/features/containerd/ https://docs.docker.com/engine/storage/containerd/ . Asking why this is not default is completely valid, but not in the scope of this repository.

[ggjulio]

If a solution is implemented it would be nice for it to work declaratively with bake.

When building multi-arch images, it would also allow to load images of current platform while making sure no layers are rebuilt.
( --load currently doesn't work if many arch are defined, which force us either to push/pull or redo a build with a single arch)

docker buildx bake --set '*.platform=linux/amd64,linux/arm64' [...]      # --load won't work if many arch defined.
docker buildx bake --load [...]     # need to load separately, it may rebuild layers when it shouldn't.

[tom-stytch]

What are your thoughts on #2718 (comment)? Cross posting here for visibility, I wonder if this is a valid approach or if it helps OP. This worked for me.

[ggjulio]

@tom-stytch not super useful using bake.

docker buildx bake [...] # build many images without push
# Before pushing, do tests, scan images, etc... 
docker buildx bake --push [..] # Push all the images and tags, layers could be invalidated and rebuilt before push.

There is still a way to do something like that with bake, but idk.
It would be better if it was supported in build/bake directly.

docker buildx bake [...] # build many images without --push, assuming we use containerd store
# Do something before push
docker buildx bake --print \
      | jq -r '.target[] | select(.tags != null) | .tags[]'  \
      | xargs -I {} docker push {}


# Note: push is not optimal as it will push each ref one by one.
# Note 2: Also images are available to the docker daemon only if they are loaded with `--load`, which takes time.