specifiy max parallelism for multi-stage build

[ktzsolt]

Description

We have a huge build that uses 8 independent/parallel stages and the artifacts are copied into the final stage, however each build stage uses a lot of CPU/disk that the builder machine can not serve and eventually some of the stages fail. Rerunning the build will succeed on the second run because the build cache is already there so those stages that are already built won't be built again so the remaining stages and the image can built.

My proposal is that there should be a --multistage-parallelmax flag so we could control how many stages are built concurrently at any given time during the docker build process, default setting would be unlimited.

[tonistiigi]

BuildKit supports max-parallelism config https://docs.docker.com/build/buildkit/configure/#max-parallelism that you can configure for your daemon.

[ktzsolt]

@tonistiigi I found this thread and this is my use case too, I don't want to create builders, I want to use the default builder in docker engine but I don't find buildkit.toml anywhere in the specified locations: #2906 (comment)

Also more granular control in docker build commands would be better, some builds we want to build with unlimited parallelism while for some we want some more control over parallelism. Setting up builders for each parallel setting would be a chore.

[tonistiigi]

As mentioned in that issue, shouldn't be a problem exposing it in dockerd config.

Also more granular control in docker build commands would be better, some builds we want to build with unlimited parallelism while for some we want some more control over parallelism. Setting up builders for each parallel setting would be a chore.

BuildKit is made to handle multiple requests in parallel and efficiently run them together. When the build step runs, there likely isn't even a specific job associated with it. So it doesn't make sense when you have one build setting a limit, but some other requests set a different limit or no limit.