Add a per-build flag to limit concurrency / parallelism

[ag-TJNII]

Description

Currently the only way to limit parallel builds is via config, added in #2049. It would be useful to be able to tune this per-build, for cases where parallel RUN commands are problematic, but only in that specific build and not system wide.

For example, I have a system where normally the default behavior is fine. However, if one stage is doing a very large download, it might choke downloads in other stages due to network congestion. This is a problem unique to the multi-stage Dockerfile that is trying to download a large model in one stage, and packages in another stage. I do not wish to retune my entire Docker daemon for this issue, so instead I'm simply disabling buildkit with DOCKER_BUILDKIT=0 docker build

This, or something very similar, has been hinted at but as far as I can tell not formally requested:

• Add a configuration item to limit parallelism #2049 (comment)
• Add ability to disable concurrency #1032 (comment)

The current solution from #2049 does not meet this need, as that implementation is daemon-wide. The follow-up proposal in #2108 does not meet this need as there are times when the user will know better than the automated system, for example when there is a bottleneck on the internet link not easily detected. Please add a flag to docker build to allow tuning this per-build. Thank you.

[jirimoravcik]

CC @tonistiigi something that would make sense to implement as a complement to #593 ?

[BtbN]

I can only second this.
Being unable to limit parallelism without creating an entire new builder has been a huge pain-point for years, which necessitates super ugly and clunky workarounds that add a lot of confusing gotchas to a multi-imaged build process, where each step should be exported as local image and then used as base image by the next step.