You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Some Packages download sources while building eg: tensorflow downloads a lot of it's dependencies using bazel. We need an way to bits to tell bits to upload them too.
thanks @akritkbehera for opening this issue. Just to add some context to this request, in cmsBuild we have a way to add a source url which is not available at the time of the build but package build itself downloads and provide the contents of that source. cmsBuild then make sure to cache the generated source for future re-use. This we have used for
openloops source: openloops download list of porocesses at build time so building openloops at two different times can end up downloaded different version of processes. We have a special openloops package which declares a source URL which initially does not have any contents but the actual package build downloads the sources and then feed them back to cmsBuild so that it can be cached for future re-builds. This allowed us to use the exact same source for openloops even if we build it after N years
We build Tensorflow from sources and it downloads/builds many ( over 70) external packages ( including llvm, protobuf, eigen etc.) . Recently we have noticed that Google cloud mirror of tensorflow sources are gone (specially the llvm-project source). This causes Bazel ( the build system of TF) to download the actual (https://github.com/llvm/llvm-project/archive/909041e4802c4b9a2223ca04099f35bf1dbbd460.tar.gz) source from github. For some strange reason the TF commit tar.gz file now have a different checksum which causes Bazel to fail [a]. So to avoid this we also make use of cmsBuild feature to cache sources which are only available after the build. A special source URL instructs the packaging system (cmsBuild) that the source url might not have any initial contents and the actual files to be cached will be generated after the build itself.
[a]
WARNING: Download from https://github.com/llvm/llvm-project/archive/909041e4802c4b9a2223ca04099f35bf1dbbd460.tar.gz failed: class com.google.devtools.build.lib.bazel.repository.downloader.UnrecoverableHttpException Checksum was 00b1077e029fa57e6f2d9ac24936a49acf23ebc051b04f487131116258be6248 but wanted 3f986184ee126677dbd77edb16d6b82c057ec869fefd7a9871979941e52e837a
INFO: Repository llvm-raw instantiated at:
/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc14/external/tensorflow-sources-cache/2.21.0-33aa2ca680813287eb131e31bc444e4c/tensorflow-2.21.0/WORKSPACE:101:14: in <toplevel>
/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc14/external/tensorflow-sources-cache/2.21.0-33aa2ca680813287eb131e31bc444e4c/tensorflow-2.21.0/tensorflow/workspace2.bzl:854:21: in workspace
/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc14/external/tensorflow-sources-cache/2.21.0-33aa2ca680813287eb131e31bc444e4c/tensorflow-2.21.0/tensorflow/workspace2.bzl:418:9: in _tf_repositories
/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc14/external/tensorflow-sources-cache/2.21.0-33aa2ca680813287eb131e31bc444e4c/build/725a8a92c18c44ca588f3bb41a3eecf4/external/xla/third_party/llvm/workspace.bzl:10:20: in repo
/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc14/external/tensorflow-sources-cache/2.21.0-33aa2ca680813287eb131e31bc444e4c/build/725a8a92c18c44ca588f3bb41a3eecf4/external/xla/third_party/repo.bzl:134:21: in tf_http_archive
Repository rule _tf_http_archive defined at:
/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc14/external/tensorflow-sources-cache/2.21.0-33aa2ca680813287eb131e31bc444e4c/build/725a8a92c18c44ca588f3bb41a3eecf4/external/xla/third_party/repo.bzl:89:35: in <toplevel>
ERROR: /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc14/external/tensorflow-sources-cache/2.21.0-33aa2ca680813287eb131e31bc444e4c/build/725a8a92c18c44ca588f3bb41a3eecf4/external/xla/third_party/repo.bzl:73:33: An error occurred during the fetch of repository 'llvm-raw':
Traceback (most recent call last):
File "/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc14/external/tensorflow-sources-cache/2.21.0-33aa2ca680813287eb131e31bc444e4c/build/725a8a92c18c44ca588f3bb41a3eecf4/external/xla/third_party/repo.bzl", line 73, column 33, in _tf_http_archive_impl
ctx.download_and_extract(
Error in download_and_extract: java.io.IOException: Error downloading [https://storage.googleapis.com/mirror.tensorflow.org/github.com/llvm/llvm-project/archive/909041e4802c4b9a2223ca04099f35bf1dbbd460.tar.gz, https://github.com/llvm/llvm-project/archive/909041e4802c4b9a2223ca04099f35bf1dbbd460.tar.gz] to /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc14/external/tensorflow-sources-cache/2.21.0-33aa2ca680813287eb131e31bc444e4c/build/725a8a92c18c44ca588f3bb41a3eecf4/external/llvm-raw/temp3870220017362556620/909041e4802c4b9a2223ca04099f35bf1dbbd460.tar.gz: Checksum was 00b1077e029fa57e6f2d9ac24936a49acf23ebc051b04f487131116258be6248 but wanted 3f986184ee126677dbd77edb16d6b82c057ec869fefd7a9871979941e52e837a
ERROR: Error computing the main repository mapping: no such package '@@llvm-raw//utils/bazel': java.io.IOException: Error downloading [https://storage.googleapis.com/mirror.tensorflow.org/github.com/llvm/llvm-project/archive/909041e4802c4b9a2223ca04099f35bf1dbbd460.tar.gz, https://github.com/llvm/llvm-project/archive/909041e4802c4b9a2223ca04099f35bf1dbbd460.tar.gz] to /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el9_aarch64_gcc14/external/tensorflow-sources-cache/2.21.0-33aa2ca680813287eb131e31bc444e4c/build/725a8a92c18c44ca588f3bb41a3eecf4/external/llvm-raw/temp3870220017362556620/909041e4802c4b9a2223ca04099f35bf1dbbd460.tar.gz: Checksum was 00b1077e029fa57e6f2d9ac24936a49acf23ebc051b04f487131116258be6248 but wanted 3f986184ee126677dbd77edb16d6b82c057ec869fefd7a9871979941e52e837a
error: Bad exit status from /data/cmsbld/jenkins/workspace/build-any-ib/w/tmp/rpm-tmp.mOUoGi (%build)
This is possible today, with no change to bits: split the fetch into its own package:
tensorflow-deps is a package whose build only fetches (with sandbox_network: off so it’s allowed network access)
tensorflow requires it and builds offline from it
The fetch package’s output is a normal build tarball in the store, keyed by its hash (recipe plus version). The first CI build uploads it, and every later build with the same hash reuses exactly those bytes, years later too, even if Google’s mirror is gone. Freezing content at first fetch is exactly what OpenLoops needs. It can be architecture: share when the fetched content is platform-independent, so one copy serves every architecture.
I do not understand - if dependency changes, you generally want to rebuild otherwise, it is not dependency. If you split your package into (presumably) immutable data part and code, you control versions and decide when data part will be refreshed (or not) and that is how you control what will be rebuild. I am missing something?
@pbuncic , yes I agree that if dependency changed then we do want to rebuilt but we also want that we re-use the sources from our cache. So yes the tensorflow-deps package will be re-build when any dependency changed but we want that if TF version remain same then it should re-use the previously cached generated sources.
Lets say, we update urllib3 version and that triggers the re-build of tensoflow-deps package. Now in that case we want tensorflow-deps to re-use the sources it had fetched and cached previously. This behaviour is same as when we change a package build recipe or dependency then that package is re-build but we still get its sources from our cache (instead of going back to upstream repository).
The directory:
$BITS_FETCH_CACHE is FETCH_CACHE/// in the work directory.
Restore:
before the build, bits fills it from the store. Every file is checked against its sha256; a missing or corrupt file only gives a warning, and the build downloads it again.
Save:
after a successful build with a write store, bits uploads what the build left there.
Storage:
each file is kept once by sha256 under SOURCES/cache/, shared by all versions. Each version and architecture has its own list of files, written once by the first build that saves it. Later builds restore exactly that list (and warn about files they downloaded that aren’t in it), so a rebuild years later sees the same inputs even after an upstream mirror has gone.
Package hash:
the cache isn’t part of it, so the first build and later rebuilds get the same identity.
Licences: nothing is saved without a write store, or for recipes whose redistributable forbids sources.
OpenLoops: OpenLoops has no checksums of its own, so the first saved copy is trusted as it is. Bazel still verifies its own sha256s on restored files.
Activity
smuzaffar commentedon Oct 14, 2026
thanks @akritkbehera for opening this issue. Just to add some context to this request, in
cmsBuildwe have a way to add a source url which is not available at the time of the build but package build itself downloads and provide the contents of that source. cmsBuild then make sure to cache the generated source for future re-use. This we have used forcmsBuildfeature to cache sources which are only available after the build. A special source URL instructs the packaging system (cmsBuild) that the source url might not have any initial contents and the actual files to be cached will be generated after the build itself.[a]
pbuncic commentedon Oct 17, 2026
This is possible today, with no change to bits: split the fetch into its own package:
The fetch package’s output is a normal build tarball in the store, keyed by its hash (recipe plus version). The first CI build uploads it, and every later build with the same hash reuses exactly those bytes, years later too, even if Google’s mirror is gone. Freezing content at first fetch is exactly what OpenLoops needs. It can be architecture: share when the fetched content is platform-independent, so one copy serves every architecture.
Would this work for you?
akritkbehera commentedon Oct 19, 2026
that would lead to rebuilding of sources when dependency changes even though we don't want that.
pbuncic commentedon Oct 19, 2026
I do not understand - if dependency changes, you generally want to rebuild otherwise, it is not dependency. If you split your package into (presumably) immutable data part and code, you control versions and decide when data part will be refreshed (or not) and that is how you control what will be rebuild. I am missing something?
smuzaffar commentedon Oct 19, 2026
@pbuncic , yes I agree that if dependency changed then we do want to rebuilt but we also want that we re-use the sources from our cache. So yes the
tensorflow-depspackage will be re-build when any dependency changed but we want that if TF version remain same then it should re-use the previously cached generated sources.Lets say, we update
urllib3version and that triggers the re-build oftensoflow-depspackage. Now in that case we want tensorflow-deps to re-use the sources it had fetched and cached previously. This behaviour is same as when we change a package build recipe or dependency then that package is re-build but we still get its sources from our cache (instead of going back to upstream repository).fetch_cache: keep the sources a build downloads itself in the store
pbuncic commentedon Oct 24, 2026
Implemented in 346e73d on orchestration.
A recipe can now ask bits to keep what its build downloads itself:
It only takes effect when the community’s defaults name the cache, so no recipe can switch it on elsewhere:
pbuncic commentedon Oct 24, 2026
How it works:
The directory:
$BITS_FETCH_CACHE is FETCH_CACHE/// in the work directory.
Restore:
before the build, bits fills it from the store. Every file is checked against its sha256; a missing or corrupt file only gives a warning, and the build downloads it again.
Save:
after a successful build with a write store, bits uploads what the build left there.
Storage:
each file is kept once by sha256 under SOURCES/cache/, shared by all versions. Each version and architecture has its own list of files, written once by the first build that saves it. Later builds restore exactly that list (and warn about files they downloaded that aren’t in it), so a rebuild years later sees the same inputs even after an upstream mirror has gone.
Package hash:
the cache isn’t part of it, so the first build and later rebuilds get the same identity.
Licences: nothing is saved without a write store, or for recipes whose redistributable forbids sources.
OpenLoops: OpenLoops has no checksums of its own, so the first saved copy is trusted as it is. Bazel still verifies its own sha256s on restored files.
pbuncic commentedon Oct 26, 2026
Does this work for you? If so, please close the issue.
smuzaffar commentedon Oct 27, 2026
@akritkbehera is away this week, once he comes back then he will test this.
Orchestration (#143)