Repository navigation
Cleaning up old nightly conda packages #394
Description
Having support for dask nightlies has been wonderful from a development standpoint
Recently we noticed that we are getting quite a lot of them tracking back over the years (back to 2021!)
As a result this means there are a lot of packages in the dask channel
This can make the channel repodata get quite large (and unfortunately a bit unwieldy)
So am wondering if we can clean this up a bit. Namely would it be reasonable to remove all nightlies after making a release?
Presumably users only care about nightlies newer than the last release. So dropping the old nightlies shouldn't matter much. Perhaps we can do this in an automated GHA script when a git tag is detected
Would be interested to hear thoughts on this 🙂
Activity
jrbourbeau commentedon Aug 6, 2024
Thanks for the issue @jakirkham. Cleaning up old nightly builds seems very reasonable to me.
I don't use these builds myself, but that seems sensible. To my knowledge the main consumers are various RAPIDS-related users / projects (I could be wrong about that), so if you / those folks are happy with that policy, then 👍
jakirkham commentedon Aug 6, 2024
Thanks James! 🙏
Cool we can look into this then
Ok it might be worth double checking at Coiled. Recall hearing these were used previously ( for example: coiled/benchmarks#311 (comment) ). Though don't have much visibility over there
jrbourbeau commentedon Aug 6, 2024
Fair point. Though I think things have changed in that repo since then (cc @hendrikmakait in case you happen to know if the
daskconda channel is used)fjetter commentedon Aug 7, 2024
we're not using nightlies
keewis commentedon Aug 8, 2024
you might want to check with the scientific-python people, as far as I can tell they have something similar working for
scientific-python-nightly-wheelsand the upload-nightly-wheels action.cc @matthewfeickert, in case you have any hints
matthewfeickert commentedon Aug 8, 2024
Dask nightlies sound pretty great to me. :) If the Dask team is interested, either give me the GitHub handle of a team member to work with and I'll setup a Issue on https://github.com/scientific-python/upload-nightly-action, or have a team member take a look at scientific-python/upload-nightly-action#87 and then self start a similar issue.
edit: Ah, I only read the last comment before posting. I guess the Dask team already has nightlies and wants a way to clean them up? If so, we have a system that works for https://anaconda.org/scientific-python-nightly-wheels using https://github.com/scientific-python/upload-nightly-action/blob/cb172cd73e045bb8b9fdaf271a183ffe4b5814d3/.github/workflows/remove-wheels.yml, but that's currently (c.f. scientific-python/upload-nightly-action#62) only for scientific-python-nightly-wheels.
charlesbluca commentedon Aug 9, 2024
Opened dask/dask#11294 with a first pass at what these changes could look like
Also given recent inactivity on dask-sql (dask-contrib/dask-sql#1344), think we should also be good to outright purge
dask/dask-sqlnow