Repository navigation
Implement a global non-maven download policy #1669
Description
The Spotless core has few dependencies. During plugin configuration, we add maven dependencies which get downloaded in the normal way.
There are some exceptions where we use non-maven formatters. For example
- npm-based formatters which grab dependencies using npm install
- eclipse-based formatters which download dependencies from p2 update sites
There are other cases we are considering, such as
- Support Rome as a formatter #1663
- Better method for managing python black package #674
- Better method for managing clang-format binaries #673
A very good thing about Spotless is that it "just works". Search the docs for the kind of formatter you want, specify that formatter, and you're ready to go. But for these non-maven-based formatters, I think it's fair for users to be surprised that their build plugin is making network requests and caching artifacts outside of the expected maven channel.
I'm proposing something like this:
spotless {
allowDownloadsFrom('https://download.eclipse.org/')As a user, it would work like this:
- I add Eclipse CDT to my spotless configuration
- I get an error like so
Eclipse CDT is not available in a maven repository, it must be downloaded from a p2 repository. You can allow this download by adding a block like this to your spotless configuration
spotless {
allowDownloadsFrom('https://download.eclipse.org/')This will allow Spotless to download p2 metadata and jar files using the Equo p2 client. The metadata and jar files will be cached at ~/.equo. Alternatively, you can specify an alternative p2 update site, and add that URL to your allowDownloadsFrom.
It's not a perfect mechanism, it relies on our own diligence in making sure that PR's such as the Rome PR follow the rules about checking the allowed URLs. But it takes away the surprise, which is important. Users might choose a different formatter based on whether it is available through a standard maven proxy or not.
Activity
blacelle commentedon Apr 7, 2023
InReading more about it, I feel it is unrelated to this matter (except if downloading from external sources would require specific additional customizable configuration (e.g. auth)).mvnleg, this relates to<server>section. Some plugins does this : spotify-maven-plugin, or azure-maven-pluginMaybe a list of hosts would be a better fit.
I feel premature to consider mirroring capabilities, but I would add it in the radar.
This is also a relevant information for people integrating Spotless in a SaaS deployment :). (For Cleanthat, how can I customize this path?)
nedtwigg commentedon Apr 7, 2023
This is different formatter-to-formatter. For example, the Equo formatters download locations are based on this (customization info), and you can change mirroring like this. I agree that just an allowed-host list would work great for Eclipse.
But for the Rome formatter, it downloads to
~/.m2/repository/com/diffplug/spotless/spotless-data, and it downloads fromhttps://github.com/rome/tools/releases/download/(lots of people willing to add that URL that would be less comfortable with justgithub.com).Equo is the latest change from non-downloading to downloading, but it's not the only place in the codebase where we break out of the expected maven-only behavior.
blacelle commentedon Apr 8, 2023
We agree Equo is just the latest example of
non-mvndownloads.I was about to argue than individual users or corporate users would be interested in URL-prefix whitelists or host-only whitelists. Writing about it, I rather feel:
github.comwould be enough)github.combeing generally not sufficient)).For my personal case (an individual and a corporate users), I'm still not fully convinced I care much about URL-prefix as:
whitelist('*')is generally fine with mem2central,npmregistry)I would suggest switching this to
~/.m2/repository/com/diffplug/spotless/spotless-data/equoblacelle commentedon Apr 17, 2023
Here is a similar request but gradle-related: #1524 (comment)
SemyonRomankov commentedon Jan 29, 2024
Did I understand you right that you just hardcode "https://download.eclipse.org/' as the only one repo from you can fetch eclipse(versions 2.4+)? If the answer is "yes", I believe you should fix that, because any other repositories(mirrors) that we will pass to maven to fetch eclipse will be ignored.
sann3 commentedon Aug 1, 2024
We initially added download.eclipse.org to
-Dhttp.nonProxyHostsingradle build, but still it is failing with following exceptionThen to find the actual issue we added
--debugtogradle buildand found that it is looking for groovy urlWe got the groovy url from
https://github.com/diffplug/spotless/blob/main/lib-extra/src/main/java/com/diffplug/spotless/extra/groovy/GrEclipseFormatterStep.java
So we added both
download.eclipse.org,groovy.jfrog.ioto-Dhttp.nonProxyHostswhich solved the issue.gregallen commentedon Jan 24, 2025
Many firms do not allow repositories to be accessed by mirrors at all. Instead we have a single corporate mirror that presents a security scanned cache of several external sites. This is preconfigured and we do not want any plugins to attempt to modify the repository list.
Ideally there would be a well documented configuration option to disable adding repos. Also be good to be clear which tasks might be affected by selecting that
gregallen commentedon Jan 24, 2025
There might be some configuration cache issue at play here. With latest gradle I see it trying to serialize the state of tasks which trigger an eclipse download attempt for spotlessGroovy task. This is unfortunate since we don't want nor use this task and so had not previously attempted to configure p2mirrors
gregallen commentedon Jan 24, 2025
Problem is due to EquoStepBuilders state() method calling jarPromise.get()