Warn packagers about files with multiple identical paths in their package tarballs #3223

Describe the feature

Today I discovered and fixed a bug in Bun's package publishing utilities:

• fix(pack): Now dedups paths in package.json bin field oven-sh/bun#41291

We know that neither the TAR standard forbids nor does NPM itself reject such archives. I accidentally created such a package myself with Bun.

packed 1.0KB package.json
packed 8.19KB README.md
packed 247.73KB dist/minified/index.mjs
packed 247.73KB dist/minified/index.mjs

Total files: 4

Packages created by bun pack/publish, as well as other package publishers, might be affected because there are no upstream guardrails. Bun users will still be affected for some time before either my PR or its modified version lands. Given the ratio of open/closed PRs on Bun's repo (5k/16k), I doubt it will happen any time soon.

I haven't gathered enough data yet to estimate how many packages with the same problem exist.

This feature request consists of essentially 3 things:

1. Show a user on a main page a warning that the package is silently corrupted and unnecessarily bloated and should be republished properly.
2. Make the stats page report the actual package install size, and not blindly trust the size reported by NPM. And maybe show both numbers: on disk vs. the data to be fetched.
3. Make the code page highlight the paths that seem duplicated, similarly to how feat(ui): highlight files commonly shipped unnecessarily in packages #2714 does it. I don't recall the exact implementation or how exactly the data is fetched. It only shows a single open file and a list of paths. I doubt you would fetch the whole package immediately just for that. And depending on whether the upstream data provider of the list of paths dedupes them or not, it might also be very easily implementable.

This would require some implementation, and I spent some time thinking about potential ways to do that.

Create a stateful backend service.

Doesn't store the actual packages. Doesn't even need to store them midflight because you only operate on the streamed metadata parts, discarding the contents. All it needs is essentially just file paths to identify if some would overwrite itself.

Variations:

1. Process replication stream

   Variations:

   • can always be running, listening for the stream of changes
   • can wake up on CRON and process in batches the latest updates
   • Not sure if possible to deploy as an NPM-triggered lambda (but there are ways to launch on schedule). Even if NPM exposes a webhook like that, does it guarantee a stable order of events to drastically reduce storage needed, and is it replayable?

   Pros:

   • Compact storage. Only needs to store quite a small (I assume) set of [scope],package,version values (and can be stored efficiently via a trie, via a small normalized SQLite DB, via a plain JSON file, maybe even via atproto lol; anyway, a bit hard to estimate right now) and the latest processed replication pointer
   • durable because can just continue pulling new work after restarts, crashes, OOMs, whatever from the latest recorded replication pointer
   • rate limits don't usually apply to this approach, and you can pull data at whatever rate the internet connection allows

   Cons:

   • takes one big prefil. A price paid once (depending on the network speed, might take between a few hours and a full day)
   • processes, and rarely stores the scope+package+versions that might not ever be seen by npmx users. Given how small I assume the number of these broken packages is out there, it might not be a problem at all.

2. Process packages on demand as they're requested via the web. The npmx visitors are the automatic trigger. I assume the set of affected packages is small and can be stored entirely without eviction.

   Variations:

   • Bug-less majority of packages is stored
     • Entirely to avoid ever refetching and recomputing. My estimation for this whole thing is about 500Mb-1Gb of data under the condition that NPMX users (scrapers/bots) will visit every single package and version that exist on NPM.
     • As a small-enough subset in a cache for popular pages with whatever eviction strategy
   • Deployment
     • Lives directly inside the server
     • A lambda that's triggered by either the npmx backend or as a separate second request from a browser

   Pros:

   • If we pretend that scrapers don't exist, you need to store only a subset of packages smaller by the same degree that NPMX's web UI traffic is smaller than NPM's.

   Cons:

   • A potential to store much more data than needed, because you also need to store whatever subset of packages NOT affected by the bug, to then avoid refetching it from NPM to not waste the server's bandwidth.
   • Might face NPM's rate limits

Pros:

• the work is done globally once per package version published/first-viewed
• no additional insane load on NPM
• isn't really computationally expensive, and isn't very bandwidth-intensive
• allows for static site generation. Can even be implemented nicely via something like a Suspense boundary, if Vue has something like this. The warning is tied to the version and will never change can be baked in

Do it client-side

Every time a user opens a page, the actual package tarball starts loading asynchronously after most of the page resources are fetched.

Variations

• Cache isn't stored
• Cache stored
  • in IndexedDB
  • in OPFS -- a possibility to choose a much more efficient format, but anyway I don't assume a single person to visit so many NPMX pages that it would ever become a problem

Pros

• No additional load on NPMX
• No need for a backend and storage to deploy and maintain by the NPMX team

Cons

• Ugly
• Wastes too much of the world's resources. On phones and metered connections, it has the potential to load VERY big packages that weigh like hundreds of megabytes and be quite costly, especially without a cache, just to get a simple boolean flag at the end of the operation.
• The same work is done multiple times across the globe
• I know NPM handles Billions of package downloads a day, but why would you still choose to hammer the non-profit that serves requests free of charge?...
• Also has to store not only the list of packages that are affected by the bug, but also the ones that aren't. Essentially logging and storing purely client-side a list of packages and versions the user has ever visited.

Find whoever solved this problem already and reports THE ACTUAL install size that accounts for files overwriting each other

Something that actually analyses the tarball's content instead of trusting the misreported field inside the package's metadata. I have no idea if anything like this exists, but I think this would probably be the best option: finding some correct data provider.

Additional information

• Would you be willing to help implement this feature?

Final checks

• Read the contribution guide.
• Check existing issues.

_{(even though this is a bunch of text, it's created and typed entirely by me)}