Repository navigation
[BUG] npm ci grows the cache with each run #3176
Description
After upgrading to npm 7 our Travis CI runs started to detect cache changes after each run.
This increases the duration of test runs, because it triggers an additional "store cache" step after each run.
Current Behavior:
Running npm ci two times changes ~/.npm/_cacache after the 2nd npm ci.
Expected Behavior:
No change to ~/.npm/_cacache after the 2nd npm ci.
Steps To Reproduce:
npm ci
mtree -c -K sha1digest -p ~/.npm > mtree.txt
npm ci
mtree -p ~/.npm < mtree.txt
Example run showing the effect of the issue: https://travis-ci.org/github/Scrivito/scrivito_example_app_js/jobs/769013652#L233-L237. (click the triangle icon on the left side to see the store build cache details)
In a bigger project with 1821 packages we have some more changed cache files:
_cacache/index-v5/02/b1/3c8b5d8a3900fba0f23e98df6cd0dc9f1d54f17900cb2fcc357bcb5a38a1 changed
size expected 1553 found 3304
modification time expected Sat May 1 21:06:27 2021.114046195 found Sat May 1 21:08:39 2021.284260556
SHA-1 expected 32f3a770c6a2513c71d99387df8c00969225387a found a02bb030e75cd339b47936a64ba36009abdffc62
_cacache/index-v5/5c/b9/d302c51607da8ad5a99afdd9049ad0b57872d28aac958e6785febe40b6cb changed
size expected 1536 found 3266
modification time expected Sat May 1 21:06:26 2021.754708347 found Sat May 1 21:08:38 2021.880260027
SHA-1 expected f0e19a9c68d074c0b6c93e1812d31bfaf42082f2 found eca3f6c65aed41fd59105316a983844f85b64e0b
_cacache/index-v5/98/64/01faddae5e2bf199c5f8953942897841c5634b1800e246226ea5489f4584 changed
size expected 364 found 546
modification time expected Sat May 1 21:06:09 2021.436763782 found Sat May 1 21:08:37 2021.244963558
SHA-1 expected b6bfefd0461fc64263703140a84bd313c9bc99d7 found 5b9c98cf685492d787c3a4189958be6b05047a0d
_cacache/index-v5/dc/8a/4dc2a1bdf5b195e99648a945ce742de5f6d0b1aac410a6fe3e7155933cc7 changed
size expected 1554 found 3308
modification time expected Sat May 1 21:06:26 2021.264559792 found Sat May 1 21:08:38 2021.263061305
SHA-1 expected d7c52609e746c05d667627edec95f6eaf978ee94 found 06a84f3afe1fea40e0ab5c755f586d08db32d610
_cacache/index-v5/b0/3e/b8a6a2861b39d6a52af270cd952366bc25cf491175c06aa8c7634e4e3f96 changed
size expected 1551 found 3303
modification time expected Sat May 1 21:06:26 2021.900409983 found Sat May 1 21:08:39 2021.125024382
SHA-1 expected 6ce841db85dc3ba2cb92cae364253034c34e31df found 921066925f4b370a7ea43a6cc76bf8a051d5e519
_cacache/index-v5/85/dc/ef67dd87104b21fe208d6c85fe1ac916fdb7b78f7c44cb07eadeb32156c1 changed
size expected 1572 found 3350
modification time expected Sat May 1 21:06:26 2021.606812258 found Sat May 1 21:08:38 2021.632563439
SHA-1 expected 664318f36adff6742884b28a86fb5bf7babfa4a1 found 93e29dcc0dc300a8a9083ae57e46f410e39972a1
The reasons for the unexpected cache entries vary, they differ in
timein~/.npm/_cacache/index-v5/98/64/01faddae5e2bf199c5f8953942897841c5634b1800e246226ea5489f4584- Different
reqHeaders, e.g.npm-session - Different
resHeaders, e.g.date,age,cf-ray,cf-request-id - Missing vs. present
resHeaders, e.g.accept-ranges,content-type,X-Local-Cache
Examples
~/.npm/_cacache/index-v5/98/64/01faddae5e2bf199c5f8953942897841c5634b1800e246226ea5489f4584
1e1b2caf31bff978092f6e85c66c8cb0a59973ff {"key":"make-fetch-happen:request-cache:https://registry.npmjs.org/-/npm/v1/security/advisories/bulk","integrity":null,"time":1619895945806}
3bed652a04c04142ab495dadcf3c225c804c93d2 {"key":"make-fetch-happen:request-cache:https://registry.npmjs.org/-/npm/v1/security/advisories/bulk","integrity":null,"time":1619895969433}
~/.npm/_cacache/index-v5/5c/b9/d302c51607da8ad5a99afdd9049ad0b57872d28aac958e6785febe40b6cb
d02c0829ba221f877d5f2751e47542c469083579 {"key":"make-fetch-happen:request-cache:https://registry.npmjs.org/fbjs","integrity":"sha512-WmmvC1kCjNlCbcAsYz+Wg/NpIFIvBnF5jYhM1haqYa1QGmY79LB7c6PXOQbtd73NqmzABo0sKAosBzwP4HMx8Q==","time":1619895986750,"size":40580,"metadata":{"url":"https://registry.npmjs.org/fbjs","reqHeaders":{"npm-in-ci":["false"],"user-agent":["npm/7.0.0 node/v14.16.1 darwin x64"],"pacote-version":["11.1.11"],"pacote-req-type":["packument"],"pacote-pkg-id":["registry:fbjs"],"accept":["application/vnd.npm.install-v1+json; q=1.0, application/json; q=0.8, */*"],"npm-session":["95498c74deac94da"],"npm-command":["ci"],"authorization":["Bearer 02e3be98-f6ea-4ba5-ac53-250710b8c19b"],"connection":["keep-alive"]},"resHeaders":{"date":["Sat, 01 May 2021 19:06:26 GMT"],"content-type":["application/vnd.npm.install-v1+json"],"content-length":["40580"],"connection":["keep-alive"],"set-cookie":["__cfduid=d6c7df5414da83d9fe0332159d84d17a61619895986; expires=Mon, 31-May-21 19:06:26 GMT; path=/; domain=.npmjs.org; HttpOnly; SameSite=Lax"],"cf-ray":["648b497cbbdc4150-HAM"],"accept-ranges":["bytes"],"age":["2624"],"cache-control":["public, max-age=300"],"etag":["\"ca47ebb6a06c244ee259fe73f1478899\""],"last-modified":["Sat, 03 Apr 2021 18:08:04 GMT"],"vary":["accept-encoding, accept"],"cf-cache-status":["HIT"],"cf-request-id":["09caec41fb000041503d1fd000000001"],"expect-ct":["max-age=604800, report-uri=\"https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct\""],"server":["cloudflare"],"x-fetch-attempts":["1"]}}}
21d2a18108a13e6a0b6b8eb75b75db1d04d1d641 {"key":"make-fetch-happen:request-cache:https://registry.npmjs.org/fbjs","integrity":"sha512-WmmvC1kCjNlCbcAsYz+Wg/NpIFIvBnF5jYhM1haqYa1QGmY79LB7c6PXOQbtd73NqmzABo0sKAosBzwP4HMx8Q==","time":1619896118862,"size":40580,"metadata":{"url":"https://registry.npmjs.org/fbjs","reqHeaders":{"npm-in-ci":["false"],"user-agent":["npm/7.0.0 node/v14.16.1 darwin x64"],"pacote-version":["11.1.11"],"pacote-req-type":["packument"],"pacote-pkg-id":["registry:fbjs"],"accept":["application/vnd.npm.install-v1+json; q=1.0, application/json; q=0.8, */*"],"npm-session":["c5d842abffa519ff"],"npm-command":["ci"],"authorization":["Bearer 02e3be98-f6ea-4ba5-ac53-250710b8c19b"]},"resHeaders":{"date":["Sat, 01 May 2021 19:08:38 GMT"],"connection":["keep-alive"],"set-cookie":["__cfduid=d7e55572ebeb0148266c609679d51be9e1619896118; expires=Mon, 31-May-21 19:08:38 GMT; path=/; domain=.npmjs.org; HttpOnly; SameSite=Lax"],"cf-ray":["648b4cb5fd8f2199-DUS"],"age":["7113"],"cache-control":["public, max-age=300"],"etag":["\"ca47ebb6a06c244ee259fe73f1478899\""],"last-modified":["Sat, 03 Apr 2021 18:08:04 GMT"],"vary":["Accept-Encoding"],"cf-cache-status":["HIT"],"cf-request-id":["09caee45b800002199ad283000000001"],"expect-ct":["max-age=604800, report-uri=\"https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct\""],"server":["cloudflare"],"x-fetch-attempts":["1"],"X-Local-Cache":["%2FUsers%2Fdave%2F.npm%2F_cacache"],"X-Local-Cache-Key":["make-fetch-happen%3Arequest-cache%3Ahttps%3A%2F%2Fregistry.npmjs.org%2Ffbjs"],"X-Local-Cache-Hash":["sha512-WmmvC1kCjNlCbcAsYz%2BWg%2FNpIFIvBnF5jYhM1haqYa1QGmY79LB7c6PXOQbtd73NqmzABo0sKAosBzwP4HMx8Q%3D%3D"],"X-Local-Cache-Time":["Sat, 01 May 2021 19:06:26 GMT"]}}}
Environment:
- OS: OSX, Ubuntu 16.04
- Node: v12.22.1, v14.16.1
- npm: 7.0.0, 7.11.2
Activity
wraithgar commentedon May 6, 2021
npm/make-fetch-happen#48 Is the issue where this is being tracked.
dcsaszar commentedon Jun 27, 2021
Now that npm/make-fetch-happen#48 is fixed and released I still see the issue, at least for
dateandtime, as mentioned in my original comment. For example:Reproduced with npm 7.19.0:
How to proceed from here?
nlf commentedon Jun 28, 2021
the behavior you're seeing is correct. it's working as intended and as required. the cache indexes will, theoretically, never contain more than 3 entries in each for packuments. 2 for tarballs.
every time we do a cache lookup, we deduplicate the existing cache index to matching request/response pairs (of which there is a maximum of 2 for packuments, due to having these in two different formats, and 1 for tarballs) and then examine that deduplicated list for something that matches the current request. we then either revalidate an existing response or fetch and store a new one, appending that latest result to the index.
the file must change over time because we need to know when we wrote something to the cache, that timestamp serves as part of the freshness check to determine if we should respond from the cache or send a new request. if we never updated it, you would never receive an updated packument and never be able to retrieve newly published versions of a package. the changes we made were simply to prevent it from growing for every single request.
if, in your build system, you want to minimize changes to the cache you can pass
--prefer-offlinein your build system, then npm will only send a network request if something is missing from the cache entirely, it will then write the missing data to the cache but will not modify anything that's already there. you will potentially get stale data, though, so be prepared for that.dcsaszar commentedon Jun 28, 2021
Allright, thanks. I'll look into
--prefer-offlinethen.Doesn't sound particularly critical in the context of an
npm ci, which will only fetch the package versions baked into the package-lock. I hope I'm not missing something here.wraithgar commentedon Jul 22, 2025
@dcsaszar There was an active token listed in this comment. It has been revoked.