Repository navigation
Cloudflare R2 Workers #3461
Description
As part of the continuous conversation we've been having on Slack (and also spread in some of the nodejs/build issues); We have long-term goals for improving the reliability of our Distribution Assets (aka /home/dist) on the DigitalOcean Server.
The main goal is to uphold a reliable way to serve our assets (binaries, docs, metrics, etc) to the public.
What's the issue?
As mentioned in a few issues such as (#3424 and #3410) and on our March 17th incident (on https://nodejs.org/ko/blog/announcements/node-js-march-17-incident) and discussed over many other places; Our DigitalOcean server is unable to serve all the traffic it gets.
Of course, a few things are set in place, such as Cloudflare-caching, so that, in theory, all requests are served by Cloudflare after the initial load.
This has been proven inefficient due to numerous factors, for example, cache purges. (Even if the cache purges were tailored to only the affected paths) It is still a risky approach that creates a gigantic load on our DigitalOcean server. The same server is stored for all Node.js Binaries and numerous other vital assets to date.
Not to mention that even for a short period, the server cannot withhold the immense traffic that goes through it.
Meaning: In the best scenario, this server should never enter a stressful situation.
What's the Plan?
Champions
Proof of Concept Repository
After numerous discussions with the Build Team, the OpenJS Foundation, and Cloudflare, we've concluded: The DigitalOcean server should not serve these files at best.
Enter the solution: Cloudflare R2
The idea is that all requests that do not go to Vercel (aka selected paths such as /download, /dist, /docs, /api) will go through a Cloudflare Worker.
This worker is responsible for:
- Serving content from a R2 Bucket
- The R2 Bucket has the contents of
/home/distfrom the DO server synced- Meaning that the root (
/) of the R2 bucket is the contents of/home/dist
- Meaning that the root (
- The sync of file is done through a script/daemon/or something else sitting inside the DO server.
- It should be reactive and only do additions/removals/updates based on
fschanges (so it is incremental)
- It should be reactive and only do additions/removals/updates based on
- The R2 Bucket has the contents of
- Mapping Requests
- Since the contents is the same from the DO server, we should respect the same paths originally created on the
NGINX nodejs.orgconfig file. - For example, requests to the following places go mapped:
/distgoes to/nodejs/release(originally on DO/home/dist/nodejs/release)/downloadgoes to/nodejs(originally on DO/home/dist/nodejs)/docsgoes to/nodejs/docs(originally on DO/home/dist/nodejs/docs)/apigoes to/nodejs/docs/latest/api(originally on DO/home/dist/nodejs/docs/latest/api)/metricsgoes to/metrics(originally on DO/home/dist/metrics)
- These mappings ensure that requests are respected the original way intended on NGINX
- Since the contents is the same from the DO server, we should respect the same paths originally created on the
- Serving Directory Listings
- Caching Access (setting cache headers)
- Serving 404's
The Worker will never make a single request to the DigitalOcean origin, because it should always be in sync with whatever is in DigitalOcean.
This also means that:
- We can disable our Load Balancer configuration on Cloudflare
- In case of emergencies, we can disable the Worker and re-enable the Load Balancer
- Cloudflare will never make requests to DigitalOcean itself
- DigitalOcean Server is still the source of truth
- Load on the Server will be close to 0
The Long Term
This solution is also long-term proof, But in the future we could create ways that Jenkins uploads directly to R2, for example, or other shenanigans.
Activity
mcollina commentedon Aug 19, 2023
What is the expected cost of this? The worker would be executed millions of times per day.
ovflowd commentedon Aug 19, 2023
0
MoLow commentedon Aug 19, 2023
can you describe why? is this donated by cloudflare?
MoLow commentedon Aug 19, 2023
added to the build adgenda for visibilty and tracking
flakey5 commentedon Aug 19, 2023
Yep! @ovflowd would know more of the specifics regarding the quotas they gave
ovflowd commentedon Aug 19, 2023
Yes. To iterate on that Cloudflare gaves us R2 and Worker quotas/credits that are enough for 2x what would be our usage.
MoLow commentedon Sep 13, 2023
@nodejs/build should we procced with this or should we continue waiting for the BuildWG meeting?
ovflowd commentedon Sep 13, 2023
Afaik @flakey5 is almost finishing their implementation.
MoLow commentedon Sep 26, 2023
Discussed on the Build WG - will work with @ovflowd and @flakey5 on getting this deployed
MoLow commentedon Sep 27, 2023
worker is up in running. see:
https://dist-worker-prod.nodejs.workers.dev/dist
https://dist-worker-prod.nodejs.workers.dev/download
https://dist-worker-prod.nodejs.workers.dev/docs
https://dist-worker-prod.nodejs.workers.dev/api/
https://dist-worker-prod.nodejs.workers.dev/metrics
workers.devdomain, so this might be slower then the final resulta manual synchronization of the data has been performed, I am now working on automating the deployment of release artifacts into the r2 bucket
ovflowd commentedon Sep 27, 2023
I wonder if the directory listing the
Modifiedfield is important? I also wonder if the fact that the new directory listing renders differently from the usual way NGINX does is somehow problematic.How does the web use directory listing to index things? Tools such as Artifactory, for example.
MoLow commentedon Sep 27, 2023
I was actually thinking we can make it look nicer in the future
ovflowd commentedon Sep 27, 2023
Sure, but I don't want to break the web 😅
12 remaining items
trivikr commentedon Mar 22, 2024
I'm visiting this issue from an old issue I'd reported on
nnode version manager tj/n#784What are the remaining action items on this request? Is the cache purging issue fixed?
If there an availability dashboard which we can refer to verify it's fixed?
flakey5 commentedon Mar 22, 2024
Right now we're waiting on reviews for a few prs (nodejs/node#51394, #3620, nodejs/release-cloudflare-worker#103). I hope when those are landed we can move forward with this.
With the current infra, no. The cache is still being purged completely with each release of Node. With the worker, selective cache purging isn't necessary since Workers and R2 can most definitely handle the traffic that comes from a complete cache purge and more. It still is something to tackle though to ensure we're being smart with our usage, but it can wait (imo at least).
If you're referring to a Node status page, there's one here https://status.nodejs.org/ but from what I can see it's used for announcing major outages?
flakey5 commentedon Jun 26, 2024
As an update: we are waiting on #3620 to be merged and deployed onto the www server. Once it is I'd like for us to do a test of the full pipeline (from release CI to promoting it). Once we see that the full pipeline is working, we can continue with rolling the worker out when y'all are ready. I'm still in favor of incrementally rolling it out like we did with the
https://nodejs.org/download/*path for a bit.The worker is ready to be deployed whenever. We're in the middle of refactoring it but it's more so for code quality than anything functionality wise. It also mimics nginx's directory listing response so we shouldn't see another nodejs/citgm#1028
trivikr commentedon Jul 3, 2024
The PR #3620 is merged.
Who's going to deploy it and test the full pipeline, so that worker can be rolled out?
targos commentedon Jul 3, 2024
I deployed it right after merge. The only way to test it is waiting for a new release to happen.
flakey5 commentedon Jul 3, 2024
Nightly build was promoted from the looks of it!
https://r2.nodejs.org/download/nightly/v23.0.0-nightly2024070366b76e24e2/
https://nodejs.org/download/nightly/v23.0.0-nightly2024070366b76e24e2/
ovflowd commentedon Jul 3, 2024
So should we do a retroactive sync of the buckets?
flakey5 commentedon Jul 3, 2024
Yeah!
flakey5 commentedon Jul 3, 2024
Just to be safe I think we should monitor the worker the next time a regular release is made (v22.5.0 or whatever's next) and make sure everything is updated correctly.
Once we see everything working as intended, I'd be in favor of rolling it out. I'd like it to be a day where I and someone else on @nodejs/web-infra are free though so we can monitor it and make any changes necessary
trivikr commentedon Jul 3, 2024
Can this be verified with Security Releases planned for Thu, Jul 4th?
flakey5 commentedon Jul 3, 2024
Should be able to!
trivikr commentedon Jul 8, 2024
Security releases are out
I don't see directory created on r2 subdomain though https://r2.nodejs.org/download/release/v18.20.4/
ovflowd commentedon Jul 8, 2024
#3620 (comment)
flakey5 commentedon Nov 4, 2024
The following paths are being routed through R2 as of now:
/api//download/chakracore-*//download/next-nightly//download/nightly//download/rc//download/test//download/v8-canary/targos commentedon Nov 7, 2024
We just enabled
/dist/*and/download/release/*(with the exception of the index page on those paths that still goes to DO because of nodejs/release-cloudflare-worker#163)