Repository navigation
Nearform can no longer host machines #3615
Description
Creating this to capture/track as opposed to email discussion which is harder to pull people into.
Nearform has let the build WG know through email that they can no longer host the machines they had in our datacenter. These include
- 2 Windows on ARM machines
- 3 OSX machines
- 2 Large benchmarking machines.
They have proposed moving then to another hoster which would cost $3856 Euros as a move cost and then $850 Euro per month as an ongoing cost.
From informal discussion so far we believe we don't need the Windows on ARM machines as they have been replaced by machines in Azure. That may make the cost a bit lower.
The options going forward at a high level would be:
- Move all the machines (minus windows on ARM), proposal from NearForm
outlined above which would require foundation Funding. - Find a new host/company willing to sponsor hosting/or host the machines for free.
- Find replacements (minus windows on ARM)
- Possibly reach out to MacInCloud for OSX machines (1 ARM and two x86)
- Find new sponsor for large benchmarking machines.
Initial discussion is that we don't believe we can/should just create the larger machines in existing hosters. As part of this process we should also confirm with the performance team what size machines are actually needed.
Given that there have been discussions with the Foundation/Linux IT team about them helping to manage machines and their stated approach of "fully owning" what they manage it would be good to see if Linux IT can take on solving this time sensitve issue for the project.
@bensternthal could to you take on getting Linux IT to give us a yes/no in terms of taking this on, ideally in a timeframe needed by Nearform.
@efrisby could you share what the required timeframe for a move is?
Activity
efrisby commentedon Jan 22, 2024
Hi Michael,
We have a Fibre line that needs to be removed that controls the fixed ip addresses that are currently on the servers. We can keep this in place for a period of time, but we would be hoping to shut down in under 2 months.
So if we could set the 1st of April as the deadline, would that give enough time to address the above?
Thanks,
Eamonn
mhdawson commentedon Jan 22, 2024
@nodejs/performance FYI
ryanaslett commentedon Jan 23, 2024
Hi Michael, Ryan Aslett from LF IT here.
I've been doing a bit of background research on this and wanted to make sure I understood what the requirements are.
If I understand the situation correctly, these are physical machines that Nearform is hosting for nodejs in their datacenter, which they can no longer continue to support having on their network.
My understanding of the 2 Windows on ARM machines, (Surface Pro)
https://github.com/nodejs/build/blob/main/ansible/inventory.yml#L227-L228 which were donated/on loan from ARM (#2540 (comment) and #2540 (comment)) and have since been decommissioned in favor of resources at Azure. Looks like there is still some question as to what state they're in (#3286). Seems prudent that those either get returned to ARM or dealt with some other way.
The OSX machines:
It also seems like theres 4(?) OSX machines, two x86 ones, with 2 VM's running on them via VMware fusion, and 2 ARM based ones that I believe are just bare metal machines (Couldn't find any info about the state of any VM's on the ARM Mac Mini's in the issue history, other than the fact that the IP's in the inventory match what @efrisby mentioned here: #3390 (comment)).
One of the x86 mac mini's VM's were split between release and test for 10.15, and the other has 2 vm's dedicated to test of 10.15-x64
The release machine was retired because 10.15 isnt able to run xcode13 and notarize.
It looks like there were some recent experiments to get 10.15 x64 tests to run on orka: https://ci.nodejs.org/computer/test%2Dorka%2Dmacos10.15%2Dx64%2D1/builds .
for the 2 ARM based mac minis, it seems like the test one has been unused for the last 11 days:
https://ci.nodejs.org/computer/test%2Dnearform%2Dmacos11.0%2Darm64%2D1/builds, and its jobs are being run on macstadium nodes already: https://ci.nodejs.org/computer/test%2Dmacstadium%2Dmacos11.0%2Darm64%2D4/builds
I do not have access to the release jenkins, so Im not sure what the status is of the nearform ARM release machine, other than it seems like we now have two functioning release machines for macos11-x64-1 #3179 (comment).
Given that, I wonder if we already have capacity at Macstadium an Orka to handle the roles these Nearform OSX machines are performing ? (Though, perhaps we might need another additional orka testrunner for 10.15/x64)
Would the goal in pursuing MacInCloud or another provider (i.e. sponsored https://aws.amazon.com/pm/ec2-mac/ ) be mostly for redundancy and resiliency against provider outages?
** Large Benchmarking Machines **
Other than the specs of the machines themselves,
#791 (comment), Im not sure how these are used, and whether or not there is a requirement to keep those specific machines as the benchmark machines, or if moving to another resource is an option.
Would changing the benchmark infra be an option? can those be virtual/cloud based machines, or is bare metal a requirement?
In any case I look forward to helping get this figured out.
mcollina commentedon Jan 23, 2024
Virtual/is really not an option. However any "bare metal" host would do. I personally use a Hetzner machine for similar purposes (it's significantly worse/slower, but we need the consistency of results, not the actual speed).
In terms of resources, we could do with similar specs (I don't have those handy), or even something a bit less powerful.
My 2 cents is that those machines are likely near end-of-life.
Uzlopak commentedon Jan 23, 2024
Is there something speaking against using github runners?
mcollina commentedon Jan 23, 2024
Anything running on VMs have too much interference and the standard deviation between runs is too high to measure bytecode level optimizations, e.g. microbenchmarks.
ryanaslett commentedon Jan 23, 2024
Github actions supports self hosted runners, even bare metal ones, but converting the Jenkins CI infrastructure to a Github Actions infrastructure is an ambitious undertaking that would be unlikely to succeed in the timeframe of this immediate need.
#2247 seems like a good place to continue discussing whether or not that's an eventual or possible outcome.
targos commentedon Jan 23, 2024
2 Windows on ARM machines
I agree it seems we don't need them anymore.
The OSX machines
Currently almost unused because the current version of Node.js doesn't support macOS 10.15. These machines could be updated to macOS 12, 13, or even 14.
I'm not sure we have enough capacity to replace them at Macstadium (we already struggle with disk space), but I would be more happy if we find other providers to donate resources (for example, Scaleway have bare metal M1 and M2 Pro mac minis).
The Intel benchmarking machines
The systems we have now are based on dual-CPU Intel Xeon E5-2699 v4.
The each have a total of 88 logical cores and 64 GB of RAM.
I'm also familiar with Hetzner machines at work and maybe we should try to ask them for sponsorship. They have machines with up to 64 cores, and datacenters in different countries.
sxa commentedon Jan 23, 2024
Thanks for confirming - that was one of the questions I had when discussing this with some members of build yesterday - I figured that was likely the case. Obviously the implication is that we won't be able to compare "old" runs with "new" runs without re-running them, but that shouldn't be too much of a problem (we can always re-run if required).
Does the performance team require two systems or would one be adequate for the capacity needs?
I believe that's the primary driver, yes. AWS should also be a viable option if they were willing to sponsor us.
ryanaslett commentedon Jan 23, 2024
Regarding OSX testing:
Based on everything I've been able to glean from issues and meeting notes, it seems like a good path forward would be to lean into what we're doing with MacStadium for the short term, with an eye on having a secondary provider longer term.
Regarding Benchmark testing:
mcollina commentedon Jan 24, 2024
In order to land any performance related PR, we run the benchmarks.
Usually benchmarks run on dev machines are not effective.
I don't have actual stats, but I'd say we run them weekly.
Some of those jobs lasts 6-8 hours, and in the most extreme cases days.
The lack of benchmarking machines would slow down progress on most things performance related.
They are not part of our release process.
I's guess a few times per week.
One of the key strategies we employ is to rely on previous runs to compare.
I'd not really trust this setup, because the actual machine would change every time.
On top, AWS spot instances cost for c5.metal (seems a good choice in terms of resources) is likely 3x (or more) compared to a provider like Hetzner.
richardlau commentedon Jan 24, 2024
Do we? I thought each Benchmark CI job that is ran runs through the requested benchmark(s) twice -- once with the base branch (i.e. what is being compared to) and once with the PR being tested.
benchmark-node-micro-benchmarks runs this shell script.
mhdawson commentedon Jan 24, 2024
Our goal across platforms has been to have at least two providers for any platform. So while we might be able to use 1 for a short period of time, the plan should be to find a second provider if at all possible.
23 remaining items
mhdawson commentedon Apr 23, 2024
@jasnell, @mcollina sounds like there is a plan for the surface pros, but what about the benchmark machines? As mentioned I don't think the Build WG was in the loop when they went into Nearform so I don't know what the agreement was in terms of end of life.
jasnell commentedon Apr 23, 2024
It's been so long that I can't remember the details and everything about the benchmark machine was in my old NearForm email inbox that I no longer have access to. I know the machine was on loan only but that's all I remember.
mhdawson commentedon Apr 23, 2024
@bensternthal, @ryanaslett maybe you can help out here in terms of the Intel machines. There seems to be no retained context in terms of the loan from Intel as the people from Neaform who worked with intel to bring the machines in are no longer at Nearform and there was nobody from the build WG who was involved in setting up the loan. I don't think Intel is a Foundation member anymore so don't know who to reach out to.
Could you two handle figuring out what to do with the machines?
mcollina commentedon Apr 23, 2024
I don't think they are on loan, more of a donation. I don't have access to those emails anymore.
bensternthal commentedon Apr 23, 2024
@mhdawson based on reading this thread I would say the intel machines can be donated.
mhdawson commentedon Apr 24, 2024
@efrisby if that is good enough for you then that's good enough for me. Is it possible for you to donate or dispose of them?
efrisby commentedon Apr 25, 2024
@mhdawson @mcollina @jasnell I will take out the intel servers and see what we can do. We have a company that recycle old equipment that I will contact as donating servers like this is harder to find a home for. If you have anyone has any suggestions of anyone local in Ireland let me know as shipping outside of Ireland will be difficult due to the size and weight. Thanks
GlenTiki commentedon Apr 25, 2024
I'd take server hardware if it's going to just be stripped for parts - could use it in a home lab.
efrisby commentedon May 2, 2024
@GlenTiki If you can make it to Tramore tomorrow, I should be onsite, else if you wish we can arrange another day perhaps. I have support person calling to help with taking out some equipment so we can hopefully take these out. It might be late in the day that we will be ready, perhaps around 4.30 however. Thanks.
GlenTiki commentedon May 3, 2024
@efrisby I'm away for a bit and won't be around until after the 11th May - gimme a time that suits after that and I'll be out :)
efrisby commentedon May 3, 2024
@GlenTiki do you still have my email, the nearform one. Just drop me an email when your back and we can arrange a time that suits us. Catch up with you soon :)
jasnell commentedon Jul 26, 2024
FYI... I now have the old surface pro machines originally hosted by nearform in my possession. We haven't been able to get back in contact with the original providers of the machines. If we still need the machines and someone is available/willing to host them for the nodejs CI, then I can send them along. Otherwise, I will hold onto them until the end of the year.
ryanaslett commentedon Apr 1, 2025
Im going to go ahead and close this as I don't believe there is anything left to do here. Feel free to reopen if I missed something.