Repository navigation
Infrastructure for MacOS 13.x #3240
Description
It would be beneficial to begin brainstorming ideas on how we can incorporate MacOS 12.x and 13.x into our current infrastructure. I anticipate that we will need to make the machines accessible for both Intel and ARM devices.
To my recollection, we can utilize some of the existing resources in Orka for the Intel machines with our current configuration. However, to accommodate ARM machines, we may require additional infrastructure resources.
What is your opinion, @nodejs/build? Should we discuss this matter in our next meeting?
Issues related:
Activity
UlisesGascon commentedon Mar 19, 2023
I was checking the availability for Orka resources, and we have some empty spots in the Node
macpro-4, as well we can potentially use port 8825 in all the nodes. In average each VM is configured to use 4 vCPU/CPU and 9.40G RAM.Current Status in Orka
VMs distribution
You can check this in Orka by running
orka vm listNote that
release-macos11-x64-1is not in the inventory as the PR still open #3185The NAT
The NAT is quite tricky to process, but let me summarize it from #3112 (comment)
Nodes Physical resources distribution
You can check this in Orka by running
orka node listFuture distribution (proposal)
We might want to add at least 2 vms per MacOS test version, and 1 for release:
test-macos12-x64-1,test-macos12-x64-2andrelease-macos12-x64-1test-macos13-x64-1,test-macos13-x64-2andrelease-macos13-x64-1Resources Distribution
Nodes Physical resources availability
We will still have room for another 3 vms if we want to squeeze the nodes.
mhdawson commentedon Mar 21, 2023
Getting some new machines into the mix for 20.x makes a lot of sense to me. I think we usualy only release using one version so in terms of release I think should likely assess if we can move up to 12 or 13 for doing releases, but I don't think we'll have resources for both (particularly on the arm side)
UlisesGascon commentedon Mar 22, 2023
Possible Solution for ARM machines in Macstadium
Seems like since Orka 2.0 we can add ARM based nodes to the existing intel Orka Cluster.
Currently, we use Bare Metal solution in Macstadium for ARM:
test-macos11.0-arm64-4: Mac mini G5A (AS/M1/8C/8G/256G/SSD/1G)test-macos11.0-arm64-3: Mac mini G5A (AS/M1/8C/8G/256G/SSD/1G)release-macos11.0-arm64-1: Mac mini G5A (AS/M1/8C/8G/256G/SSD/1G)And I believe that the machines are bigger than our needs, but if we convert this resources into Orka Nodes. For Intel we use Orka C1 Build (24 vCPU + 60 GB) Node, so I assume that for ARM Nodes the resources are similar. Then we can extend the current Orka by converting the existing Bare Metal Resources and add 12.x and 13.x for ARM specific:
Resources Distribution
Obviously we need to check with Macstadium if this is possible. I think that we are in Orka v2.0 or higher based on #3112 (comment)
We can check also with Nearform if we can extend the current nodes in their infra to support 12.x and 13.x
AshCripps commentedon Mar 22, 2023
Our Orka cluster is 2.3.1 according to the support ticket when they upgraded.
mhdawson commentedon Mar 22, 2023
Using orka makes sense to me. Maybe the next stop is to open a ticket with Mac Stadium asking if we can get an ARM based machine added to our free account?
UlisesGascon commentedon Mar 24, 2023
I will start working in
test-macos12-x64-1andtest-macos13-x64-1(REF: #3240 (comment)) so I can update the Ansible scripts and create the base VMs in Orka. I will need some support to add the new machines to Jekins as I am not an adminThanks @AshCripps!
@mhdawson do you want me to contact support?
feat: added macos12-x64-1
UlisesGascon commentedon Mar 26, 2023
Current progress
I created the VM
test-macos12-x64-1from scratch using the official images provided by Orka. This VM is using imagemacos12-x64that was a clean installation with a password change only to match the setup instruction in the private repo. The base imagemacos12-x64can be used to create release or test VMs as there are no ssh key associated. This #3258 will include thetest-macos12-x64-1to the inventory. Next step will the to Ansible the machine and create the tags and setting in Jenkins.Regarding
test-macos13-x64-1I had some issues with the startup of the Operative system using the Orka Ventura official image and also by creating a new VM based onmacos12-x64with manual upgrade to Ventura. I created this ticket asking for support in MacStadium.mhdawson commentedon Mar 27, 2023
sure
UlisesGascon commentedon Mar 28, 2023
They confirmed that the current Intel Nodes (Macpro) are not supported for Ventura in this environment due Apple restrictions, so I am checking if we can use different Node type to run Ventura in Intel Arch with the current Orka settings, as well I am validating with them also if the ARM Nodes can run Ventura and Monterey
UlisesGascon commentedon Mar 29, 2023
If we migrate the existing nodes to mac minis then we can add Ventura VMs in Intel Arch. I am checking the specs with Support so I can update the
Resources Distributiontable. Venture is full supported in Mac Mini ARM too.I asume that we will have some downtime period while migrating between nodes as we will need to redeploy the existing VMs in the correct spots inside the New nodes to keep the same IPs and ports. I assume this will be very similar to #3112. We can plan it ahead and try to minimize the impact.
In the meantime we can continue working on adding Macos 12.x vms to the existing resources, so we are not blocked before the nodes migration.
As a potential date for this migration, I propose April 15-16 as it is weekend and I will be more available so I can deep focus on redeploying the VMs and solve any additional issue that might appear.
I assume that the builds will continue working with the available resources in Nearform, so we might slowdown a bit the pipelines during the migration but we will not block the builds 🤞
UlisesGascon commentedon Mar 31, 2023
TL;DR:
No changes have been made to any machine in the inventory yet. I have made some changes to the distribution of resources and node ideas based on feedback from customer support. The discussion with customer support is now complete, and we can move on to the next steps. Please refer to the "Next Steps" heading at the bottom.
Current Status
Response from Macstadium support:
I closed the support ticket SERVICE-157786. The recommendation from support team will be to discuss with the customer success team (not sure if we have a direct contact already, we can check it?), we can do that as a separate action before we start making changes in the nodes.
Based on the last discussions with the support, most likely we can face this scenario with some minor changes in terms of Orka nodes or VM distribution inside the nodes:
Macstadium resources:
test-macos11.0-arm64-4: Mac mini G5A (AS/M1/8C/8G/256G/SSD/1G). Delete resource (VM will replace it in Orka)test-macos11.0-arm64-3: Mac mini G5A (AS/M1/8C/8G/256G/SSD/1G). Delete resource (VM will replace it in Orka)release-macos11.0-arm64-1: Mac mini G5A (AS/M1/8C/8G/256G/SSD/1G). Delete resource (VM will replace it in Orka)macpro-4(24 vCPU/60G). Remindsmacpro-5(24 vCPU/60G). Remindsmacpro-6(24 vCPU/60G). Remindsmacmini-intel-1(TBC: 8vCPU/16GB/~900G NVMe SSD). New addition for Macos13 Intel VMsmacmini-intel-1(TBC: 8vCPU/16GB/~900G NVMe SSD). New addition for Macos13 Intel VMsmacmini-arm-1(TBC: 8vCPU/16GB/~900G NVMe SSD). New addition for ARM support (11.x,12.x,13.x)macmini-arm-2(TBC: 8vCPU/16GB/~900G NVMe SSD). New addition for ARM support (11.x,12.x,13.x)macmini-arm-3(TBC: 8vCPU/16GB/~900G NVMe SSD). New addition for ARM support (11.x,12.x,13.x)macmini-arm-4(TBC: 8vCPU/16GB/~900G NVMe SSD). New addition for ARM support (11.x,12.x,13.x)macmini-arm-5(TBC: 8vCPU/16GB/~900G NVMe SSD). New addition for ARM support (11.x,12.x,13.x)The NAT
Resources Distribution
Opportunity
The port 8826 won't be used for macpro-*, but it can be used if needed.
The impact
With this setup the impact in machines availability is lower than in previous comments, but still an impact.
Future notes
When we remove the support for macos10.14, we will be able to move the VMs from
macpro-4tomacpro-5andmacpro-6, so we can decomission themacpro-4node.VM Resources
emptyspots that we can use for other purposesNext steps
Important: Do we have any critical deadline? This can impact in Node 20 release?
9 remaining items
UlisesGascon commentedon May 26, 2023
Resources Distribution in Orka
[-]Infrastructure for MacOS 12.x[/-][+]Infrastructure for MacOS 13.x[/+]UlisesGascon commentedon Jan 12, 2024
As discussed in our last meeting (#3604) we will skip macos 12.x and jump to macos 13.x
UlisesGascon commentedon Jan 29, 2024
I created the first VM with Ventura (macos13) image and I am getting issues to login via VNC, so I opened the ticket SERVICE-177494.
UlisesGascon commentedon Feb 21, 2024
The login problem was solved already
targos commentedon Apr 16, 2024
What's the status? Can I do anything to help move this forward?
targos commentedon Apr 16, 2024
I'm asking because
osx11-x64is a problematic bottleneck with only 2 machines and often space issues (currently 1 machine is offline)UlisesGascon commentedon Apr 18, 2024
[-]Infrastructure for MacOS 13.x[/-][+]Infrastructure for Orka (2024 and beyond)[/+][-]Infrastructure for Orka (2024 and beyond)[/-][+]Infrastructure for MacOS 13.x[/+]UlisesGascon commentedon Apr 19, 2024
I close this is favor of #3686 that centralize the Orka changes including MacOS 13 addition in Orka. Feel free to reopen this issue if needed.