Repository navigation
Infrastructure for Orka (2024 and beyond) #3686
Description
I plan to work on it during the weekend, so I can provide a good overview on the next build meeting on Tuesday.
Current tasks on MacOS infra
- Cleaned up all the macos machines that are available via SSH
- Open discussion to apply upgrade in the Firewall: Schedule Maintenance for Orka CISCO Firewall #3642
- Review Orka state: Infrastructure for Orka (2024 and beyond) #3686 (comment)
- Plan the final allocation for resources in Orka (see: Infrastructure for Orka (2024 and beyond) #3686 (comment))
- MacOS 13
- Nearform MacOS Machines (see: Replacement for NearForm Mac OS machines #3638)
- Provide access for @ryanaslett to the infra-mac related resources: LFIT access to secrets #3658
- Prepare and deliver and onboarding session for @ryanaslett to the Orka infrastructure
- Add new VMs for MacOS 13 Intel (@ryanaslett)
Blocked until ARM nodes are provided
- Confirm org decision regarding new ARM nodes (discussion ongoing in the mailing list)
- Add new VMs for MacOS 13 ARM
- Add new VMs for MacOS 11 ARM
Activity
UlisesGascon commentedon Apr 19, 2024
Current Orka state
updated on April 19, 2024
UlisesGascon commentedon Apr 19, 2024
Next Orka state
updated on April 22, 2024
Intel Nodes
ARM Nodes
We assume that ARM Nodes can handle only 2 VMs and not +4 as Intel in the past due license limitations. This needs to be confirmed with support AFAIK?
How Nearform machines are "relocated"?
release-nearform-macos11.0-arm64-1->release-orka-macos11-arm64-1test-nearform-macos11.0-arm64-1->test-orka-macos11-arm64-1targos commentedon Apr 22, 2024
I don't think it's necessary to have two identical release machines.
targos commentedon Apr 22, 2024
Are these typos?
UlisesGascon commentedon Apr 22, 2024
Great feedback @targos! I updated the tables
We have space for redundancy, but let's remove them for now.
I made a better reference for the "relocated" machines
targos commentedon May 2, 2024
Actually, I think we should have one x64 and two arm64 machines, because there are two jobs that run on macos-arm64 during a release (osx11-release-pkg and osx11-arm64-release-tar).
51 remaining items
ryanaslett commentedon Jan 30, 2025
Ah, yes. I see that. They were not using that on the tests I ran, so not sure what's different between the 'real jobs' and my smoke test setup.
I'm looking into why its not respecting the ccache env vars. I think I may need to use a .ccache config instead of env vars.
Also, we're only getting half the instances for our arm builds as they were set to 6 cpu's and those nodes only had 10 available. I've reset those to 5 but the queue will have to clear out before it shuts down the 6 cpu vm's and re-creates 2 5cpu vms.
mhdawson commentedon Jan 30, 2025
I think we should target moving to sequoia, and since we have depended on the flag to set the lowest target OS to run on, versus building on the lowest target OS for macOS successfully over the year I don't think we expect any surprises when we make the move.
In the interests of unblocking the CIs, I'm also ok running on osx13 as you have it set up until new images can be built and deployed.
targos commentedon Jan 31, 2025
The builds generally look good, but yeah they definitely need ccache. At least for x64, which takes around 3 hours (vs 20 minutes for arm64!)
ryanaslett commentedon Jan 31, 2025
I had set up the x64 machines to use the ccache earlier today, and when I checked the shared cache directory, it had created all of the hash prefix subdirs, so I assumed it was working.
But I just dug deeper with the ccache debug logs and turns out it created those dirs without group perms, but the processes can only write as group members.
I just did a chmod g+w on the whole cache dir and now all the files are starting to cache. Hopefully this speeds things up a bit.
There does still seem to be something awry with the x64 machines connecting to the shared drive, as doing simple things like an ls take 22 seconds (wheras on the arm machines it acts like a normal HD)
build: test macos-13 on GitHub actions
ryanaslett commentedon Feb 5, 2025
Status update on MacOS testing/Release infra.
Our Orka3 cluster is now fully responsible for the x86/arm jobs and is running images that are macos13 with Xcode16 on them.
The testing images are configured to use a permanent shared storage for ccache such that new ephemeral instances will all see the same warmed cache.
There seems to be an issue with differences in how each architecture accesses this storage: as such, the x86 machines are running builds very slowly, averaging about 2 hours for a test, whereas the ARM machines are done in about 15min.
I have an open ticket with Macstadium to investigate this to see if there's another way to establish the mount to that shared storage on the x86 machines that is more performant.
The infrastructure can support four x86 instances, and four arm instances. I have configured the test infra to only use a maximum of 3 so that a burst of PR testing doesnt block releases.
Next steps:
UlisesGascon commentedon Feb 5, 2025
@ryanaslett there are some PRs that can save u time with the decommission process:
ryanaslett commentedon Feb 5, 2025
Thanks @UlisesGascon . I had just done those locally, came here to post and saw these PR's :/ . But, still saved time on review waits.
I also went ahead and removed any osx 'testing' related jobs we had created during this process, as well as converted any ci testing job that was using the osx10.15 or osx11 labels to use the osx13-{arch} labels.
ryanaslett commentedon Nov 18, 2025
Now that #4163 is merged in, Im comfortable closing this out. The whole architecture is in place and refined. There is one outstanding upgrade coming, #4180 But that can be tracked on its own when Macstadium gives us the additional machine.
Fantastic work on setting up the foundation and getting the majority of the problemspace contained @UlisesGascon !