macos11.0 release hosts are both offline #3179

Activity

1. targos

   added

   platform:macos

   incident

   ci-release

   on Feb 5, 2023
2. richardlau commented on Feb 6, 2023

   

   richardlau

   on Feb 6, 2023

   Member

   More actions

   I've sent an email to our NearForm contacts regarding the NearForm machine. (FYI @efrisby).
3. mhdawson commented on Feb 6, 2023

   

   mhdawson

   on Feb 6, 2023

   Member

   More actions

   @UlisesGascon is

   release-macstadium-macos11.0-arm64-1 is offline and #3084

   Something you can help with?
4. UlisesGascon commented on Feb 7, 2023

   

   UlisesGascon

   on Feb 7, 2023

   Member

   More actions

   I can help and see if I can restore the machine, but I will need some support to re-ansible as we did for release-orka-10.15-x64-1 in the past
5. targos commented on Feb 7, 2023

   

   targos

   on Feb 7, 2023

   MemberAuthor

   More actions

   Before re-ansibling, maybe there is a secondary way to access the machine from Macstadium's infra/network?

   👍React with 👍1Reacted by Ulises Gascón
6. UlisesGascon commented on Feb 7, 2023

   

   UlisesGascon

   on Feb 7, 2023

   Member

   More actions

   I will check the machine and share the status before I do a destructive or non-reversible action 👍
7. UlisesGascon commented on Feb 7, 2023

   

   UlisesGascon

   on Feb 7, 2023

   Member

   More actions

   Here is an update on what I have tried. Thanks to @targos for the great support:

   • I reviewed the internal documents about MacStadium but did not find a new way to access the machines.
   • The MacStadium web UI does not offer any option to access the machine, but the stats confirmed that they are connected and running.
   • I hard rebooted all the bare metal machines (including the test ones, macstadium macos11 immediate closed connection  #3084), but even after the machines were back online, none of them had improved connectivity.
   • I tried to VNC connect with them following an official tutorial (from 10 years ago) but had no luck, possibly because the username and password were not the same as in the internal documentation or the VNC Server is disabled (as expected).
   • I also reviewed the Bare Metal official documentation but had no luck.
   • I attempted to communicate to the machines from the Orka VPN but was unsuccessful.

   I see two options: we can raise a ticket and ask for support, pointing to this issue, or we can add one or several additional release VMs in the Orka platform using the backups I made. We would only need to register the new machines in the inventory and re-Ansible them.
8. richardlau commented on Feb 7, 2023

   

   richardlau

   on Feb 7, 2023

   Member

   More actions

   I see two options: we can raise a ticket and ask for support

   Am I misremembering or did @AshCripps try that before (or perhaps it was a different VM)?
9. UlisesGascon commented on Feb 7, 2023

   

   UlisesGascon

   on Feb 7, 2023

   Member

   More actions

   In the meantime I will add a new macos11 in Orka for releases
10. UlisesGascon

    added a commit that references this issue on Feb 7, 2023

    feat: added machine release-orka-macos11-x64-1

    ...

    5c3c4e4
11. AshCripps commented on Feb 7, 2023

    

    AshCripps

    on Feb 7, 2023

    Member

    More actions

    I see two options: we can raise a ticket and ask for support

    Am I misremembering or did @AshCripps try that before (or perhaps it was a different VM)?

    Yes it was a very similar issue and took some effort to get them to acknowledge it - should still be in the system somewhere of what I asked for.
12. richardlau commented on Feb 7, 2023

    

    richardlau

    on Feb 7, 2023

    Member

    More actions

    In the meantime I will add a new macos11 in Orka for releases

    Ah, forgot that these are arm64 machines/VMs and therefore not in Orka.
13. UlisesGascon commented on Feb 7, 2023

    

    UlisesGascon

    on Feb 7, 2023

    Last edited by UlisesGascon

    Member

    More actions

    🎉 Good News! I created a new bare metal instance, and the machine is up and running.

    So... here is the plan:

    • [ ] I will create a PR to replace the IP for release-macstadium-macos11.0-arm64-1 in the inventory, and I will ask you to reansible the machine for releases and to include it in the secret repo. Current status: Chore: IP replacement for macstadium release arm machine #3187
    • [ ] I will create 2 new machines, reansible them to replace the test-macstadium--arm64-. Current status: IP replacement for test-macstadium-macos11.0-arm64-3 #3189 and IP replacement for test-macstadium-macos11.0-arm64-4 #3188
    • [ ] Once we are happy with the new machines I will remove the old ones from macstadium

    So... here is the NEW plan

    • I will ask support to restore the credentials for the arm machines (ticket)
    • Check that SSH is working in all the machines with the new credentials
    • Add build SSH key to test-macstadium-macos11.0-arm64-3
    • Re-ansible test-macstadium-macos11.0-arm64-3
    • Re-ansible test-macstadium-macos11.0-arm64-4
    • I will ask release team to reansible the release one (release-macstadium-macos11.0-arm64-1)
14. UlisesGascon

    added 2 commits that reference this issue on Feb 7, 2023

    chore: ip replacement for macstadium release arm machine

    ...

    bf54d26

    chore: ip replacement for test-macstadium-macos11.0-arm64-4

    ...

    e2f11b2
15. UlisesGascon

    self-assigned this

    on Feb 8, 2023
16. UlisesGascon

    added a commit that references this issue on Feb 8, 2023

    chore: ip replacement for test-macstadium-macos11.0-arm64-3

    ...

    672fcc5
17. richardlau commented on Feb 8, 2023

    

    richardlau

    on Feb 8, 2023

    Member

    More actions

    FWIW the Nearform machine is now back online. (Thank you @efrisby.)
18. richardlau commented on Feb 9, 2023

    

    richardlau

    on Feb 9, 2023

    Member

    More actions

    I've reansibled the release and two test macos11 arm64 macstadium machines.
    Currently running a release rebuild on today's nightly (osx11-tar happened to fail) with just osx11-tar and osx11-pkg: https://ci-release.nodejs.org/job/iojs+release/9138/

    👍React with 👍2Reacted by Ulises Gascón and Michaël Zasso
19. richardlau commented on Feb 9, 2023

    

    richardlau

    on Feb 9, 2023

    Member

    More actions

    Currently running a release rebuild on today's nightly (osx11-tar happened to fail) with just osx11-tar and osx11-pkg: https://ci-release.nodejs.org/job/iojs+release/9138/

    Build passed. Release machine looks good.

    👍React with 👍1Reacted by Ulises Gascón
20. UlisesGascon commented on Feb 10, 2023

    

    UlisesGascon

    on Feb 10, 2023

    Member

    More actions

    Thanks for the conformation @richardlau. I will close the issue then 😉
21. UlisesGascon

    closed this as completedon Feb 10, 2023
22. targos

    added a commit that references this issue on Apr 12, 2023

    ansible: add Orka release machine macos11 (#3185)

    ...

    f45d157