Repository navigation
Multiple infra failures on arm machines #2205
Description
https://ci.nodejs.org/job/node-test-commit-arm/
https://ci.nodejs.org/job/node-test-commit-arm/29801/
https://ci.nodejs.org/job/node-test-commit-arm/29797/
https://ci.nodejs.org/job/node-test-commit-arm/29796/
All of them failing due to socket hangups
18:35:28 Error: socket hang up
18:35:28 at connResetException (internal/errors.js:613:14)
18:35:28 at TLSSocket.socketCloseListener (_http_client.js:406:25)
18:35:28 at TLSSocket.emit (events.js:328:22)
18:35:28 at net.js:672:12
18:35:28 at TCP.done (_tls_wrap.js:563:7) {
18:35:28 code: 'ECONNRESET'
18:35:28 }
Same errors as #2178
Activity
MylesBorins commentedon Mar 5, 2020
Only happening on these three hosts, both from packetnet
https://ci.nodejs.org/computer/test-packetnet-centos7-arm64-1/builds
https://ci.nodejs.org/computer/test-packetnet-centos7-arm64-2/builds
https://ci.nodejs.org/computer/test-packetnet-ubuntu1604-arm64-1/builds
It concerns me that it is 3 different machines all having the same error... two different operating systems. They do share the same architecture as well as provider... so perhaps they are having network issues?
rvagg commentedon Mar 5, 2020
It's tricky trying to interpret jenkins errors like this, they often hide the real issue and all you see is something that makes it look like something inside jenkins is at fault.
I'm tempted to blame the JRE on this since one of the common themes is arm64 but there's some variation in the version installed (1.8.0 222 vs 1.8.0 242).
On all 4 packet.net arm64 machines I've done a system upgrade, upgraded slave.jar (it's nearly a year old on all of them), cleaned out the workspaces and rebooted. Let's leave this issue open as a reminder to monitor for a few days to see if it's fixed.
jkleinsc commentedon Mar 6, 2020
I'd like to try getting GitHub actions running on arm64. I have an arm64 server on packet.net for Electron that I can use for testing a POC.
jkleinsc commentedon Mar 6, 2020
POC here: https://github.com/jkleinsc/node/runs/490669108
rvagg commentedon Mar 9, 2020
We haven't done any self-hosted github acitons builders yet @jkleinsc, but it'd be great to start exploring that. If you think this is something we should consider adopting here then maybe we can wire it up on our packet.net infra; as long as runs aren't going to conflict with our existing CI runs or we could entirely replace them. We currently test on Ubuntu 16.04 and CentOS 7.
jkleinsc commentedon Mar 24, 2020
@rvagg I think moving over to GitHub actions could be a great way to reduce the maintenance load on the build WG as it is infrastructure you do not have to maintain. The self hosted runners would have to still be maintained, but I'm pretty sure the agent that runs on the self host runners is self updating. The only caveat that might be an issue is that GitHub doesn't recommend using self hosted runners on public repos: https://help.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners#self-hosted-runner-security-with-public-repositories. I do wonder though if docker containers were used if it would mitigate the security issue.
mmarchini commentedon Mar 24, 2020
richardlau commentedon Mar 24, 2020
mmarchini commentedon Mar 24, 2020
Moved the discussion on Action runners to a separate issue and marked the comments here as "off-topic" to avoid diverging from the issue at hand.