Skip to content

Improve concurrency of build processes on AIX machines. #4226

Open
Open
Improve concurrency of build processes on AIX machines.#4226
Labels
platform:aix

Description

@sxa
sxa
opened on Jan 29, 2026

Many of our current AIX machines have significant processing power and RAM (Most are 4-core with SMT8) and 32GiB if RAM. We are currently throttling some of them to -j8 or lower which means the machines are likely to be underutilised relative to their ability:

build/ansible/inventory.yml

Line 122 in 7d8af5b

server_jobs: 8

This issue will cover experimenting to see if we can increase this, as duering some recent testing I have been running directly on the machine with values of -j16 without issues (Noting that it's also possible that running through jenkins has different effects...)

The limitations were initially put in due to the analysis in #2566 and then added to our automation in #2574 - it's not clear which versions of Node were affected but I will also try node.js 22 to cover both ends of what we are supporting. I will also note that the specific machine which was listed as failing with anything greater than 3 is currently offline.

Activity

  1. sxa
    transferred this issue fromnodejs/nodeon Jan 29, 2026

  2. richardlau commented on Jan 29, 2026

    @richardlau
    richardlau
    on Jan 29, 2026
    Member

    Just to note, we need to do the testing for the worst case scenario -- no populated ccache (or near to 100% miss rate). Problems usually arose with V8-canary or V8 update PRs.

  3. richardlau
    added
    platform:aix
    on Jan 29, 2026

  4. sxa commented on Jan 29, 2026

    @sxa
    sxa
    on Jan 29, 2026
    MemberAuthor

    Just to note, we need to do the testing for the worst case scenario -- no populated ccache (or near to 100% miss rate). Problems usually arose with V8-canary or V8 update PRs.

    Yep I'm doing all of the testing with ccache disabled for now in order to stretch the machine resources. but initially only with the node builds.

  5. sxa commented on Jan 29, 2026

    @sxa
    sxa
    on Jan 29, 2026
    MemberAuthor

    Unfortunately I lost my earlier table as firefox crashed on me ...

    At present osuosl-be-4 and ibm-aix73-3 are giving me linker problems (stopping with return code 8) and it's not clear why. I've got one ccached build through on osuosl-be-4 and still diagnosing why it doesn't always work.

    These times are from a clean build without a populated ccache:

    System Processors default Node26-j16 Node26-j20 Node20-j20 Node25-j20
    ibm-be-2 4*P9,SMT8 99m (-j8) 69m/75m 69m 36m 67m
    osuosl-be-4 4*P8,SMT8 151m (-j6) 140m/138m 159m 56m Stuck? [*]
    ibm-aix73-3 (8G) 4*P9,SMT8 174m (-j5) 58m (-j8) 174m (-j4)
    ibm-aix73-3 (16G) 4*P8,SMT8 88m (-j12) 87m OOM 45m

    [*] - It got a bit stuck while at the stage where it was creating the libnode.a archive. Probably best not to push it this high. It hadn't ran out of memory but the make process stuck

    Note: I also tried -j12 on ibm-be-2 and it took 77 minutes
    Also tried another run with -j20 on main and it stalled or otherwise caused a disconnect on my ssh session after 33 minutes, so with that and the apparent lack of improvement over -j16 shown in the table above when it does complete I suspect that's the limit of how far we should push this.

      g++-12 -o /home/iojs/sxa/node/out/Release/obj.target/v8_base_without_compiler/deps/v8/src/handles/global-handles.o ../deps/v8/src/handles/global-handles.cc '-D_GLIBCXX_USE_CXX11_ABI=1' '-D_FILE_OFFSET_BITS=64' '-DNODE_OPENSSL_CONF_NAME=nodejs_conf' '-DICU_NO_USER_DATA_OVERRIDE' '-DV8_GYP_BUILD' '-DV8_TYPED_ARRAY_MAX_SIZE_IN_HEAP=64' '-D__STDC_FORMAT_MACROS' '-DOPENSSL_NO_PINSHARED' '-DOPENSSL_THREADS' '-DV8_TARGET_ARCH_PPC64' '-D_LINUX_SOURCE_COMPAT=1' '-D_ALL_SOURCE=1' '-DV8_EMBEDDER_STRING="-node.11"' '-DENABLE_DISASSEMBLER' '-DV8_PROMISE_INTERNAL_FIELD_COUNT=1' '-DOBJECT_PRINT' '-DV8_INTL_SUPPORT' '-DV8_ATOMIC_OBJECT_FIELD_WRITES' '-DV8_ENABLE_LAZY_SOURCE_POSITIONS' '-DV8_USE_SIPHASH' '-DNDEBUG' '-DV8_WIN64_UNWINDING_INFO' '-DV8_ENABLE_REGEXP_INTERPRETER_THREADED_DISPATCH' '-DV8_USE_ZLIB' '-DV8_ENABLE_LEAPTIERING' '-DV8_ENABLE_SPARKPLUG' '-DV8_ENABLE_TURBOFAN' '-DV8_ENABLE_WEBASSEMBLY' '-DV8_ENABLE_JAVASCRIPT_PROMISE_HOOKS' '-DV8_ENABLE_CONTINUATION_PRESERVED_EMBEDDER_DATA' '-DV8_ALLOCATION_FOLDING' '-DV8_ALLOCATION_SITE_TRACKING' '-DV8_ADVANCED_BIGINT_ALGORITHMS' '-DICU_UTIL_DATA_IMPL=ICU_UTIL_DATA_STATIC' '-DUCONFIG_NO_SERVICE=1' '-DU_ENABLE_DYLOAD=0' '-DU_STATIC_IMPLEMENTATION=1' '-DU_HAVE_STD_STRING=1' '-DUCONFIG_NO_BREAK_ITERATION=0' '-DTOOLCHAIN_MISS_ASM_HWCAP_H' '-DHWY_BROKEN_EMU128=0' -I../deps/v8 -I../deps/v8/include -I/home/iojs/sxa/node/out/Release/obj/gen/inspector-generated-output-root -I../deps/v8/third_party/inspector_protocol -I/home/iojs/sxa/node/out/Release/obj/gen -I/home/iojs/sxa/node/out/Release/obj/gen/inspector-generated-output-root/include -I/home/iojs/sxa/node/out/Release/obj/gen/generate-bytecode-output-root -I../deps/icu-small/source/i18n -I../deps/icu-small/source/common -I../deps/v8/third_party/zlib -I../deps/v8/third_party/zlib/google -I../deps/v8/third_party/fp16/src/include -I../deps/v8/third_party/highway/src -I../deps/v8/third_party/simdutf -I../deps/v8/third_party/abseil-cpp  -pthread -Wno-unused-parameter -maix64 -Wno-attributes -Wno-strict-overflow -Wno-return-type -Wno-int-in-bool-context -Wno-deprecated -Wno-stringop-overflow -Wno-stringop-overread -Wno-restrict -Wno-array-bounds -Wno-nonnull -Wno-dangling-pointer -flax-vector-conversions -ffp-contract=off -mcpu=power5+ -mfprnd -mno-popcntb -maix64 -fdollars-in-identifiers -fno-extern-tls-init -O3 -fno-omit-frame-pointer -fdata-sections -ffunction-sections -O3 -fno-rtti -fno-exceptions -fno-strict-aliasing -std=gnu++20 -Wno-invalid-offsetof -MMD -MF /home/iojs/sxa/node/out/Release/.deps//home/iojs/sxa/node/out/Release/obj.target/v8_base_without_compiler/deps/v8/src/handles/global-handles.o.d.raw   -c
Timeout, server 169.54.113.165 not responding.
sxa:nodebuild$

    The AIX 7.3 machine with 8GiB of RAM struggled with -j8 and caused processes to be killed (actualyl most of the subjobs disappeared and then the underlying make froze, which was a bit odd...) -j5 seems like it might be safe though so that's what I've used for the times on that machine in the table above.

  6. sxa commented on Jan 30, 2026

    @sxa
    sxa
    on Jan 30, 2026
    MemberAuthor

    Just to note, we need to do the testing for the worst case scenario -- no populated ccache (or near to 100% miss rate). Problems usually arose with V8-canary or V8 update PRs.

    We don't have any separate V8 build jobs in our CI for AIX do we? Confirmed answer as "No" so that should not affect what we set these values to on the machines.

  7. sxa commented on Sep 28, 2026

    @sxa
    sxa
    on Sep 28, 2026
    MemberAuthor

    Noting also that in #4230 (comment) we hit problems with some tests on AIX machines with "only" 8GB of RAM so that should be taken into account when defining the specifications.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    platform:aix

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions