Proposal: run SmartOS and AIX in node-daily-master instead of node-test-pull-request

I pulled the latest 450 node-test-pull-request runs out of the Jenkins API (167 PRs, 2026-08-27 to 2026-09-05) to see where PR CI time goes. Most of it is waiting for an executor on two platforms rather than running tests, so i'd like to propose running SmartOS and AIX from node-daily-master only and dropping them from the PR job.

job	runs	median	p90	of which waiting for an executor (median / p90)	slowest configuration's own run (median)
node-test-commit-smartos	362	246 min	453 min	~215 / ~409 min	46 min (smartos23-x64)
node-test-commit-aix	407	152 min	334 min	~87 / ~261 min	69 min (aix73-power9)
node-test-commit-linux (10 configs)	421	94 min	305 min	~22 / ~194 min	62 min (rhel8-x64)
node-test-commit-osx	433	91 min	188 min	~5 / ~105 min	65 min (macos15-x64)
node-test-commit-linux-containered	435	66 min	154 min	~17 / ~91 min	43 min
node-test-commit-windows-fanned	428	58 min	192 min
node-test-commit-arm	431	52 min	112 min	~0 / ~12 min	45 min (rhel8-arm64)
node-test-commit-plinux	442	35 min	69 min	~0 / ~22 min	33 min
node-test-commit-linuxone	447	30 min	51 min	~0 / ~6 min	30 min
node-test-linter	445	15 min	20 min
node-test-pull-request, end to end	345	238 min	507 min

"Waiting for an executor" is the matrix job's duration minus its slowest configuration's execution time (the API doesn't expose queue time for matrix children directly); aborted runs are excluded. Over the same window 13% of PR runs finished SUCCESS and another 18% UNSTABLE, the average PR needed 2.7 runs, and when exactly one child job was non-green in a run it was linuxone 34 times (nearly all yellow from marked-flaky tests), aix 19, osx 17, linux 17, windows 9, smartos 4. Raw per-configuration numbers (exec median/p90, failure rate) are available if useful.

What the change would mean in practice: SmartOS and AIX stay Tier 2 exactly as BUILDING.md defines it (full test coverage maintained, failures block releases), keep running in node-daily-master and release CI, and failures there get filed the way other daily-only jobs are handled. PRs stop holding a jenkins-workspace executor for 2-3 h while a single smartos23 executor works through the queue, which by these numbers moves the median PR run from ~4 h to ~1.5 h (osx/linux become the long pole) with no new hardware, and takes some pressure off the executor deadlocks discussed in #4435. ppc64le and s390x are also Tier 2 but have no queueing to speak of, so i'd leave them in the PR job.

Mechanically i think this is either a parameter on node-test-commit that node-test-pull-request sets to skip those two phases, or a second multijob without them for the PR trigger; whichever the WG prefers. Two questions before this goes on an agenda: is there history on why these two gate PRs specifically rather than the daily, and would anyone want a lighter variant first (e.g. keep AIX and move only SmartOS, which is the larger share of the wait)?

Refs: #4435
Refs: nodejs/node#65826 (separate, cuts console log volume from the same jobs)

While I'm on vacation at the moment I think AIX will have primarily been a problem caused by a test which was consistently failing which will have resulted in a lot of rei-runs. Having said that I'm a little surprised the median value there is quite so high. Before I left I thought tthe queues seemed to have returned to about normal after a week of chaos. There's a pinned post in this respiratory about SmartOS being down to one executor and I have on a couple of occasions recently disabled it by deactivating the job in the PR CIs

At this point I would agree on SmartOS but will the AIX being a release platform I would definitely prefer to leave it active for now. Typically two executors for each platform is adequate although are in the process of configuring an extra AIX7.3 one at the moment and should be able to add another 7 2 if required (although I think we might have 3 there already)

Having said that I was talking about whether we could have some platforms only run on the daily builds for other reasons so I'm in favour of having that as an option :-)

On another note do you have the script you used to generate the data? I was going to look at doing something similar to keep an eye on things when I returned.

(Apologies for any typos etc. in here - writing this on my phone with almost no connectivity just now)

@sxa

AIX being a release platform I would definitely prefer to leave it active for now. Typically two executors for each platform is adequate although are in the process of configuring an extra AIX7.3 one at the moment and should be able to add another 7 2 if required (although I think we might have 3 there already)

+1

We only have 2 machines for each AIX axis, (but have provisioned more machines, see notes):

AIX 7.2 Power 8 machines (used for Node.js < 26 since Node.js >= 26 requires Power 9)

We should be seeing less PRs for Node.js 24 and 22 compared to main/v26.x, so I don't think we necessarily need more of these.

AIX 7.2 Power 9 machines

We have provisioned (but not enabled for the regular CI) two more AIX 7.2 Power 9 machines:

test-ibm-aix72-ppc64_be-3
test-ibm-aix72-ppc64_be-5 (this one is marked SXA TEST SYSTEM FOR ANSIBLE - NOT FOR PRODUCTION USE)

I can try labelling test-ibm-aix72-ppc64_be-3 with aix72-power9 on Monday (since I'd want to monitor so I can remove it if it isn't ready) to see if it's okay to be in the pool of AIX 7.2 Power 9 machines for the CI.

AIX 7.3

test-ibm-aix73-ppc64_be-4 exists but is marked Labels disabled by sxa : POtential failures on node 24: https://ci.nodejs.org/job/node-test-commit-aix/nodes=aix73-power9/62792/ (and of course that CI run is too old and no longer exists). I'll see if this can be added to the pool.

As @sxa mentioned we are also in the process of getting another AIX 7.3 machine added.

@richardlau

@richardlau I was just logging in to say that aix73-4 and aix72-5 (the one named as mine) are probably worth retesting as they can maybe be enabled. 73-4 is smaller than the others IIRC but with testing. Just check that disk space on them is ok before enabling again.

If you need space on aix72-5 find my node checkout and clear it out 😃

Another proposal (not sure if it's actually doable, but here it is anyway): keep AIX and smartos on node-test-commit only for non-main PRs and daily runs. That way we can still surface bugs affecting those platforms when working on releases, hopefully the daily builds should help with bisecting (or even catch problems even before we backport, but that only works if someone is monitoring the results), and it should solve the availability problem (because it would run only a handful of jobs per day).

There are a number of proposals that would aleviate the executor long wait times in some capacity.

#4458 - SmartOS - seems like the only objection is the inclusion of AIX in there, not to SmartOS, this would be immediately noticeable (https://ci.nodejs.org/label/smartos23-x64/load-statistics)
#4457 (comment) - more AIX executors are on the way
#4447 - all matrix jobs - Reuse successful configurations from the matrix being resumed - we could avoid running the full matrix when just one of its configurations flakes

@panva

@panva On the matrix thing that's potentially non trivial. The resume builds on the higher level jobs is done using a now unsupported Jenkins plugin so it may not be ready to make it correctly talk to the "standard" platform matrix jobs to rerun only the failing ones

@sxa

Correction: the numbers above cover the latest 450 PR runs, which is 9 days (Aug 27 - Sep 5), not the month i wrote. i've fixed the description.

That window caught the rough AIX stretch @sxa mentioned. Same measurement since, counting only the sub-builds each PR run started (median minutes, waiting for an executor in parens):

job	Aug 27 - Sep 5	Sep 10 - Oct 1
node-test-commit-smartos	244 (204)	133 (93)
node-test-commit-aix	191 (126)	83 (0)
node-test-pull-request	238	172

@sxa here's the script:

node-ci-timing.py

#!/usr/bin/env python3
"""Where does node-test-pull-request time go?

Pulls recent build history for node-test-pull-request, node-test-commit and
every job below it from the ci.nodejs.org JSON API, then prints duration,
executor wait and failure statistics per job.

Credentials: JENKINS_USER and JENKINS_TOKEN, or the "username" and
"jenkins_token" keys of the ncurc file that node-core-utils uses.

  ./node-ci-timing.py                          # last 30 days, Markdown tables
  ./node-ci-timing.py --days 7 --per-config
  ./node-ci-timing.py --save raw.json          # keep the API responses
  ./node-ci-timing.py --load raw.json          # re-analyse without fetching
"""

import argparse
import base64
import collections
import datetime
import json
import os
import re
import statistics
import sys
import time
import urllib.error
import urllib.request

BASE = 'https://ci.nodejs.org/'
TOP = 'node-test-pull-request'
COMMIT = 'node-test-commit'
BUILD = ('number,timestamp,duration,result,building,'
         'subBuilds[jobName,buildNumber,duration,result]')
RUNS = 'runs[number,timestamp,duration,result,builtOn,url]'


def credentials():
    user, token = os.environ.get('JENKINS_USER'), os.environ.get('JENKINS_TOKEN')
    if user and token:
        return user, token
    config = os.environ.get('XDG_CONFIG_HOME', os.path.expanduser('~/.config'))
    for path in (os.path.join(config, 'ncurc'), os.path.expanduser('~/.ncurc')):
        try:
            with open(path) as f:
                rc = json.load(f)
        except OSError:
            continue
        if rc.get('username') and rc.get('jenkins_token'):
            return rc['username'], rc['jenkins_token']
    sys.exit('Set JENKINS_USER and JENKINS_TOKEN, or add jenkins_token to ncurc.')


def fetcher(user, token):
    auth = 'Basic ' + base64.b64encode(f'{user}:{token}'.encode()).decode()

    def get(job, tree, count, tries=6):
        url = f'{BASE}job/{job}/api/json?tree=allBuilds[{tree}]{{0,{count}}}'
        print(f'fetching {job}', file=sys.stderr)
        for attempt in range(tries):
            try:
                req = urllib.request.Request(url, headers={'Authorization': auth})
                with urllib.request.urlopen(req, timeout=120) as res:
                    return json.load(res)['allBuilds']
            except urllib.error.HTTPError as err:
                if err.code in (401, 403, 404):
                    sys.exit(f'{job}: HTTP {err.code}')
                error = err
            except (urllib.error.URLError, TimeoutError) as err:
                error = err
            print(f'  retry {attempt + 1}: {error}', file=sys.stderr)
            time.sleep(4 + 3 * attempt)
        print(f'  giving up on {job}', file=sys.stderr)
        return []

    return get


def fetch(count):
    get = fetcher(*credentials())
    data = {TOP: get(TOP, f'{BUILD},actions[parameters[name,value]]', count)}
    # Walk down through the multijobs (windows-fanned, arm-fanned, ...).
    pending = [TOP]
    while pending:
        linked = collections.defaultdict(set)
        for job in pending:
            for build in data[job]:
                for sub in build.get('subBuilds') or []:
                    linked[sub['jobName']].add(sub['buildNumber'])
        pending = sorted(set(linked) - set(data))
        for job in pending:
            tree = BUILD if job == COMMIT else f'{BUILD},{RUNS}'
            data[job] = get(job, tree, count)
            numbers = [build['number'] for build in data[job]]
            # Other pipelines use these jobs too, so the oldest build that a
            # PR run links to can be further back than `count` builds.
            depth = max(numbers) - min(linked[job]) + 1 if numbers else 0
            if depth > len(numbers) == count:
                data[job] = get(job, tree, min(depth, 5 * count))
    return data


def minutes(ms):
    return ms / 60000


def summary(values):
    values = sorted(values)
    if not values:
        return None
    return {
        'n': len(values),
        'median': round(statistics.median(values)),
        'p90': round(values[min(len(values) - 1, int(0.9 * len(values)))]),
        'max': round(values[-1]),
    }


def config_name(url):
    match = re.search(r'/([^/]+)/\d+/$', url)
    return re.sub(r'.*nodes=', '', match.group(1)) if match else url


def own_runs(build):
    # runs[] also lists older builds of configurations this build skipped.
    return [run for run in build.get('runs') or []
            if run['number'] == build['number'] and run['result'] != 'ABORTED']


def table(header, rows):
    print('| ' + ' | '.join(header) + ' |')
    print('|' + '|'.join(' --- ' if i == 0 else ' ---: '
                         for i in range(len(header))) + '|')
    for row in rows:
        print('| ' + ' | '.join(str(cell) for cell in row) + ' |')
    print()


def report(data, days, per_config):
    newest = max(build['timestamp'] for build in data[TOP])
    wanted = newest - days * 86400 * 1000
    cutoff = max(wanted, min(build['timestamp'] for build in data[TOP]))

    # Only count builds that a PR run in the window links to, directly or not.
    linked = collections.defaultdict(set)
    linked[TOP] = {build['number'] for build in data[TOP]
                   if build['timestamp'] >= cutoff}
    pending = [TOP]
    while pending:
        found = set()
        for job in pending:
            for build in data.get(job) or []:
                if build['number'] not in linked[job]:
                    continue
                for sub in build.get('subBuilds') or []:
                    linked[sub['jobName']].add(sub['buildNumber'])
                    found.add(sub['jobName'])
        pending = sorted(found)

    def finished(job, aborted=False):
        return [build for build in data.get(job) or []
                if build['number'] in linked[job] and not build.get('building')
                and (aborted or build['result'] != 'ABORTED')]

    def day(ms):
        return datetime.datetime.fromtimestamp(ms / 1000, datetime.UTC).date()

    pr_runs = finished(TOP, aborted=True)
    per_pr = collections.Counter()
    for build in pr_runs:
        params = {param['name']: param['value']
                  for action in build.get('actions') or []
                  for param in action.get('parameters') or []}
        per_pr[params.get('PR_ID')] += 1
    print(f'## {TOP}, {day(cutoff)} to {day(newest)} '
          f'({(newest - cutoff) / 86400000:.1f} days)\n')
    if cutoff > wanted:
        print(f'Note: {len(data[TOP])} builds do not reach back {days} days. '
              'Raise --builds, unless Jenkins has already discarded them.\n')
    gone = {job: len(numbers - {build['number'] for build in data.get(job) or []})
            for job, numbers in linked.items()}
    if any(gone.values()):
        print('Note: linked builds that Jenkins no longer has, or that are '
              'further back than this fetched: ' + ', '.join(
                  f'{job} {count}' for job, count in gone.items() if count) + '.\n')
    print(f'{len(pr_runs)} finished runs across {len(per_pr)} PRs, '
          f'{len(pr_runs) / len(per_pr):.1f} runs per PR.')
    results = collections.Counter(build['result'] for build in pr_runs)
    print('Results: ' + ', '.join(
        f'{result} {count} ({100 * count / len(pr_runs):.0f}%)'
        for result, count in results.most_common()) + '.\n')

    # A matrix child's timestamp equals its parent's, so queue time is not
    # exposed; estimate it as the job's duration minus its slowest child's.
    rows = []
    for job in data:
        builds = finished(job)
        total = summary(minutes(build['duration']) for build in builds)
        if job in (TOP, COMMIT) or not total:
            continue
        waits, slowest, names = [], [], collections.Counter()
        for build in builds:
            runs = own_runs(build)
            if not runs:
                continue
            longest = max(runs, key=lambda run: run['duration'])
            waits.append(minutes(max(0, build['duration'] - longest['duration'])))
            slowest.append(minutes(longest['duration']))
            names[config_name(longest['url'])] += 1
        wait, run = summary(waits), summary(slowest)
        rows.append((total['median'], [
            job, total['n'], total['median'], total['p90'], total['max'],
            f"{wait['median']} / {wait['p90']}" if wait else '',
            f"{run['median']} ({names.most_common(1)[0][0]})" if run else '',
        ]))
    rows = [row for _, row in sorted(rows, key=lambda r: r[0], reverse=True)]
    end_to_end = summary(minutes(build['duration']) for build in finished(TOP))
    rows.append([f'**{TOP}**', end_to_end['n'], end_to_end['median'],
                 end_to_end['p90'], end_to_end['max'], '', ''])
    print('Minutes, aborted runs excluded.\n')
    table(['job', 'runs', 'median', 'p90', 'max',
           'waiting for an executor (median / p90)',
           'slowest configuration (median, most often)'], rows)

    alone, at_all = collections.Counter(), collections.Counter()
    for build in finished(COMMIT, aborted=True):
        bad = [sub['jobName'] for sub in build.get('subBuilds') or []
               if sub.get('result') not in ('SUCCESS', None)]
        at_all.update(bad)
        if len(bad) == 1:
            alone[bad[0]] += 1
    print(f'## Non-green children of {COMMIT}\n')
    table(['job', 'non-green', 'the only non-green job in the run'],
          [[job, count, alone[job]] for job, count in at_all.most_common()])

    if not per_config:
        return
    configs = collections.defaultdict(
        lambda: {'exec': [], 'results': collections.Counter()})
    for job in data:
        for build in finished(job, aborted=True):
            for run in own_runs(build):
                entry = configs[f"{job}/{config_name(run['url'])}"]
                entry['exec'].append(minutes(run['duration']))
                entry['results'][run['result']] += 1
    rows = []
    for name, entry in configs.items():
        stats = summary(entry['exec'])
        rows.append([name, stats['n'], stats['median'], stats['p90'], stats['max'],
                     f"{100 * entry['results']['FAILURE'] / stats['n']:.0f}%"])
    print('## Per configuration (execution time only, minutes)\n')
    table(['configuration', 'runs', 'median', 'p90', 'max', 'FAILURE'],
          sorted(rows, key=lambda row: row[2], reverse=True))


def main():
    parser = argparse.ArgumentParser(
        description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
    parser.add_argument('--days', type=int, default=30,
                        help='window, counted back from the newest run (default 30)')
    parser.add_argument('--builds', type=int, default=450,
                        help='builds to request per job (default 450)')
    parser.add_argument('--per-config', action='store_true',
                        help='also print one row per matrix configuration')
    parser.add_argument('--save', metavar='FILE', help='write the API data to FILE')
    parser.add_argument('--load', metavar='FILE', help='read API data from FILE')
    args = parser.parse_args()

    if args.load:
        with open(args.load) as f:
            data = json.load(f)
    else:
        data = fetch(args.builds)
    if args.save:
        with open(args.save, 'w') as f:
            json.dump(data, f)
    report(data, args.days, args.per_config)


if __name__ == '__main__':
    main()

We had a discussion in the build WG about implementing "daily only" builds and it's worth noting that we do have a way of doing that at the moment. For example the current node-daily-master runs a number of other jobs outside node-test-commit. Specifically:

The last of those I added recently. I would also say that it would be nice to add in the reproducibility test job into that set.

There are now two potential approaches here which could achieve the same result of making it run on a daily cadence rather than on each PR:

The existing method used by those four jobs: Have them explicitly added to node-daily-master
Using a "daily" selector as proposed in jenkins: skip SmartOS and AIX for pull request test runs #4458

I'd personally lean slightly towards the second as it (a) puts the selectors all under version control, making it easier to switch things on and off in the main runs and (b) would increase the visibility of some of the other builds like ibmi/riscv/pointer-compression by allowing them to be added into the node-test-commit job.

Navigation Menu

Repository navigation

Proposal: run SmartOS and AIX in node-daily-master instead of node-test-pull-request #4457

AIX 7.2 Power 8 machines (used for Node.js < 26 since Node.js >= 26 requires Power 9)

AIX 7.2 Power 9 machines

AIX 7.3

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Proposal: run SmartOS and AIX in node-daily-master instead of node-test-pull-request #4457

Description

Activity

sxa commented on Sep 6, 2026

richardlau commented on Sep 6, 2026

AIX 7.2 Power 8 machines (used for Node.js < 26 since Node.js >= 26 requires Power 9)

AIX 7.2 Power 9 machines

AIX 7.3

sxa commented on Sep 6, 2026

aduh95 commented on Sep 7, 2026

panva commented on Sep 11, 2026

sxa commented on Sep 13, 2026

codebytere commented on Oct 1, 2026

sxa commented on Oct 6, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions