Repository navigation
Draft Statement of Work - Test reliability lead #1629
Description
The test flakiness lead will be expected to:
- lead a test reliability strategic initiative, rallying and supporting contributors who work to reduce flaky tests. This might include running regular test team meetings, documentation, tools, or whatever strategy works to achieve more than they can do on their own
- build tools and improve automation that allows the project to effectively manage flaky tests to reduce their impact on the CI
- Investigate and fix existing tests being marked as flaky in the status files
Duration
- 6 months
Success looks like
- maintain good test coverage
- reduced number of tests marked as flaky in the status files
- Running node-test-commit on the main branch will pass more often (hopefully always).
- critical mass of contributors/collaborators dedicating time to addressing flaky tests that persists beyond the strategic initiative
Activity
mhdawson commentedon Oct 2, 2024
@nodejs/tsc as discussed in the TSC meeting today a first cut at what a statement of work for a test flakiness lead might look like.
mhdawson commentedon Oct 7, 2024
Adding to agenda so that we review/get feedback in a future meeting.
joyeecheung commentedon Oct 7, 2024
I think we should explicitly add:
Otherwise this might just optimize towards marking all the tests as flaky and let them rot in the status files, which isn't ideal.
mhdawson commentedon Oct 8, 2024
@joyeecheung updated, thanks for the suggestion.
mcollina commentedon Oct 24, 2024
I think we should have a more measurable success criteria, such as:
I would structure the agreement as:
Alternatively, if the person is a member of the TSC, I would put at:
daily rate of XXX * number of days worked, capped at zzz.
joyeecheung commentedon Oct 24, 2024
That would be assuming the list of tests doesn't change while this is happening, which is unlikely to be true. Other contributors can always alter the tests as necessary or mark tests as flaky as they see fit, or add more tests while all these are happening, and could use some eyes watching the status of the new tests or otherwise new flakes still come up and we won't be much better off. What we care about is whether the overall situation improves, while the situation isn't always static without the hired individual doing anything.
Also, identifying this list is also non-trivial work, especially when it could be challenging to triage and identify a correct list. It wouldn't be too meaningful if the list contains a lot of false positives, yet eliminating false positives can already be difficult enough.
If we want a quantitative measurement, then I think the rate of a passing node-test-commit CI on the main branch is already enough (which has been around 0% for some time).
mhdawson commentedon Oct 28, 2024
I also think we want somebody who will do more than just fix specific tests, helping to improve how we manage and resolve flaky tests though automation and tools is just as important as fixing specific flaky tests.
Trott commentedon Nov 12, 2024
I've not been privy to this conversation so I might be missing the mark with this comment, but I think it will be more understandable and more professional-sounding if you call it a "test reliability lead" rather than a "test flakiness lead". In a formal/professional document, I wouldn't refer to "flakiness" but instead refer to "reliability" (or "unreliability").
[-]Draft Statement of Work - Test flakiness lead[/-][+]Draft Statement of Work - Test reliability lead[/+]mhdawson commentedon Nov 12, 2024
@Trott like that suggestion, incorporated.
mhdawson commentedon Nov 18, 2024
Sent email today to the Foundation executive directory with @mcollina as our CPC rep on CC to ask what might be possible in terms of funding from the OpenJS foundation.
bnoordhuis commentedon Apr 16, 2025
@mhdawson (or @mcollina) seeing how half a year has passed I'm guessing the answer is 'no' but did you guys ever receive a reply?
mcollina commentedon Apr 16, 2025
The answer is that there is no money for this right now. We are trying to assess if we can find the funding to make it happen. Specifically, the amount of money that we will receive from HeroDevs. Having said that, most of the funds are spent on keeping the lights up of the Jenkins CI.
3 remaining items
mcollina commentedon May 4, 2025
That's revenue, not profit. That'd be fantastic if sustainable, but it's not.
Almost all that projected revenue is tied into:
The Foundation is also doing its best to help with build/infra (specifically Macs).
jasnell commentedon Jun 24, 2025
We have no oversight or say in what funds the Foundation may or may not make available to the project. We can ask if they are able to help with something and if they say they can't then they can't. Any feedback back to the foundation on that is correctly directed through the CPC.
Node.js is not the foundations only project. That said, we already get the majority of focus and an overwhelming majority of the available resources. The foundation is doing what it can. What would be most helpful in this conversation, for those of you wishing there was more money available for Node.js to use to fund various efforts, would be to encourage new member organizations to join the foundation (bringing in new member revenue), existing members to increase their investment (e.g. silvers change to gold), or by helping to promote services such as the extended support program.
joyeecheung commentedon Jun 24, 2025
I think this request is targeting a very ineffective audience - the same group of people who are capable of doing that work if they have the time. For example, if it takes more than 2 hour per week to reach out to companies to send money to the foundation in the hope that a fraction of it may be spent on fixing CI flakes in maybe a year (which is not guaranteed), it would be way more effective for us to just spend that 2 hour per week to actually work on CI flakes ourselves (which is immediate and guaranteed work). The efficiency in this model would be in negative correlation to how capable the audience is in doing that work themselves.
(I have a vague impression that the collective hours in the back and forth of trying to get money to work on this so far already exceeded what would be needed to bring the CI green rate up significantly if we just used those hours to fix CI flakes ourselves instead.)
RaisinTen commentedon Jun 25, 2025
There was already an attempt to document a way to allow for more funds for Node.js in nodejs/admin#955 but the Foundation said no with unclear reasoning, so it's been blocked with no observable progress for nearly 4 months.
jasnell commentedon Jun 25, 2025
Then let's just close this and not worry about asking the Foundation to pay for anything here. I see no reason to keep this issue open.
gireeshpunathil commentedon Aug 14, 2025
reopening this per TSC discussion on 13th Aug to revisit the draft
joyeecheung commentedon Oct 22, 2025
Swapping the TSC agenda label of #1614 with this one because this one is more actionable
mcollina commentedon May 1, 2026
After over 1 and a half years, I think we have the budget to start this.
mcollina commentedon May 27, 2026
At the last TSC meeting it didn't seem we had a strong consensus on this. @nodejs/tsc please chime in if you strongly believe this should be done.
panva commentedon May 27, 2026
Absolutely.
marco-ippolito commentedon May 27, 2026
From the collaborator survey CI flakiness was one of the top cause of frustration, so +1