v8 test coverage in jest is incorrect when upgrading to 20.10.0

@joyeecheung

Version

20.10.0

Platform

Macbook Pro M1/M2 / Ubuntu 22.04 x64 (GitHub runner)

Subsystem

V8

What steps will reproduce the bug?

See jestjs/jest#14766

How often does it reproduce? Is there a required condition?

Found a consistent repro, but initially it was flaky because of the order jest was running tests.

What is the expected behavior? Why is that the expected behavior?

20.9.0 just works
20.10.0 fails

So this is a regression.

What do you see instead?

Coverage information is wrong in jest, unclear why, but based what we know Node is the place where this will most likely be fixed.

Additional information

Based on https://nodejs.org/en/blog/release/v20.10.0, cc @joyeecheung as you might be interested/know what's wrong.

Based on jestjs/jest#14764, cc @kwmhp as you experienced the same problem as us, and figure out the version upgrade is what triggers it.

@SimenB

Can you provide a repro without third-party modules such as jest? Or maybe someone like @SimenB who's more familiar with Jest knows how to minimize the repro.

I can't, sorry, we spent a long time minimizing this much, and have no clue how to go further.

@kwmhp

@kwmhp here (work/private account)
I am trying to reproduce this with my test case, but I can't so far
node --test --experimental-test-coverage
return 100% coverage.
I will do some more experiments, as the problem in jest occurred only when executing tests in certain order/in-line.
The issue only occurs with v20.10.0, but the problem might be further downstream with aggregation of test coverage.

Data returned from session.post('Profiler.takePreciseCoverage') (see https://github.com/SimenB/collect-v8-coverage/blob/2dd5d7aad3a8a1a07e33f9bf331f675cb5b9b480/index.js#L25-L27) a bit different for 20.9.0 and 21.5.0:

You can find that data at line jest/v8-flaky-coverage/node_modules/jest-runtime/build/index.js:1165, in this._v8CoverageResult.

Because c8 jest --runInBand shows 100% coverage despite of Node version (== NODE_V8_COVERAGE produces proper data), it looks like the root cause is somewhere near inspector and/or how it used by jest-runtime / collect-v8-coverage packages.

I am also getting this issue after updating to 20.10.0 from 20.9.0. Is there any workaround?

Our code coverage failure seem to be related to tests that assert error exceptions which are not being included in the code coverage.

#51251 (comment) mentioned c8 produces correct results, so you could try that if that works. Otherwise it's difficult to tell whether this is a Node.js issue without a repro that only uses Node.js and not third-party modules.

@joyeecheung

@joyeecheung

@koshic seems to be right with his diagnosis. I can verify that the following fixes the flaky code coverage jestjs/jest#14766 (comment)

The code changes relate to this repo https://github.com/SimenB/collect-v8-coverage/blob/main/index.js

Are you able to provide any insight why this might be the case?

I don't know much about how the jest package implements coverage collection but if commenting out the stop command makes a difference it might be that the package was stopping coverage collection too early or the ordering of start and stop commands it uses is wrong somehow. C8 spawns child processes with NODE_V8_COVERAGE which is implemented by Node.js to start the collection very early and stop the collection when the process is about to exit, that could be why it is more reliable. Maybe the jest approach could be updated to use NODE_V8_COVERAGE instead of using the inspector API too.

Seems it can be fixed with await this.postSession('Debugger.enable');
see https://github.com/SimenB/collect-v8-coverage/pull/235/files

I have been facing the same issue: my coverage report was 100% with 20.9.0, but as of 20.10.0, it now reports less than 100% coverage and somewhat randomly flags lines not covered.

I noticed that I get the same coverage reporting flakiness with 18.20.2 (I initially saw this issue when my GitHub actions started to fail). Coverage works fine with 18.19.1.

So there seems to be a change between 18.19.1 and 18.20.0 that causes the issue as well as between 20.9.0 to 20.10.0.

This is now causing my GitHub CI/CD actions to fail since GitHub is using 18.20.2.

Use the exact same node version locally as in CI

test: switch back to `babel`-based coverage

Hi all,
Same for us, we have actually ~50 nodeJS services stuck on node 20.9.0 because of this issue.

Not 100% sure about this, but after checking jestjs/jest#14766 and istanbuljs/v8-to-istanbul#236 (comment), it seems that the problem is related to how Jest handles the V8 profiler output.

I’ll take a look. In the meantime, is this happening for all 20.x versions starting from 20.9?

@pmarchini

@pmarchini Presumably - I just checked with v20.18.0 and v22.11.0, and both still had this issue.

@pmarchini

@pmarchini Presumably - I just checked with v20.18.0 and v22.11.0, and both still had this issue.

Perfect, thanks @Vinnl

Here's a minimal reproduction:

'use strict';
console.log(process.version);
const { Session } = require('node:inspector');
const { promisify } = require('node:util');
const vm = require('node:vm');
const code = `
  if (condition) {
    return trueFunction();
  } else {
    return falseFunction();
  }
  function trueFunction() {
    return 1;
  }
  function falseFunction() {
    return 0;
  }
`;

async function runTests(condition) {
  const session = new Session();
  const postSession = promisify(session.post.bind(session));

  session.connect();

  await postSession('Profiler.enable');
  await postSession('Profiler.startPreciseCoverage', {
    callCount: true,
    detailed: true,
  });

  const context = vm.createContext({});
  const func = vm.compileFunction(code, ['condition'], {
    filename: 'conditionalFunction',
    parsingContext: context
  });
  func(condition);

  const {result} = await postSession(
    'Profiler.takePreciseCoverage',
  );

  await postSession('Profiler.stopPreciseCoverage');
  await postSession('Profiler.disable');
  session.disconnect();
  return result;
}

async function runTwice() {
  const result1 = await runTests(false);
  const result2 = await runTests(true);

  for (const result of [result1, result2]) {
    console.log();
    for (const item of result.filter(item => item.url === 'conditionalFunction')) {
      console.log(JSON.stringify(item.functions));
    }
  }
}
runTwice();

This only reproduces when using the node:vm module. From some experimentation, the behavior change here is that in old versions of Node (e.g. 18.19.1), the v8 compile cache appears to have been isolated per VM Context (i.e. compiling a function in a new context received a fresh, empty code cache). In newer versions (e.g. 18.20+, 20.10+, 22), new contexts reuse the existing v8 compile cache results.

My guess would be that after the call to Profiler.stopPreciseCoverage, v8's optimizing compiler gets reenabled, sees the new contents of the cache, and optimizes them, which causes them to lose the ability to collect precise coverage data when that cached data is used in subsequent test cases.

Edit: In order to work around this, the ideal solution would be a way to specify in vm.createContext to disable the optimizing compiler for code used in that context, which would probably require support from v8.

Still issue in v23.6.1

I facing the same issue when using with ts-jest.

node 22.15.0
typescript 5.8.3
jest 29.7.0
ts-jest 29.3.2

I have tried to use babel instead of v8, but didn't fix this issue.

Minimal reproduction: V8 coverage branch map inconsistency

This issue has been marked as stale due to 210 days of inactivity.
It will be automatically closed in 30 days if no further activity occurs. If this is still relevant, please leave a comment or update it to keep it open.

This issue has been marked as stale due to 90 days of inactivity.
It will be automatically closed in 30 days if no further activity occurs. If this is still relevant, please leave a comment or update it to keep it open.

.

Navigation Menu

Repository navigation

v8 test coverage in jest is incorrect when upgrading to 20.10.0 #51251

Version

Platform

Subsystem

What steps will reproduce the bug?

How often does it reproduce? Is there a required condition?

What is the expected behavior? Why is that the expected behavior?

What do you see instead?

Additional information

2 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

v8 test coverage in jest is incorrect when upgrading to 20.10.0 #51251

Description

Version

Platform

Subsystem

What steps will reproduce the bug?

How often does it reproduce? Is there a required condition?

What is the expected behavior? Why is that the expected behavior?

What do you see instead?

Additional information

Activity

joyeecheung commented on Dec 21, 2023

TWiStErRob commented on Dec 21, 2023

Snafuh commented on Dec 21, 2023

koshic commented on Dec 23, 2023

ritchieanesco commented on Jan 10, 2024

joyeecheung commented on Jan 11, 2024

ritchieanesco commented on Jan 29, 2024

joyeecheung commented on Jan 30, 2024

cenfun commented on Mar 19, 2024

kmccammon commented on Apr 16, 2024

2 remaining items

raphaelboukara commented on Oct 31, 2024

pmarchini commented on Oct 31, 2024

Vinnl commented on Oct 31, 2024

pmarchini commented on Oct 31, 2024

dmichon-msft commented on Jan 17, 2025

jancajthaml commented on Jan 22, 2025

krishnabrq commented on May 7, 2025

github-actions commented on May 24, 2026

github-actions commented on Aug 24, 2026

TWiStErRob commented on Aug 24, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions