Skip to content

codecov/project fails on unrelated PRs: coverage-report does not wait for ha-integration-tests #5701

Description

@lvca

What happens

codecov/project reports a whole-project coverage drop on PRs that cannot have caused one, so the check is red on
work that never touched the affected code. On #5685 (a 29-line change confined to
engine/src/main/java/com/arcadedb/index/hash/) Codecov reported:

Files          1770     1770
Lines        150057   150086      +29
Hits         101236   100126    -1110     <-- 1,110 covered lines disappear

A 29-line change cannot delete 1,110 hits. Every one of the 34 files that lost coverage is in ha-raft/:

Hits lost File base -> head
-295 RaftReplicatedDatabase.java 68.16% -> 29.19%
-229 RaftHAServer.java 57.71% -> 38.07%
-116 ArcadeStateMachine.java 66.66% -> 54.35%
-97 PostVerifyDatabaseHandler.java 60.58% -> 3.52%

Cause

In .github/workflows/mvn-test.yml, coverage-report is wired to consume HA coverage but never waits for it:

coverage-report:
  needs: [ unit-tests, integration-tests, slow-unit-tests ]   # ha-integration-tests is absent
  ...
  - name: Download HA integration test coverage reports        # line ~757
    uses: actions/download-artifact@...
    continue-on-error: true                                    # failure is swallowed
    with:
      name: ha-integration-coverage-reports

ha-integration-tests does upload that artifact (line ~340, if: success() || failure()), so the intent is clearly
that HA coverage be included. But because the job is not in needs, coverage-report starts as soon as the three
listed jobs finish, while ha-integration-tests runs for ~55 minutes - by far the longest job in the workflow. The
artifact does not exist yet, continue-on-error: true hides that, and the merged report silently omits the whole
ha-raft module.

So the project total is computed against a base that included HA coverage and a head that does not. The gap is
roughly -0.76pp, which is enough to trip the threshold.

Why it matters

This is not specific to one PR - it is structural, and it fires on every PR whose other jobs finish before the HA
suite. The practical effect is that codecov/project is red for reasons unrelated to the diff, which trains
reviewers to ignore it. A coverage gate that is always red is worse than no gate: a real coverage regression would
look exactly like the noise.

Note codecov/patch is unaffected and remains meaningful (it scored the #5685 diff at 87.75% and passed), as do
Codacy Diff Coverage and Codacy Coverage Variation.

Suggested fix

Add the missing dependency:

coverage-report:
  needs: [ unit-tests, integration-tests, slow-unit-tests, ha-integration-tests ]

coverage-report already has if: success() || failure(), so adding the dependency does not make a failing HA suite
block the coverage report - it only makes it wait, which is what the existing download step assumes.

Two things worth deciding alongside:

  1. continue-on-error: true on the download hides exactly this class of bug. Consider failing the job when an
    expected coverage artifact is missing, or at minimum logging which artifacts were skipped, so an incomplete
    upload is visible rather than silently reported as a coverage drop.
  2. ha-integration-tests is currently failing on main most of the time (filed separately). While that is true,
    its coverage artifact still uploads (if: success() || failure()), so this fix works regardless - but the two
    interact, and fixing only this one will surface partial HA coverage from partially-failed runs.

Found while diagnosing a spurious codecov/project failure on #5685 (issue #5677).

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions