check-coverage passes when there is no coverage data to check #610

Activity

1. mahirhir commented on Sep 12, 2026

   

   mahirhir

   on Sep 12, 2026

   Author

   More actions

   Re-ran this on c8@latest, which is still 12.0.0, so nothing has moved under it. Same result, and this time with the control in the same table so the zeros mean something:

   === c8 12.0.0 / node v24.15.0 ===
   CONTROL: real coverage under threshold   exit=1  threshold_error=1  All files | 75 | 100 | 50 | 75 |
   empty: no coverage data at all           exit=0  threshold_error=0  All files |  0 |   0 |  0 |  0 |
   empty: --per-file                        exit=0  threshold_error=0  All files |  0 |   0 |  0 |  0 |
   empty: include glob matches nothing      exit=0  threshold_error=0  All files |  0 |   0 |  0 |  0 |
   empty: check-coverage over empty dir     exit=0  threshold_error=0  <no table>
   

   Every one of those is --lines 100. The control shows the threshold is alive in that exact invocation; the four below it are the same flag against nothing.

   Two reporters describe the same empty run differently, which is the part I find most useful for locating it:

   text table    All files | 0 | 0 | 0 | 0
   text-summary  Statements : Unknown% ( 0/0 )
                 Lines      : Unknown% ( 0/0 )
   

   text-summary is right. The table prints 0, which reads as a real measurement of zero, and the threshold check sees the 'Unknown' that text-summary shows:

   > 'Unknown' < 100
   false
   > 0 < 100
   true
   > NaN < 100
   false

   So summary[key].pct < thresholds[key] is false for an empty map, and false means "not below threshold", which means pass. It is the same shape as a NaN comparison: the value that means we could not measure this is ordered against the threshold rather than excluded from the comparison.

   I am not asking for a particular fix and I know the ecosystem-compatibility question here is real (istanbul's pct: 'Unknown' is longstanding, and people do run check-coverage in matrix jobs where some shards legitimately produce nothing). If it helps, the options as I see them, cheapest first:

   1. Treat a non-numeric pct as a distinct third outcome and fail the check with its own message, rather than routing it through the same < that numeric percentages use.
   2. Fail only when the coverage map is empty and a threshold was explicitly requested, leaving pct: 'Unknown' alone for anyone parsing reports.
   3. Leave the exit code and make the table stop printing 0 for an unmeasured run, so the two reporters at least agree.

   Happy to send a patch for whichever of those you would take, or none if this is intended. Mostly I wanted the reproduction on record with its control attached, since "exit 0" and "exit 0 because there was nothing to check" are indistinguishable in CI.