Sudden and significant drop in coverage

Support Bug Fixes
1

Before submitting a topic, please confirm the following

I have searched for similar issues before creating this topic.
I have verified that my repository is using the Codecov GitHub app, if using GitHub
I have validated my codecov.yaml configuration file.
I have filled out the below sections to the best of my ability.
I understand that the community boards are a free, best-effort tool. While we hope to have someone on the Codecov team resolve your problem quickly, this is not always possible.

Description

We have recently updated our node CI to run on node 18. No other changes that I can think that would be relevant PR here

We’re uploading with codecov-action@v3. The coverage is collected by c8 running mocha.

CI/CD URL

Example run after the drop: build(deps-dev): update linting packages · zazuko/cube-creator@08ee8ef · GitHub

Codecov Output

==> linux OS detected
https://uploader.codecov.io/latest/linux/codecov.SHA256SUM
==> SHASUM file signed by key id 806bb28aed779869
==> Uploader SHASUM verified (671cf0d89d1c149f57e1a9a31f3fb567ab4209e4d5829f13ff7b8c104db7131f  codecov)
==> Running version latest
==> Running version v0.4.0
/home/runner/work/_actions/codecov/codecov-action/v3/dist/codecov -n  -Q github-action-3.1.1
[2023-03-28T10:39:12.852Z] ['info'] 
     _____          _
    / ____|        | |
   | |     ___   __| | ___  ___ _____   __
   | |    / _ \ / _` |/ _ \/ __/ _ \ \ / /
   | |___| (_) | (_| |  __/ (_| (_) \ V /
    \_____\___/ \__,_|\___|\___\___/ \_/

  Codecov report uploader 0.4.0
[2023-03-28T10:39:12.872Z] ['info'] => Project root located at: /home/runner/work/cube-creator/cube-creator
[2023-03-28T10:39:12.876Z] ['info'] ->  Token found by environment variables
[2023-03-28T10:39:12.886Z] ['info'] Searching for coverage files...
[2023-03-28T10:39:13.021Z] ['info'] Warning: Some files located via search were excluded from upload.
[2023-03-28T10:39:13.022Z] ['info'] If Codecov did not locate your files, please review https://docs.codecov.com/docs/supported-report-formats
[2023-03-28T10:39:13.022Z] ['info'] => Found 6 possible coverage files:
  coverage/apis/lcov.info
  coverage/publish/lcov.info
  coverage/queries/lcov.info
  coverage/apis/tmp/coverage-19197-1679999857514-0.json
  coverage/publish/tmp/coverage-19969-1679999950594-0.json
  coverage/queries/tmp/coverage-19358-1679999930825-0.json
[2023-03-28T10:39:13.023Z] ['info'] Processing /home/runner/work/cube-creator/cube-creator/coverage/apis/lcov.info...
[2023-03-28T10:39:13.025Z] ['info'] Processing /home/runner/work/cube-creator/cube-creator/coverage/publish/lcov.info...
[2023-03-28T10:39:13.026Z] ['info'] Processing /home/runner/work/cube-creator/cube-creator/coverage/queries/lcov.info...
[2023-03-28T10:39:13.027Z] ['info'] Processing /home/runner/work/cube-creator/cube-creator/coverage/apis/tmp/coverage-19197-1679999857514-0.json...
[2023-03-28T10:39:13.093Z] ['info'] Processing /home/runner/work/cube-creator/cube-creator/coverage/publish/tmp/coverage-19969-1679999950594-0.json...
[2023-03-28T10:39:13.135Z] ['info'] Processing /home/runner/work/cube-creator/cube-creator/coverage/queries/tmp/coverage-19358-1679999930825-0.json...
[2023-03-28T10:39:13.917Z] ['info'] Detected GitHub Actions as the CI provider.
[2023-03-28T10:39:13.918Z] ['info'] Pinging Codecov: https://codecov.io/upload/v4?package=github-action-3.1.1-uploader-0.4.0&token=*******&branch=node-18-support&build=4542055940&build_url=https%3A%2F%2Fgithub.com%2Fzazuko%2Fcube-creator%2Factions%2Fruns%2F4542055940&commit=08ee8ef25b7b20f27f9b7b96cba5a7bf52491eae&job=CI&pr=&service=github-actions&slug=zazuko%2Fcube-creator&name=&tag=&flags=&parent=
[2023-03-28T10:39:14.269Z] ['info'] https://app.codecov.io/github/zazuko/cube-creator/commit/08ee8ef25b7b20f27f9b7b96cba5a7bf52491eae
https://storage.googleapis.com/codecov/v4/raw/2023-03-28/979C502059786D3B4D90CB0C797E8F46/08ee8ef25b7b20f27f9b7b96cba5a7bf52491eae/980516a8-2d75-4034-b43c-b7765c199af7.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=GOOG1EJOGFN2JQ4OCTGA2MU5AEIT7OT5Z7HTFOAN2SPG4NWSN2UJYOY5U6LZQ%2F20230328%2FUS%2Fs3%2Faws4_request&X-Amz-Date=20230328T103914Z&X-Amz-Expires=10&X-Amz-SignedHeaders=host&X-Amz-Signature=3d72e7d07e1146f1f2855daa08a829464b193a583a3b9d810a1f875111cc690b
[2023-03-28T10:39:14.269Z] ['info'] Uploading...
[2023-03-28T10:39:14.820Z] ['info'] {"status":"success","resultURL":"https://app.codecov.io/github/zazuko/cube-creator/commit/08ee8ef25b7b20f27f9b7b96cba5a7bf52491eae"}

Expected Results

Exact same coverage given no code changes

Actual Results

The coverage dropped from ~80% to <1%

Additional Information

I just today found that if I run with flag c8 --reporter lcovonly, the coverage came back. The output does not look all that dissimilar does it?

The weird thing is that with or without the --reporter option the lcov.info file is indeed produced.

==> linux OS detected
https://uploader.codecov.io/latest/linux/codecov.SHA256SUM
==> SHASUM file signed by key id 806bb28aed779869
==> Uploader SHASUM verified (32cb14b5f3aaacd67f4c1ff55d82f037d3cd10c8e7b69c051f27391d2e66e15c  codecov)
==> Running version latest
==> Running version v0.4.1
/home/runner/work/_actions/codecov/codecov-action/v3/dist/codecov -n  -Q github-action-3.1.1
[2023-04-06T08:21:20.707Z] ['info'] 
     _____          _
    / ____|        | |
   | |     ___   __| | ___  ___ _____   __
   | |    / _ \ / _` |/ _ \/ __/ _ \ \ / /
   | |___| (_) | (_| |  __/ (_| (_) \ V /
    \_____\___/ \__,_|\___|\___\___/ \_/

  Codecov report uploader 0.4.1
[2023-04-06T08:21:20.743Z] ['info'] => Project root located at: /home/runner/work/cube-creator/cube-creator
[2023-04-06T08:21:20.747Z] ['info'] ->  Token found by environment variables
[2023-04-06T08:21:20.768Z] ['info'] Searching for coverage files...
[2023-04-06T08:21:20.974Z] ['info'] Warning: Some files located via search were excluded from upload.
[2023-04-06T08:21:20.975Z] ['info'] If Codecov did not locate your files, please review https://docs.codecov.com/docs/supported-report-formats
[2023-04-06T08:21:20.975Z] ['info'] => Found 6 possible coverage files:
  coverage/apis/lcov.info
  coverage/publish/lcov.info
  coverage/queries/lcov.info
  coverage/apis/tmp/coverage-19713-1680769140258-0.json
  coverage/publish/tmp/coverage-20862-1680769277926-0.json
  coverage/queries/tmp/coverage-20030-1680769249084-0.json
[2023-04-06T08:21:20.976Z] ['info'] Processing /home/runner/work/cube-creator/cube-creator/coverage/apis/lcov.info...
[2023-04-06T08:21:20.980Z] ['info'] Processing /home/runner/work/cube-creator/cube-creator/coverage/publish/lcov.info...
[2023-04-06T08:21:20.982Z] ['info'] Processing /home/runner/work/cube-creator/cube-creator/coverage/queries/lcov.info...
[2023-04-06T08:21:20.984Z] ['info'] Processing /home/runner/work/cube-creator/cube-creator/coverage/apis/tmp/coverage-19713-1680769140258-0.json...
[2023-04-06T08:21:21.064Z] ['info'] Processing /home/runner/work/cube-creator/cube-creator/coverage/publish/tmp/coverage-20862-1680769277926-0.json...
[2023-04-06T08:21:21.114Z] ['info'] Processing /home/runner/work/cube-creator/cube-creator/coverage/queries/tmp/coverage-20030-1680769249084-0.json...
[2023-04-06T08:21:22.048Z] ['info'] Detected GitHub Actions as the CI provider.
[2023-04-06T08:21:22.675Z] ['info'] Pinging Codecov: https://codecov.io/upload/v4?package=github-action-3.1.1-uploader-0.4.1&token=*******&branch=ci-coverage-merge&build=4626843642&build_url=https%3A%2F%2Fgithub.com%2Fzazuko%2Fcube-creator%2Factions%2Fruns%2F4626843642%2Fjobs%2F8184052097&commit=8be314e7bc6f8b38d199c576e342b72e410e51fb&job=CI&pr=&service=github-actions&slug=zazuko%2Fcube-creator&name=&tag=&flags=&parent=
[2023-04-06T08:21:22.885Z] ['info'] https://app.codecov.io/github/zazuko/cube-creator/commit/8be314e7bc6f8b38d199c576e342b72e410e51fb
https://storage.googleapis.com/codecov/v4/raw/2023-04-06/979C502059786D3B4D90CB0C797E8F46/8be314e7bc6f8b38d199c576e342b72e410e51fb/668deff8-df88-4b05-ac3a-19dff34659ee.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=GOOG1EJOGFN2JQ4OCTGA2MU5AEIT7OT5Z7HTFOAN2SPG4NWSN2UJYOY5U6LZQ%2F20230406%2FUS%2Fs3%2Faws4_request&X-Amz-Date=20230406T082122Z&X-Amz-Expires=10&X-Amz-SignedHeaders=host&X-Amz-Signature=68d5ad70dd7eccfdfbee949a9b577691f6f3e89a59996b10693dea7a443e018e
[2023-04-06T08:21:22.885Z] ['info'] Uploading...
[2023-04-06T08:21:23.169Z] ['info'] {"status":"success","resultURL":"https://app.codecov.io/github/zazuko/cube-creator/commit/8be314e7bc6f8b38d199c576e342b72e410e51fb"}
2

@tpluscode, it looks to me like this was something inherent with the way c8 is collecting coverage. If you take a look at some of the reports for a commit that returned as ~0% coverage like this one, you’ll see blocks like

SF:apis/core/lib/ChangelogDataset.ts
FN:1,(empty-report)
FNF:1
FNH:0
FNDA:0,(empty-report)
DA:1,0
DA:2,0
DA:3,0
DA:4,0
DA:5,0
DA:6,0
DA:7,0
DA:8,0
DA:9,0
DA:10,0
DA:11,0
DA:12,0
DA:13,0
DA:14,0
DA:15,0
DA:16,0
DA:17,0

To me, this looks like all of those lines are uncovered. I don’t know why the addition of --reporter lcovonly is properly producing coverage.

3

I compared both those lcov reports and the looked exactly equal.

The solution is surprising and the fact that updating to node 18 seems to have caused this is weird too

4

@tpluscode that is really strange. Just to be clear, adding --reporter lcovonly does fix the issue. And the only change you can tell is a migration to node 18?

5

Indeed. I cannot see anything else in that PR’s changes that should affect c8 or codecov :man_shrugging:

6

@tpluscode I’m sorry I still don’t have a good answer for you here. I tried digging into the code, but I haven’t found anything promising. Are you blocked on this or is adding --reporter lcovonly a suitable solution for now?

7

Yes, I am able to get the coverage so this is not a blocker, thanks

1 Like