Codecov bot coverage shows -0.25% decrease for flag, even when the bot also shows "will not change coverage"

Please search for similar issues before creating a ticket.
All fields below are required.

Description

The codecov PR bot shows a decrease in coverage when there is no decrease in coverage.

CI/CD URL

Codecov Output

Please provide the full output of running the uploader on your CI/CD. This will typically have the Codecov logo as ASCII.

Run codecov/codecov-action@v2
==> linux OS detected
https://uploader.codecov.io/latest/linux/codecov.SHA256SUM
==> SHASUM file signed by key id 806bb28aed779869
==> Uploader SHASUM verified (330066ecce62f2cdcfd2fe1845095475f51bbad9649dce81ac7b663dc884bbef  codecov)
==> Running version latest
==> Running version v0.1.15
/home/runner/work/_actions/codecov/codecov-action/v2/dist/codecov -n  -Q github-action-2.1.0 -Z -f ./cov-api.xml -F api-tests -C 7605b9e21599a848fc78d8fa26732a0a9258fc61
[2022-02-02T16:37:17.705Z] ['info'] 
     _____          _
    / ____|        | |
   | |     ___   __| | ___  ___ _____   __
   | |    / _ \ / _` |/ _ \/ __/ _ \ \ / /
   | |___| (_) | (_| |  __/ (_| (_) \ V /
    \_____\___/ \__,_|\___|\___\___/ \_/

  Codecov report uploader 0.1.15
[2022-02-02T16:37:17.712Z] ['info'] => Project root located at: /home/runner/work/spinalcordtoolbox/spinalcordtoolbox
[2022-02-02T16:37:17.714Z] ['info'] -> No token specified or token is empty
[2022-02-02T16:37:17.723Z] ['info'] Searching for coverage files...
[2022-02-02T16:37:18.313Z] ['info'] => Found 1 possible coverage files:
  ./cov-api.xml
[2022-02-02T16:37:18.313Z] ['info'] Processing ./cov-api.xml...
[2022-02-02T16:37:18.330Z] ['info'] Detected GitHub Actions as the CI provider.
[2022-02-02T16:37:18.331Z] ['info'] Pinging Codecov: https://codecov.io/upload/v4?package=github-action-2.1.0-uploader-0.1.15&token=*******&branch=jn%2F3669-test-release-path-in-CI&build=1784986182&build_url=https%3A%2F%2Fgithub.com%2Fspinalcordtoolbox%2Fspinalcordtoolbox%2Factions%2Fruns%2F1784986182&commit=7605b9e21599a848fc78d8fa26732a0a9258fc61&job=Tests&pr=3678&service=github-actions&slug=spinalcordtoolbox%2Fspinalcordtoolbox&name=&tag=&flags=api-tests&parent=
[2022-02-02T16:37:18.776Z] ['info'] Uploading...
[2022-02-02T16:37:18.969Z] ['info'] {"status":"success","resultURL":"https://codecov.io/github/spinalcordtoolbox/spinalcordtoolbox/commit/7605b9e21599a848fc78d8fa26732a0a9258fc61"}

Expected Results

The “flag” table should show 0.00% change.

Actual Results

See Test release path in CI by joshuacwnewton · Pull Request #3678 · spinalcordtoolbox/spinalcordtoolbox · GitHub

Codecov Report

Merging #3678 (7605b9e) into master (5d422d9) will not change coverage.
The diff coverage is n/a.

Flag Coverage Δ
api-tests 24.20% <ø> (-0.25%) :arrow_down:
cli-tests 56.84% <ø> (ø)

Flags with carried forward coverage won’t be shown. Click here to find out more.

Even more strangely, the web UI shows a coverage increase? Codecov

Screenshot from 2022-02-02 12-20-32

While the changes page shows no change at all: Codecov

Additional Information


My best guess is that maybe the cause of this is because the changes happened outside of the monitored folder:

  • The file changed is: ./.github/workflows/create-release.yml
  • However we’re only monitoring ./spinalcordtoolbox (the package directory)

So, maybe the bot is including other files in how it calculates coverage %?

Even still, it’s odd that the changes are only showing for the api-tests flag and not the cli-tests flag. :thinking:

@joshuacwnewton I’m not totally sure the answer to this, except that it might have been in a middle state. I took a look now and it looks like coverage is right, unless the 0.01% decrease is wrong. Let me know if it is still wrong.

My apologies for linking an example on an active pull request.

I’m realizing that isolating issues like this is tricky given the evolving nature of coverage reports… an issue might pop up for one state of a PR then disappear when more changes are made. :sweat_smile:

@joshuacwnewton not a problem, I feel like these things are always tricky to nail down. But if you see it again, post here and I’ll take a look as quickly as I can.