Update to codecov/codecov-action@v2 causes coverage change

Support Bug Fixes
1

Description

First of all, thanks for the amazing work the Codecov team is doing! Really appreciate that!
We’re trying to update the Codecov GitHub Action from v1 to v2 at Mautic (PR is here), but that change alone seems to change our overall coverage by 0.77%.

I didn’t expect that coverage change to pop up and couldn’t find any notes related to it in the blog post. Does anyone have an idea what could be causing this?

Commit SHAs

Repository

CI/CD or Build URL

GitHub Actions:

Uploader

GitHub Actions

Codecov Output

==> linux OS detected
https://uploader.codecov.io/latest/linux/codecov.SHA256SUM
==> SHASUM file signed by key id 806bb28aed779869
==> Uploader SHASUM verified (d8fe17b4f8cf960b7b02a44067aab92681c7a983bdb9542c64624fc8281dfb80  codecov)
==> Running version latest
==> Running version v0.1.5
/home/runner/work/_actions/codecov/codecov-action/v2/dist/codecov -n  -Q github-action-2.1.0 -Z -f ./coverage.xml -C c407733a0f1a18b0b224849ed8ad44c0d5bf58a7
[2021-10-13T18:00:32.265Z] ['info'] 
     _____          _
    / ____|        | |
   | |     ___   __| | ___  ___ _____   __
   | |    / _ \ / _` |/ _ \/ __/ _ \ \ / /
   | |___| (_) | (_| |  __/ (_| (_) \ V /
    \_____\___/ \__,_|\___|\___\___/ \_/

  Codecov report uploader 0.1.5
[2021-10-13T18:00:32.273Z] ['info'] => Project root located at: /home/runner/work/mautic/mautic
[2021-10-13T18:00:32.276Z] ['info'] -> No token specified or token is empty
[2021-10-13T18:00:32.286Z] ['info'] Searching for coverage files...
[2021-10-13T18:00:32.513Z] ['info'] => Found 1 possible coverage files:
  ./coverage.xml
[2021-10-13T18:00:32.513Z] ['info'] Processing ./coverage.xml...
[2021-10-13T18:00:32.648Z] ['info'] Detected GitHub Actions as the CI provider.
[2021-10-13T18:00:32.649Z] ['info'] Pinging Codecov: https://codecov.io/upload/v4?package=github-action-2.1.0-uploader-0.1.5&token=*******&branch=update-codecov-4.0&build=1338627072&build_url=https%3A%2F%2Fgithub.com%2Fmautic%2Fmautic%2Factions%2Fruns%2F1338627072&commit=c407733a0f1a18b0b224849ed8ad44c0d5bf58a7&job=Mautic+tests+and+validations&pr=10530&service=github-actions&slug=mautic%2Fmautic&name=&tag=&flags=&parent=
[2021-10-13T18:00:33.088Z] ['info'] Uploading...
[2021-10-13T18:00:33.394Z] ['info'] {"status":"success","resultURL":"https://codecov.io/github/mautic/mautic/commit/c407733a0f1a18b0b224849ed8ad44c0d5bf58a7"}

Expected Results

No change to the code coverage when moving to v2 of the official GitHub Action.

Actual Results

Code coverage change of 0.77% project-wide occurs when moving to v2 of the official GitHub Action.

Additional Information

We’re not using any specific configuration :slight_smile:

Please let me know in case more information is needed from our side. Thanks in advance!

2

@dennisameling, thanks for the message. tl;dr, v1 makes adjustments to the coverage report before uploading that v2 doesn’t (yet) do.

The v1 uploader uses the underlying bash uploader. In that uploader, we make adjustments to the coverage reports to account for things like lines with just a closed bracket. Some coverage utilities will report on those lines.

For this PR, we can see the difference (and why you see a 0.77% change) between the base and HEAD commits. If we take a look at the AuthorizeController.php file,

base shows line 103 as not a line of interest, while
HEAD shows line 103 as uncovered (per the coverage report).

This functionality of adjustments is currently being re-evaluated by the Codecov team.

2 Likes