Incorrect coverage changes reported - base commit not uploading?

Support
1

Before submitting a topic, please confirm the following

I have searched for similar issues before creating this topic.
I have verified that my repository is using the Codecov GitHub app, if using GitHub
I have validated my codecov.yaml configuration file.
I have filled out the below sections to the best of my ability.

Description

PRs on GitHub - typescript-eslint/typescript-eslint: Monorepo for all the tooling which enables ESLint to support TypeScript consistently have incorrect amounts of coverage reported. We often have reports like ā€œdecrease coverage by 2.81%ā€ for PR sthat change only package.json & yarn.lock, for example.

CI/CD URL

Codecov Output

Please provide the full output of running the uploader on your CI/CD. This will typically have the Codecov logo as ASCII.

==> linux OS detected
https://uploader.codecov.io/latest/linux/codecov.SHA256SUM
==> SHASUM file signed by key id 806bb28aed779869
==> Uploader SHASUM verified (20f9c9d78483fce977b6cc39e231a734a23bcd36f4d536bb7355222fb88d02bc  codecov)
==> Running version latest
==> Running version v0.3.2
/home/runner/work/_actions/codecov/codecov-action/v3/dist/codecov -n codecov -Q github-action-3.1.1 -f coverage/**/lcov.info -F unittest -C ea0da9ec92e74b960bf769d82d2e85733d1fd5dd
[2022-10-02T09:05:22.578Z] ['info'] 
     _____          _
    / ____|        | |
   | |     ___   __| | ___  ___ _____   __
   | |    / _ \ / _` |/ _ \/ __/ _ \ \ / /
   | |___| (_) | (_| |  __/ (_| (_) \ V /
    \_____\___/ \__,_|\___|\___\___/ \_/

  Codecov report uploader 0.3.2
[2022-10-02T09:05:22.588Z] ['info'] => Project root located at: /home/runner/work/typescript-eslint/typescript-eslint
[2022-10-02T09:05:22.592Z] ['info'] -> No token specified or token is empty
[2022-10-02T09:05:22.604Z] ['info'] Searching for coverage files...
[2022-10-02T09:05:22.886Z] ['info'] Warning: Some files passed via the -f flag would normally be excluded from search.
[2022-10-02T09:05:22.886Z] ['info'] If Codecov encounters issues processing your reports, please review https://docs.codecov.com/docs/supported-report-formats
[2022-10-02T09:05:22.886Z] ['info'] => Found 9 possible coverage files:
  coverage/eslint-plugin-coverage/lcov.info
  coverage/eslint-plugin-internal-coverage/lcov.info
  coverage/eslint-plugin-tslint-coverage/lcov.info
  coverage/parser-coverage/lcov.info
  coverage/scope-manager-coverage/lcov.info
  coverage/type-utils-coverage/lcov.info
  coverage/typescript-estree-coverage/lcov.info
  coverage/utils-coverage/lcov.info
  coverage/visitor-keys-coverage/lcov.info
[2022-10-02T09:05:22.887Z] ['info'] Processing /home/runner/work/typescript-eslint/typescript-eslint/coverage/eslint-plugin-coverage/lcov.info...
[2022-10-02T09:05:22.889Z] ['info'] Processing /home/runner/work/typescript-eslint/typescript-eslint/coverage/eslint-plugin-internal-coverage/lcov.info...
[2022-10-02T09:05:22.890Z] ['info'] Processing /home/runner/work/typescript-eslint/typescript-eslint/coverage/eslint-plugin-tslint-coverage/lcov.info...
[2022-10-02T09:05:22.891Z] ['info'] Processing /home/runner/work/typescript-eslint/typescript-eslint/coverage/parser-coverage/lcov.info...
[2022-10-02T09:05:22.891Z] ['info'] Processing /home/runner/work/typescript-eslint/typescript-eslint/coverage/scope-manager-coverage/lcov.info...
[2022-10-02T09:05:22.892Z] ['info'] Processing /home/runner/work/typescript-eslint/typescript-eslint/coverage/type-utils-coverage/lcov.info...
[2022-10-02T09:05:22.892Z] ['info'] Processing /home/runner/work/typescript-eslint/typescript-eslint/coverage/typescript-estree-coverage/lcov.info...
[2022-10-02T09:05:22.893Z] ['info'] Processing /home/runner/work/typescript-eslint/typescript-eslint/coverage/utils-coverage/lcov.info...
[2022-10-02T09:05:22.894Z] ['info'] Processing /home/runner/work/typescript-eslint/typescript-eslint/coverage/visitor-keys-coverage/lcov.info...
[2022-10-02T09:05:22.927Z] ['info'] Detected GitHub Actions as the CI provider.
[2022-10-02T09:05:22.929Z] ['info'] Pinging Codecov: https://codecov.io/upload/v4?package=github-action-3.1.1-uploader-0.3.2&token=*******&branch=member-ordering-natural-order&build=3167939340&build_url=https%3A%2F%2Fgithub.com%2Ftypescript-eslint%2Ftypescript-eslint%2Factions%2Fruns%2F3167939340&commit=ea0da9ec92e74b960bf769d82d2e85733d1fd5dd&job=CI&pr=5662&service=github-actions&slug=typescript-eslint%2Ftypescript-eslint&name=codecov&tag=&flags=unittest&parent=
[2022-10-02T09:05:23.496Z] ['info'] https://app.codecov.io/github/typescript-eslint/typescript-eslint/commit/ea0da9ec92e74b960bf769d82d2e85733d1fd5dd
https://storage.googleapis.com/codecov/v4/raw/2022-10-02/FB67E67CBA915F6C68E09A5DD9FD2D70/ea0da9ec92e74b960bf769d82d2e85733d1fd5dd/f0d34772-f43a-421b-a887-244f936d5d03.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=GOOG1EJOGFN2JQ4OCTGA2MU5AEIT7OT5Z7HTFOAN2SPG4NWSN2UJYOY5U6LZQ%2F20221002%2FUS%2Fs3%2Faws4_request&X-Amz-Date=20221002T090523Z&X-Amz-Expires=10&X-Amz-SignedHeaders=host&X-Amz-Signature=113055635430db28f33d635ccee62c1502e62e9a5352002391f2e608e05985ab
[2022-10-02T09:05:23.498Z] ['info'] Uploading...
[2022-10-02T09:05:23.724Z] ['info'] {"status":"success","resultURL":"https://app.codecov.io/github/typescript-eslint/typescript-eslint/commit/ea0da9ec92e74b960bf769d82d2e85733d1fd5dd"}

Expected Results

Coverage numbers should be accurate to each PR.

Actual Results

They are not.

Additional Information

Sent here from Twitter thanks to the friendly Tom Hu :avocado:: https://twitter.com/thomasrockhu/status/1577417093472985089

the tl;dr is that it seems like the base commit is not uploading ā€œcoverage/eslint-plugin-internal-coverage/lcov.infoā€, while the head of the PR does.

Out of curiosity, how did you determine this? Experience from Codecov, or is there some log, orā€¦?

2

Hi @JoshuaKGoldberg, Iā€™m sorry I lost track of this one. But hereā€™s something I found when digging into this recent PR (#5825).

Clicking into the Codecov report shows us the HEAD and BASE

image
image1484Ɨ342 49.1 KB

If there is a change, then we should see a difference between the two reports. Because this is happening on a lot of files, Iā€™m going to check to see that weā€™re uploading the same things first.

So for the BASE CI run on GitHub, we see only 1 coverage report being uploaded

[2022-10-13T19:16:25.763Z] ['info'] => Found 1 possible coverage files:
  coverage/scope-manager-coverage/lcov.info

while for the HEAD CI run on GitHub, there are 9 coverage reports

[2022-10-14T02:25:18.694Z] ['info'] => Found 9 possible coverage files:
  coverage/eslint-plugin-coverage/lcov.info
  coverage/eslint-plugin-internal-coverage/lcov.info
  coverage/eslint-plugin-tslint-coverage/lcov.info
  coverage/parser-coverage/lcov.info
  coverage/scope-manager-coverage/lcov.info
  coverage/type-utils-coverage/lcov.info
  coverage/typescript-estree-coverage/lcov.info
  coverage/utils-coverage/lcov.info
  coverage/visitor-keys-coverage/lcov.info

So, this gives me a pretty good idea why the coverage is increasing/decreasing. Letā€™s make sure this is right. Iā€™ll pick a file thatā€™s changed from the report, no-extraneous-class.ts

On the BASE commit thereā€™s no coverage! But on the HEAD commit itā€™s fully covered.

This is strange, but that means to me that coverage wasnā€™t uploaded for that file, and it probably exists in one of the 8 coverage files that werenā€™t uploaded. I can check that by looking at the reports uploaded.

BASE reports
image
shows to coverage information, but
HEAD reports shows data

SF:src/rules/no-extraneous-class.ts
FN:16,(anonymous_0)
FN:66,(anonymous_4)
...

this data seems to be coming from path=coverage/eslint-plugin-coverage/lcov.info which is not getting uploaded in BASE.

So, to wrap up, I think somewhere in your CI pipeline, you are not uploading coverage reports to Codecov which is why you are seeing the discrepancy

3

Fantastic, thanks so much for looking into this Tom!

Your pointers were spot on. Weā€™d changed our CI to cache test results, but never added coverage data to the list of files restored from cache. And since PR branches have to be up-to-date with the main branch, the main branch was always failing to upload coverage data. Kind of funny!

I still think this indicates a bug / feature failure in Codecov, though. Weā€™d had this issue for months without any indication on PR comments or codecov.io reports that the coverage data was very out of date. Thatā€™s not a good user experience.

That being said, this thread was quite a good user experience for me :smile:. Thanks again!

1 Like
4

@JoshuaKGoldberg thanks for the feedback here, the main problem to summarize is that it was unclear that the number of coverage files uploaded was not the same. Iā€™ll let the product team know about it.

5

Wait, sorry, just making sure weā€™re on the same page - for me itā€™s that Codecov was comparing coverage numbers against an outdated number. As in, the main branch coverage percentage was computed from not the latest commit to main.

6

@JoshuaKGoldberg ooo looks like we have some confusion. Can you point to a specific example with SHAs?

What I think youā€™re saying is that we are comparing to a base commit that you are not expecting

7

Iā€™m not confident that Iā€™m correct here :smile:, itā€™s possible youā€™re right and Iā€™m blatantly wrongā€¦ But, looking at the codecov report from the OP: Codecov

Coverage data based on HEAD 5e9d356 compared to BASE 9ffdb05

9ffdb05 is the right commit on main.

8

@JoshuaKGoldberg Iā€™m not sure I understand that scenario. 9ffdb05 is the correct BASE and we are showing it too?

9

Greetings to all, We have the same problem with the wrong coverage diff calculation within Codecov Coverage bot.
Codecov Bot always takes the very outdated merge commit from base branch that was made in the last July. As we expect, the coverage should compare the last PR head branch commit and the last base (main, parent) branch merge commit.
We tried to play around with this problem and added some additional options to Codecov YAML config file (like allow_coverage_offsets and allow_pseudo_compare). Also, since we use GitHub Actions for CI, we added commit_parent feature to codecov/codecov-action@v3 Action.
From codecov/codecov-action@v3 logs here only one available report to be uploaded:

Run codecov/codecov-action@v3
  with:
    files: lcov.info
    fail_ci_if_error: true
    commit_parent: 64078706a9ef27763a826916f38b942d9b8dfbdf
    override_pr: 2911
  env:
    CARGO_TERM_COLOR: always
    NIGHTLY: nightly-2022-08-15
    CACHE_ON_FAILURE: false
    CARGO_INCREMENTAL: 0
/usr/bin/docker exec  8983c094317397e3ab9502280426d30f2df7c060e914fb900afcf7e02eb4dc08 sh -c "cat /etc/*release | grep ^ID"
==> linux OS detected
https://uploader.codecov.io/latest/linux/codecov.SHA256SUM
==> SHASUM file signed by key id 806bb28aed779869
==> Uploader SHASUM verified (20f9c9d78483fce977b6cc39e231a734a23bcd36f4d536bb7355222fb88d02bc  codecov)
==> Running version latest
==> Running version v0.3.2
/__w/_actions/codecov/codecov-action/v3/dist/codecov -n  -Q github-action-3.1.1 -N 64078706a9ef27763a826916f38b942d9b8dfbdf -Z -f lcov.info -C 743769678615c44d85e454d50dd3f470d3a13f27 -P 2911
[2022-10-25T13:12:12.463Z] ['info'] 
     _____          _
    / ____|        | |
   | |     ___   __| | ___  ___ _____   __
   | |    / _ \ / _` |/ _ \/ __/ _ \ \ / /
   | |___| (_) | (_| |  __/ (_| (_) \ V /
    \_____\___/ \__,_|\___|\___\___/ \_/

  Codecov report uploader 0.3.2
[2022-10-25T13:12:12.475Z] ['info'] => Project root located at: /__w/iroha/iroha
[2022-10-25T13:12:12.483Z] ['info'] -> No token specified or token is empty
[2022-10-25T13:12:13.440Z] ['info'] Searching for coverage files...
[2022-10-25T13:12:13.798Z] ['info'] => Found 1 possible coverage files:
  lcov.info
[2022-10-25T13:12:13.798Z] ['info'] Processing /__w/iroha/iroha/lcov.info...
[2022-10-25T13:12:14.015Z] ['info'] Detected GitHub Actions as the CI provider.
[2022-10-25T13:12:14.017Z] ['info'] Pinging Codecov: https://codecov.io/upload/v4?package=github-action-3.1.1-uploader-0.3.2&token=*******&branch=feature%2FDOPS-2045%2Fcargo-caching&build=3320982669&build_url=https%3A%2F%2Fgithub.com%2Fhyperledger%2Firoha%2Factions%2Fruns%2F3320982669&commit=743769678615c44d85e454d50dd3f470d3a13f27&job=I2%3A%3ADev%3A%3ATests&pr=2911&service=github-actions&slug=hyperledger%2Firoha&name=&tag=&flags=&parent=64078706a9ef27763a826916f38b942d9b8dfbdf
[2022-10-25T13:12:14.803Z] ['info'] https://app.codecov.io/github/hyperledger/iroha/commit/743769678615c44d85e454d50dd3f470d3a13f27
https://storage.googleapis.com/codecov/v4/raw/2022-10-25/B5A7D58AA6C150C0DA3976D28D834AFB/743769678615c44d85e454d50dd3f470d3a13f27/5df76c5e-45cc-456a-b91a-2ce68defbc67.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=GOOG1EJOGFN2JQ4OCTGA2MU5AEIT7OT5Z7HTFOAN2SPG4NWSN2UJYOY5U6LZQ%2F20221025%2FUS%2Fs3%2Faws4_request&X-Amz-Date=20221025T131214Z&X-Amz-Expires=10&X-Amz-SignedHeaders=host&X-Amz-Signature=926f4e7095ae344a2d47c0056107a86cfbcb40e4df4d7afddd886781f196b643
[2022-10-25T13:12:14.804Z] ['info'] Uploading...
[2022-10-25T13:12:15.227Z] ['info'] {"status":"success","resultURL":"https://app.codecov.io/github/hyperledger/iroha/commit/743769678615c44d85e454d50dd3f470d3a13f27"}

Codecov report link from the particular PR.

Here is the example of the wrong base branch commit taking by Coverage Bot.

Can anyone provide an information how it could be fixed? Thank you.

10

@BAStos525 my guess is that you are making commits from a fork that is really behind the upstream repo. Thatā€™s probably why it needs to go so far back to find a commit that exists or that has coverage.

11

Thank you for the answer. Actually if we are talking about the main branch - itā€™s true. But the thing here that we create a PRs to the iroha2-dev branch specifically. A branch from fork is always rebased against the particular base upstream branch before to create a PR to it. Head and base branches are different from each other only by PR commits(s).
Moreover, since we make a commits to the upstream repository from forks for a long time, we have noticed this issue relatively recently.

12

Hi @tom! Iā€™m also seeing this issue. We disabled codecov a year ago due to the security issue, and weā€™re reenabling with the new uploader. Codecov is comparing to a report from last year (back when codecov was enabled) instead of the proper base on main. Weā€™re not on a fork or using any flags.
BASE commit chosed by codecov and the HEAD commit. Codecov is showing recent pulls but not their results commits to main.

Maybe this issue is unrelated to the original post (please let me know if I should create a separate ticket!).

13

(Responded here)