Coverage is comparing with incorrect base branch

Before submitting a topic, please confirm the following

I have searched for similar issues before creating this topic.
I have verified that my repository is using the Codecov GitHub app, if using GitHub
I have validated my codecov.yaml configuration file. (no yaml is used)
I have filled out the below sections to the best of my ability.

Description

The coverage report shows the incorrect coverage on the pull requests due to the coverage is comparing to incorrect base commit.

The incorrect coverage is occurred in multiple pull requests.

After checking the basic commit on the Github, it shows that This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.

CI/CD URL

Codecov Output

/usr/bin/docker exec  1c432ee65d1f0425013e16c108454f996084010e510a9395bda27ed4dabc5008 sh -c "cat /etc/*release | grep ^ID"
==> linux OS detected
https://uploader.codecov.io/latest/linux/codecov.SHA256SUM
==> SHASUM file signed by key id 806bb28aed779869
==> Uploader SHASUM verified (20f9c9d78483fce977b6cc39e231a734a23bcd36f4d536bb7355222fb88d02bc  codecov)
==> Running version latest
==> Running version v0.3.2
/__w/_actions/codecov/codecov-action/v3/dist/codecov -n  -Q github-action-3.1.1 -f ./coverage.out -C a5dfbeb015d28d8b900d29076c8fb07414c98ec7
[2023-02-13T07:21:57.259Z] ['info'] 
     _____          _
    / ____|        | |
   | |     ___   __| | ___  ___ _____   __
   | |    / _ \ / _` |/ _ \/ __/ _ \ \ / /
   | |___| (_) | (_| |  __/ (_| (_) \ V /
    \_____\___/ \__,_|\___|\___\___/ \_/

  Codecov report uploader 0.3.2
[2023-02-13T07:21:57.277Z] ['info'] => Project root located at: /__w/vald/vald
[2023-02-13T07:21:57.278Z] ['info'] ->  Token found by environment variables
[2023-02-13T07:21:57.291Z] ['info'] Searching for coverage files...
[2023-02-13T07:21:57.438Z] ['info'] => Found 1 possible coverage files:
  ./coverage.out
[2023-02-13T07:21:57.442Z] ['info'] Processing ./coverage.out...
[2023-02-13T07:21:57.468Z] ['info'] Using manual override from args.
[2023-02-13T07:21:57.469Z] ['info'] Detected GitHub Actions as the CI provider.
[2023-02-13T07:21:57.471Z] ['info'] Pinging Codecov: https://codecov.io/upload/v4?package=github-action-3.1.1-uploader-0.3.2&token=*******&branch=refactor%2Fremove-empty-test-stub&build=4159390414&build_url=https%3A%2F%2Fgithub.com%2Fvdaas%2Fvald%2Factions%2Fruns%2F4159390414&commit=a5dfbeb015d28d8b900d29076c8fb07414c98ec7&job=Coverage&pr=1950&service=github-actions&slug=vdaas%2Fvald&name=&tag=&flags=&parent=
[2023-02-13T07:21:58.037Z] ['info'] https://app.codecov.io/github/vdaas/vald/commit/a5dfbeb015d28d8b900d29076c8fb07414c98ec7
https://storage.googleapis.com/codecov/v4/raw/2023-02-13/D0FF395BC3712B41960AC7811420E6DB/a5dfbeb015d28d8b900d29076c8fb07414c98ec7/c099fd9b-0dfd-4924-8215-301def08a4c8.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=GOOG1EJOGFN2JQ4OCTGA2MU5AEIT7OT5Z7HTFOAN2SPG4NWSN2UJYOY5U6LZQ%2F20230213%2FUS%2Fs3%2Faws4_request&X-Amz-Date=20230213T072157Z&X-Amz-Expires=10&X-Amz-SignedHeaders=host&X-Amz-Signature=1284cc211b04a67b8e301a1c05f258a8a1e58fef0771545f2ec9e0ce9a91f6b4
[2023-02-13T07:21:58.037Z] ['info'] Uploading...
[2023-02-13T07:21:58.238Z] ['info'] {"status":"success","resultURL":"https://app.codecov.io/github/vdaas/vald/commit/a5dfbeb015d28d8b900d29076c8fb07414c98ec7"}

Expected Results

The coverage of the pull request should compare to the commit of the base branch of the pull request.

Actual Results

The coverage of the pull request is comparing to the incorrect commit.

Additional Information

The parent of the incorrect base commit, is actually belongs to the PR/commit which is already merged to the main branch, but we are not sure about how is it merged to the PR/commit.

@kevindiu can you provide the SHA that you expect it to be comparing to?

Hi @tom
Thank you for your reply.
I expect it to compare with the main branch, which is 96afa5afbc7a9d7ebd0bcd97ca34533977b68fc9.

Got it, it is comparing to an earlier commit than the one expected, because that is the last commit on main where we have coverage. You can see that here. There are no commits on main that have coverage in the past 6 months.

Thank you @tom !!!
I will find out why the coverage is not updated on the main branch for a while, and I may ask if I have further questions, thank you!!!

1 Like

We have 1 PR merged to main branch recently, and the coverage is updated on main branch.

I think the cause of not updating the coverage on main branch is that we forgot to update the default branch setting here when we update the main branch name before.
I will keep update if we find anything new, thank you :slight_smile:

1 Like

@kevindiu ahh! Nice catch. Let me know if anything else is wrong!