Wrong coverage number in badge with flags

Support
,
1

Before submitting a topic, please confirm the following

I have searched for similar issues before creating this topic.
I have verified that my repository is using the Codecov GitHub app, if using GitHub
I have validated my codecov.yaml configuration file.
I have filled out the below sections to the best of my ability.

Description

In our project:

we use codecov already since quite some time but have recently added the coverage result of mypy (static type checking coverage) using flags.
Now, we have two flags currently active: unittests and mypy

In the browser overview under flags (Codecov) I can see the correct coverage results of ~90% and ~40%, respectively. In the main overview (Codecov) I get ~68%, so far so good.

The problem now is that the badges don’t seem to work.

CI/CD URL

This is a CI run of a PR that tried to fix this issue:

But also happens on the main branch…

Codecov Output

     _____          _
    / ____|        | |
   | |     ___   __| | ___  ___ _____   __
   | |    / _ \ / _` |/ _ \/ __/ _ \ \ / /
   | |___| (_) | (_| |  __/ (_| (_) \ V /
    \_____\___/ \__,_|\___|\___\___/ \_/

  Codecov report uploader 0.3.2
[2022-11-18T16:27:41.879Z] ['info'] => Project root located at: /home/runner/work/xarray/xarray
[2022-11-18T16:27:41.885Z] ['info'] -> No token specified or token is empty
[2022-11-18T16:27:41.895Z] ['info'] Searching for coverage files...
[2022-11-18T16:27:41.961Z] ['info'] => Found 1 possible coverage files:
  ./coverage.xml
[2022-11-18T16:27:41.962Z] ['info'] Processing ./coverage.xml...
[2022-11-18T16:27:41.977Z] ['info'] Detected GitHub Actions as the CI provider.
[2022-11-18T16:27:41.980Z] ['info'] Pinging Codecov: https://codecov.io/upload/v4?package=github-action-3.1.1-uploader-0.3.2&token=*******&branch=codecov&build=3498487060&build_url=https%3A%2F%2Fgithub.com%2Fpydata%2Fxarray%2Factions%2Fruns%2F3498487060&commit=ad5d35b74585cd352b73d684252ebd9feeff7b46&job=CI&pr=7142&service=github-actions&slug=pydata%2Fxarray&name=codecov-umbrella&tag=&flags=unittests&parent=
[2022-11-18T16:27:42.717Z] ['info'] https://app.codecov.io/github/pydata/xarray/commit/ad5d35b74585cd352b73d684252ebd9feeff7b46
https://storage.googleapis.com/codecov/v4/raw/2022-11-18/774C80CCC165E4FFB9396325BF84EB05/ad5d35b74585cd352b73d684252ebd9feeff7b46/f9635bcb-d3f0-4057-a433-6dd16690ad91.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=GOOG1EJOGFN2JQ4OCTGA2MU5AEIT7OT5Z7HTFOAN2SPG4NWSN2UJYOY5U6LZQ%2F20221118%2FUS%2Fs3%2Faws4_request&X-Amz-Date=20221118T162742Z&X-Amz-Expires=10&X-Amz-SignedHeaders=host&X-Amz-Signature=3ac7735b27a93731087159d2307ae331baac8249cd20caf5200dc5b130838b44
[2022-11-18T16:27:42.718Z] ['info'] Uploading...
[2022-11-18T16:27:42.909Z] ['info'] {"status":"success","resultURL":"https://app.codecov.io/github/pydata/xarray/commit/ad5d35b74585cd352b73d684252ebd9feeff7b46"}

Expected Results

I expected that the badge with the unittests flag displays ~90%.

Actual Results

If I use a badge without a flag (https://codecov.io/gh/pydata/xarray/branch/main/graph/badge.svg) it only displays 56%.
If I use the badge with the unittests flag (https://codecov.io/gh/pydata/xarray/branch/main/graph/badge.svg?flag=unittests) I get 68%, which strangely is the average percentage in the overview?.
If I use the badge with the mypy flag (https://codecov.io/gh/pydata/xarray/branch/main/graph/badge.svg?flag=mypy) I get 38%, which is weirdly correct.

Additional Information

Maybe it is an issue that the unittests are part of the package but they are type checked. Meaning the coverage of these files is only given with the mypy flag and not with the unittests flag…

2

@mick there seems to be a mismatch between the branch you’re showing above main and the default branch on Codecov master. I would try changing your default branch first and seeing if you are getting the right badge values.

3

We have changed the default branch to main, now it matches GitHub.
I’m still seeing the same results though :frowning:

4

@mick, this got away from me a little bit, is this still a problem for you?

5

Yes, we still couldn’t figure it out. Our next “solution” would be to undo the changes and stop uploading the Mypy coverage.

6

@mick, ok I dug into this a bit. Let me clarify a few things.

  1. The badge shows the current coverage on a particular branch for the latest commit.
  2. The flag trends display in the UI shows the average coverage on a particular flag over the day.

So in this case (and at the time of this writing), I’m seeing this commit as the latest on main. I don’t have the tooling to investigate why the unittests flag is showing as 37% on the badge, but given that the coverage for this commit is 36.66%, it makes sense to me.

Calling the Codecov API, I’m seeing that the coverage trend for the main branch is showing 60.31%, which reflects in the Codecov UI.

    {
      "timestamp": "2023-08-23T00:00:00Z",
      "min": 36.56395,
      "max": 92.73322,
      "avg": 60.31205
    }

image
Source

This is the call

curl --request GET \
     --url 'https://api.codecov.io/api/v2/github/pydata/repos/xarray/flags/unittests/coverage/?branch=main&interval=1d&page=9&page_size=100' \
     --header 'accept: application/json' \
     --header 'authorization: Bearer {{CODECOV_API_TOKEN}}'

It is worth noting that the min is 36.56% here.

The tl;dr here is that the UI is showing the average trend by day, while the badge is showing the latest commit.

7

Thanks for digging into this. The main problem still is the same, the unit test coverage should be ~95%, while the Mypy coverage should be ~35%.
Somehow the splitting into the flags does not seems to work properly.

I have the suspicion that part of the problem is that the test directory is covered by Mypy but not by unit test (obviously because the tests don’t test themself). Maybe excluding this on one flag only does not work?

8

@mick, I’m not really following the problem here. What are you expecting the coverage to be and is it on the UI or the badge?

9

Now I am really confused…
Without changing anything the coverage increased to the expected 93% two weeks ago. However, no change was made that should reflect such a change?
It is still very unstable (sometimes it reports 37%, which is the mypy coverage. Maybe this is a race condition?).

10

@mick without digging too much into your CI pipeline, if you want to make sure that the xarray/tests/ directory is excluded from coverage results, you can always do

ignore:
  - "xarray/tests"

in the codecov.yml file

11

I’d also like to point out that there has been no coverage report with the mypy flag in the past 14 days on the main branch.