# Wrong coverage number in badge with flags

**URL:** <https://community.codecov.com/t/wrong-coverage-number-in-badge-with-flags/3975>\
**Category:** Support\
**Tags:** python, github\
**Created:** [November 18, 2022, 4:41pm UTC](https://community.codecov.com/t/wrong-coverage-number-in-badge-with-flags/3975 "2022-11-18T16:41:06Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![mick](https://yyz1.discourse-cdn.com/flex029/user_avatar/community.codecov.com/mick/32/2671_2.png) [@mick](https://community.codecov.com/u/mick)\
**Post date:** [November 18, 2022, 4:41pm UTC](https://community.codecov.com/t/wrong-coverage-number-in-badge-with-flags/3975/1 "2022-11-18T16:41:06Z")

</div>

## Before submitting a topic, please confirm the following

I have searched for similar issues before creating this topic.  
 I have verified that my repository is using the [Codecov GitHub app](https://github.com/apps/codecov), if using GitHub  
 I have [validated](https://docs.codecov.com/docs/codecov-yaml#validate-your-repository-yaml) my `codecov.yaml` configuration file.  
 I have filled out the below sections to the best of my ability.

### Description

In our project:

> **[GitHub - pydata/xarray: N-D labeled arrays and datasets in Python](https://github.com/pydata/xarray)**
>
> N-D labeled arrays and datasets in Python

> **[Code coverage done right.](https://app.codecov.io/gh/pydata/xarray/)**
>
> Hosted coverage report highly integrated with GitHub, Bitbucket and GitLab. Awesome pull request comments to enhance your QA.

we use codecov already since quite some time but have recently added the coverage result of mypy (static type checking coverage) using flags.  
Now, we have two flags currently active: unittests and mypy

In the browser overview under flags ([Codecov](https://app.codecov.io/gh/pydata/xarray/flags)) I can see the correct coverage results of ~90% and ~40%, respectively. In the main overview ([Codecov](https://app.codecov.io/gh/pydata/xarray/)) I get ~68%, so far so good.

The problem now is that the badges don’t seem to work.

### CI/CD URL

This is a CI run of a PR that tried to fix this issue:  
[https://github.com/pydata/xarray/actions/runs/3498487060/jobs/5858915980](https://github.com/pydata/xarray/actions/runs/3498487060/jobs/5858915980)  
But also happens on the main branch…

### Codecov Output

```auto
     _____ _
    / ____ | | |
   | | _____ | | ______  _______
   | | / _ \ / _` |/ _ \/ __/ _ \ \ / /
   | |___| (_) | (_| |__/ (_| (_) \ V /
    \ _____\___ / \ __,_|\___ |\ ___\___ / \_/

  Codecov report uploader 0.3.2
[2022-11-18T16:27:41.879Z] ['info'] => Project root located at: /home/runner/work/xarray/xarray
[2022-11-18T16:27:41.885Z] ['info'] -> No token specified or token is empty
[2022-11-18T16:27:41.895Z] ['info'] Searching for coverage files...
[2022-11-18T16:27:41.961Z] ['info'] => Found 1 possible coverage files:
  ./coverage.xml
[2022-11-18T16:27:41.962Z] ['info'] Processing ./coverage.xml...
[2022-11-18T16:27:41.977Z] ['info'] Detected GitHub Actions as the CI provider.
[2022-11-18T16:27:41.980Z] ['info'] Pinging Codecov: https://codecov.io/upload/v4?package=github-action-3.1.1-uploader-0.3.2&token= ******* &branch=codecov&build=3498487060&build_url=https%3A%2F%2Fgithub.com%2Fpydata%2Fxarray%2Factions%2Fruns%2F3498487060&commit=ad5d35b74585cd352b73d684252ebd9feeff7b46&job=CI&pr=7142&service=github-actions&slug=pydata%2Fxarray&name=codecov-umbrella&tag=&flags=unittests&parent=
[2022-11-18T16:27:42.717Z] ['info'] https://app.codecov.io/github/pydata/xarray/commit/ad5d35b74585cd352b73d684252ebd9feeff7b46
https://storage.googleapis.com/codecov/v4/raw/2022-11-18/774C80CCC165E4FFB9396325BF84EB05/ad5d35b74585cd352b73d684252ebd9feeff7b46/f9635bcb-d3f0-4057-a433-6dd16690ad91.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=GOOG1EJOGFN2JQ4OCTGA2MU5AEIT7OT5Z7HTFOAN2SPG4NWSN2UJYOY5U6LZQ%2F20221118%2FUS%2Fs3%2Faws4_request&X-Amz-Date=20221118T162742Z&X-Amz-Expires=10&X-Amz-SignedHeaders=host&X-Amz-Signature=3ac7735b27a93731087159d2307ae331baac8249cd20caf5200dc5b130838b44
[2022-11-18T16:27:42.718Z] ['info'] Uploading...
[2022-11-18T16:27:42.909Z] ['info'] {"status":"success","resultURL":"https://app.codecov.io/github/pydata/xarray/commit/ad5d35b74585cd352b73d684252ebd9feeff7b46"}

```

### Expected Results

I expected that the badge with the unittests flag displays ~90%.

### Actual Results

If I use a badge without a flag ([https://codecov.io/gh/pydata/xarray/branch/main/graph/badge.svg](https://codecov.io/gh/pydata/xarray/branch/main/graph/badge.svg)) it only displays 56%.  
If I use the badge with the unittests flag ([https://codecov.io/gh/pydata/xarray/branch/main/graph/badge.svg?flag=unittests](https://codecov.io/gh/pydata/xarray/branch/main/graph/badge.svg?flag=unittests)) I get 68%, which strangely is the average percentage in the overview?.  
If I use the badge with the mypy flag ([https://codecov.io/gh/pydata/xarray/branch/main/graph/badge.svg?flag=mypy](https://codecov.io/gh/pydata/xarray/branch/main/graph/badge.svg?flag=mypy)) I get 38%, which is weirdly correct.

### Additional Information

Maybe it is an issue that the unittests are part of the package but they are type checked. Meaning the coverage of these files is only given with the mypy flag and not with the unittests flag…

---

<div class="post-metadata">

**Author:** ![tom](https://yyz1.discourse-cdn.com/flex029/user_avatar/community.codecov.com/tom/32/8_2.png) [@tom](https://community.codecov.com/u/tom)\
**Post date:** [November 19, 2022, 4:10am UTC](https://community.codecov.com/t/wrong-coverage-number-in-badge-with-flags/3975/2 "2022-11-19T04:10:34Z")

</div>

@mick there seems to be a mismatch between the branch you’re showing above `main` and the default branch on Codecov `master`. I would try changing your default branch first and seeing if you are getting the right badge values.

---

<div class="post-metadata">

**Author:** ![mick](https://yyz1.discourse-cdn.com/flex029/user_avatar/community.codecov.com/mick/32/2671_2.png) [@mick](https://community.codecov.com/u/mick)\
**Post date:** [November 22, 2022, 3:31pm UTC](https://community.codecov.com/t/wrong-coverage-number-in-badge-with-flags/3975/3 "2022-11-22T15:31:04Z")

</div>

We have changed the default branch to main, now it matches GitHub.  
I’m still seeing the same results though ☹

---

<div class="post-metadata">

**Author:** ![tom](https://yyz1.discourse-cdn.com/flex029/user_avatar/community.codecov.com/tom/32/8_2.png) [@tom](https://community.codecov.com/u/tom)\
**Post date:** [August 23, 2023, 2:54am UTC](https://community.codecov.com/t/wrong-coverage-number-in-badge-with-flags/3975/4 "2023-08-23T02:54:12Z")

</div>

@mick, this got away from me a little bit, is this still a problem for you?

---

<div class="post-metadata">

**Author:** ![mick](https://yyz1.discourse-cdn.com/flex029/user_avatar/community.codecov.com/mick/32/2671_2.png) [@mick](https://community.codecov.com/u/mick)\
**Post date:** [August 23, 2023, 6:31am UTC](https://community.codecov.com/t/wrong-coverage-number-in-badge-with-flags/3975/5 "2023-08-23T06:31:59Z")

</div>

Yes, we still couldn’t figure it out. Our next “solution” would be to undo the changes and stop uploading the Mypy coverage.

---

<div class="post-metadata">

**Author:** ![tom](https://yyz1.discourse-cdn.com/flex029/user_avatar/community.codecov.com/tom/32/8_2.png) [@tom](https://community.codecov.com/u/tom)\
**Post date:** [August 24, 2023, 1:12am UTC](https://community.codecov.com/t/wrong-coverage-number-in-badge-with-flags/3975/6 "2023-08-24T01:12:29Z")

</div>

@mick, ok I dug into this a bit. Let me clarify a few things.

1. The badge shows the **current** coverage on a particular branch for the latest commit.
2. The flag trends display in the UI shows the **average** coverage on a particular flag over the day.

So in this case (and at the time of this writing), I’m seeing [this commit](https://app.codecov.io/gh/pydata/xarray/commit/83c2919b27b4b2d8a01bfa380226134c71321aa0) as the latest on main. I don’t have the tooling to investigate why the `unittests` flag is showing as 37% on the badge, but given that the coverage for this commit is 36.66%, it makes sense to me.

Calling the [Codecov API](https://docs.codecov.com/reference/overview), I’m seeing that the coverage trend for the `main` branch is showing 60.31%, which reflects in the Codecov UI.

```auto
    {
      "timestamp": "2023-08-23T00:00:00Z",
      "min": 36.56395,
      "max": 92.73322,
      "avg": 60.31205
    }

```

![image](https://canada1.discourse-cdn.com/flex029/uploads/codecov/original/2X/f/fe24cd95826bdfecf1142ae884263e37f44a06c7.png)  
[Source](https://app.codecov.io/gh/pydata/xarray/flags?historicalTrend=LAST_7_DAYS&flags%5B0%5D=unittests)

This is the call

```auto
curl --request GET \
     --url 'https://api.codecov.io/api/v2/github/pydata/repos/xarray/flags/unittests/coverage/?branch=main&interval=1d&page=9&page_size=100' \
     --header 'accept: application/json' \
     --header 'authorization: Bearer {{CODECOV_API_TOKEN}}'

```

It is worth noting that the `min` is 36.56% here.

* * *

The tl;dr here is that the UI is showing the average trend by day, while the badge is showing the latest commit.

---

<div class="post-metadata">

**Author:** ![mick](https://yyz1.discourse-cdn.com/flex029/user_avatar/community.codecov.com/mick/32/2671_2.png) [@mick](https://community.codecov.com/u/mick)\
**Post date:** [August 24, 2023, 7:15am UTC](https://community.codecov.com/t/wrong-coverage-number-in-badge-with-flags/3975/7 "2023-08-24T07:15:30Z")

</div>

Thanks for digging into this. The main problem still is the same, the unit test coverage should be ~95%, while the Mypy coverage should be ~35%.  
Somehow the splitting into the flags does not seems to work properly.

I have the suspicion that part of the problem is that the test directory is covered by Mypy but not by unit test (obviously because the tests don’t test themself). Maybe excluding this on one flag only does not work?

---

<div class="post-metadata">

**Author:** ![tom](https://yyz1.discourse-cdn.com/flex029/user_avatar/community.codecov.com/tom/32/8_2.png) [@tom](https://community.codecov.com/u/tom)\
**Post date:** [August 29, 2023, 10:23pm UTC](https://community.codecov.com/t/wrong-coverage-number-in-badge-with-flags/3975/8 "2023-08-29T22:23:41Z")

</div>

@mick, I’m not really following the problem here. What are you expecting the coverage to be and is it on the UI or the badge?

---

<div class="post-metadata">

**Author:** ![mick](https://yyz1.discourse-cdn.com/flex029/user_avatar/community.codecov.com/mick/32/2671_2.png) [@mick](https://community.codecov.com/u/mick)\
**Post date:** [August 30, 2023, 6:49pm UTC](https://community.codecov.com/t/wrong-coverage-number-in-badge-with-flags/3975/9 "2023-08-30T18:49:13Z")

</div>

Now I am really confused…  
Without changing anything the coverage increased to the expected 93% two weeks ago. However, no change was made that should reflect such a change?  
It is still very unstable (sometimes it reports 37%, which is the mypy coverage. Maybe this is a race condition?).

---

<div class="post-metadata">

**Author:** ![tom](https://yyz1.discourse-cdn.com/flex029/user_avatar/community.codecov.com/tom/32/8_2.png) [@tom](https://community.codecov.com/u/tom)\
**Post date:** [August 31, 2023, 8:44pm UTC](https://community.codecov.com/t/wrong-coverage-number-in-badge-with-flags/3975/10 "2023-08-31T20:44:44Z")

</div>

@mick without digging too much into your CI pipeline, if you want to make sure that the `xarray/tests/` directory is excluded from coverage results, you can always do

```auto
ignore:
  - "xarray/tests"

```

in the `codecov.yml` file

---

<div class="post-metadata">

**Author:** ![tom](https://yyz1.discourse-cdn.com/flex029/user_avatar/community.codecov.com/tom/32/8_2.png) [@tom](https://community.codecov.com/u/tom)\
**Post date:** [August 31, 2023, 8:45pm UTC](https://community.codecov.com/t/wrong-coverage-number-in-badge-with-flags/3975/11 "2023-08-31T20:45:43Z")

</div>

I’d also like to point out that there has been no coverage report with the `mypy` flag in the past 14 days on the `main` branch.
