Codecov reporting coverage that is nonsensical

Support Bug Fixes
1

Description

Codecov is reporting coverage that is nonsensical because it has lines like comment report as missing.

CI/CD URL

Codecov Output

Expected Results

This file should show as 100% coverage.

Actual Results

It shows as 15% coverage.

2

I should note, this is happening on many builds. This has impacted the last 3 builds on main.

3

@alex can you share the CI runs between a 100% and not 100% covered instance? In particular, did you change anything in your setup like the uploader you are using?

4

Nope, no change. You can see our commit history here: Commits · pyca/cryptography · GitHub the commit that this started with had no behavioral changes at all, it was a new github action workflow.

5

This appears to be impacting every single build at this point. Did codecov deploy a change yesterday?

6

@alex Looking at the commits on Codecov, the last commit at 100% is 311a7d9 and the first one to have not 100% is 924926d.

You’ll find on the builds tab the build named tox -e backend-import boringssl-1285d5305ad69ceb519de76cd74e743aed1efd89-shared no-ssl2 no-ssl3

For 311a7d9, you’ll see

<class name="utils.py" filename="tests/utils.py" complexity="0" line-rate="0" branch-rate="0">
    <methods/>
    <lines>
        ...
        <line number="14" hits="0"/>

and 924926d shows

<class name="utils.py" filename="utils.py" complexity="0" line-rate="0" branch-rate="0">
    <methods/>
    <lines>
        ...
        <line number="14" hits="0"/>

You’ll see that showing line 14 is one of the lines showing as uncovered (and is blank anyways)
image

The difference between the two is in the file names. The first has test prepended, while the 2nd doesn’t. This means the file is ambiguous and we do our best to try to guess which file it belongs to.

Did you update how you collect coverage or upgrade any dependencies? This seems to be an error in your coverage package.

7

Thanks Tom, this debugging was incredibly helpful.

Based on what I can see, I think the following happened:

Based on the fact that the build where we added the new job was not the one where the coverage started being incorrect, I think codecov is not deterministic (perhaps with respect to which jobs finish first?), is that right?

Is it possible to disable codecov’s guessing algorithm? We’d have much preferred if the PR where this job was added simply had failing codecov, rather than this non-deterministic missing coverage.

8

@alex I’ll be honest, it SHOULD be deterministic, but there were too many uploads for me to parse through. More so, I wasn’t if you uploaded one coverage report per upload or multiple.

If this is something we want to get to the bottom of, I can try uploading that coverage report on a fork and track down why that file is getting wonky. Let me know if that’s interesting enough to you.

9

At this point, thanks to your help, I think we’ve nailed this down and our codecov situation is back to normal.

I’m not sure further debugging of this is valuable, however I do still think an option to disable the heuristics would be valuable.

1 Like