Codecov reporting old coverage data for new builds in a PR


Codecov seems to be using old data on this PR (from earlier in the PR) despite new data seemingly being uploaded successfully.

2020-07-29T03:54:32.3813490Z /__t/Python/3.8.3/x64/lib/python3.8/site-packages/qtrio/                                   21      0      4      0   100.0%

The log says to see reports at:

The Ubuntu 3.8 PySide2 build data shows two hits="0" for the file.




bash: bash -n ‘Ubuntu (3.8, PySide2)’

Commit SHAs


Codecov YAML

I do not have a Codecov YAML file

Codecov Output

2020-07-29T03:54:32.5085766Z + bash -n 'Ubuntu (3.8, PySide2)'
2020-07-29T03:54:32.5140591Z   _____          _
2020-07-29T03:54:32.5140720Z  / ____|        | |
2020-07-29T03:54:32.5140848Z | |     ___   __| | ___  ___ _____   __
2020-07-29T03:54:32.5140982Z | |    / _ \ / _` |/ _ \/ __/ _ \ \ / /
2020-07-29T03:54:32.5141109Z | |___| (_) | (_| |  __/ (_| (_) \ V /
2020-07-29T03:54:32.5141227Z  \_____\___/ \__,_|\___|\___\___/ \_/
2020-07-29T03:54:32.5141756Z                               Bash-20200707-353aa93
2020-07-29T03:54:32.5153639Z e[0;90m==>e[0m GitHub Actions detected.
2020-07-29T03:54:32.5254178Z     e[0;90mproject root:e[0m /__w/qtrio/qtrio
2020-07-29T03:54:32.5348779Z     e[0;32mYaml not found, that's ok! Learn more ate[0m e[0;36m[0m
2020-07-29T03:54:32.5676268Z e[0;90m==>e[0m Running gcov in /__w/qtrio/qtrio e[0;90m(disable via -X gcov)e[0m
2020-07-29T03:54:32.5730847Z e[0;90m==>e[0m Python coveragepy exists e[0;90mdisable via -X coveragepye[0m
2020-07-29T03:54:32.5785807Z     e[0;90m->e[0m Running coverage xml
2020-07-29T03:54:32.8746842Z e[0;90m==>e[0m Searching for coverage reports in:
2020-07-29T03:54:32.8747253Z     e[0;32m+e[0m /__w/qtrio/qtrio
2020-07-29T03:54:32.8855258Z     e[0;90m->e[0m Found 1 reports
2020-07-29T03:54:32.8855594Z e[0;90m==>e[0m Detecting git/mercurial file structure
2020-07-29T03:54:32.8934477Z e[0;90m==>e[0m Reading reports
2020-07-29T03:54:32.9002199Z     e[0;32m+e[0m /__w/qtrio/qtrio/empty/coverage.xml e[0;90mbytes=40042e[0m
2020-07-29T03:54:32.9035112Z e[0;90m==>e[0m Appending adjustments
2020-07-29T03:54:32.9035514Z     e[0;36m[0m
2020-07-29T03:54:32.9232007Z     e[0;90m->e[0m No adjustments found
2020-07-29T03:54:32.9238558Z e[0;90m==>e[0m Gzipping contents
2020-07-29T03:54:32.9297584Z e[0;90m==>e[0m Uploading reports
2020-07-29T03:54:32.9297964Z     e[0;90murl:e[0m
2020-07-29T03:54:32.9298571Z     e[0;90mquery:e[0m branch=test_timeout_parameter&commit=74253edd4ea5cb6e04ab29bd25648791d282a520&build=186608969&
2020-07-29T03:54:32.9339474Z e[0;90m->e[0m  Pinging Codecov
2020-07-29T03:54:33.3253194Z e[0;90m->e[0m  Uploading to
2020-07-29T03:54:33.3335899Z   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
2020-07-29T03:54:33.3336656Z                                  Dload  Upload   Total   Spent    Left  Speed
2020-07-29T03:54:33.5420559Z   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
2020-07-29T03:54:33.5421376Z 100  3527    0     0  100  3527      0  16956 --:--:-- --:--:-- --:--:-- 16956
2020-07-29T03:54:33.5473316Z     e[0;32m->e[0m View reports at e[0;36m[0m

Hi @altendky, just some questions to clarify. You are saying this file has incorrect coverage on it based on an older report. Can you specify between which commit SHAs you see the issue? What do you expect the coverage to be?

Hi Tom, Thanks for the quick reply. Yes, qtrio/ is the file of interest. Specifically the lines with ... in the @typing.overload decorated functions. They simply will never be run so they won’t be covered But… :]

That commit configures coverage to exclude the functions. In the gist you can see that coverage is reporting, at least via stdout, that qtrio/ has no misses.

You asked for the pair of SHAs. This is the parent of the above.

I’m getting pretty confused though as I created new PRs including fully rebasing to get entirely new commit hashes for all commits and that still has the same issue.

Alrighty, I added an explicit coverage xml call and printed out the contents and what I see is a discrepancy in the two coverage generated outputs.

2020-07-30T13:35:28.7722736Z /__t/Python/3.8.5/x64/lib/python3.8/site-packages/qtrio/                                   21      0      4      0   100.0%
2020-07-30T13:35:29.1352239Z 				<class name="" filename="/__t/Python/3.8.5/x64/lib/python3.8/site-packages/qtrio/" complexity="0" line-rate="0.9259" branch-rate="1">
2020-07-30T13:35:29.1352386Z 					<methods/>
2020-07-30T13:35:29.1352489Z 					<lines>
2020-07-30T13:35:29.1352606Z 						<line number="3" hits="1"/>
2020-07-30T13:35:29.1352727Z 						<line number="4" hits="1"/>
2020-07-30T13:35:29.1352846Z 						<line number="6" hits="1"/>
2020-07-30T13:35:29.1352963Z 						<line number="7" hits="1"/>
2020-07-30T13:35:29.1353083Z 						<line number="8" hits="1"/>
2020-07-30T13:35:29.1353200Z 						<line number="9" hits="1"/>
2020-07-30T13:35:29.1353304Z 						<line number="11" hits="1"/>
2020-07-30T13:35:29.1353425Z 						<line number="14" hits="1"/>
2020-07-30T13:35:29.1353543Z 						<line number="15" hits="1"/>
2020-07-30T13:35:29.1353667Z 						<line number="18" hits="0"/>
2020-07-30T13:35:29.1353785Z 						<line number="21" hits="1"/>
2020-07-30T13:35:29.1353904Z 						<line number="22" hits="1"/>
2020-07-30T13:35:29.1354023Z 						<line number="27" hits="0"/>

Or… I just don’t know how to read this stuff. Anyways, at this point I’m going to explore the issue in the coverage direction rather than Codecov so I am not waiting on you to dig into this now. I will follow up when I have more info to document the resolution (hopefully).

1 Like

Hi @altendky, thanks for such a detailed writeup here. Yeah, that is strange that you are getting two different valuesets. The most obvious thing is that file has 27 statements marked in the xml part, but only 21 statements + 4 branches. It makes me wonder if the ... lines aren’t considered statements or branches.

tl;dr the exclusions are applied when reporting not when collecting. This means that if they are specified in the configuration file that file must be found when generating the reports. In the case of QTrio they are run from a subdirectory and so the file is not found by default and was only manually specified for the run. Codecov did not know to look up a directory and I failed to specify the configuration file when manually calling coverage xml.

Thanks @tom or your time and effort and my apologies for using them up on this.


1 Like

No worries @altendky! I’m really glad you were able to get it working!