Codecov seems to be using old data on this PR (from earlier in the PR) despite new data seemingly being uploaded successfully.
2020-07-29T03:54:32.3813490Z /__t/Python/3.8.3/x64/lib/python3.8/site-packages/qtrio/_pytest.py 21 0 4 0 100.0%
The log says to see reports at:
The Ubuntu 3.8 PySide2 build data shows two hits="0" for the _pytest.py file.
altendky:master ← altendky:test_timeout_parameter
opened 04:57PM - 26 Jul 20 UTC
altendky:master ← altendky:test_timeout_parameter
opened 04:57PM - 26 Jul 20 UTC
bash: bash codecov.sh -n ‘Ubuntu (3.8, PySide2)’
0caf8da4df054bfbf700cb6efe9a8af2cb729d46
I do not have a Codecov YAML file
2020-07-29T03:54:32.5085766Z + bash codecov.sh -n 'Ubuntu (3.8, PySide2)'
2020-07-29T03:54:32.5140360Z
2020-07-29T03:54:32.5140591Z _____ _
2020-07-29T03:54:32.5140720Z / ____| | |
2020-07-29T03:54:32.5140848Z | | ___ __| | ___ ___ _____ __
2020-07-29T03:54:32.5140982Z | | / _ \ / _` |/ _ \/ __/ _ \ \ / /
2020-07-29T03:54:32.5141109Z | |___| (_) | (_| | __/ (_| (_) \ V /
2020-07-29T03:54:32.5141227Z \_____\___/ \__,_|\___|\___\___/ \_/
2020-07-29T03:54:32.5141756Z Bash-20200707-353aa93
2020-07-29T03:54:32.5141869Z
2020-07-29T03:54:32.5141931Z
2020-07-29T03:54:32.5153639Z e[0;90m==>e[0m GitHub Actions detected.
2020-07-29T03:54:32.5254178Z e[0;90mproject root:e[0m /__w/qtrio/qtrio
2020-07-29T03:54:32.5348779Z e[0;32mYaml not found, that's ok! Learn more ate[0m e[0;36mhttp://docs.codecov.io/docs/codecov-yamle[0m
2020-07-29T03:54:32.5676268Z e[0;90m==>e[0m Running gcov in /__w/qtrio/qtrio e[0;90m(disable via -X gcov)e[0m
2020-07-29T03:54:32.5730847Z e[0;90m==>e[0m Python coveragepy exists e[0;90mdisable via -X coveragepye[0m
2020-07-29T03:54:32.5785807Z e[0;90m->e[0m Running coverage xml
2020-07-29T03:54:32.8746842Z e[0;90m==>e[0m Searching for coverage reports in:
2020-07-29T03:54:32.8747253Z e[0;32m+e[0m /__w/qtrio/qtrio
2020-07-29T03:54:32.8855258Z e[0;90m->e[0m Found 1 reports
2020-07-29T03:54:32.8855594Z e[0;90m==>e[0m Detecting git/mercurial file structure
2020-07-29T03:54:32.8934477Z e[0;90m==>e[0m Reading reports
2020-07-29T03:54:32.9002199Z e[0;32m+e[0m /__w/qtrio/qtrio/empty/coverage.xml e[0;90mbytes=40042e[0m
2020-07-29T03:54:32.9035112Z e[0;90m==>e[0m Appending adjustments
2020-07-29T03:54:32.9035514Z e[0;36mhttps://docs.codecov.io/docs/fixing-reportse[0m
2020-07-29T03:54:32.9232007Z e[0;90m->e[0m No adjustments found
2020-07-29T03:54:32.9238558Z e[0;90m==>e[0m Gzipping contents
2020-07-29T03:54:32.9297584Z e[0;90m==>e[0m Uploading reports
2020-07-29T03:54:32.9297964Z e[0;90murl:e[0m https://codecov.io
2020-07-29T03:54:32.9298571Z e[0;90mquery:e[0m branch=test_timeout_parameter&commit=74253edd4ea5cb6e04ab29bd25648791d282a520&build=186608969&build_url=http%3A%2F%2Fgithub.com%2Faltendky%2Fqtrio%2Factions%2Fruns%2F186608969&name=Ubuntu%20%283.8%2C%20PySide2%29&tag=&slug=altendky%2Fqtrio&service=github-actions&flags=&pr=105&job=&cmd_args=n
2020-07-29T03:54:32.9339474Z e[0;90m->e[0m Pinging Codecov
2020-07-29T03:54:32.9340283Z https://codecov.io/upload/v4?package=bash-20200707-353aa93&token=secret&branch=test_timeout_parameter&commit=74253edd4ea5cb6e04ab29bd25648791d282a520&build=186608969&build_url=http%3A%2F%2Fgithub.com%2Faltendky%2Fqtrio%2Factions%2Fruns%2F186608969&name=Ubuntu%20%283.8%2C%20PySide2%29&tag=&slug=altendky%2Fqtrio&service=github-actions&flags=&pr=105&job=&cmd_args=n
2020-07-29T03:54:33.3253194Z e[0;90m->e[0m Uploading to
2020-07-29T03:54:33.3254567Z https://storage.googleapis.com/codecov/v4/raw/2020-07-29/382B25014ADF4901720497EAD40C1DE3/0caf8da4df054bfbf700cb6efe9a8af2cb729d46/fd877bfd-cdae-4e0b-9fa1-ed132528c3c9.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=GOOG1EQX6OZVJGHKK3633AAFGLBUCOOATRACRQRQF6HMSMLYUP6EAD6XSWAAY%2F20200729%2FUS%2Fs3%2Faws4_request&X-Amz-Date=20200729T035433Z&X-Amz-Expires=10&X-Amz-SignedHeaders=host&X-Amz-Signature=0ef7acc4b73d2ca9f49a78f9c0f7460831bde559b218d7611a7b62f7b2a64ee1
2020-07-29T03:54:33.3335899Z % Total % Received % Xferd Average Speed Time Time Time Current
2020-07-29T03:54:33.3336656Z Dload Upload Total Spent Left Speed
2020-07-29T03:54:33.3336812Z
2020-07-29T03:54:33.5420559Z 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
2020-07-29T03:54:33.5421376Z 100 3527 0 0 100 3527 0 16956 --:--:-- --:--:-- --:--:-- 16956
2020-07-29T03:54:33.5473316Z e[0;32m->e[0m View reports at e[0;36mhttps://codecov.io/github/altendky/qtrio/commit/0caf8da4df054bfbf700cb6efe9a8af2cb729d46e[0m
tom
July 29, 2020, 2:40pm
2
Hi @altendky , just some questions to clarify. You are saying this file has incorrect coverage on it based on an older report. Can you specify between which commit SHAs you see the issue? What do you expect the coverage to be?
Hi Tom, Thanks for the quick reply. Yes, qtrio/_pytest.py is the file of interest. Specifically the lines with ... in the @typing.overload decorated functions. They simply will never be run so they won’t be covered But… :]
That commit configures coverage to exclude the functions. In the gist you can see that coverage is reporting, at least via stdout, that qtrio/_pytest.py has no misses.
You asked for the pair of SHAs. This is the parent of the above.
I’m getting pretty confused though as I created new PRs including fully rebasing to get entirely new commit hashes for all commits and that still has the same issue.
altendky:master ← altendky:test_timeout_parameter_rebase
opened 01:13AM - 29 Jul 20 UTC
Alrighty, I added an explicit coverage xml call and printed out the contents and what I see is a discrepancy in the two coverage generated outputs.
2020-07-30T13:35:28.7722736Z /__t/Python/3.8.5/x64/lib/python3.8/site-packages/qtrio/_pytest.py 21 0 4 0 100.0%
2020-07-30T13:35:29.1352239Z <class name="_pytest.py" filename="/__t/Python/3.8.5/x64/lib/python3.8/site-packages/qtrio/_pytest.py" complexity="0" line-rate="0.9259" branch-rate="1">
2020-07-30T13:35:29.1352386Z <methods/>
2020-07-30T13:35:29.1352489Z <lines>
2020-07-30T13:35:29.1352606Z <line number="3" hits="1"/>
2020-07-30T13:35:29.1352727Z <line number="4" hits="1"/>
2020-07-30T13:35:29.1352846Z <line number="6" hits="1"/>
2020-07-30T13:35:29.1352963Z <line number="7" hits="1"/>
2020-07-30T13:35:29.1353083Z <line number="8" hits="1"/>
2020-07-30T13:35:29.1353200Z <line number="9" hits="1"/>
2020-07-30T13:35:29.1353304Z <line number="11" hits="1"/>
2020-07-30T13:35:29.1353425Z <line number="14" hits="1"/>
2020-07-30T13:35:29.1353543Z <line number="15" hits="1"/>
2020-07-30T13:35:29.1353667Z <line number="18" hits="0"/>
2020-07-30T13:35:29.1353785Z <line number="21" hits="1"/>
2020-07-30T13:35:29.1353904Z <line number="22" hits="1"/>
2020-07-30T13:35:29.1354023Z <line number="27" hits="0"/>
Or… I just don’t know how to read this stuff. Anyways, at this point I’m going to explore the issue in the coverage direction rather than Codecov so I am not waiting on you to dig into this now. I will follow up when I have more info to document the resolution (hopefully).
1 Like
tom
July 30, 2020, 4:43pm
5
Hi @altendky , thanks for such a detailed writeup here. Yeah, that is strange that you are getting two different valuesets. The most obvious thing is that file has 27 statements marked in the xml part, but only 21 statements + 4 branches. It makes me wonder if the ... lines aren’t considered statements or branches.
opened 12:18AM - 31 Jul 20 UTC
closed 02:09AM - 31 Jul 20 UTC
bug
**Describe the bug**
Exclusions may not work depending on the working directory… from which `coverage run` is launched.
**To Reproduce**
OS: Ubuntu Linux 20.04
Python: 3.8.3
coverage: 5.2.1
Example repository:
https://github.com/altendky/covtest/tree/1071c7f999a9ed51154a48148819a01ebc5cdd11
```bash
git clone https://github.com/altendky/covtest
cd covtest
git checkout 1071c7f999a9ed51154a48148819a01ebc5cdd11
bash go.sh
```
<details> <summary>Complete run with diagnostic info</summary>
```bash
altendky@p1:~/repos/covtest/blue$ git clone https://github.com/altendky/covtest
Cloning into 'covtest'...
remote: Enumerating objects: 11, done.
remote: Counting objects: 100% (11/11), done.
remote: Compressing objects: 100% (7/7), done.
remote: Total 11 (delta 2), reused 11 (delta 2), pack-reused 0
Unpacking objects: 100% (11/11), 1.11 KiB | 228.00 KiB/s, done.
altendky@p1:~/repos/covtest/blue$ cd covtest
altendky@p1:~/repos/covtest/blue/covtest$ git checkout 1071c7f999a9ed51154a48148819a01ebc5cdd11
Note: switching to '1071c7f999a9ed51154a48148819a01ebc5cdd11'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:
git switch -c <new-branch-name>
Or undo this operation with:
git switch -
Turn off this advice by setting config variable advice.detachedHead to false
HEAD is now at 1071c7f more info
altendky@p1:~/repos/covtest/blue/covtest$ bash go.sh
rm -rf venv
+ rm -rf venv
python3 -m venv venv
+ python3 -m venv venv
venv/bin/pip install --upgrade pip setuptools wheel
+ venv/bin/pip install --upgrade pip setuptools wheel
Collecting pip
Using cached https://files.pythonhosted.org/packages/36/74/38c2410d688ac7b48afa07d413674afc1f903c1c1f854de51dc8eb2367a5/pip-20.2-py2.py3-none-any.whl
Collecting setuptools
Using cached https://files.pythonhosted.org/packages/8e/11/9e10f1cad4518cb307b484c255cae61e97f05b82f6d536932b1714e01b47/setuptools-49.2.0-py3-none-any.whl
Collecting wheel
Using cached https://files.pythonhosted.org/packages/8c/23/848298cccf8e40f5bbb59009b32848a4c38f4e7f3364297ab3c3e2e2cd14/wheel-0.34.2-py2.py3-none-any.whl
Installing collected packages: pip, setuptools, wheel
Found existing installation: pip 19.2.3
Uninstalling pip-19.2.3:
Successfully uninstalled pip-19.2.3
Found existing installation: setuptools 41.2.0
Uninstalling setuptools-41.2.0:
Successfully uninstalled setuptools-41.2.0
Successfully installed pip-20.2 setuptools-49.2.0 wheel-0.34.2
venv/bin/pip install .
+ venv/bin/pip install .
Processing /home/altendky/repos/covtest/blue/covtest
Collecting coverage
Using cached coverage-5.2.1-cp38-cp38-manylinux1_x86_64.whl (231 kB)
Building wheels for collected packages: covtest
Building wheel for covtest (setup.py) ... done
Created wheel for covtest: filename=covtest-0.0.0-py3-none-any.whl size=1393 sha256=3105c0587d6e21f6130dee46d99e475af34ef24bcfd9af4f3ec6265a03cab417
Stored in directory: /tmp/pip-ephem-wheel-cache-0eb7ph9w/wheels/db/d9/36/53374d0c3d6fbc08d9281d3236f942159b62b3b821efa6ebc8
Successfully built covtest
Installing collected packages: coverage, covtest
Successfully installed coverage-5.2.1 covtest-0.0.0
echo
+ echo
venv/bin/python --version --version
+ venv/bin/python --version --version
Python 3.8.3 (default, Jun 4 2020, 17:51:25)
[GCC 9.3.0]
git show --no-patch HEAD
+ git show --no-patch HEAD
commit 1071c7f999a9ed51154a48148819a01ebc5cdd11 (HEAD, origin/main, origin/HEAD, main)
Author: Kyle Altendorf <sda@fstab.net>
Date: Thu Jul 30 20:01:07 2020 -0400
more info
venv/bin/pip freeze --all
+ venv/bin/pip freeze --all
coverage==5.2.1
covtest @ file:///home/altendky/repos/covtest/blue/covtest
pip==20.2
setuptools==49.2.0
wheel==0.34.2
venv/bin/coverage debug sys
+ venv/bin/coverage debug sys
-- sys -------------------------------------------------------
version: 5.2.1
coverage: /home/altendky/repos/covtest/blue/covtest/venv/lib/python3.8/site-packages/coverage/__init__.py
tracer: -none-
CTracer: available
plugins.file_tracers: -none-
plugins.configurers: -none-
plugins.context_switchers: -none-
configs_attempted: .coveragerc
configs_read: /home/altendky/repos/covtest/blue/covtest/.coveragerc
config_file: /home/altendky/repos/covtest/blue/covtest/.coveragerc
config_contents: b'[report]\nexclude_lines =\n typing\\.overload\n'
data_file: -none-
python: 3.8.3 (default, Jun 4 2020, 17:51:25) [GCC 9.3.0]
platform: Linux-5.4.0-40-generic-x86_64-with-glibc2.2.5
implementation: CPython
executable: /home/altendky/repos/covtest/blue/covtest/venv/bin/python3
def_encoding: utf-8
fs_encoding: utf-8
pid: 846734
cwd: /home/altendky/repos/covtest/blue/covtest
path: /home/altendky/repos/covtest/blue/covtest/venv/bin
/home/altendky/.pyenv/versions/3.8.3/lib/python38.zip
/home/altendky/.pyenv/versions/3.8.3/lib/python3.8
/home/altendky/.pyenv/versions/3.8.3/lib/python3.8/lib-dynload
/home/altendky/repos/covtest/blue/covtest/venv/lib/python3.8/site-packages
environment: PYENV_ROOT = /home/altendky/.pyenv
PYENV_SHELL = fish
PYENV_VIRTUALENV_INIT = 1
PYTHONDONTWRITEBYTECODE = 1
PYTHON_CONFIGURE_OPTS = --enable-shared
command_line: venv/bin/coverage debug sys
sqlite3_version: 2.6.0
sqlite3_sqlite_version: 3.31.1
sqlite3_temp_store: 0
sqlite3_compile_options: COMPILER=gcc-9.3.0
ENABLE_COLUMN_METADATA
ENABLE_DBSTAT_VTAB
ENABLE_FTS3
ENABLE_FTS3_PARENTHESIS
ENABLE_FTS3_TOKENIZER
ENABLE_FTS4
ENABLE_FTS5
ENABLE_JSON1
ENABLE_LOAD_EXTENSION
ENABLE_PREUPDATE_HOOK
ENABLE_RTREE
ENABLE_SESSION
ENABLE_STMTVTAB
ENABLE_UNLOCK_NOTIFY
ENABLE_UPDATE_DELETE_LIMIT
HAVE_ISNAN
LIKE_DOESNT_MATCH_BLOBS
MAX_SCHEMA_RETRY=25
MAX_VARIABLE_NUMBER=250000
OMIT_LOOKASIDE
SECURE_DELETE
SOUNDEX
TEMP_STORE=1
THREADSAFE=1
USE_URI
venv/bin/coverage erase
+ venv/bin/coverage erase
rm -rf htmlcov
+ rm -rf htmlcov
venv/bin/coverage run --rcfile .coveragerc -m covtest
+ venv/bin/coverage run --rcfile .coveragerc -m covtest
venv/bin/coverage html
+ venv/bin/coverage html
echo ------- root
+ echo ------- root
------- root
venv/bin/coverage report
+ venv/bin/coverage report
Name Stmts Miss Cover
-----------------------------------------
covtest/__init__.py 0 0 100%
covtest/__main__.py 4 0 100%
-----------------------------------------
TOTAL 4 0 100%
echo
+ echo
grep -r missing htmlcov/*__main__*
+ grep -r missing htmlcov/covtest___main___py.html
<button type="button" class="mis show_mis shortkey_m button_toggle_mis" title="Toggle lines missing">0 missing</button>
echo
+ echo
echo
+ echo
rm -rf empty
+ rm -rf empty
mkdir empty
+ mkdir empty
cd empty
+ cd empty
../venv/bin/coverage erase
+ ../venv/bin/coverage erase
rm -rf htmlcov
+ rm -rf htmlcov
../venv/bin/coverage run --rcfile ../.coveragerc -m covtest
+ ../venv/bin/coverage run --rcfile ../.coveragerc -m covtest
../venv/bin/coverage html
+ ../venv/bin/coverage html
echo ------- empty/
+ echo ------- empty/
------- empty/
../venv/bin/coverage report
+ ../venv/bin/coverage report
Name Stmts Miss Cover
--------------------------------------------------------------------------------------------------------------------
/home/altendky/repos/covtest/blue/covtest/venv/lib/python3.8/site-packages/covtest/__init__.py 0 0 100%
/home/altendky/repos/covtest/blue/covtest/venv/lib/python3.8/site-packages/covtest/__main__.py 7 1 86%
--------------------------------------------------------------------------------------------------------------------
TOTAL 7 1 86%
echo
+ echo
grep -r missing htmlcov/*__main__*
+ grep -r missing htmlcov/_home_altendky_repos_covtest_blue_covtest_venv_lib_python3_8_site-packages_covtest___main___py.html
<button type="button" class="mis show_mis shortkey_m button_toggle_mis" title="Toggle lines missing">1 missing</button>
cd -
+ cd -
/home/altendky/repos/covtest/blue/covtest
```
</details>
**Expected behavior**
The `...` lines in the `@typing.overload` function are excluded rather than missing.
**Additional context**
This was observed while working on QTrio. Also reported to Codecov before deciding it seemed to be a `coverage` issue. Given that Codecov was reporting the lines as missed that suggests to me that all tested platforms (Windows, macOS, Linux) and all tested Python versions (3.6, 3.7, and 3.8) are affected.
https://github.com/altendky/qtrio/pull/105
https://community.codecov.io/t/codecov-reporting-old-coverage-data-for-new-builds-in-a-pr/1814
I'll throw my guess here that there are paths associated with the coverage measurement and separate paths associated with the exclusions and that in the case of running from the `empty/` directory these end up not agreeing and so the excluded lines are not considered present in the coverage results and thus not excluded. But, that's only based on the HTML reports showing different paths for the same `covtest` files.
I'll try to take a look at this and see what I can learn about `coverage`.
tl;dr the exclusions are applied when reporting not when collecting. This means that if they are specified in the configuration file that file must be found when generating the reports. In the case of QTrio they are run from a subdirectory and so the file is not found by default and was only manually specified for the run. Codecov did not know to look up a directory and I failed to specify the configuration file when manually calling coverage xml.
Thanks @tom or your time and effort and my apologies for using them up on this.
Cheers,
1 Like
tom
July 31, 2020, 2:23am
7
No worries @altendky ! I’m really glad you were able to get it working!