I have several coverage reports from a Python project generated on Travis CI and Appveyor; when uploaded and processed, the Appveyor reports are missing some code that was actually covered. It looks like the missing coverage is the code that’s run in a subprocess in the tests (it’s running the application and collecting coverage for it).
Appveyor (plus Travis CI, but that appears to work fine).
Example job, with command-line output showing the coverage report for verification: https://ci.appveyor.com/project/jimporter/bfg9000/build/job/s4st0i2sq0iclndn
Using codecov-python, invoked with
codecov --name "%APPVEYOR_JOB_NAME%"
comment: false coverage: status: project: default: informational: true patch: default: informational: true
_____ _ / ____| | | | | ___ __| | ___ ___ _____ __ | | / _ \ / _ |/ _ \/ __/ _ \ \ / / | |___| (_) | (_| | __/ (_| (_) \ V / \_____\___/ \____|\___|\___\___/ \_/ v2.1.8 ==> Detecting CI provider AppVeyor Detected ==> Preparing upload ==> Processing gcov (disable by -X gcov) ==> Collecting reports XX> Searching for reports disabled Generating coverage xml reports for Python + C:\projects\bfg9000\coverage.xml bytes=368247 ==> Uploading .url https://codecov.io .query commit=b8e2f4523c37a591e8c15d8d8628761e292b8f3d&branch=refactor-cc-msvc&job=jimporter%2Fbfg9000%2Frefactor-cc-msvc-2264&service=appveyor&build=s4st0i2sq0iclndn&slug=jimporter%2Fbfg9000&name=Image%3A%20Visual%20Studio%202013%3B%20Environment%3A%20COMPILER%3Dvs&yaml=codecov.yml&package=py2.1.8 Gzipping contents.. Compressed contents to 20136 bytes Pinging Codecov... Uploading to S3... https://codecov.io/github/jimporter/bfg9000/commit/b8e2f4523c37a591e8c15d8d8628761e292b8f3d
Steps to Reproduce
- Inject the following
sitecustomize.pyto collect coverage for Python subprocesses:
import coverage; coverage.process_startup()
coverage run setup.py test
- Submit coverage
Expected behavior: Full coverage; in particular, note the coverage for the file
bfg9000/tools/msvc/__init__.py. In the Appveyor log, it shows 99% coverage.
Actual behavior: In the Codecov report,
bfg9000/tools/msvc/__init__.py only shows 86% coverage: https://codecov.io/gh/jimporter/bfg9000/src/b8e2f4523c37a591e8c15d8d8628761e292b8f3d/bfg9000/tools/msvc/__init__.py. I’ve verified that the raw uploaded report looks ok.
Flakiness? This appears to happen all the time, since refactoring this part of the code to be in multiple files; formerly it lived in
bfg9000/tools/mscv.py and worked fine.