Parallel coverage in Python handles @click function definitions differently than single run coverage

Support
,
1

Before submitting a topic, please confirm the following

[x ] I have searched for similar issues before creating this topic.
[x ] I have verified that my repository is using the Codecov GitHub app, if using GitHub
[n/a ] I have validated my codecov.yaml configuration file.
[x ] I have filled out the below sections to the best of my ability.
[x ] I understand that the community boards are a free, best-effort tool. While we hope to have someone on the Codecov team resolve your problem quickly, this is not always possible.

Description

We have a repo that gets coverage analysis in CI through GitHub Actions to codecov.io:

We used to run coverage analysis with a single coverage run command, but have recently started using parallel mode and several uploads to cover the code base more comprehensively.

With our previous configuration, the definitions of functions that are decorated with @click decorators were not considered executable lines of code, and thus they didn’t count in coverage statistics. With our new configuration, the @click-decorated function definitions are now considered executable code that has never been executed, and thus count as a bunch of misses in the coverage statistics, despite the fact that those functions are in fact called multiple times.

CI/CD URL

Original configuration (single coverage run call): feat!: g2p convert now tokenizes by default · roedoejet/g2p@9d4163d · GitHub

New configuration (multiple coverage run calls combined)
Upload 1: refactor: rename get_langs->get_arpabet_langs to make purpose clearer · roedoejet/g2p@7c5222e · GitHub
Upload 2: refactor: rename get_langs->get_arpabet_langs to make purpose clearer · roedoejet/g2p@7c5222e · GitHub

Codecov Output

Please provide the full output of running the uploader on your CI/CD. This will typically have the Codecov logo as ASCII.

Original config:

Run codecov/codecov-action@v3
==> linux OS detected
https://uploader.codecov.io/latest/linux/codecov.SHA256SUM
==> SHASUM file signed by key id 806bb28aed779869
==> Uploader SHASUM verified (02b055414cc124203ab061ffc45f5497ca39ede5e63a06717a6981c32d7b425f  codecov)
==> Running version latest
==> Running version v0.5.0
/home/runner/work/_actions/codecov/codecov-action/v3/dist/codecov -n  -Q github-action-3.1.3
[2023-04-27T20:58:20.154Z] ['info'] 
     _____          _
    / ____|        | |
   | |     ___   __| | ___  ___ _____   __
   | |    / _ \ / _` |/ _ \/ __/ _ \ \ / /
   | |___| (_) | (_| |  __/ (_| (_) \ V /
    \_____\___/ \__,_|\___|\___\___/ \_/

  Codecov report uploader 0.5.0
[2023-04-27T20:58:20.163Z] ['info'] => Project root located at: /home/runner/work/g2p/g2p
[2023-04-27T20:58:20.164Z] ['info'] ->  Token found by environment variables
[2023-04-27T20:58:20.337Z] ['info'] Running coverage xml...
[2023-04-27T20:58:21.020Z] ['info'] Searching for coverage files...
[2023-04-27T20:58:21.082Z] ['info'] Warning: Some files located via search were excluded from upload.
[2023-04-27T20:58:21.082Z] ['info'] If Codecov did not locate your files, please review https://docs.codecov.com/docs/supported-report-formats
[2023-04-27T20:58:21.082Z] ['info'] => Found 1 possible coverage files:
  coverage.xml
[2023-04-27T20:58:21.082Z] ['info'] Processing /home/runner/work/g2p/g2p/coverage.xml...
[2023-04-27T20:58:21.089Z] ['info'] Detected GitHub Actions as the CI provider.
[2023-04-27T20:58:21.550Z] ['info'] Pinging Codecov: https://codecov.io/upload/v4?package=github-action-3.1.3-uploader-0.5.0&token=*******&branch=dev.main_g2p_functions&build=4824341430&build_url=https%3A%2F%2Fgithub.com%2Froedoejet%2Fg2p%2Factions%2Fruns%2F4824341430%2Fjobs%2F8594001098&commit=9d4163d5346a9f3f7356ab55c630ff45a1f6dfbf&job=Run+Tests&pr=&service=github-actions&slug=roedoejet%2Fg2p&name=&tag=&flags=&parent=
[2023-04-27T20:58:21.999Z] ['info'] https://app.codecov.io/github/roedoejet/g2p/commit/9d4163d5346a9f3f7356ab55c630ff45a1f6dfbf
https://storage.googleapis.com/codecov/v4/raw/2023-04-27/0FFB541019E762F89D43E6C40CB8B5D9/9d4163d5346a9f3f7356ab55c630ff45a1f6dfbf/7aa85ee1-bc9b-4705-86ce-db500982752d.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=GOOG1EJOGFN2JQ4OCTGA2MU5AEIT7OT5Z7HTFOAN2SPG4NWSN2UJYOY5U6LZQ%2F20230427%2FUS%2Fs3%2Faws4_request&X-Amz-Date=20230427T205821Z&X-Amz-Expires=10&X-Amz-SignedHeaders=host&X-Amz-Signature=516f08260211728e9ec91735a5519724db4f940e74801be3238773a4915ef51f
[2023-04-27T20:58:22.000Z] ['info'] Uploading...
[2023-04-27T20:58:22.277Z] ['info'] {"status":"success","resultURL":"https://app.codecov.io/github/roedoejet/g2p/commit/9d4163d5346a9f3f7356ab55c630ff45a1f6dfbf"}

New config, listing one of several uploads - all were successful:

Run codecov/codecov-action@v3
==> linux OS detected
https://uploader.codecov.io/latest/linux/codecov.SHA256SUM
==> SHASUM file signed by key id 806bb28aed779869
==> Uploader SHASUM verified (32cb14b5f3aaacd67f4c1ff55d82f037d3cd10c8e7b69c051f27391d2e66e15c  codecov)
==> Running version latest
==> Running version v0.4.1
/home/runner/work/_actions/codecov/codecov-action/v3/dist/codecov -n  -Q github-action-3.1.2 -C 7c5222e72727364acabc8bd6d2c61293b030dacb
[2023-04-17T19:01:45.762Z] ['info'] 
     _____          _
    / ____|        | |
   | |     ___   __| | ___  ___ _____   __
   | |    / _ \ / _` |/ _ \/ __/ _ \ \ / /
   | |___| (_) | (_| |  __/ (_| (_) \ V /
    \_____\___/ \__,_|\___|\___\___/ \_/

  Codecov report uploader 0.4.1
[2023-04-17T19:01:45.769Z] ['info'] => Project root located at: /home/runner/work/g2p/g2p
[2023-04-17T19:01:45.770Z] ['info'] ->  Token found by environment variables
[2023-04-17T19:01:45.882Z] ['info'] Running coverage xml...
[2023-04-17T19:01:46.367Z] ['info'] Searching for coverage files...
[2023-04-17T19:01:46.411Z] ['info'] Warning: Some files located via search were excluded from upload.
[2023-04-17T19:01:46.411Z] ['info'] If Codecov did not locate your files, please review https://docs.codecov.com/docs/supported-report-formats
[2023-04-17T19:01:46.411Z] ['info'] => Found 1 possible coverage files:
  coverage.xml
[2023-04-17T19:01:46.411Z] ['info'] Processing /home/runner/work/g2p/g2p/coverage.xml...
[2023-04-17T19:01:46.416Z] ['info'] Using manual override from args.
[2023-04-17T19:01:46.416Z] ['info'] Detected GitHub Actions as the CI provider.
[2023-04-17T19:01:46.786Z] ['info'] Pinging Codecov: https://codecov.io/upload/v4?package=github-action-3.1.2-uploader-0.4.1&token=*******&branch=dev.get_arpabet_langs&build=4724774383&build_url=https%3A%2F%2Fgithub.com%2Froedoejet%2Fg2p%2Factions%2Fruns%2F4724774383%2Fjobs%2F8382412398&commit=7c5222e72727364acabc8bd6d2c61293b030dacb&job=Run+Tests+for+Studio&pr=249&service=github-actions&slug=roedoejet%2Fg2p&name=&tag=&flags=&parent=
[2023-04-17T19:01:47.057Z] ['info'] https://app.codecov.io/github/roedoejet/g2p/commit/7c5222e72727364acabc8bd6d2c61293b030dacb
https://storage.googleapis.com/codecov/v4/raw/2023-04-17/0FFB541019E762F89D43E6C40CB8B5D9/7c5222e72727364acabc8bd6d2c61293b030dacb/0aba487c-1f7d-4c81-aad9-8033c49d7d4c.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=GOOG1EJOGFN2JQ4OCTGA2MU5AEIT7OT5Z7HTFOAN2SPG4NWSN2UJYOY5U6LZQ%2F20230417%2FUS%2Fs3%2Faws4_request&X-Amz-Date=20230417T190147Z&X-Amz-Expires=10&X-Amz-SignedHeaders=host&X-Amz-Signature=8ab1cb4f98631748d76da12bbfe402c842aca580751aaa51d15f392dbff51c26
[2023-04-17T19:01:47.058Z] ['info'] Uploading...
[2023-04-17T19:01:47.234Z] ['info'] {"status":"success","resultURL":"https://app.codecov.io/github/roedoejet/g2p/commit/7c5222e72727364acabc8bd6d2c61293b030dacb"}

Expected Results

Please provide what you expect to have happened (e.g. a file that has missing coverage on a particular line).

The line with the function definitions that are decorated with @click are either not considered executable, like in the original configuration’s output, see line 181 here:

or, those lines are considered executable and they are counted as having been called, which is what happens when I run coverage locally on my own computer.

Actual Results

Please provide what actually happened.

The function definitions that are decorated with @click are considered executable lines that were not called, e.g., see line 180 here:

To be more precise, my expected result would have been that line 180 here is considered either covered or not executable.

Additional Information

Please provide applicable commit SHAs or file names that are extraneous or missing. Any additional information, configuration, or data that might be necessary to reproduce the issue.

2

@joanise I tried to dig into this a little bit, but this seems to be coming from coverage.py, and I can’t figure out why parallelism with it is marking new lines as uncovered. You’ll notice for this commit that the first upload (ending in 11) shows this line in the raw coverage report

<line number="181" hits="0"/>

which is why we are showing it as uncovered. I would probably ping on the coverage.py repo as to why this is happening.

3

Excellent spotting, thank you! With that hint, I was just able to reproduce the problem on my own computer. I’ll troubleshoot and report back once I figure it out, but you are correct, the output of coverage run and then coverage xml includes that line 181 with a 0 hit count.

It’s the coverage test instance that runs the micro-service API where this happens, whereas the instance that runs regular unit testing is fine, and our old non-parallel build only ran the unit testing. So that answers the question as to why parallel seemed to make a difference: it didn’t, it’s just that we added coverage for this API test which is causing the problem.

1 Like
4

Oh interesting! I would love to know why there is the difference in profiling/instrumentation from the coverage.py side when adding in the microservice!

5

So far I’m not there yet, but the micro-service API is a Flask app launched via gunicorn in production or SocketIO in dev.

The cli.py file is (not too surprisingly) the CLI entry point into the same code base, and it is not executed when the micro-service is launched, which is why all the lines have 0 hits when exercising the micro-services. It only gets hits when exercising the regular unit tests.

Another difference I noticed between CI and my computer is that the regular unit test suites count these function definition lines as executable and covered on my PC, but as non-executable in CI. Again, not a codecov difference, but different coverage xml output that I have not yet figured out how to explain. I might need to run an environment that is closer to what happens in CI.

Unfortunately, I’m running out of time to investigate this further, but thank you for your support!

1 Like