Codecov becomes flaky

Please check the troubleshooting guide before creating a ticket.
All fields below are required.

Description

From earlier today, we are seeing Codecov coverage becomes flaky. We saw two things

  1. the coverage rate becomes up and down for exactly the same PR
  2. The coverage does not seem to reflect the real coverage

for 2, after re-run sometimes, it becomes back to normal. However it’s extremely frustrating since the coverage rate cannot be used as shipping criteria

Seeing

Commit SHAs

Please include the commit SHA(s)

https://codecov.io/gh/kalacloud-inc/kalacloud-client/commit/b5326c25e9c194e409d1b2846cc6f6d10cbcc048/

Uploader

Please provide which uploader you are using and command used to run it (e.g. Bash, Node, Python, GitHub Actions, etc…). If you are not using the bash uploader, we recommend switching to it before opening an issue here.

Github action (using github app)

Codecov Output

Please provide the full output of running the uploader on your CI/CD. This will typically have the Codecov logo as ASCII.

Expected Results

Please provide what you expect to have happened (e.g. a file that has missing coverage on a particular line).
the coverage rate should remain 100%

Actual Results

it becames 99.7% and sometimes even drop to 98% but it came back after re-run

Additional Information

Any additional information, configuration or data that might be necessary to reproduce the issue.

BTW: these two SHAs with the same code produced different results:

https://codecov.io/gh/kalacloud-inc/kalacloud-client/commit/d5d9c839de200f9ebf7c9351e82f8f7dc71c9d19/

https://codecov.io/gh/kalacloud-inc/kalacloud-client/commit/996f5e3a3b960c1f0c7b0fa24ca1f9696169f0f8/

@Edx, what I notice is that those two commits send a different number of uploads. The first sends 9, while the second sends 16. This could lead to coverage changes if there is additional coverage information being provided.

my apologies. Found a bug in the test-splitting code and that’s causing our tests to be randomly run now and then. Thus the different coverage rate. Sorry about that

1 Like