No change to relative scope but coverage decreases


From time-to-time, a commit that doesn’t change the relative scope (for example, dependencies upgrade) creates a big coverage decrease on our repository, such as:
Screenshot 2020-11-05 at 15.33.36


Private repository (Rails application)





Commit SHAs


Codecov YAML

Default one, no override

Codecov Output

  _____          _
 / ____|        | |
| |     ___   __| | ___  ___ _____   __
| |    / _ \ / _\`|/ _ \/ __/ _ \ \ / /
| |___| (_) | (_| |  __/ (_| (_) \ V /
 \_____\___/ \__,_|\___|\___\___/ \_/
==> Semaphore CI detected
==> Appending file network
==> Gzipping contents
==> Uploading reports
    query: token=secret&flags&package=ruby-0.2.12&service=semaphore&branch=master&commit=***&build&job&slug
->  Pinging Codecov***&build&job&slug
->  Uploading to***
    View reports at***

Steps to Reproduce

It’s flacky, so it’s really hard to reproduce, unfortunately.

Expected behavior: A commit that doesn’t change the tracked files shouldn’t change drastically the code coverage (a 0.x% change is definitely acceptable though).

Actual behavior: A commit that doesn’t change the tracked files change drastically the code coverage (> 1%).

Flakiness? Happens only sometimes.

Additional Information


@BastienL, unfortunately without a commit SHA, I’m not really able to help here. Can you DM me one?

Confirming receipt of SHA

Not idempotent tests were the cause of the coverage difference.

Thanks @tom for pointing this out!

Absolutely! Will you let us know where the issue in idempotent tests ends up being? We’d love to try to understand how to help users solve this problem.

In our case, integration tests that interact with 3rd parties are the cause. Mocks would fix that issue, but on the other hand those tests would be less reliable. So we’ve decided to be fine with small changes on coverage from one build to another.

1 Like