No change to relative scope but coverage decreases

Description

From time-to-time, a commit that doesn’t change the relative scope (for example, dependencies upgrade) creates a big coverage decrease on our repository, such as:
Screenshot 2020-11-05 at 15.33.36

Repository

Private repository (Rails application)

CI/CD

SemaphoreCI

Uploader

codecov-ruby

Commit SHAs

Private

Codecov YAML

Default one, no override

Codecov Output

  _____          _
 / ____|        | |
| |     ___   __| | ___  ___ _____   __
| |    / _ \ / _\`|/ _ \/ __/ _ \ \ / /
| |___| (_) | (_| |  __/ (_| (_) \ V /
 \_____\___/ \__,_|\___|\___\___/ \_/
                               
==> Semaphore CI detected
==> Appending file network
==> Gzipping contents
==> Uploading reports
    url:   https://codecov.io
    query: token=secret&flags&package=ruby-0.2.12&service=semaphore&branch=master&commit=***&build&job&slug
->  Pinging Codecov
https://codecov.io/upload/v4?token=secret&flags&package=ruby-0.2.12&service=semaphore&branch=master&commit=***&build&job&slug
->  Uploading to https://storage.googleapis.com/codecov/v4/raw/***
    View reports at https://codecov.io/github/***

Steps to Reproduce

It’s flacky, so it’s really hard to reproduce, unfortunately.

Expected behavior: A commit that doesn’t change the tracked files shouldn’t change drastically the code coverage (a 0.x% change is definitely acceptable though).

Actual behavior: A commit that doesn’t change the tracked files change drastically the code coverage (> 1%).

Flakiness? Happens only sometimes.

Additional Information

None

@BastienL, unfortunately without a commit SHA, I’m not really able to help here. Can you DM me one?

Confirming receipt of SHA

Not idempotent tests were the cause of the coverage difference.

Thanks @tom for pointing this out!

Absolutely! Will you let us know where the issue in idempotent tests ends up being? We’d love to try to understand how to help users solve this problem.

In our case, integration tests that interact with 3rd parties are the cause. Mocks would fix that issue, but on the other hand those tests would be less reliable. So we’ve decided to be fine with small changes on coverage from one build to another.

1 Like