Description
From time-to-time, a commit that doesn’t change the relative scope (for example, dependencies upgrade) creates a big coverage decrease on our repository, such as:
Repository
Private repository (Rails application)
CI/CD
SemaphoreCI
Uploader
Commit SHAs
Private
Codecov YAML
Default one, no override
Codecov Output
_____ _
/ ____| | |
| | ___ __| | ___ ___ _____ __
| | / _ \ / _\`|/ _ \/ __/ _ \ \ / /
| |___| (_) | (_| | __/ (_| (_) \ V /
\_____\___/ \__,_|\___|\___\___/ \_/
==> Semaphore CI detected
==> Appending file network
==> Gzipping contents
==> Uploading reports
url: https://codecov.io
query: token=secret&flags&package=ruby-0.2.12&service=semaphore&branch=master&commit=***&build&job&slug
-> Pinging Codecov
https://codecov.io/upload/v4?token=secret&flags&package=ruby-0.2.12&service=semaphore&branch=master&commit=***&build&job&slug
-> Uploading to https://storage.googleapis.com/codecov/v4/raw/***
View reports at https://codecov.io/github/***
Steps to Reproduce
It’s flacky, so it’s really hard to reproduce, unfortunately.
Expected behavior: A commit that doesn’t change the tracked files shouldn’t change drastically the code coverage (a 0.x% change is definitely acceptable though).
Actual behavior: A commit that doesn’t change the tracked files change drastically the code coverage (> 1%).
Flakiness? Happens only sometimes.
Additional Information
None