We are using Codecov for generating coverage reports and for the commit status in Oppia(GitHub - oppia/oppia: A free, online learning platform to make quality education accessible for all.). We are facing a problem of false coverage reports. Codecov seems to compare the coverage with the base of the PR and not the master branch. This results in false decrease of coverage reports which can be fixed by merging master.
For example,
- PR “A” is opened first and it does not affect coverage.
- PR “B” is then opened which increased the coverage of the code-base and is merged.
- The commit status in PR “A” will now fail by showing decreased coverage which in reality it does not.
- This is fixed after merging the new master branch to PR “A”.
This is really problematic and we are wondering if there’s a way to configure codecov somehow to always compare against the master branch?