Decrease in code coverage reported without any related code changes on GitHub push/PR

Support
,
1

Before submitting a topic, please confirm the following

I have searched for similar issues before creating this topic.
I have verified that my repository is using the Codecov GitHub app, if using GitHub
I have validated my codecov.yaml configuration file.
I have filled out the below sections to the best of my ability.
I understand that the community boards are a free, best-effort tool. While we hope to have someone on the Codecov team resolve your problem quickly, this is not always possible.

Description

The codecov/project action fails without a decrease of -0.03% in code coverage but the report says No Files covered by tests were changed.

CI/CD URL

Codecov Output

Please provide the full output of running the uploader on your CI/CD. This will typically have the Codecov logo as ASCII.

Expected Results

No change in coverage is reported or the code changes which caused the regression are being shown.

Actual Results

No actual change in coverage is being shown in the

Additional Information

I omitted some checkmarks because we do not have the App installed and we do not have a configuration file.

1 Like
2

@firewave, I think this might be a bug on our side, I’ll sync with product team about it.

Which is actually correct?
So, it looks like there is a diff between the two commits, and you can see the only changes on externals/simplecpp/simplecpp.cpp.

On Codecov, we are showing No files changed, but the coverage drops by -0.03%. So, looking to see what lines are changed, I see that the only file that has changed #lines and #missed is externals/simplecpp/simplecpp.cpp

image
image1018×738 165 KB

So from my understanding, the coverage number is correct. We just aren’t doing a diff properly.

3

Hope it is okay if I chime in, we are also having an issue like that and looking at this it seems to me it has the same cause.

The clue is to look at this history:

We can see a commit on main that reduced the code coverage by this amount
If you look at the base of the PR, it is slightly before that commit that reduced the code coverage:

image
image1669×350 63.9 KB

Before running any CI, Github will merge it back to the target branch, so the -0.03 commit is included in the commit, that the CI is run on.

The workaround is to rebase or merge the PR, to change the base that codecov is looking at: Comparing Coverage Across Commits

or someone here pointed out, that you can tell github not to merge the commit before running the CI (but this also reduces your confidence in your CI): Patch Status should compare to merge-base instead of HEAD · Issue #83 · codecov/codecov-bash · GitHub

But this is quite annoying of course, so I would really like if codecov would fix this.

4

@StefanLobbenmeierObj I see, thanks for bringing this up. To be clear, this would probably be outside of my domain, and I’d recommend making a feature request to be able to compare to a different commit, or maybe re-run comparisons.