Codecov seems to find changes where there were none

Description

For some changesets there seem to be files marked as changed and coverage changes show on them when no source code changed at all, such as a markdown file.

Repository

Steps to Reproduce

  1. Check in a non-source code file change and submit a PR.
  2. Codecov will comment on the PR, and sometimes will have changes listed. When inspecting the changeset on Codecov, it seems to indicate files have changed when they have not.

Expected behavior: [What you expect to happen]
Codecov should not indicate there’s a coverage change for changesets where the change did not effect the coverage.

Actual behavior: [What actually happens]
Codecov indicates a change in code coverage when there was not one.

Flakiness? [Does this happen all the time or only sometimes?]
It doesn’t seem to happen all the time, it seems intermittent.

Versions

N/A

Additional Information

None that I can think of!

Depending on your branching model an CI strategy, this issue might be related to the unability of Codecov to pick the right base commit to compare with.

I had the same problem of not changed files being reported due to the base commit issue. I would recommend you to take a close look into the git tree and the commit used as base to double check if your problem is related with that.

When I look at this PR in Codecov https://codecov.io/gh/grafana/loki/pull/1675/commits, it looks like https://github.com/grafana/loki/commit/6bbb61eb2f52288038ae0112a3b577a1b7861d4d is the detected parent.

Is this correct? If so, then confirming this manually agrees with the PR: https://codecov.io/gh/grafana/loki/compare/6bbb61eb2f52288038ae0112a3b577a1b7861d4d...255a140451d7aa0c3d579ca69f11e43b3aa8992a

If these are right commits we can dig into the raw reports together to determine why we are finding coverage changes.

I believe you’re right @drazisil, that’s the right parent. Is there a way for me to look at the coverage file inside of Codecov that was uploaded for each of these? I would imagine if we could see that, it should show no changes, but perhaps that isn’t the reality.

Thanks for your help!

Yep, you should be able to get the raw uploaded report from the builds tag for each commit on Codecov.