Selected base commit when PR is based on an older commit of the target branch

Support
1

Before submitting a topic, please confirm the following

I have searched for similar issues before creating this topic.
I have verified that my repository is using the Codecov GitHub app, if using GitHub
I have validated my codecov.yaml configuration file.
I have filled out the below sections to the best of my ability.

Description

Say your main branch is on commit A and then two things happen. First, commit B lands in your main branch in a way that affects its coverage data. Coverage data for both commits is uploaded to codecov. Second, a PR is opened against main, but the first common commit between the PR and main is the commit A. In other words, the PR isn’t up-to-date with its target branch.

In this case, for a PR build, most CI systems would fetch pull/<PR num>/merge from GitHub which is the result of merging that PR into the current state of its target branch. In this case, it would merge the PR into main at commit B and pick up the coverage changes from that commit.

CI/CD URL

I set up this scenario in a dummy repo at change something by bmw · Pull Request #3 · bmw/codecov-test · GitHub.

Expected Results

In the codecov report for the pull request, I’d expect only coverage changes that would take place as a result of that pull request to be reported.

Actual Results

When codecov computes coverage changes, it selects commit A as the base commit since it’s the first common commit between main and the PR branch. As a result, codecov will report that the PR will cause the change in coverage caused by commit B despite the PR having nothing to do with commit B.

Additional Information

It seems to me like codecov’s uploader should report that main was on commit B when CI ran and then compare the coverage change between the PR and commit B.

Assuming that’s not going to happen anytime soon, what’s the best way to solve this? Is it to manually set the --parent option on the codecov uploader? Is it to force all PRs to be up-to-date with the target branch?

2

@bmw, we use commit A in all of these cases because we cannot always make assumptions on the merge of PR and commit B. It is possible to use the --parent option to spoof it, but honestly, I think it’s more of a pain to handle, and will cause you headaches down the road if you’re not careful.

3

Thanks for the quick response!

Can you tell or link me to more info about the kind of problems I may have if I set --parent?

Alternatively, can you tell me another way to solve the problem of unrelated coverage changes being reported in PRs like this? Is the best option to force PRs to be up-to-date with the target branch? Is there anything else?

4

@bmw, I don’t have a resource or page to point you at, this is just a behavior that we don’t necessarily encourage and won’t cleanly solve your problem. To be clear, we will always use the first commit before a branching that has coverage as the BASE. As an example,

---A---B (main)
    \
     ----C (feature)

We will never use B as the base commit to compare against C. You will need to keep the PR up to date with B if you want to use that as the BASE commit.

5

Got it. Thanks.

Sorry for continuing to push this issue, but can you help me understand what kind of problems can occur if --parent is set to the latest commit on the target branch? In the past, my team and I weren’t enthusiastic about requiring all PRs to be up-to-date so understanding the pitfalls here would help us.

6

It is likely to really mess with your commit tree (because we take --parent as the correct one going forward) further down the road. If you do not upload coverage for every commit, you will probably see wildly inaccurate coverage for some commits.

This is really a, yes, we support --parent as a feature, but you’re on your own in knowing your codebase the best.

1 Like