Flag -N to set the base report does not work

[rolgalan]

Description

We have several issues with codecov trying to guess the base commit to compare against, so we decided to use the -N flag to tell it ourselves.

I’ve seen several other people having the base report problem (one, two, three). All suggestions to fix this are using the -N flag… but we cannot make it work.

We first had an issue with codecov identifying the head commit, but we solved it by using the -C flag. This one works fine. We tried doing the same with -N, but unsuccessfully.

Codecov bash script is properly setting the parent parameter when uploading the reports to codecov… but the report I see is wrongly using the incorrect parent, and not the one provided.

Repository

Private repo

Steps to Reproduce

1. Create a branch feature from a base branch develop
2. Commit anything into base branch develop
3. Open a pull request from “delayed” branch feature
4. Configure your CI to do a virtual merge on the feature with recent develop changes
5. Run your tests on this virtual merge (I call it virtual, because is not pushed)
6. Since codecov is not picking correctly the base commit to compare, find it yourself and provide it with -N

Expected behavior: Get the report comparing against the commit passed with -N

Actual behavior: The report is comparing about whatever codecov script picks without the parameter

Flakiness? Always

Versions

Github and jenkins. Using the bash uploader.
We allow outdated PRs, but have the Merging the pull request with the current target branch revision enabled.

Additional Information

Let me show a git tree to visualize better the problem.

*   5555 Virtual Merge commit develop into feature branch
|\  
| *   0022 Latest develop commit  
| *  0011 Another develop commit
* |  1122 Second commit on feature branch
* |  1111 First commit on feature branch
|/   
*    0000 First develop commit

We have a base commit 0000 in develop branch.
We create a branch feature from there and push two commits, 1111 and 1122. The head of the branch is 1122 and this is the commit we want to bind the report. We use -C to tell this to the bash codecov script.
While we open our PR, other developer pushes two more commits to develop: 0011 and 0022. The head of develop is 0022.

Now, we would like to compare our test report with develop. If we do nothing, codecov picks 0000, which is outdated, producing a fake and useless report.

If we tell bash codecov script to use 0022 with -N flag to have the most updated develop version… codecov ignores us

Console output

curl -s codecov.io/bash | bash -s – -t token -C 1122 -N 0022
[ … ignoring … ]
==> Gzipping contents
==> Uploading reports
url: codecov.io

query: parent=0022&branch=PR-1206&commit=1122&build=24&build_url=ignore&name=&tag=&slug=ignore&service=jenkins&flags=&pr=1206&job=

→ Pinging Codecov

codecov.io/upload/v4?package=ignore&token=secret&parent=0022&branch=PR-1206&commit=1122&build=24&build_url=ignore&name=&tag=&slug=ignore&service=jenkins&flags=&pr=1206&job=

→ Uploading

→ View reports at codecov.io/github/company/commit/1122

Up to here it looks ok… but when I open the report, it’s comparing with the wrong commit 000 instead!

Appreciation

Thanks!

[drazisil]

Hi @rolgalan

I apologize, there was some confusion about how -N works, including myself, so I’ve given out some bad information.

The -N flag allows you to set the parent commit, not the base commit, as I believed. This means it’s not helpful in overriding our automatic PR base picking system.

I believe the alibity to override this is a feature request at this point, I am double-checking with product and engineering to see if there is a current way about this.

[rolgalan]

Hi @drazisil!

But the parent of what? What would be the purpose of setting a parent commit if is not to be used as the base report?

I understood exactly the same thing when read the documentation of the bash script.

codecov/codecov-bash/blob/e06ebeaf72bab0bb22ab46215e6fa7cae85c7fc8/codecov#L129

Thanks for your answer, really curious to know if there is other way to provide this.

[drazisil]

The parent of the commit being uploaded. I’m very fuzzy on the use case of this, myself, It’s apparently helpful in very large and complex high-speed pipelines where the commits may change faster then GitHub can keep up.

I totally thought it was PR base picking

Will let you know if I find a solution.

[scott]

I also need this functionality and added it to the feature request section. Set base commit

[jerrod]

Hey y’all, just clarifying a couple things for this thread and all future readers.

1.) We have a feature to manually pick parents of commits (docs)

2.) We do not have a feature to manually pick base of a PR.

3.) Codecov will always pick the base of the PR as where the branch happened (or prior). If you want to have the head of master as a comparison:

At that point, the only way to get this to work would be merge master into the feature branch so the head commit of master becomes the base commit of the feature branch.

Codecov should correctly show the diff of the feature branch with master merged into it.

Let me know any questions!

Jerrod