How to make codecov branch status checks conditional

Support
1

Description

I have the codecov/project and codecov/patch status
set as required for protected branches in GitHub.
I have tried every combination of the codecov.yml
settings to achieve the following:

  • If for a given commit coverage is uploaded
    all codecov checks must pass. If no coverage
    is uploaded the status checks should be set
    to “success” automatically. (IE. updating docs, README, etc).

In the codecov.yml reference
it says about the “if_not_found” setting:

" success : the status will pass if there is no report for the head. Use this on commits / PRs where you won’t be uploading coverage but still want codecov status checks to pass. "

So what I’m expecting is that codecov status checks
are set to success if no coverage is uploaded after
all actions are run. However, its always stays stuck
at "Expected — Waiting for status to be reported "

I’m thinking that maybe it has something to
do with “skipped” actions not being recognized as
completed.

CI/CD URL

(ci(monorepo): wait_for_ci = false, n_builds =1 by calvinalkan · Pull Request #112 · snicco/snicco · GitHub)

Codecov Output

No output. Uploading coverage is skipped for
changes outside of src/*.php

Expected Results

codcov/project and codecov/patch
should be set to true in the linked PR.

Actual Results

codcov/project and codecov/patch
are stuck waiting for a status to be reported.

Additional Information

This is the codecov.yml currently used for this PR.

But I tried every other imaginable combination.

2

hi @calvinalkan I’m not sure this is quite possible. I see that the if_not_found reference is misleading, as it is more specific to flags.

The reason that it’s still waiting on a status, is that Codecov is not made aware of the commit (since no uploads are sent up). So Codecov doesn’t send up notifications. A way around this would be to send up a blank coverage report.

3

Do you have an example of how to generate and upload a blank coverage report?

4

There’re both if_not_found (Status Checks) and if_no_uploads (seen in some public gists), and both are not making status check green. If both references are just wrong, and the only way is to upload a blank report, I think there’s definitely something to fix inside codecov so we can finally update stuff not influencing code coverage.

5

@FallMonkey definitely, this is something we’re looking into doing. Unfortunately, right now it’s not possible.

@extesy If you were to create a fake coverage report, I would create a file coverage-automatic.json to be uploaded that might look like

{"files":[{"name":"{{ filename }}","coverage":[1]}],"uploader":"empty"}

This implies that some file (that exists in the git repo) named {{ filename }} has coverage on the first line. You may need to change it as appropriate.

1 Like
6

Thx for the followup!

Seems there’s one more thing until we get this working. After uploading the fake report json to codecov, my codecov will be stuck like below. Pushing another commit will get this to pass, but that new one will be stuck again. Forcing pushing notification will also work but it doesnt seem right.

image
image1267×114 10.4 KB

image
image1723×133 13.1 KB

In this case, what CI completion exactly is codecov waiting for? The fake report upload workflow is the only CI running here and it’s already complete. Note: in normal PRs where real report will be uploaded, this works totally fine so our GHE integration with codecov app is functioning.

7

@FallMonkey strange, can you share the commit SHA for one of these instances?

8

a0e3f308bb72f184daa4145709df4b14ab6b93ff

It’s a GHE repo and using self-hosted codecov so not sure if you could access that.

9

Any update here? Lemme know if I need to provide additional information.

10

@FallMonkey given you are using us self-hosted, we will have to take a different approach to support. I spoke with your colleague and they will be tapping you in on a separate thread.

1 Like
11

For all Codecov users:

Medium term: I agree this is a problem and a good feature request. It had been in the backlog from a couple months back but is starting to get more attention now as a feature to better integrate with Github.

Short term: As far as remediation before we can build around this, I’m posting a response from our team:

A) You need to be using carryforward flags. That way, we won’t have to worry about any large swings in code coverage. You need the entire set of their coverage reports uploaded under one flag

B) We still need a coverage report for our notification system to send out the PR comment/status check. So “blank report” is actually a misnomer, because it has to contain data. What I would suggest is to find a file that will always exist where the first line will always be covered, and upload that file like this named coverage.xml

/home/runner/work/{{ owner name }}/{{ repo name }}

alternatively you could use this format and call it something like coverage.json
{“files”:[{“name”:“{{ filename }}”,“coverage”:[1]}],“uploader”:“R”}

12

Here is an example report that will work for this purpose. Create and check the following three lines into the root of your repo as lcov.info

SF:lcov.info
DA:1,1
end_of_record
1 Like