If a project that has less than 100% coverage and some of the code that is covered is deleted codecov/project reports a decrease in coverage and sets the status as failed even if all the lines that are not covered were not covered before either. See this commit for example Update test_hello.py · hexagonrecursion/tmp-cc@a140892 · GitHub
It would be better to use the change in the number of lines not covered instead of the change in the coverage percentage to decide if the check should fail
Hey @hexagonrecursion, great point here. And thanks for making the POC repo to show what you meant.
A few points here:
Yes, though often removing code is a very healthy choice for a project, removing covered code can have the impact of lowering code coverage due to the math of numerator / denominator of code coverage.
—> 3/4 covered lines = 75%, 4/5 covered lines = 80%. Boo, percentages. We do try to make up for this using something we call “offset coverage”: https://docs.codecov.io/docs/coverage-offset
If you only want to look at the added lines, that’s where patch coverage comes from. Status Checks
In your example commit you sent, the patch status would be 100% as you added only covered lines.
Codecov will set the Project Status and Patch Status to success when it detects that the coverage was decreased due to the offset of the removed lines . Clearly, the coverage is lower, but not due to a developer fault.
This is not what I’m experiencing. The project status was set to fail. I got a red x in the github UI where according to the document you linked a green checkmark should be.
Hey, I got the same issue on this. Has this problem been fixed? It seems like the coverage offset was not working as expected when the code changed because of the deletion of lines.
Hi @JasonZhongZexin and future viewers, we have currently pulled offset coverage as it was causing more heartache than it was solving. We do not have a timeline for when this will be re-implemented.
It’s not a great solution, but I would suggest setting the threshold on status checks to something small like 0.01% or similar to help mitigate these issues.
@JasonZhongZexin, we recently released a flag in the codecov.yml that should help fix this issue. It’s turned on by default, so no action is required on your part.
If there are no unexpected changes and the patch check is 100% (all “new” code is covered), the project status check will also pass. So if you have a PR that removes a bunch of unused code, it should no longer fail the status check.
when it detects that the coverage was decreased due to the offset of the removed lines . Clearly, the coverage is lower, but not due to a developer fault.