If a project that has less than 100% coverage and some of the code that is covered is deleted codecov/project reports a decrease in coverage and sets the status as failed even if all the lines that are not covered were not covered before either. See this commit for example Update test_hello.py · hexagonrecursion/tmp-cc@a140892 · GitHub
It would be better to use the change in the number of lines not covered instead of the change in the coverage percentage to decide if the check should fail
A real world example: Fix unescaped } breaking expansions that follow it by hexagonrecursion · Pull Request #1961 · tox-dev/tox · GitHub
Hey @hexagonrecursion, great point here. And thanks for making the POC repo to show what you meant.
A few points here:
- Yes, though often removing code is a very healthy choice for a project, removing covered code can have the impact of lowering code coverage due to the math of numerator / denominator of code coverage.
—> 3/4 covered lines = 75%, 4/5 covered lines = 80%. Boo, percentages. We do try to make up for this using something we call “offset coverage”: Coverage Offset
- If you only want to look at the added lines, that’s where
patch coverage comes from. Status Checks
In your example commit you sent, the
patch status would be 100% as you added only covered lines.
Nonetheless, let me make sure the
coverage offset concept is working as expected from #1.
Codecov will set the Project Status and Patch Status to
success when it detects that the coverage was decreased due to the offset of the removed lines . Clearly, the coverage is lower, but not due to a developer fault.
This is not what I’m experiencing. The project status was set to fail. I got a red x in the github UI where according to the document you linked a green checkmark should be.
Hey, I got the same issue on this. Has this problem been fixed? It seems like the coverage offset was not working as expected when the code changed because of the deletion of lines.
Hi @JasonZhongZexin and future viewers, we have currently pulled
offset coverage as it was causing more heartache than it was solving. We do not have a timeline for when this will be re-implemented.
It’s not a great solution, but I would suggest setting the threshold on status checks to something small like
0.01% or similar to help mitigate these issues.
@JasonZhongZexin I can confirm that a better treatment of declining coverage % is prioritized over the next few months