A decrease in the coverage percentage is not a reliable metric

If a project that has less than 100% coverage and some of the code that is covered is deleted codecov/project reports a decrease in coverage and sets the status as failed even if all the lines that are not covered were not covered before either. See this commit for example Update test_hello.py · hexagonrecursion/tmp-cc@a140892 · GitHub

It would be better to use the change in the number of lines not covered instead of the change in the coverage percentage to decide if the check should fail

A real world example: Fix unescaped } breaking expansions that follow it by hexagonrecursion · Pull Request #1961 · tox-dev/tox · GitHub

1 Like

Hey @hexagonrecursion, great point here. And thanks for making the POC repo to show what you meant.

A few points here:

  1. Yes, though often removing code is a very healthy choice for a project, removing covered code can have the impact of lowering code coverage due to the math of numerator / denominator of code coverage.

—> 3/4 covered lines = 75%, 4/5 covered lines = 80%. Boo, percentages. We do try to make up for this using something we call “offset coverage”: Coverage Offset

  1. If you only want to look at the added lines, that’s where patch coverage comes from. Status Checks

In your example commit you sent, the patch status would be 100% as you added only covered lines.

Nonetheless, let me make sure the coverage offset concept is working as expected from #1.

Codecov will set the Project Status and Patch Status to success :white_check_mark: when it detects that the coverage was decreased due to the offset of the removed lines . Clearly, the coverage is lower, but not due to a developer fault.

This is not what I’m experiencing. The project status was set to fail. I got a red x in the github UI where according to the document you linked a green checkmark should be.

1 Like

Hey, I got the same issue on this. Has this problem been fixed? It seems like the coverage offset was not working as expected when the code changed because of the deletion of lines.

Hi @JasonZhongZexin and future viewers, we have currently pulled offset coverage as it was causing more heartache than it was solving. We do not have a timeline for when this will be re-implemented.

It’s not a great solution, but I would suggest setting the threshold on status checks to something small like 0.01% or similar to help mitigate these issues.

@JasonZhongZexin I can confirm that a better treatment of declining coverage % is prioritized over the next few months