PHP: Codecov shows non-executable lines as not covered and wrong execution counts from clover.xml

Support Bug Fixes
1

Description

I upload coverage reports in clover.xml format for a PHPUnit. Coverage data is created with phpunit.

Issue 1: Lines not listed in the clover.xml file should be considered non-executable (as opposed to executable lines with an execution count value of 0). For example, the “} else {” line of an if-else conditional is considered such a line, that codecov counts as uncovered even though all statements of the else case have been executed. For comparison, I uploaded the same XML files to coveralls, which show these lines as non-executable as expected (white). So does the HTML report generated by PHPUnit itself.

Issue 2: As I understand the codecov docs, the number in green circle next to each line should show the number of test cases that executed this line, so I would expect the number from the clover file there. However, in my reports I always see 1 instead. Additionally, I have two merged reports with different flags. In the initial view, both flags are enabled and the green circles show 1. If I disable one of the flags in the filter and then reenable it, the numbers change to 2.

Commit SHAs

ce0fd04

Repository

CI/CD or Build URL

Github actions used as CI system
https://github.com/mstilkerich/rcmcarddav/runs/1671395698?check_suite_focus=true

Uploader

I use the codecov/codecov-action@v1 action to upload.

I upload two reports for each build using different flags.

Codecov Output

Omitted because of verbosity, please see link to github actions build. The output shows no errors.

Expected Results

For example, see the file src/DelayedPhotoLoader.php line 120. The line should be either white or green. (From the clover files, I would expect white, as a user I would expect green).

Actual Results

The lines are shown as uncovered (red) and full coverage thus cannot be achieved.

2

Hi @mstilkerich,

For issue 1, the reason line 120 is showing coverage is that we are mirroring what is shown in the dbinterop report:

image
image843×576 167 KB

We try to match as closely as possible to the source matter here, especially because this would affect branch coverage.

For issue 2, it’s not actually the number of test cases. In general, it’s the number of builds that have that line. There is currently a bug with that element, but as we don’t get information on execution numbers, we cannot display it unfortunately.

3

Hello, thanks for the information. WIth that I was able to fix the issue: For anyone who has the same problem, you need to instruct PHPUnit to process uncovered files (processUncoveredFiles=true) in order to get these lines marked as non-executable also for test runs that have no coverage at all for the file.

Concerning the execution counts, I find your docs a bit confusing:

Lines that are hit (covered and executed). The number in the green label represents execute count.

In interpreted “execute count” as the count given in the clover.xml, not the number of builds. I think it would also be more helpful than the number of builds, since I can filter those already by their flag.

1 Like
4

@mstilkerich, yes this is on us for having confusing documentation. I’ll work with the team to update this.