Incorrect coverage number reported by codecov

[ramachandranms]

Description

Incorrect coverage number reported by codecov for this java class generated by jacoco

Repository

GitHub - ramachandranms/incubator-hudi: Upserts And Incremental Processing on Big Data

Steps to Reproduce

1. Run the travis-ci build, which will publish the report to codecov
2. Observe the code coverage for hudi-common/src/main/java/org/apache/hudi/common/model/HoodieWriteStat.java

Expected behavior:

Code coverage to be around 85% as given by a local jacoco report

Actual behavior:
Codecov reports only show ~20% as covered

Flakiness?

This happens all the time

Versions

Happens across all OS and browser versions

Additional Information

[drazisil]

Hi @ramachandranms

jacoco reports sometimes have expiry issues, can you share a commit SHA so I can check the logs and see what’s up?

[ramachandranms]

Hi @drazisil. I restarted a build for master in travis. The commit SHA is 2d040145810b8b14c59c5882f9115698351039d1. The build should take about 30 mins to complete.

Thanks for looking into this.

[drazisil]

I only have one report from 3 March when you uploaded it before. Do you normally upload a single report?

[ramachandranms]

This is my private fork. if you are interested in frequent updates, you can checkout the apache project. GitHub - apache/hudi: Upserts, Deletes And Incremental Processing on Big Data.

Sample commit for that file (uploaded less than 3 hours ago)

Code coverage done right.

[ramachandranms]

hi @drazisil , did you get a chance to look into this issue ? please let me know if you need any further information to troubleshoot this issue

[drazisil]

@ramachandranms

2d040145810b8b14c59c5882f9115698351039d1

I didn’t, looking now…

EDIT: I’m not seeing an issues with 2d040145810b8b14c59c5882f9115698351039d1, did you have problems, or did it work as expected?

[ramachandranms]

@drazisil

This is the file i am looking at in that SHA.

codecov is reporting 20.38% coverage and 293 lines missing coverage. But the file have very good amount of coverage and only about 15 lines missing coverage.

The local jacoco report (screenshot below) shows about 85% coverage. The problem is codecov reporting only 20% while jacoco reports 85%.

[drazisil]

Hmm. Here’s the uploaded coverage report, will have to look though it to see if the lines are reported covered. https://codecov.io/codecov/v4/raw/2020-03-03/87AC6CA37FD62FF9A5E96EDF8BC725C6/2d040145810b8b14c59c5882f9115698351039d1/b6f33b40-6aec-43c7-9b1b-733c5672d950.txt

Do you have an example line that you believe is covered that Codecov says is not that I can look in the report for?

[ramachandranms]

The covered/uncovered seems to be correct visually (if we got by green and red highlighting). It’s just that the counting of lines seem wrong. codecov is including variables declarations in the uncovered count it seems.

[roniemartinez]

Same issue here: # pragma: no cover is still being covered

Reporting is incorrect for file with 100% coverage.

[ramachandranms]

@roniemartinez our problem seems slightly different. the stats looks like the following from jacoco report

     <counter type="INSTRUCTION" missed="147" covered="182"/>
     <counter type="BRANCH" missed="8" covered="0"/>
     <counter type="LINE" missed="12" covered="74"/>
     <counter type="COMPLEXITY" missed="7" covered="49"/>
     <counter type="METHOD" missed="3" covered="49"/>
     <counter type="CLASS" missed="0" covered="2"/>

The problem seems to be on calculating the number of tracked lines, which is not present in the report. ti is calculated as 370+ whereas it should only be around 90 lines. rest are all variable declarations, which should not be part of code coverage.

@drazisil any update on this ?

[drazisil]

Let me make sure I understand:

When looking at a file overlay, the coverage lines are correct, but the number is not? From looking at the report, are you able to guess what might be being counted?

I’m trying to determine if this is a backed or frontend issue. You can add api/ in front of the gh part of most any URL on Codecov to view the underlying data passed to the frontend.

If you can give me an example file URL on Codecov we can take a look together.

[ramachandranms]

@drazisil this is the data for the file i am looking at. I don’t think i understand the internals of codecov to make any assumptions. lmk your thoughts

[drazisil]

@ramachandranms

I’m sorry for sending you to that rabbit hole, I remembered the API data being a bit saner then that.

Lets take a look here referencing the reference (which I had to look up, don’t worry, I don’t have this memorized) https://docs.codecov.io/v4.4.0/reference#totals

We have 368 lines in total, with 293 hit, calculating at "20.38043" percent coverage for the HoodieWriteStat.java file.

We are getting a count of 368 files, because that is how many lines the report has coverage for

If any of those lines should not have coverage data, you will need to find a way to tweak your coverage reporter to not include them on the report. When pasing the report, we only have line nukbers and hit counts, like so

We have no way of knowing the contents of those lines.

[ramachandranms]

thanks @drazisil seems like the best course of action is to ignore coverage for this file, since it is only a POJO and some generated code. will try that out.