Different results when using the node package compared to the bash script

1

Hey,

Following the recent security issue with the bash uploader, I wanted to switch to using the node package instead of fetching+executing the bash script like before.
And I was surprised that the result in codecov is notably different.

In our project we’re using jest with these options for the coverage system:

This looks pretty standard to me. As a result Jest generates 3 files:

  • clover.xml
  • coverage-final.json
  • lcov.info
  • lcov-report/ (directory)

That’s where I noticed a difference, when looking at the CI log:

  • the bash script uploads all 3 clover.xml coverage-final.json lcov.info
  • the node script seems to upload only lcov.info and clover.xml

Could that make a difference ? It’s not clear to me what different informations all these files are, until then I supposed they were containing the same information really.

Some more information:

Codecov’s UI doesn’t make it any easier to analyze…
For example, see this file: Codecov
+10% as coverage, but the 2 lines that are shown are red as “not covered”; they were already not covered previously.
But when looking closer:

we see that 2 lines (42 and 43) were previously shown as “uncovered” and now they are just blank.

Another example is: Codecov. Again the UI is quite bad… let’s look at the files directly:

We clearly see a lot of the case disjunctions in the JSX code are missing in the new coverage.

I believe the “previous” information is correct in these 2 cases but haven’t looked at every cases.

All that to say that the behavior is different. Which might be OK :slight_smile: I guess I mostly need some guidance from you experts on how to get the same behavior as with the bash script. Happy to help with more data if needed!

Thanks!

2

@julienwaj looks like we don’t search directly for coverage-final.json in the node uploader, but we do in bash. I’ll make a PR and look to get it released next week.

3

Thanks so much!

But is it expected that these various files give different information in the first place?

4

@julienwaj I’ll be honest I’m not completely sure. Clearly it looks like there is a difference from our processing of the file, but I haven’t dug into the specifics of how jest creates coverage files to be able to answer authoritatively.

5

I think Jest just uses istanbul behind the scene nowadays.

6

@julienwaj, I’ve updated the node uploader to 3.8.2. Let me know if that works for you!

7

Yeah this seems to work fine now :slight_smile: Thanks for the fix!

I’m still curious about the differences between all these files.

8

@julienwaj one day I will do a deep dive into what the underpinning files are doing for these coverage tools. I will be sure to write it up and post it if that is helpful.