Coverage report not showing any "green" lines (executed code) on app.codecov.io but works on codecov.io

Description

A file coverage report on the new app.codecov.io is not not showing any “green” lines, i.e., not marking any lines as “code that is being covered”, despite there being hundreds; it works in another view of the same data on the old codecov.io

Commit SHAs

E.g. ff4358f9ab0d01e8e194b2651d62ee6bbd985fed

Repository

CI/CD or Build URL

Uploader

We are using the bash uploader, i.e., https://codecov.io/bash

Codecov Output

If you really need this, I guess we can extract it from the GitHub Action CI logs above (but there are like … 18 or so ?)

Expected Results

Code which is executed should be marked with green lines; code not executed with red lines; non-code stays white. As is the case on e.g. https://codecov.io/gh/gap-system/gap/src/ff4358f9ab0d01e8e194b2651d62ee6bbd985fed/src/io.c

Actual Results

Looking at Codecov there are no green lines, only a couple red ones – however, that code is really being executed: For starters in a few places like line 498 there are these little green circles with a hover tool tip saying “this line appeared in 4 coverage reports”, despite that line being shown as white. Also if I directly go to the same page but now for the commit, not the PR (i.e. Code coverage done right.) then I do get those missing green lines. Note how that URL is on codecov.io while the broken one is on app.codecov.io

Additional Information

None.

Hmmm @fingolfin, I think something strange happened through your CI.

These are the commits for that PR. You’ll notice the most recent commit is only showing 4 builds. It also is showing that this commits was deleted or rebased on. For reference, a previous commit had 8 uploads to Codecov.

Hmm, actually, the “most recent” commit was not most recent, so I ignored it, considering it yet another fluke in the codecov UI (to which I’ve become quite used over the years, so I don’t really think about them anymore). And yeah, something weird is going on, because a full CI will upload more like 15 build reports, see e.g. Code coverage done right.

But whether it is 4, 8 or 15 reports: isn’t it still weird that Codecov shows no code executed code lines? Something is really weird here.

BTW, I see many oddities with codecov all the time. E.g on another PR it reports a massive coverage drop that I can’t explain at all. I know that these things often are fixed by re-running the CI, though. I tried to see if Codecov there works better, but it just shows me a spin wheel “forever”

Anyway, I’ve just rebased that PR and triggered a full rebuild for it this way, let’s wait and see if that helps.

Latest commit (the automatic merge commit created by GitHub) is this one and it has 14 build results uploaded as I write this. But the PR specific file view of io.c appears to still be bogus, albeit in a different way (it now has both red and green lines, but they cover a seemingly random subset of lines). While the commit specific file view of io.c seems fine (well, modulo the issue I describe in another post).

I note that one is on the app. subdomain of codecov, the other on the top domain, so maybe this is a matter of the new versus old website code?

I just observed a similar problem on a completely different project, even using a different programming language (Julia). The CI report for this GitHub pull request reports increased coverage. Looking at the PR specific file coverage report for mpoly.jl (or fo I also only see

  • white lines
  • red lines
  • lines that changed from red to green
  • lines marked with a green circle but otherwise white (but only a few of them)

But no purely green lines, even though I know they must be there; and indeed, the lines with the green circles should be green (but also many more). This screen shot might illustrate the issue. Note that this is on app.codecov.io once again.

I then went to the commit status page on Codecov, and searched for the relevant file there and ended up at this commit specific file coverage report for mpoly.jl – note that it is on codecov.io, the old site. Here is a screenshot from the same function; it looks reasonable here.

Note how there are green lines; and more green circles.

My impression thus is quite strongly that this is a bug on the “new” site.

@fingolfin, we made some changes to the commit details page that should have fixed this. I see on both versions of the site the 2nd screenshot you posted. Are you still seeing a difference?

@tom unfortunately I see no change. Both on the page from my original report but also for the same file in a more recent (from today) PR.

It still looks like this:

While on the old site, looking at a commit specific file report (again, from my original issue report), I see this:

Thanks @fingolfin, I’ll ping the product team again here.

1 Like

@fingolfin we’ve been working on updating our UI to incorporate the above changes, but I just wanted to circle back and see if this was still an issue. If so, do you have a more recent instance of the bug?

Still happens exactly the same way.

Bad: Codecov

Good: Code coverage done right.