Codecov server 500 error and endless "processing"

Description

Codecov_io is very much useless for our repo, no two runs give the same coverage, even with a repeated force-push with no content changes, variance is about ±2%.

  • Clicking a PR link [1] either says the report could not be found in the archives, or it gives a 500 error.
  • Checking the commits [2] on codecov_io often shows some of the commits in an endless “Reports are queued for processing…” state (Even many months later).
  • Github comment [3] only very very very rarely (1% or so?) shows a correct looking report for each flag. Usual behavior is it only shows a coverage percentage for 5-6 of the flags (~30 flags total), or it shows correct-looking percentage for 5-6 flags and 100% for all the rest (and we know that is impossible). Sometimes 5-6 correct and a couple 100% with the rest empty.

1: https://codecov.io/gh/zlib-ng/zlib-ng/pull/523
2: https://codecov.io/gh/zlib-ng/zlib-ng/commit/af399ea6c309dd8fbb2ce107115fa7020c388e6d/graphs
3: https://github.com/zlib-ng/zlib-ng/pull/523#issuecomment-585378227

We have made sure we don’t have any codecov.yaml file, and not anything filled out for that in the project config either.
We have tried deleting all project data on codecov_io (that is why there is still very little data there now).
We have often had upload errors, to the point we forked codecov-python, but even with the rare occasion of all uploads succeeding the things above don’t work.
All permissions in github etc have been checked repeatedly.

We currently use Github Actions for CI builds, but used to use Travis CI with the exact same problems (In fact we switched because we thought it might fix the upload problems).
Our CI builds run from Linux, MacOS and Windows. gcc and clang. x86, x86_64, arm, aarch64, ppc, ppc64 and s390x.

We first tried to solve these problems ~a year ago, but figured it was probably a codecov_io bug that might be fixed in their next release, but it seems it is rare enough that it hasn’t been fixed yet. That was when we changed over from Travis to Github Actions too actually.

Less important, but likely to be related; The sourcegraph integration does not seem to work properly for our repo. It works when viewing other repos using codecov on github, but in ours it won’t show coverage data in the header or hit counts. Browsing our code through sourcegraph_com shows hit counts per line perhaps once in a blue moon (I have seen it work once, but then it broke when I switched to another file, going back to the first file was then also broken).
We have followed the docs repeatedly and checked all permissions etc.
I am not allowed to post another link.

Repository

Versions

Not relevant, but Win10+Chrome, Win10+Waterfox, Fedora 30+Chrome, Fedora 30+Waterfox

I do wish codecov would whitelist codecov.io from their “No more than 4 urls in a post” rule…

The sourcegraph link is: https://sourcegraph.com/github.com/zlib-ng/zlib-ng/-/blob/deflate_medium.c

Hi @Dead2

I thought I had whitelisted codecov.io, I’ll double-check.

Regarding your issues, I think you may have a couple, please give me some time to look though this.

Regarding the Sourcegraph extension, this is managed by Sourcegraph, and only hosted by us. If yiu open an issue on the repo or contact their support they are very responsive and will likely be able to assist you in solving that part promptly.

I think the sourcegraph issue is probably closely related to the others in this post. Simply that sourcegraph cannot get the data, for the same reason I cannot get the data directly at codecov.io. Will follow up with sourcegraph if the issue continues after the other problems have been solved.

1 Like

I believe you are hitting an edge case where the reports are getting corrupted in storage. We have prioritized to investigate and fix this.

As a workaround, uploading the report again should fix it.

The workaround of uploading the report again would in our case require a full CI re-build, and it seems we have something like a 99% chance of each CI run to have at least one corrupt report (I assume it is one or more of the ~30 reports that get corrupted, ruining the whole PR/commit report).

@drazisil Thank you for looking into this and prioritizing it, we think that codecov.io will be very beneficial for zlib-ng as a project if we can get it working properly. :slight_smile:

While we still have a 500 error for the above link, most of the new pull requests and new commits work, and most reports seem to get properly processed (previously 80~-90% loss, now something like 0-5%).
Now codecov.io is often giving reports of unchanged coverage, we never saw that before.
@drazisil Thanks for getting this fixed :slight_smile:

1 Like