GitHub PR not being updated with final coverage data

Okay, first PR after the change to the yaml already shows one of the problems:

It currently has two comments from a single CI-run, appearing within 4 minutes of each other.
There is still 3 jobs queued, so another comment might still appear when those finally finish.

First comment came when 5 jobs were still pending, and the second came when 3 jobs were pending.

1 Like

@tom Here is another that got 2 comments after we added after_n_builds

And this one is up to 7 comments now:

@tom This one is also up to 7.
3 comments from the first run, 4 from the second run.

I am removing after_n_builds from the yaml now, because it causes the PR comment section to be very messy.

@tom Looks like we are getting extra comments after removing the after_n_builds parameter too.
See: Adjust cmake to install same .h and .pc files as configure by dankegel · Pull Request #642 · zlib-ng/zlib-ng · GitHub
To me, this looks like codecov.io somehow forgets about its previous comments and opts to make a new one. What this instance tells me is that after_n_builds causes the forgetting (or messes up so it cannot be recognized later) so that even after the parameter has been removed we still get the duplication.
IE: after_n_builds does not cause the extra comment directly, but rather by forgetting/messing it up, so the effect is that there will be new comments later.

Not sure whether I managed to explain what I mean well enough, just ask if anything is unclear.

I’ll try to explain using pseudocode. This is how it should work:

global has_comment = 0
---
while(reports):
    if not has_comment:
        post_new_comment()
        has_comment = 1
    else:
        update_comment()

But this is how it looks like to me now:

global has_comment = 0
---
while(reports):
    if not has_comment:
        post_new_comment()
        if not after_n_builds:
            has_comment = 1
    else:
        update_comment()

Hi @Dead2, this clearly got away from me. I checked your most recent PRs, and the values seem to be right. Is this still an issue you’re facing?

@tom
Hi, this problem seems to have gotten better during the summer.
But we still see some issues with updates.
Such as this one Harmonize the CPU architecture preprocessor definitions. by nmoinvaz · Pull Request #719 · zlib-ng/zlib-ng · GitHub

3 builds failed, all of them failed at uploading coverage data after about 4.5 hours of retrying, that is fine in regards of the comment, we can’t expect those to appear.

But one additional build (the #ubuntu_gcc_osb build) is missing from the comment, it also failed to upload coverage data at first, but finally succeeded after 3.9 hours. The website report shows the correct coverage data, but the comment is lacking that builds data from its report. Why wasn’t the comment updated?

So, something is still not quite right with the comment system.

(And uploading is still broken for us on about 1-4 builds per CI run, requiring re-runs of all CI jobs, but that is another issue)

Hi @Dead2, apologies, this is my fault. I wasn’t able to pull logs for this PR, do you have something more recent?

Well, you could have a look at these two: The first had 3 comments in a row, I deleted the two first that had outdated info, then after a force push it made another comment, so I had to delete 3 of 4 comments.
The second had a total of 6 comments…
I have not checked whether the last comment is up-to-date with the numbers shown at the website.

Hi @Dead2, we made some changes to our infrastructure today which should help with the multi-comment issue. Please let me know if you run into it again in case we missed an edge case here.

@tom Looks like you missed something, because the multiple-comments problem is still there:

We also still see different info in the (newest) comment and what the codecov website says for the same PR, where the comment seems not to have been updated.

Thanks @Dead2! I think we are seeing a related problem with checks not completing which is causing this issue. We are working on getting this resolved quickly.

@tom see Remove FORCE_STATIC and FORCE_STORED by Dead2 · Pull Request #744 · zlib-ng/zlib-ng · GitHub for an example of different info in the comment compared to the codecov website (-1.60% vs -1.65% in this example).

That PR is also missing a bunch of data from several CI runs that failed to upload their coverage data.
Could some of these bugs be related to the uploader first registering the upload slot on codecov.io but then failing to actually upload the data to the amazon storage (causing the codecov.io code to look for upload data that is missing)?

@Dead2, we’ve made infra changes on our side that should help with this. I checked some recent PRs on your repo and it looks ok. Can you let me know if this isn’t true?

@Dead2 just wanted to circle back on this and see if you were still getting the issue

@tom Yes, it looks to me like this issue has been fixed. :+1:

1 Like

Hi Tom,

We seem to be getting this issue again where the code coverage bot is not commenting on our PRs. When looking at the CI builds, it successfully uploads the coverage data. I’m wondering if it is something we did or if the service is having problems. Thanks!

As Nathan said, we have had a problem for the past several days where Codecov statistics looks perfectly fine when looking at PRs on codecov.io, but no codecov comment in github, and also no longer any codecov “checks” appear in the CI list.
Nothing has really changed on our end, and I have verified that codecov still has access and is enabled etc according to GitHub.

@Dead2, @nmoinvaz it looks like that particular PR never finished CI
image
We didn’t send a comment because we were waiting for CI to finish. Does this sound right to you?

@tom No, I don’t think that is it unfortunately.

Codecov always used to send comment after first report was uploaded, so unless you changed the default, it should still be doing that. Codecov is also no longer listed in the checks list, it used to do so even before any checks had begun (But showed as a failure due to too low coverage).

Also, our Buildkite server was taken offline several days before codecov stopped working properly, and Buildkite has never been used to upload coverage reports to codecov.

If you check PR 1160, I cancelled the Buildkite check, but still nothing from Codecov unfortunately.

Thanks @Dead2, I’m pretty confused about this. I went through a bunch of your most recently merged PRs, and the reason we don’t send a comment is because the Buildkite check hasn’t returned back. Specifically, we don’t send notifications if there is a status check that isn’t complete.

The PR you posted is showing that that status check failed, and we also do not send notifications in this case.

If you want to disable this behavior, you can add this to a codecov.yml

codecov:
  require_ci_to_pass: false
  notify:
    wait_for_ci: false