Codecov not properly processing coverage data

Support Bug Fixes
1

Description

I saw some new flakiness with codecov this morning which I wanted to report in hopes of getting it fixed either in codecov or in a workaround on our end.

This morning codecov was reporting a large drop in reported code coverage in the GitHub status check for our windows flag for one of our PRs. All CI builds successfully uploaded coverage data which did not show a large drop in coverage.

I reran CI which caused a new set of coverage reports to be uploaded and the status check passed. The reported coverage between the first and second CI run are identical. Any idea what may have happened?

You can see the builds for the commit at https://codecov.io/gh/certbot/certbot/commit/ba31a111bf924b409933097ba4395a41b93da2e6/build.

Repository

2

Hey @bmw the builds/uploads are likely coming through at different times, and depending on when the check runs, perhaps not all the uploads have reached Codecov from the CI pipeline.

My recommendation is to use after_n_builds feature in the YAML to not post statuses until all builds are confirmed uploaded.

Let me know if this helps!

3

Does after_n_builds also affect commit statuses or just pull request comments? We have pull request comments disabled.