Currently the PR comment is posted / updated as and when new reports come in (subject to config settings).
I would be happy for the comment to be posted and updated if it showed it as “in progress” or similar to highlight that it is subject to change.
Description
Include some details in the PR comment about the “latest details” when the comment was last updated – e.g. a commit hash, number of uploads received (ideally compared to the “normal” or expected number usually received), and perhaps “last updated” text in the comment itself.
Regarding the “status update”, text similar to this would work well:
“This comment is based on commit xyz987 but it is now outdated - 4 coverage reports for the commit abc123 have been received, and this comment will be updated once 25 (or more) coverage reports have been uploaded.”
Examples
At the moment, the only way to see when the comment was last updated is to click through to the edit history.
The thing is, isn’t it always subject to change? At any moment, new coverage reports can be uploaded that will cause the comment to be revised. As long as a PR branch is editable (and, it’s always editable as long as the PR is open), the coverage data is always in flux.
OK, that’s already there, right in your screenshot in fact:
As new commits are pushed to the PR branch and analyzed, that ref is updated to point to the one the comment is based on. But all the coverage data from a given build run is going to be aggregated under the same hash, so it only changes the first time new data comes in, which seems like the opposite of what you say you’re looking for.
The thing is… how would they know that?
In some of the repos I contribute to, the build matrix — and therefore how the coverage data is split up among the various jobs — is constantly in flux. It can change based on what branch is being targeted, what parts of the tree a PR touches, or what set of dependencies are currently configured to test against (a list that changes from week to week). That’s fine if the number goes up, Codecov can just notice the extra data sets and adjust accordingly, but what happens when the number goes down, and they’re expecting more data that never comes? (Keep in mind, webhooks trigger in response to events, not the absence of events, so it’s hard to trigger a process on “did not receive more data” because that’s literally a non-event.)
I… suppose that could be a way to go. But I can tell you right now it’s going to annoy you, because the time won’t be localized to your time zone. (Or if there’s some configuration variable you can set so that it is, then it’ll annoy your collaborators by always being in your specific time zone.)
But that’s not correct. Or at least, it’s not how Codecov currently works. Right now, the comment is updated every time new data comes in to Codecov, with the most current information as they have it. You’re asking them to delay updating the information (but still update the comment?) until some theoretical “finished” state is reached. Even though that state is, as I’ve said, largely imaginary.
I honestly doubt most users would really want that. I know I don’t.
But, if you do for some reason object to the incremental nature of Codecov’s updates, you could solve that issue for your repo by not sending them data incrementally. It wouldn’t be that hard to locally aggregate the coverage reports, and send them all in one shot when they’ve reached a state that you can declare as “finished”, much more easily than Codecov is able to.
Are you using Github Actions to run your CI? Then it’d be a snap to change your workflow so that, instead of uploading coverage reports to Codecov at the end of each build job, you use actions/upload-artifact to publish each set of coverage data as a Build Artifact.
Then, when the build workflow is complete, add another stage to the workflow that uses actions/download-artifact to retrieve all of the artifacts from the workflow, and send them to Codecov in one consolidated upload.
I don’t know why you’d want that, but it should be totally achievable with local customizations to the coverage submission process.
Hi @MysterAitch, a lot of what @ferdnyc is true. There would be no way for Codecov to know when all uploads have been received as you could always upload more. We have used after_n_builds as a workaround for this since we can alert after that number of builds has been received. You can set that for both status checks and the comment.
