Merging Reports states that “Codecov will delay notifications such as Slack, Hipchat, and Pull Request Comments until all reports are uploaded and merged. This prevents posting invalid coverage metrics when the first report is uploaded.”
However, we are experiencing invalid coverage metrics due to early reporting, before our python test finish and python coverage is uploaded.
(our Javascript tests complete in about a minute but python tests take about 8 minutes)
Is there any way to lengthen the notification delay so we don’t get invalid reports?
Thanks for pinging this out to us. @gambreu is investigating why your paid ticket didn’t get picked up.
On the early reporting front, you are 100% right, we discovered the same issue a couple days ago and have noted it as a bug. Will looking into it and let you know what we find, sorry for the error.
Thanks for the response. It turns out we are still on free plan (I thought we were on paid because private repo), sorry. My boss will be buying the paid plan.
We are experiencing the exact same issue.
We have 5 parallel jobs and the PR comment gets created after the first job is finished. It is then updated, but frequently reports wrong coverage %, especially for flags.
Sometimes, the last job posts the correct flag coverage %, sometimes it doesn’t and leaves the PR comment with incorrect numbers.
It really seems that Codecov doesn’t delay the PR notification.
From what I understood from your documentation, Codecov is supposed to wait until the build is finished. But how does it know is finished? Are you using the Github PR build status?
FYI, we are using Semaphore 2 and the Github PR status looks like that:
Thanks @lakim – we actually take our queue from the CI tool itself. It is posting early no doubt and we have this in our product board to investigate. Thanks for your patience.
