Inaccurate coverage metrics

https://docs.codecov.io/docs/merging-reports states that “Codecov will delay notifications such as Slack, Hipchat, and Pull Request Comments until all reports are uploaded and merged. This prevents posting invalid coverage metrics when the first report is uploaded.”

However, we are experiencing invalid coverage metrics due to early reporting, before our python test finish and python coverage is uploaded.

(our Javascript tests complete in about a minute but python tests take about 8 minutes)

Is there any way to lengthen the notification delay so we don’t get invalid reports?

Before: https://i.imgur.com/ORtC55N.png

After: https://i.imgur.com/ctL9z8J.png

We are using circleci with codecov orb:

version: 2.1
orbs:
  codecov: codecov/codecov@1.0.4

Python upload:

      - codecov/upload:
          flags: backend

Javascript upload:

      - codecov/upload:
          flags: frontend

I sent a email about this weeks ago but didn’t receive a response. We are on the paid plan with a private repo.

Hey Caleb,

Thanks for pinging this out to us. @gambreu is investigating why your paid ticket didn’t get picked up.

On the early reporting front, you are 100% right, we discovered the same issue a couple days ago and have noted it as a bug. Will looking into it and let you know what we find, sorry for the error.

Best,
Jerrod

1 Like

Thanks for the response. It turns out we are still on free plan (I thought we were on paid because private repo), sorry. My boss will be buying the paid plan.

Cool! Glad to have 15five onboard. I started using and loving your product a few years back.

1 Like

Hello,

We are experiencing the exact same issue.
We have 5 parallel jobs and the PR comment gets created after the first job is finished. It is then updated, but frequently reports wrong coverage %, especially for flags.

Sometimes, the last job posts the correct flag coverage %, sometimes it doesn’t and leaves the PR comment with incorrect numbers.

Do you think it is related to this bug?

Cheers,
Louis

It really seems that Codecov doesn’t delay the PR notification.
From what I understood from your documentation, Codecov is supposed to wait until the build is finished. But how does it know is finished? Are you using the Github PR build status?

FYI, we are using Semaphore 2 and the Github PR status looks like that:

2019-04-10_15-03

Cheers,
Louis

Thanks @lakim – we actually take our queue from the CI tool itself. It is posting early no doubt and we have this in our product board to investigate. Thanks for your patience.

Jerrod