Github status / notification isn't delayed (should wait all CI jobs to finish)

[Delgan]

Before submitting a topic, please confirm the following

I have searched for similar issues before creating this topic.
I have verified that my repository is using the Codecov GitHub app, if using GitHub
I have validated my codecov.yaml configuration file.
I have filled out the below sections to the best of my ability.

Description

Hi.

I’m using multiple Github jobs and Codecov merges the reports which is perfect.

However, Codecov updates GIthub status before receiving all reports, causing a misleading “Coverage failed” status for a few minutes.

According to the documentation, Codecov should wait before sending the notification, but it seems it’s not working as expected.

I specified require_ci_to_pass: yes and wait_for_ci: yes but it changed nothing. The only solution I found so far to prevent premature status update is to explicitly set the after_n_builds parameter.

Why isn’t automatic detection of CI completion working in my case?

CI/CD URL

I created a minimal reproducible example in this repository: GitHub - Delgan/codecov-status-test: Just a testing repository

Here is an example of the pipeline (one job is made longer than the other to test delayed coverage): Add testing files · Delgan/codecov-status-test@41ef463 · GitHub

Codecov Output

2022-06-12T08:22:23.6523462Z ##[group]Run codecov/codecov-action@v2
2022-06-12T08:22:23.6523803Z with:
2022-06-12T08:22:23.6524022Z   flags: unittests
2022-06-12T08:22:23.6524290Z   verbose: true
2022-06-12T08:22:23.6524602Z   fail_ci_if_error: true
2022-06-12T08:22:23.6524863Z env:
2022-06-12T08:22:23.6525211Z   pythonLocation: /opt/hostedtoolcache/Python/3.10.4/x64
2022-06-12T08:22:23.6525613Z   LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.10.4/x64/lib
2022-06-12T08:22:23.6525890Z ##[endgroup]
2022-06-12T08:22:23.7792914Z ==> linux OS detected
2022-06-12T08:22:26.6738580Z https://uploader.codecov.io/latest/linux/codecov.SHA256SUM
2022-06-12T08:22:26.8103664Z ==> SHASUM file signed by key id 806bb28aed779869
2022-06-12T08:22:26.9703307Z ==> Uploader SHASUM verified (432553f03714a83ab5cac78be2e7fe6a6050767f78f77e9c7a84bdb1c34a0bd3  codecov)
2022-06-12T08:22:26.9704203Z ==> Running version latest
2022-06-12T08:22:27.0638836Z ==> Running version v0.2.4
2022-06-12T08:22:27.0663940Z [command]/home/runner/work/_actions/codecov/codecov-action/v2/dist/codecov -n  -Q github-action-2.1.0 -Z -F unittests -v
2022-06-12T08:22:27.2651351Z [2022-06-12T08:22:27.263Z] ['verbose'] Start of uploader: 1655022147263...
2022-06-12T08:22:27.2653509Z [2022-06-12T08:22:27.264Z] ['info'] 
2022-06-12T08:22:27.2653797Z      _____          _
2022-06-12T08:22:27.2654015Z     / ____|        | |
2022-06-12T08:22:27.2654262Z    | |     ___   __| | ___  ___ _____   __
2022-06-12T08:22:27.2654514Z    | |    / _ \ / _` |/ _ \/ __/ _ \ \ / /
2022-06-12T08:22:27.2654750Z    | |___| (_) | (_| |  __/ (_| (_) \ V /
2022-06-12T08:22:27.2655004Z     \_____\___/ \__,_|\___|\___\___/ \_/
2022-06-12T08:22:27.2655163Z 
2022-06-12T08:22:27.2655290Z   Codecov report uploader 0.2.4
2022-06-12T08:22:27.2712040Z [2022-06-12T08:22:27.270Z] ['info'] => Project root located at: /home/runner/work/codecov-status-test/codecov-status-test
2022-06-12T08:22:27.2748935Z [2022-06-12T08:22:27.274Z] ['info'] -> No token specified or token is empty
2022-06-12T08:22:27.2750002Z [2022-06-12T08:22:27.274Z] ['verbose'] Start of network processing...
2022-06-12T08:22:27.2750588Z [2022-06-12T08:22:27.274Z] ['verbose'] Searching for files in /home/runner/work/codecov-status-test/codecov-status-test
2022-06-12T08:22:27.4050448Z [2022-06-12T08:22:27.404Z] ['info'] Running coverage xml...
2022-06-12T08:22:27.5018926Z [2022-06-12T08:22:27.501Z] ['verbose'] Wrote XML report to coverage.xml
2022-06-12T08:22:27.5019684Z [2022-06-12T08:22:27.501Z] ['info'] Searching for coverage files...
2022-06-12T08:22:27.5390505Z [2022-06-12T08:22:27.538Z] ['info'] Warning: Some files located via search were excluded from upload.
2022-06-12T08:22:27.5391663Z [2022-06-12T08:22:27.538Z] ['info'] If Codecov did not locate your files, please review https://docs.codecov.com/docs/supported-report-formats
2022-06-12T08:22:27.5392755Z [2022-06-12T08:22:27.538Z] ['verbose'] Preparing to clean the following coverage paths: coverage.xml
2022-06-12T08:22:27.5395282Z [2022-06-12T08:22:27.538Z] ['info'] => Found 1 possible coverage files:
2022-06-12T08:22:27.5395981Z   coverage.xml
2022-06-12T08:22:27.5396553Z [2022-06-12T08:22:27.538Z] ['verbose'] End of network processing
2022-06-12T08:22:27.5397267Z [2022-06-12T08:22:27.538Z] ['info'] Processing /home/runner/work/codecov-status-test/codecov-status-test/coverage.xml...
2022-06-12T08:22:27.5416918Z [2022-06-12T08:22:27.541Z] ['info'] Detected GitHub Actions as the CI provider.
2022-06-12T08:22:27.5418869Z [2022-06-12T08:22:27.541Z] ['verbose'] -> Using the following env variables:
2022-06-12T08:22:27.5420103Z [2022-06-12T08:22:27.541Z] ['verbose']      GITHUB_ACTION: __codecov_codecov-action
2022-06-12T08:22:27.5420698Z [2022-06-12T08:22:27.541Z] ['verbose']      GITHUB_HEAD_REF: 
2022-06-12T08:22:27.5421217Z [2022-06-12T08:22:27.541Z] ['verbose']      GITHUB_REF: refs/heads/main
2022-06-12T08:22:27.5422878Z [2022-06-12T08:22:27.541Z] ['verbose']      GITHUB_REPOSITORY: Delgan/codecov-status-test
2022-06-12T08:22:27.5424903Z [2022-06-12T08:22:27.541Z] ['verbose']      GITHUB_RUN_ID: 2482829602
2022-06-12T08:22:27.5426138Z [2022-06-12T08:22:27.541Z] ['verbose']      GITHUB_SERVER_URL: https://github.com
2022-06-12T08:22:27.5426790Z [2022-06-12T08:22:27.541Z] ['verbose']      GITHUB_SHA: 41ef463106bf1ab8b5d84971de94c5baeef8619e
2022-06-12T08:22:27.5427365Z [2022-06-12T08:22:27.541Z] ['verbose']      GITHUB_WORKFLOW: Tests
2022-06-12T08:22:27.5427993Z [2022-06-12T08:22:27.542Z] ['verbose'] Using the following upload parameters:
2022-06-12T08:22:27.5428501Z [2022-06-12T08:22:27.542Z] ['verbose'] branch
2022-06-12T08:22:27.5428929Z [2022-06-12T08:22:27.542Z] ['verbose'] build
2022-06-12T08:22:27.5429383Z [2022-06-12T08:22:27.542Z] ['verbose'] buildURL
2022-06-12T08:22:27.5429825Z [2022-06-12T08:22:27.542Z] ['verbose'] commit
2022-06-12T08:22:27.5430268Z [2022-06-12T08:22:27.542Z] ['verbose'] job
2022-06-12T08:22:27.5430686Z [2022-06-12T08:22:27.542Z] ['verbose'] pr
2022-06-12T08:22:27.5431131Z [2022-06-12T08:22:27.542Z] ['verbose'] service
2022-06-12T08:22:27.5431577Z [2022-06-12T08:22:27.542Z] ['verbose'] slug
2022-06-12T08:22:27.5432011Z [2022-06-12T08:22:27.542Z] ['verbose'] name
2022-06-12T08:22:27.5432445Z [2022-06-12T08:22:27.542Z] ['verbose'] tag
2022-06-12T08:22:27.5432881Z [2022-06-12T08:22:27.542Z] ['verbose'] flags
2022-06-12T08:22:27.5433325Z [2022-06-12T08:22:27.542Z] ['verbose'] parent
2022-06-12T08:22:27.5437114Z [2022-06-12T08:22:27.543Z] ['info'] Pinging Codecov: https://codecov.io/upload/v4?package=github-action-2.1.0-uploader-0.2.4&token=*******&branch=main&build=2482829602&build_url=https%3A%2F%2Fgithub.com%2FDelgan%2Fcodecov-status-test%2Factions%2Fruns%2F2482829602&commit=41ef463106bf1ab8b5d84971de94c5baeef8619e&job=Tests&pr=&service=github-actions&slug=Delgan%2Fcodecov-status-test&name=&tag=&flags=unittests&parent=
2022-06-12T08:22:27.5438422Z [2022-06-12T08:22:27.543Z] ['verbose'] Passed token was 0 characters long
2022-06-12T08:22:27.5450783Z [2022-06-12T08:22:27.543Z] ['verbose'] https://codecov.io/upload/v4?package=github-action-2.1.0-uploader-0.2.4&branch=main&build=2482829602&build_url=https%3A%2F%2Fgithub.com%2FDelgan%2Fcodecov-status-test%2Factions%2Fruns%2F2482829602&commit=41ef463106bf1ab8b5d84971de94c5baeef8619e&job=Tests&pr=&service=github-actions&slug=Delgan%2Fcodecov-status-test&name=&tag=&flags=unittests&parent=
2022-06-12T08:22:27.5451684Z         Content-Type: 'text/plain'
2022-06-12T08:22:27.5452038Z         Content-Encoding: 'gzip'
2022-06-12T08:22:27.5452389Z         X-Reduced-Redundancy: 'false'
2022-06-12T08:22:28.0481101Z [2022-06-12T08:22:28.047Z] ['info'] https://codecov.io/github/Delgan/codecov-status-test/commit/41ef463106bf1ab8b5d84971de94c5baeef8619e
2022-06-12T08:22:28.0482542Z https://storage.googleapis.com/codecov/v4/raw/2022-06-12/619D8D713C2BD5ACC13C74FB4C5A0998/41ef463106bf1ab8b5d84971de94c5baeef8619e/082d5412-2e9b-49f7-9059-479ce2fd0ca3.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=GOOG1EKKHVKCKHW7KBCGM7IHR55T63V2PAVJWLVFNITJHDU5G6R5IRN3LMWJA%2F20220612%2FUS%2Fs3%2Faws4_request&X-Amz-Date=20220612T082228Z&X-Amz-Expires=10&X-Amz-SignedHeaders=host&X-Amz-Signature=9dbdae1dfe93d554676892e84d151a228e5a868c9a94b96684e2401e069b5295
2022-06-12T08:22:28.0484800Z [2022-06-12T08:22:28.047Z] ['verbose'] Returned upload url: https://storage.googleapis.com/codecov/v4/raw/2022-06-12/619D8D713C2BD5ACC13C74FB4C5A0998/41ef463106bf1ab8b5d84971de94c5baeef8619e/082d5412-2e9b-49f7-9059-479ce2fd0ca3.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=GOOG1EKKHVKCKHW7KBCGM7IHR55T63V2PAVJWLVFNITJHDU5G6R5IRN3LMWJA%2F20220612%2FUS%2Fs3%2Faws4_request&X-Amz-Date=20220612T082228Z&X-Amz-Expires=10&X-Amz-SignedHeaders=host&X-Amz-Signature=9dbdae1dfe93d554676892e84d151a228e5a868c9a94b96684e2401e069b5295
2022-06-12T08:22:28.0485704Z [2022-06-12T08:22:28.047Z] ['info'] Uploading...
2022-06-12T08:22:28.2379244Z [2022-06-12T08:22:28.237Z] ['info'] {"status":"success","resultURL":"https://codecov.io/github/Delgan/codecov-status-test/commit/41ef463106bf1ab8b5d84971de94c5baeef8619e"}
2022-06-12T08:22:28.2380270Z [2022-06-12T08:22:28.237Z] ['verbose'] End of uploader: 974 milliseconds
2022-06-12T08:22:28.2697619Z Post job cleanup.

Expected Results

The Codecov Gtihub status is marked as “pending” (or not displayed at all) until all jobs are completed.

Actual Results

The Codecov Github status is updated as soon as the 1st job is received, possibly causing misleading failure status.

Additional Information

Here is the .codecov.yml I’m using:

comment: false

coverage:
  status:
    patch: no
    changes: no
    project:
      default:
        target: 100%
        threshold: null


codecov:
  require_ci_to_pass: yes
  notify:
    wait_for_ci: yes

[tom]

@Delgan I’m sorry for the delay here. Do you have a commit SHA of this happening recently? I’m really not sure why that happened and our logs don’t go back far enough.

[Delgan]

Hi @tom, thanks for the reply.

I was able to reproduce it without any problem last time, but this time it seems a bit more random.

It happened with this commit for example: Add testing files · Delgan/codecov-status-test@67ddd76 · GitHub

[tom]

@Delgan I believe that commit you posted is missing

codecov:
  require_ci_to_pass: yes
  notify:
    wait_for_ci: yes

so it didn’t wait for CI to finish.

[Delgan]

@tom I removed it for simplicity, because yes is the default value anyway according to the documentation: codecov.yml Reference

[tom]

@Delgan ok, I think this might have been an issue on our server-side processing. We made a few small fixes, are you still seeing the issue?

[Delgan]

Hi @tom, unfortunately it seems there is still an issue.

I made another commit and the coverage was temporarily marked as failed and updated like ~2 min later.

Another empty commit

[tom]

@Delgan I’m sorry I’m a little bit stumped on this. In our codebase, all signs point to wait_for_ci defaulting to true. But that doesn’t seem to be the case in your scenario. Do you mind adding it into the codecov.yml just to help debug this down a little further?

[Delgan]

@tom Sure, I did it, and the problem is still there:

Empty commit

It’s weird because the failure is only temporary and happens 1 minute after the two jobs are finished.

[tom]

@Delgan, well that time it failed because wait_for_ci is in the wrong section. I made a PR here

[Delgan]

@tom Oops, sorry…
Thanks for the PR!

I merged it, which triggered the error:

Update `wait_for_ci` to correct section (#3)

[tom]

@Delgan ok! I think I understand a bit better what’s happening. Let me sync up with the product team to see what we can do. I believe this is the timeline

1. First CI pipeline successfully uploads coverage report
2. Second CI pipeline successfully uploads coverage report
3. Codecov processes first report
4. Codecov checks to see if CI has completed (which it has) and sends the check to GitHub
5. Codecov processes second report
6. Codecov checks to see if CI has completed (which it has) and sends the updated check

[tom]

Hi @Delgan, we’ve been taking a look at this and made some changes. Can you check to see if it’s happening again for you?

[Delgan]

Hi @tom, thanks for the update.

Unfortunately, no change.

After ~1 min;

After ~2 mins:

Commtit:
https://github.com/Delgan/codecov-status-test/commit/2088269f923d6ecbdb119c1a7c2ca027975b495b

[tom]

@Delgan taking a look at this a little more, it looks like that Codecov was sent 1 report and reported back on it, and then it was sent the 2nd. The way around this is to implement after_n_builds

codecov:
  notify:
    after_n_builds: 2

Something like that should do the job

[Delgan]

@tom I know about after_n_builds and I mentioned it in my initial post. The whole point of this thread is to understand why I’m not observing the behavior described in the documentation:

How does Codecov know when to send notifications?

Codecov will check, every minute, the status of each CI provider. If the status of any provider is pending, we will delay the notifications. If the status of any provider is error or failure, Codecov will cancel the notification entirely. If the status of all providers is success, the notifications will be submitted.

Using after_n_build is not ideal because it can become out-of-sync with the listed Github actions.

[tom]

@Delgan I know this is 3 months later, but clearly, our documentation is wrong on this. I’m going to edit the docs today and see what we can do on the product side to implement.

Thanks for bringing this up, though. I would have never seen the error if you hadn’t pointed it out.

[Delgan]

That’s fine, thanks for the update and final answer.