Github status / notification isn't delayed (should wait all CI jobs to finish)

Before submitting a topic, please confirm the following

I have searched for similar issues before creating this topic.
I have verified that my repository is using the Codecov GitHub app, if using GitHub
I have validated my codecov.yaml configuration file.
I have filled out the below sections to the best of my ability.

Description

Hi.

I’m using multiple Github jobs and Codecov merges the reports which is perfect.

However, Codecov updates GIthub status before receiving all reports, causing a misleading “Coverage failed” status for a few minutes.

According to the documentation, Codecov should wait before sending the notification, but it seems it’s not working as expected.

I specified require_ci_to_pass: yes and wait_for_ci: yes but it changed nothing. The only solution I found so far to prevent premature status update is to explicitly set the after_n_builds parameter.

Why isn’t automatic detection of CI completion working in my case?

CI/CD URL

I created a minimal reproducible example in this repository: GitHub - Delgan/codecov-status-test: Just a testing repository

Here is an example of the pipeline (one job is made longer than the other to test delayed coverage): Add testing files · Delgan/codecov-status-test@41ef463 · GitHub

Codecov Output

2022-06-12T08:22:23.6523462Z ##[group]Run codecov/codecov-action@v2
2022-06-12T08:22:23.6523803Z with:
2022-06-12T08:22:23.6524022Z   flags: unittests
2022-06-12T08:22:23.6524290Z   verbose: true
2022-06-12T08:22:23.6524602Z   fail_ci_if_error: true
2022-06-12T08:22:23.6524863Z env:
2022-06-12T08:22:23.6525211Z   pythonLocation: /opt/hostedtoolcache/Python/3.10.4/x64
2022-06-12T08:22:23.6525613Z   LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.10.4/x64/lib
2022-06-12T08:22:23.6525890Z ##[endgroup]
2022-06-12T08:22:23.7792914Z ==> linux OS detected
2022-06-12T08:22:26.6738580Z https://uploader.codecov.io/latest/linux/codecov.SHA256SUM
2022-06-12T08:22:26.8103664Z ==> SHASUM file signed by key id 806bb28aed779869
2022-06-12T08:22:26.9703307Z ==> Uploader SHASUM verified (432553f03714a83ab5cac78be2e7fe6a6050767f78f77e9c7a84bdb1c34a0bd3  codecov)
2022-06-12T08:22:26.9704203Z ==> Running version latest
2022-06-12T08:22:27.0638836Z ==> Running version v0.2.4
2022-06-12T08:22:27.0663940Z [command]/home/runner/work/_actions/codecov/codecov-action/v2/dist/codecov -n  -Q github-action-2.1.0 -Z -F unittests -v
2022-06-12T08:22:27.2651351Z [2022-06-12T08:22:27.263Z] ['verbose'] Start of uploader: 1655022147263...
2022-06-12T08:22:27.2653509Z [2022-06-12T08:22:27.264Z] ['info'] 
2022-06-12T08:22:27.2653797Z      _____          _
2022-06-12T08:22:27.2654015Z     / ____|        | |
2022-06-12T08:22:27.2654262Z    | |     ___   __| | ___  ___ _____   __
2022-06-12T08:22:27.2654514Z    | |    / _ \ / _` |/ _ \/ __/ _ \ \ / /
2022-06-12T08:22:27.2654750Z    | |___| (_) | (_| |  __/ (_| (_) \ V /
2022-06-12T08:22:27.2655004Z     \_____\___/ \__,_|\___|\___\___/ \_/
2022-06-12T08:22:27.2655163Z 
2022-06-12T08:22:27.2655290Z   Codecov report uploader 0.2.4
2022-06-12T08:22:27.2712040Z [2022-06-12T08:22:27.270Z] ['info'] => Project root located at: /home/runner/work/codecov-status-test/codecov-status-test
2022-06-12T08:22:27.2748935Z [2022-06-12T08:22:27.274Z] ['info'] -> No token specified or token is empty
2022-06-12T08:22:27.2750002Z [2022-06-12T08:22:27.274Z] ['verbose'] Start of network processing...
2022-06-12T08:22:27.2750588Z [2022-06-12T08:22:27.274Z] ['verbose'] Searching for files in /home/runner/work/codecov-status-test/codecov-status-test
2022-06-12T08:22:27.4050448Z [2022-06-12T08:22:27.404Z] ['info'] Running coverage xml...
2022-06-12T08:22:27.5018926Z [2022-06-12T08:22:27.501Z] ['verbose'] Wrote XML report to coverage.xml
2022-06-12T08:22:27.5019684Z [2022-06-12T08:22:27.501Z] ['info'] Searching for coverage files...
2022-06-12T08:22:27.5390505Z [2022-06-12T08:22:27.538Z] ['info'] Warning: Some files located via search were excluded from upload.
2022-06-12T08:22:27.5391663Z [2022-06-12T08:22:27.538Z] ['info'] If Codecov did not locate your files, please review https://docs.codecov.com/docs/supported-report-formats
2022-06-12T08:22:27.5392755Z [2022-06-12T08:22:27.538Z] ['verbose'] Preparing to clean the following coverage paths: coverage.xml
2022-06-12T08:22:27.5395282Z [2022-06-12T08:22:27.538Z] ['info'] => Found 1 possible coverage files:
2022-06-12T08:22:27.5395981Z   coverage.xml
2022-06-12T08:22:27.5396553Z [2022-06-12T08:22:27.538Z] ['verbose'] End of network processing
2022-06-12T08:22:27.5397267Z [2022-06-12T08:22:27.538Z] ['info'] Processing /home/runner/work/codecov-status-test/codecov-status-test/coverage.xml...
2022-06-12T08:22:27.5416918Z [2022-06-12T08:22:27.541Z] ['info'] Detected GitHub Actions as the CI provider.
2022-06-12T08:22:27.5418869Z [2022-06-12T08:22:27.541Z] ['verbose'] -> Using the following env variables:
2022-06-12T08:22:27.5420103Z [2022-06-12T08:22:27.541Z] ['verbose']      GITHUB_ACTION: __codecov_codecov-action
2022-06-12T08:22:27.5420698Z [2022-06-12T08:22:27.541Z] ['verbose']      GITHUB_HEAD_REF: 
2022-06-12T08:22:27.5421217Z [2022-06-12T08:22:27.541Z] ['verbose']      GITHUB_REF: refs/heads/main
2022-06-12T08:22:27.5422878Z [2022-06-12T08:22:27.541Z] ['verbose']      GITHUB_REPOSITORY: Delgan/codecov-status-test
2022-06-12T08:22:27.5424903Z [2022-06-12T08:22:27.541Z] ['verbose']      GITHUB_RUN_ID: 2482829602
2022-06-12T08:22:27.5426138Z [2022-06-12T08:22:27.541Z] ['verbose']      GITHUB_SERVER_URL: https://github.com
2022-06-12T08:22:27.5426790Z [2022-06-12T08:22:27.541Z] ['verbose']      GITHUB_SHA: 41ef463106bf1ab8b5d84971de94c5baeef8619e
2022-06-12T08:22:27.5427365Z [2022-06-12T08:22:27.541Z] ['verbose']      GITHUB_WORKFLOW: Tests
2022-06-12T08:22:27.5427993Z [2022-06-12T08:22:27.542Z] ['verbose'] Using the following upload parameters:
2022-06-12T08:22:27.5428501Z [2022-06-12T08:22:27.542Z] ['verbose'] branch
2022-06-12T08:22:27.5428929Z [2022-06-12T08:22:27.542Z] ['verbose'] build
2022-06-12T08:22:27.5429383Z [2022-06-12T08:22:27.542Z] ['verbose'] buildURL
2022-06-12T08:22:27.5429825Z [2022-06-12T08:22:27.542Z] ['verbose'] commit
2022-06-12T08:22:27.5430268Z [2022-06-12T08:22:27.542Z] ['verbose'] job
2022-06-12T08:22:27.5430686Z [2022-06-12T08:22:27.542Z] ['verbose'] pr
2022-06-12T08:22:27.5431131Z [2022-06-12T08:22:27.542Z] ['verbose'] service
2022-06-12T08:22:27.5431577Z [2022-06-12T08:22:27.542Z] ['verbose'] slug
2022-06-12T08:22:27.5432011Z [2022-06-12T08:22:27.542Z] ['verbose'] name
2022-06-12T08:22:27.5432445Z [2022-06-12T08:22:27.542Z] ['verbose'] tag
2022-06-12T08:22:27.5432881Z [2022-06-12T08:22:27.542Z] ['verbose'] flags
2022-06-12T08:22:27.5433325Z [2022-06-12T08:22:27.542Z] ['verbose'] parent
2022-06-12T08:22:27.5437114Z [2022-06-12T08:22:27.543Z] ['info'] Pinging Codecov: https://codecov.io/upload/v4?package=github-action-2.1.0-uploader-0.2.4&token=*******&branch=main&build=2482829602&build_url=https%3A%2F%2Fgithub.com%2FDelgan%2Fcodecov-status-test%2Factions%2Fruns%2F2482829602&commit=41ef463106bf1ab8b5d84971de94c5baeef8619e&job=Tests&pr=&service=github-actions&slug=Delgan%2Fcodecov-status-test&name=&tag=&flags=unittests&parent=
2022-06-12T08:22:27.5438422Z [2022-06-12T08:22:27.543Z] ['verbose'] Passed token was 0 characters long
2022-06-12T08:22:27.5450783Z [2022-06-12T08:22:27.543Z] ['verbose'] https://codecov.io/upload/v4?package=github-action-2.1.0-uploader-0.2.4&branch=main&build=2482829602&build_url=https%3A%2F%2Fgithub.com%2FDelgan%2Fcodecov-status-test%2Factions%2Fruns%2F2482829602&commit=41ef463106bf1ab8b5d84971de94c5baeef8619e&job=Tests&pr=&service=github-actions&slug=Delgan%2Fcodecov-status-test&name=&tag=&flags=unittests&parent=
2022-06-12T08:22:27.5451684Z         Content-Type: 'text/plain'
2022-06-12T08:22:27.5452038Z         Content-Encoding: 'gzip'
2022-06-12T08:22:27.5452389Z         X-Reduced-Redundancy: 'false'
2022-06-12T08:22:28.0481101Z [2022-06-12T08:22:28.047Z] ['info'] https://codecov.io/github/Delgan/codecov-status-test/commit/41ef463106bf1ab8b5d84971de94c5baeef8619e
2022-06-12T08:22:28.0482542Z https://storage.googleapis.com/codecov/v4/raw/2022-06-12/619D8D713C2BD5ACC13C74FB4C5A0998/41ef463106bf1ab8b5d84971de94c5baeef8619e/082d5412-2e9b-49f7-9059-479ce2fd0ca3.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=GOOG1EKKHVKCKHW7KBCGM7IHR55T63V2PAVJWLVFNITJHDU5G6R5IRN3LMWJA%2F20220612%2FUS%2Fs3%2Faws4_request&X-Amz-Date=20220612T082228Z&X-Amz-Expires=10&X-Amz-SignedHeaders=host&X-Amz-Signature=9dbdae1dfe93d554676892e84d151a228e5a868c9a94b96684e2401e069b5295
2022-06-12T08:22:28.0484800Z [2022-06-12T08:22:28.047Z] ['verbose'] Returned upload url: https://storage.googleapis.com/codecov/v4/raw/2022-06-12/619D8D713C2BD5ACC13C74FB4C5A0998/41ef463106bf1ab8b5d84971de94c5baeef8619e/082d5412-2e9b-49f7-9059-479ce2fd0ca3.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=GOOG1EKKHVKCKHW7KBCGM7IHR55T63V2PAVJWLVFNITJHDU5G6R5IRN3LMWJA%2F20220612%2FUS%2Fs3%2Faws4_request&X-Amz-Date=20220612T082228Z&X-Amz-Expires=10&X-Amz-SignedHeaders=host&X-Amz-Signature=9dbdae1dfe93d554676892e84d151a228e5a868c9a94b96684e2401e069b5295
2022-06-12T08:22:28.0485704Z [2022-06-12T08:22:28.047Z] ['info'] Uploading...
2022-06-12T08:22:28.2379244Z [2022-06-12T08:22:28.237Z] ['info'] {"status":"success","resultURL":"https://codecov.io/github/Delgan/codecov-status-test/commit/41ef463106bf1ab8b5d84971de94c5baeef8619e"}
2022-06-12T08:22:28.2380270Z [2022-06-12T08:22:28.237Z] ['verbose'] End of uploader: 974 milliseconds
2022-06-12T08:22:28.2697619Z Post job cleanup.

Expected Results

The Codecov Gtihub status is marked as “pending” (or not displayed at all) until all jobs are completed.

Actual Results

The Codecov Github status is updated as soon as the 1st job is received, possibly causing misleading failure status.

Additional Information

Here is the .codecov.yml I’m using:

comment: false

coverage:
  status:
    patch: no
    changes: no
    project:
      default:
        target: 100%
        threshold: null


codecov:
  require_ci_to_pass: yes
  notify:
    wait_for_ci: yes
1 Like