CI Status is always passed despite of too low overall coverage

Support
,
1

Hi,
I’m trying to setup CI/CD pipeline (on this repo: GitHub - ptrkalm/hello-shuttle) with Github Actions and I created following configuration for github jobs:

name: CI/CD

on:
  push:
    branches:
      - "main"

jobs:
  check:
    name: Check
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - uses: actions-rs/toolchain@v1
        with:
          profile: minimal
          toolchain: stable
          override: true
      - uses: actions-rs/cargo@v1
        with:
          command: check
  test:
    name: Tests
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - uses: actions-rs/toolchain@v1
        with:
          profile: minimal
          toolchain: stable
          override: true
      - uses: actions-rs/cargo@v1
        with:
          command: test
  coverage:
    name: Coverage
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: taiki-e/install-action@cargo-llvm-cov
      - run: cargo llvm-cov --all-features --lcov --ignore-filename-regex main.rs --output-path lcov.info
      - uses: codecov/codecov-action@v3
        env:
          CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
        with:
          fail_ci_if_error: true   
  deploy:
    needs: [check, test, coverage]
    name: Deploy
    runs-on: ubuntu-latest
    steps:
      - uses: shuttle-hq/deploy-action@main
        with:
          deploy-key: ${{ secrets.SHUTTLE_DEPLOY_KEY }}

As you can see in step “coverage” I generate coverage info with llvm-cov for Rust and then I’m trying to upload it ro codecov.io, which seems to work just fine since I can see on dashboards at codecov.io all of my commits with correctly calculated coverage.

image
image1684×673 68.1 KB

However, I on purpose commited changes without tests to lower the coverage (which worked), but I expected that since I configured in codecov.yaml that I want 100% coverage, that the status will change to Failure, but it stays as passed all the time.

My configuration of codecov.yml that is in root directory:

coverage:
  status:
    project:
      default:
        target: 100%
        threshold: 0%

Response from codecov.io in Github Actions console:

[2023-07-21T00:56:00.114Z] ['info'] 
     _____          _
    / ____|        | |
   | |     ___   __| | ___  ___ _____   __
   | |    / _ \ / _` |/ _ \/ __/ _ \ \ / /
   | |___| (_) | (_| |  __/ (_| (_) \ V /
    \_____\___/ \__,_|\___|\___\___/ \_/

  Codecov report uploader 0.6.1
[2023-07-21T00:56:00.123Z] ['info'] => Project root located at: /home/runner/work/hello-shuttle/hello-shuttle
[2023-07-21T00:56:00.126Z] ['info'] ->  Token found by environment variables
[2023-07-21T00:56:00.133Z] ['info'] Searching for coverage files...
[2023-07-21T00:56:00.275Z] ['info'] Warning: Some files located via search were excluded from upload.
[2023-07-21T00:56:00.275Z] ['info'] If Codecov did not locate your files, please review https://docs.codecov.com/docs/supported-report-formats
[2023-07-21T00:56:00.276Z] ['info'] => Found 1 possible coverage files:
  lcov.info
[2023-07-21T00:56:00.276Z] ['info'] Processing /home/runner/work/hello-shuttle/hello-shuttle/lcov.info...
[2023-07-21T00:56:00.280Z] ['info'] Detected GitHub Actions as the CI provider.
[2023-07-21T00:56:00.667Z] ['info'] Pinging Codecov: https://codecov.io/upload/v4?package=github-action-3.1.4-uploader-0.6.1&token=*******&branch=main&build=5617376383&build_url=https%3A%2F%2Fgithub.com%2Fptrkalm%2Fhello-shuttle%2Factions%2Fruns%2F5617376383&commit=5367e7f3297718ad1df15de098255e1c336af4f7&job=CI%2FCD&pr=&service=github-actions&slug=ptrkalm%2Fhello-shuttle&name=&tag=&flags=&parent=
[2023-07-21T00:56:01.049Z] ['info'] https://app.codecov.io/github/ptrkalm/hello-shuttle/commit/5367e7f3297718ad1df15de098255e1c336af4f7
https://storage.googleapis.com/codecov/v4/raw/2023-07-21/4860789E6ED39C81BD52AEB9AFA147CA/5367e7f3297718ad1df15de098255e1c336af4f7/7a4817bb-5f94-41db-b1c0-dfa59f479e88.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=GOOG1EJOGFN2JQ4OCTGA2MU5AEIT7OT5Z7HTFOAN2SPG4NWSN2UJYOY5U6LZQ%2F20230721%2FUS%2Fs3%2Faws4_request&X-Amz-Date=20230721T005601Z&X-Amz-Expires=10&X-Amz-SignedHeaders=host&X-Amz-Signature=20fac72b1bc47de6a4647e5d8e5a989b9d10f9211ddf8f3883f8169121b92c32
[2023-07-21T00:56:01.050Z] ['info'] Uploading...
[2023-07-21T00:56:01.257Z] ['info'] {"status":"success","resultURL":"https://app.codecov.io/github/ptrkalm/hello-shuttle/commit/5367e7f3297718ad1df15de098255e1c336af4f7"}

Current behaviour:
Codecov.io fetches report generated and uploaded with github actions, but passes all commits despite lower coverage than expected.

Expected:

Status is not success if coverage after commit (on whole project, not just new code) and since I added this to job:

with:
          fail_ci_if_error: true

it will stop my pipeline and not deploy it.

I must do some stupid mistake, but don’t know what to do anymore xd

Thanks in advance!
Piotrek

2

Hi @ptrkalm, this is actually a result of removed code behavior.

By default, if the patch status check is 100% and there are no unexpected coverage changes, it will automatically pass the project status check. This is at least true for the commit 5367e7f3.

For commit e817af1ac, we are showing a failing project status check

image
image755×425 91.6 KB

Let me know if this makes sense.