How to prevent codecov from using a commit older than the base of the PR to do comparison?

Support
1

Description

I was working in a PR in a repository that disabled codecov a while ago (it seems that they were having problems with codecov failing a lot with small decrements). When I tried to re-activate the Github Action for codecov to see if it still works, my build failed because codecov is comparing it to a very old commit (I guess it was the last time coverage data was uploaded?).

This comparison is unfair. Is there a way of preventing codecov from comparing coverage changes with commits other than the actual PR base?

I had a look on previous discussions, specially Flag -N to set the base report does not work - #6 by jerrod, but I donā€™t think it is applicable.

CI/CD URL

Please provide a link to the CI/CD run.

Codecov Output

Please provide the full output of running the uploader on your CI/CD. This will typically have the Codecov logo as ASCII.

/usr/bin/bash codecov.sh -n  -F Linux, 3.10 -Q github-action-v1.5.2 -C 9ce6385df46d6aa897c9d7905d0f9fc660d390bc

  _____          _
 / ____|        | |
| |     ___   __| | ___  ___ _____   __
| |    / _ \ / _` |/ _ \/ __/ _ \ \ / /
| |___| (_) | (_| |  __/ (_| (_) \ V /
 \_____\___/ \__,_|\___|\___\___/ \_/
                              Bash-1.0.3


==> git version 2.35.1 found
==> curl 7.68.0 (x86_64-pc-linux-gnu) libcurl/7.68.0 OpenSSL/1.1.1f zlib/1.2.11 brotli/1.0.7 libidn2/2.2.0 libpsl/0.21.0 (+libidn2/2.2.0) libssh/0.9.3/openssl/zlib nghttp2/1.40.0 librtmp/2.3
Release-Date: 2020-01-08
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp 
Features: AsynchDNS brotli GSS-API HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP UnixSockets
==> GitHub Actions detected.
    Env vars used:
      -> GITHUB_ACTIONS:    true
      -> GITHUB_HEAD_REF:   disable-coverage-pypy
      -> GITHUB_REF:        refs/pull/3112/merge
      -> GITHUB_REPOSITORY: pypa/setuptools
      -> GITHUB_RUN_ID:     1855218487
      -> GITHUB_SHA:        1e690451bef88aad5c323fb5eb77e565a06bfe92
      -> GITHUB_WORKFLOW:   tests
    project root: .
    Yaml found at: .codecov.yml
==> Running gcov in . (disable via -X gcov)
==> Searching for coverage reports in:
    + .
    -> Found 1 reports
==> Detecting git/mercurial file structure
==> Reading reports
    + ./coverage.xml bytes=1061907
==> Appending adjustments
    https://docs.codecov.io/docs/fixing-reports
    -> No adjustments found
==> Gzipping contents
        76K	/tmp/codecov.yXkurD.gz
==> Uploading reports
    url: https://codecov.io
    query: branch=disable-coverage-pypy&commit=9ce6385df46d6aa897c9d7905d0f9fc660d390bc&build=1855218487&build_url=http%3A%2F%2Fgithub.com%2Fpypa%2Fsetuptools%2Factions%2Fruns%2F1855218487&name=&tag=&slug=pypa%2Fsetuptools&service=github-actions&flags=Linux,3.10&pr=3112&job=tests&cmd_args=n,F,Q,C
->  Pinging Codecov
https://codecov.io/upload/v4?package=github-action-v1.5.2-1.0.3&token=secret&branch=disable-coverage-pypy&commit=9ce6385df46d6aa897c9d7905d0f9fc660d390bc&build=1855218487&build_url=http%3A%2F%2Fgithub.com%2Fpypa%2Fsetuptools%2Factions%2Fruns%2F1855218487&name=&tag=&slug=pypa%2Fsetuptools&service=github-actions&flags=Linux,3.10&pr=3112&job=tests&cmd_args=n,F,Q,C
->  Uploading to
https://storage.googleapis.com/codecov/v4/raw/2022-02-16/19BC5522785AF0C590C8AE9698BF7D5D/9ce6385df46d6aa897c9d7905d0f9fc660d390bc/cde37ce3-eae7-4177-92d9-bf584debb847.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=GOOG1EKKHVKCKHW7KBCGM7IHR55T63V2PAVJWLVFNITJHDU5G6R5IRN3LMWJA%2F20220216%2FUS%2Fs3%2Faws4_request&X-Amz-Date=20220216T211259Z&X-Amz-Expires=10&X-Amz-SignedHeaders=host&X-Amz-Signature=a813d5a85a3c381470e1a3f43869365c069257b56905ce1b3bdc99a86226b0f3
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 73784    0     0  100 73784      0   324k --:--:-- --:--:-- --:--:--  324k
    -> Reports have been successfully queued for processing at https://codecov.io/github/pypa/setuptools/commit/9ce6385df46d6aa897c9d7905d0f9fc660d390bc

Expected Results

I would expect codecov to not try to compare coverage changes if it cannot find the base commit for the PR

Actual Results

Codecov failed (in a very unfair manner) when comparing coverage results more than 200 commits old.

2

hi @abravalheri, really appreciate you bringing Codecov back up. the only way for Codecov to compare against is by the coverage reported from a commitsā€™ parents. If those donā€™t exist, Codecov will keep going backwards until it hits a commit that has coverage.

I realize that in your case, itā€™s causing some heartache, but thereā€™s no way to turn that off. What you could do is change the status checks to be informational until the repository stabilizes.

coverage:
  status:
    project:
      default:
        informational: true
    patch:
      default:
        informational: true

Also of note, the Action is on v1, but it is recommended to upgrade to v2

1 Like