Unusable report after separating Codecov upload into a different job on GH Actions

Before submitting a topic, please confirm the following

I have searched for similar issues before creating this topic.
I have verified that my repository is using the Codecov GitHub app, if using GitHub
I have validated my codecov.yaml configuration file.
I have filled out the below sections to the best of my ability.
I understand that the community boards are a free, best-effort tool. While we hope to have someone on the Codecov team resolve your problem quickly, this is not always possible.


We used to use the Codecov GitHub Action for uploading reports on every job in the matrix. I tried separating this out into a different job that would run after all jobs in the matrix finished, and upload all the reports in one go.

This works by uploading the report as an artifact at the end of each matrix job, then downloading all artifacts in the Codecov job, so the coverage files are downloaded like this

`-- coverage.xml
`-- coverage.xml`

instead of just one coverage.xml file in the root dir.

However, now I get “unusable report” when I view the uploads section of the commit in the Codecov website.


Downloading the report in question shows that it seems to be perfectly valid. The only differences are

  • The list of files at the top only shows the coverage files, not the repo files because we don’t checkout the repo in the Codecov job
  • The report contains the contents of all the coverage.xmls instead of just one

I have had a look at the documentation for path fixing and tried out several path fixes but none worked. None of the paths in the report have changed anyway, other than the coverage.xml files.


Commit showing the changes: Upload coverage to Codecov in separate job by MetRonnie · Pull Request #5459 · cylc/cylc-flow · GitHub

GH Actions run: Upload coverage to Codecov in separate job · cylc/cylc-flow@146e923 · GitHub

Codecov page for the commit: Codecov

Codecov Output

Please provide the full output of running the uploader on your CI/CD. This will typically have the Codecov logo as ASCII.

[2023-04-12T14:18:31.147Z] ['verbose'] Start of uploader: 1681309111147...
[2023-04-12T14:18:31.149Z] ['info'] 
     _____          _
    / ____|        | |
   | |     ___   __| | ___  ___ _____   __
   | |    / _ \ / _` |/ _ \/ __/ _ \ \ / /
   | |___| (_) | (_| |  __/ (_| (_) \ V /
    \_____\___/ \__,_|\___|\___\___/ \_/

  Codecov report uploader 0.4.1
[2023-04-12T14:18:31.162Z] ['info'] => Project root located at: /home/runner/work/cylc-flow/cylc-flow
[2023-04-12T14:18:31.163Z] ['info'] -> No token specified or token is empty
[2023-04-12T14:18:31.164Z] ['verbose'] Start of network processing...
[2023-04-12T14:18:31.164Z] ['verbose'] Searching for files in /home/runner/work/cylc-flow/cylc-flow
[2023-04-12T14:18:31.176Z] ['verbose'] coveragepy is not installed
[2023-04-12T14:18:31.176Z] ['info'] Searching for coverage files...
[2023-04-12T14:18:31.272Z] ['info'] Warning: Some files located via search were excluded from upload.
[2023-04-12T14:18:31.272Z] ['info'] If Codecov did not locate your files, please review https://docs.codecov.com/docs/supported-report-formats
[2023-04-12T14:18:31.273Z] ['verbose'] Preparing to clean the following coverage paths: coverage_ubuntu-latest_py-3.10/coverage.xml,coverage_macos-latest_py-3.7/coverage.xml,coverage_ubuntu-latest_py-3.7/coverage.xml,coverage_ubuntu-latest_py-3.8/coverage.xml,coverage_ubuntu-latest_py-3.9/coverage.xml
[2023-04-12T14:18:31.274Z] ['info'] => Found 5 possible coverage files:
[2023-04-12T14:18:31.274Z] ['verbose'] End of network processing
[2023-04-12T14:18:31.275Z] ['info'] Processing /home/runner/work/cylc-flow/cylc-flow/coverage_ubuntu-latest_py-3.10/coverage.xml...
[2023-04-12T14:18:31.280Z] ['info'] Processing /home/runner/work/cylc-flow/cylc-flow/coverage_macos-latest_py-3.7/coverage.xml...
[2023-04-12T14:18:31.284Z] ['info'] Processing /home/runner/work/cylc-flow/cylc-flow/coverage_ubuntu-latest_py-3.7/coverage.xml...
[2023-04-12T14:18:31.287Z] ['info'] Processing /home/runner/work/cylc-flow/cylc-flow/coverage_ubuntu-latest_py-3.8/coverage.xml...
[2023-04-12T14:18:31.291Z] ['info'] Processing /home/runner/work/cylc-flow/cylc-flow/coverage_ubuntu-latest_py-3.9/coverage.xml...
[2023-04-12T14:18:31.383Z] ['info'] Detected GitHub Actions as the CI provider.
[2023-04-12T14:18:31.385Z] ['verbose'] -> Using the following env variables:
[2023-04-12T14:18:31.386Z] ['verbose']      GITHUB_ACTION: __codecov_codecov-action
[2023-04-12T14:18:31.386Z] ['verbose']      GITHUB_HEAD_REF: codecov
[2023-04-12T14:18:31.387Z] ['verbose']      GITHUB_REF: refs/pull/5459/merge
[2023-04-12T14:18:31.387Z] ['verbose']      GITHUB_REPOSITORY: cylc/cylc-flow
[2023-04-12T14:18:31.387Z] ['verbose']      GITHUB_RUN_ID: 4669416371
[2023-04-12T14:18:31.388Z] ['verbose']      GITHUB_SERVER_URL: https://github.com
[2023-04-12T14:18:31.388Z] ['verbose']      GITHUB_SHA: 27aa21ed3319058edbeb18510bd7c2c78aab90fa
[2023-04-12T14:18:31.390Z] ['verbose']      GITHUB_WORKFLOW: fast tests
[2023-04-12T14:18:31.968Z] ['verbose'] Using the following upload parameters:
[2023-04-12T14:18:31.968Z] ['verbose'] branch
[2023-04-12T14:18:31.968Z] ['verbose'] build
[2023-04-12T14:18:31.968Z] ['verbose'] buildURL
[2023-04-12T14:18:31.968Z] ['verbose'] commit
[2023-04-12T14:18:31.968Z] ['verbose'] job
[2023-04-12T14:18:31.968Z] ['verbose'] pr
[2023-04-12T14:18:31.968Z] ['verbose'] service
[2023-04-12T14:18:31.968Z] ['verbose'] slug
[2023-04-12T14:18:31.968Z] ['verbose'] name
[2023-04-12T14:18:31.968Z] ['verbose'] tag
[2023-04-12T14:18:31.969Z] ['verbose'] flags
[2023-04-12T14:18:31.969Z] ['verbose'] parent
[2023-04-12T14:18:31.969Z] ['info'] Pinging Codecov: https://codecov.io/upload/v4?package=github-action-3.1.2-uploader-0.4.1&token=*******&branch=codecov&build=4669416371&build_url=https%3A%2F%2Fgithub.com%2Fcylc%2Fcylc-flow%2Factions%2Fruns%2F4669416371%2Fjobs%2F8289304048&commit=146e923f405da354e2d3116f08e356e98b2d4e38&job=fast+tests&pr=5459&service=github-actions&slug=cylc%2Fcylc-flow&name=fast+tests&tag=&flags=fast-tests&parent=
[2023-04-12T14:18:31.970Z] ['verbose'] Passed token was 0 characters long
[2023-04-12T14:18:31.970Z] ['verbose'] https://codecov.io/upload/v4?package=github-action-3.1.2-uploader-0.4.1&branch=codecov&build=4669416371&build_url=https%3A%2F%2Fgithub.com%2Fcylc%2Fcylc-flow%2Factions%2Fruns%2F4669416371%2Fjobs%2F8289304048&commit=146e923f405da354e2d3116f08e356e98b2d4e38&job=fast+tests&pr=5459&service=github-actions&slug=cylc%2Fcylc-flow&name=fast+tests&tag=&flags=fast-tests&parent=
        Content-Type: 'text/plain'
        Content-Encoding: 'gzip'
        X-Reduced-Redundancy: 'false'
[2023-04-12T14:18:32.365Z] ['info'] https://app.codecov.io/github/cylc/cylc-flow/commit/146e923f405da354e2d3116f08e356e98b2d4e38
[2023-04-12T14:18:32.365Z] ['verbose'] Returned upload url: https://storage.googleapis.com/codecov/v4/raw/2023-04-12/83DC0162D5C62AC131649C363CB7562E/146e923f405da354e2d3116f08e356e98b2d4e38/c6747987-7fed-40f6-8445-da6509c3dc57.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=GOOG1EJOGFN2JQ4OCTGA2MU5AEIT7OT5Z7HTFOAN2SPG4NWSN2UJYOY5U6LZQ%2F20230412%2FUS%2Fs3%2Faws4_request&X-Amz-Date=20230412T141832Z&X-Amz-Expires=10&X-Amz-SignedHeaders=host&X-Amz-Signature=c39d4ce86c16b43dde4d579907c8c98026bee810d77fd209a03bafb64bd6ea40
[2023-04-12T14:18:32.365Z] ['info'] Uploading...
[2023-04-12T14:18:32.714Z] ['info'] {"status":"success","resultURL":"https://app.codecov.io/github/cylc/cylc-flow/commit/146e923f405da354e2d3116f08e356e98b2d4e38"}
[2023-04-12T14:18:32.715Z] ['verbose'] End of uploader: 1568 milliseconds

Expected Results

Coverage report is usable. If path fixing is required it should be handled automatically as before.

Actual Results

Coverage report is unusable for no apparent reason.

Additional Information

Also, there is no Codecov check showing up on GitHub when the reports are unusable. I would expect a failed check to be visible on the PR.

Ok so I’ve figured out that the report is unusable if you didn’t checkout the repo in the job that uploads the report. But why?

@MetRonnie I didn’t dig into this so this is my intuition on what is happening.

We checkout the repo in order to get a list of the git files. We also do a few other git-like things before we do the upload. As a result, not having the repo in question makes it difficult for us to do things like map the coverage report back to files necessary, apply any fixes to files, or correct the commit SHA due to the way GitHub Actions checks out as a detached head.