Something broke ca. 25 days ago. How can I find out what exactly?

Support
, ,
1

Description

A month ago, my codecov setup was working properly.
It includes both C++ tests (with CMake, Catch2, gcov) and Python tests.
In a commit 25 days ago, that did not include any changes to CI or CMake or gcov or codecov.yaml, the setup stopped working: only the Python tests are reported.

Screenshot 2022-12-21 at 20.00.02
Screenshot 2022-12-21 at 20.00.021295×280 45.6 KB

Now, I see that I might should be reporting this is the “But Fixes” rather than the Support category, but I can also imagine that my initial setup was only working because of a bug that was fixed; or that GitHub changed something about the directory structure that I did not notice.

CI/CD URL

Unfortunately, for now, it is a private GitHub repository (https://github.com/GenieTim/pylimer-tools). The relevant codecov upload (GitHub action workflow file) looks like this:

name: Run Tests

on:
  push:
    branches:
      - master
      - main

jobs:
  build-deploy:
    runs-on: ubuntu-latest

    name: Run Python + C++ Tests
    steps:
      - uses: actions/checkout@v3
        with: 
          submodules: "recursive"

      - uses: actions/cache@v2
        with:
          path: |
            ./_skbuild
            ./vendor/igraph
            ./vendor/nlopt
            ./tests/build
          key: ${{ runner.os }}-py-3.10.6-${{ hashFiles('**/requirements.txt') }}
          restore-keys: |
            ${{ runner.os }}-py-3.10.6-
      - uses: actions/setup-python@v4
        with:
          python-version: '3.10.6'
          cache: 'pip'

      - name: Install Dependencies
        run: pip install -r requirements.txt
      
      - name: Install dev dependencies
        run: sudo apt-get install bison flex lcov libeigen3-dev
      
      - name: Run Tests
        run: ./bin/run-tests.sh
      
      # Generate Coverage Badge
      - name: Install Badge Generator
        run: pip install genbadge[coverage]

      - name: Coverage Badge
        run: genbadge coverage -i coverage.xml -o ".github/coverage.svg"
      
      - name: Upload to CodeCov
        uses: codecov/codecov-action@v3
        with:
          token: ${{ secrets.CODECOV_TOKEN }} # not required for public repos
          fail_ci_if_error: false # optional (default = false)
          verbose: true # optional (default = false)
          # gcov: true

Codecov Output

The full output of running the uploader on your CI/CD.

==> SHASUM file signed by key id 806bb28aed779869
==> Uploader SHASUM verified (20f9c9d78483fce977b6cc39e231a734a23bcd36f4d536bb7355222fb88d02bc  codecov)
==> Running version latest
==> Running version v0.3.2
/home/runner/work/_actions/codecov/codecov-action/v3/dist/codecov -n  -Q github-action-3.1.1 -v
[2022-12-21T15:13:55.562Z] ['verbose'] Start of uploader: 1671635635562...
[2022-12-21T15:13:55.564Z] ['info'] 
     _____          _
    / ____|        | |
   | |     ___   __| | ___  ___ _____   __
   | |    / _ \ / _` |/ _ \/ __/ _ \ \ / /
   | |___| (_) | (_| |  __/ (_| (_) \ V /
    \_____\___/ \__,_|\___|\___\___/ \_/

  Codecov report uploader 0.3.2
[2022-12-21T15:13:55.570Z] ['info'] => Project root located at: /home/runner/work/pylimer-tools/pylimer-tools
[2022-12-21T15:13:55.574Z] ['info'] ->  Token found by environment variables
[2022-12-21T15:13:55.575Z] ['verbose'] Start of network processing...
[2022-12-21T15:13:55.575Z] ['verbose'] Searching for files in /home/runner/work/pylimer-tools/pylimer-tools
[2022-12-21T15:13:55.712Z] ['info'] Running coverage xml...
[2022-12-21T15:13:55.986Z] ['verbose'] Wrote XML report to coverage.xml
[2022-12-21T15:13:55.986Z] ['info'] Searching for coverage files...
[2022-12-21T15:13:56.337Z] ['info'] Warning: Some files located via search were excluded from upload.
[2022-12-21T15:13:56.337Z] ['info'] If Codecov did not locate your files, please review https://docs.codecov.com/docs/supported-report-formats
[2022-12-21T15:13:56.337Z] ['verbose'] Preparing to clean the following coverage paths: codecov.yaml,coverage.xml,tests/build/#home#runner#work#pylimer-tools#pylimer-tools#src#pylimer_tools_cpp#MyClass.cpp.gcov 
// ...etc., very many .gcov files redacted
[2022-12-21T15:13:56.341Z] ['info'] => Found 489 possible coverage files:
// long list of gcov files again, redacted for brevity
[2022-12-21T15:13:56.341Z] ['verbose'] End of network processing
[2022-12-21T15:13:56.341Z] ['info'] Processing /home/runner/work/pylimer-tools/pylimer-tools/codecov.yaml...
[2022-12-21T15:13:56.343Z] ['info'] Processing /home/runner/work/pylimer-tools/pylimer-tools/coverage.xml...
[2022-12-21T15:13:56.343Z] ['info'] Processing /home/runner/work/pylimer-tools/pylimer-tools/tests/build/#home#runner#work#pylimer-tools#pylimer-tools#src#pylimer_tools_cpp#MyClass.cpp.gcov 
// etc., redacted for brevity
[2022-12-21T15:13:57.166Z] ['info'] Detected GitHub Actions as the CI provider.
[2022-12-21T15:13:57.166Z] ['verbose'] -> Using the following env variables:
[2022-12-21T15:13:57.166Z] ['verbose']      GITHUB_ACTION: __codecov_codecov-action
[2022-12-21T15:13:57.166Z] ['verbose']      GITHUB_HEAD_REF: 
[2022-12-21T15:13:57.166Z] ['verbose']      GITHUB_REF: refs/heads/main
[2022-12-21T15:13:57.166Z] ['verbose']      GITHUB_REPOSITORY: GenieTim/pylimer-tools
[2022-12-21T15:13:57.166Z] ['verbose']      GITHUB_RUN_ID: ...
[2022-12-21T15:13:57.166Z] ['verbose']      GITHUB_SERVER_URL: https://github.com
[2022-12-21T15:13:57.166Z] ['verbose']      GITHUB_SHA: ...
[2022-12-21T15:13:57.166Z] ['verbose']      GITHUB_WORKFLOW: Run Tests
[2022-12-21T15:13:57.166Z] ['verbose'] Using the following upload parameters:
[2022-12-21T15:13:57.166Z] ['verbose'] branch
[2022-12-21T15:13:57.166Z] ['verbose'] build
[2022-12-21T15:13:57.166Z] ['verbose'] buildURL
[2022-12-21T15:13:57.166Z] ['verbose'] commit
[2022-12-21T15:13:57.166Z] ['verbose'] job
[2022-12-21T15:13:57.166Z] ['verbose'] pr
[2022-12-21T15:13:57.166Z] ['verbose'] service
[2022-12-21T15:13:57.166Z] ['verbose'] slug
[2022-12-21T15:13:57.166Z] ['verbose'] name
[2022-12-21T15:13:57.166Z] ['verbose'] tag
[2022-12-21T15:13:57.166Z] ['verbose'] flags
[2022-12-21T15:13:57.166Z] ['verbose'] parent
[2022-12-21T15:13:57.168Z] ['info'] Pinging Codecov: https://codecov.io/upload/v4?package=github-action-3.1.1-uploader-0.3.2&token=*******&branch=main&build=3750309808&build_url=https%3A%2F%2Fgithub.com%2FGenieTim%2Fpylimer-tools%2Factions%2Fruns%2F3750309808&commit=...&job=Run+Tests&pr=&service=github-actions&slug=GenieTim%2Fpylimer-tools&name=&tag=&flags=&parent=
[2022-12-21T15:13:57.169Z] ['verbose'] Passed token was 36 characters long
[2022-12-21T15:13:57.169Z] ['verbose'] https://codecov.io/upload/v4?package=github-action-3.1.1-uploader-0.3.2&branch=main&build=3750309808&build_url=https%3A%2F%2Fgithub.com%2FGenieTim%2Fpylimer-tools%2Factions%2Fruns%2F3750309808&commit=...&job=Run+Tests&pr=&service=github-actions&slug=GenieTim%2Fpylimer-tools&name=&tag=&flags=&parent=
        Content-Type: 'text/plain'
        Content-Encoding: 'gzip'
        X-Reduced-Redundancy: 'false'
[2022-12-21T15:13:57.828Z] ['info'] https://app.codecov.io/github/GenieTim/pylimer-tools/commit/...
https://storage.googleapis.com/codecov/v4/raw/2022-12-21/.../.../....-ed21-4dac-91a4-baa1639ea048.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=...&X-Amz-Date=20221221T151357Z&X-Amz-Expires=10&X-Amz-SignedHeaders=host&X-Amz-Signature=...
[2022-12-21T15:13:57.829Z] ['verbose'] Returned upload url: https://storage.googleapis.com/codecov/v4/raw/2022-12-21/.../.../...-ed21-4dac-91a4-baa1639ea048.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=...&X-Amz-Date=20221221T151357Z&X-Amz-Expires=10&X-Amz-SignedHeaders=host&X-Amz-Signature=...
[2022-12-21T15:13:57.829Z] ['info'] Uploading...
[2022-12-21T15:13:58.141Z] ['info'] {"status":"success","resultURL":"https://app.codecov.io/github/GenieTim/pylimer-tools/commit/..."}
[2022-12-21T15:13:58.141Z] ['verbose'] End of uploader: 2579 milliseconds

Expected Results

Coedecoverage being reported consistently.
As I understand the logs, all the required gcov files are uploaded.

Actual Results

C++ test’s coverage is no longer listed.

Additional Information

And this is the content of my codecov.yaml:

codecov:
  # strict_yaml_branch: "yaml-config"
  # max_report_age: 24
  disable_default_path_fixes: no
  require_ci_to_pass: yes
  # notify:
  #   after_n_builds: 2
  #   wait_for_ci: yes
  
fixes:
  - "/home/runner/work/pylimer-tools/pylimer-tools/::/"
2

@GenieTim, it seems like there is something going wrong with how you are collecting coverage for C++ files. If you take a look at the coverage files for 19224f3d compared to bef924e0, you’ll see that some files are showing covered lines previously, while those recently are showing no coverage. I’m being a little vague with details on this because it’s a private repo.

Have you changed versions of gcov or how you collect coverage reports? Codecov looks to be showing the coverage as reported.

3

Thank you for your response, tom. Sorry for answering late, but I needed some time to figure things out; I have not yet managed to understand, what it is, that changed, maybe it was a version of a program in the GitHub Action, or something, but at least I got back to good coverage.

Your point of looking into the files that arrive in Codecov was a good one. They look different from what is generated on my local computer.

By outputting the contents of an example gcov file after each step in the CI, I found that the point where my coverage was reset was when running cmake pylimer_tools-gcov (as suggested by the used cmake-gcov integration), and that the gcov file was already populated after running the tests before that.

By just not running the cmake pylimer_tools-gcov step I got back to a nice amount of coverage.

4

@GenieTim absolutely, thanks for replying with all the juicy details. I wonder why cmake pylimer_tools-gcov reset the coverage files. I’ll have to dig in a little at some point.

Just wanted to double-check that things are looking good for you now?

5

Yes, I am wondering as well, especially since it does not make sense to me that this would change that way. Maybe there are other factors at play as well that I have not yet understood.
As it works for now, I will not prioritize any further investigations on my part until I get problems again.

Yes, things are looking good for me now, I guess I could accept any of our answers as the solution.

1 Like