Coverage report shows different files on random changes

escopecz · February 22, 2023, 7:35am

Before submitting a topic, please confirm the following

I have searched for similar issues before creating this topic.
I have verified that my repository is using the Codecov GitHub app, if using GitHub
I have validated my codecov.yaml configuration file.
I have filled out the below sections to the best of my ability.

Description

I’m building a GitHub Actions config for a HelloWorld Mautic plugin so it could be re-used by other open source plugins.

The tricky part is that the CI must firstly install Mautic and then the plugin to the dir plugins/HelloWorldBundle so the files I need to cover are not in the root. I found that I have to configure Path Fixing and it took be a day of trials and errors to get it somehow working.

CI/CD URL

Take a look at this CI run:

It resulted to this nice coverage report:

However, I then removed comments from the GitHub Actions config with this commit:

It resulted to this CI run:

And it got me back to the problematic report that shows 2 files that are not in the GitHub repository at all:

Codecov Output

Please provide the full output of running the uploader on your CI/CD. This will typically have the Codecov logo as ASCII.

{
  execArgs: [
    '-n',
    '',
    '-Q',
    'github-action-3.1.1',
    '-Z',
    '-C',
    '2344664aa35ace78c877dee90895da128a448ac1',
    '-v'
  ]
}
==> linux OS detected
https://uploader.codecov.io/latest/linux/codecov.SHA256SUM
Received SHA256SUM 080b43eaec3434326bb0f61653a82d27aba15c311ddde9d3f68cb364314f7aae  codecov
Received SHA256SUM signature -----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEEJwNOf9uFDgu8LGL/gGuyiu13mGkFAmPuYsYACgkQgGuyiu13
mGn5PQ//RI9iwWNANzlN6IVk5DZ0ZSK4H9OBzUbkoZenHwe/45h3uUFqiv40PUHG
nVMTDBwk9ygmSKJ5LgOC354/SKBBdeuEpex0RA4+Wc/K6dDblmvgYVLt4iAG+scy
tKwowB42AHMGJeaBXDJa2nc7AtaQEXw8tlfcZBgZhzdipsjV2fq9/muZF0BtaWCf
Cn80eoSvIhrPvaJ+8i5tYQPi5jPD54isRpcQB8piVjGXcvlrh5CE3V1L6kijSRti
BK8Ox+6/TR5xgesuriFfNOD/92yXuRJ8HrS8dmHChDLt++LjMSsCcDG3mpU3qlnh
1Ao5aJrNcxLVyYAjQ7MXIOuNy3dw0NjnHSVKk/UIbk6tHS/tYDiwRCNx+ssVuUW/
v96uR2qfs2Zb+qfz4QeM6Zfl9i7ZnTNh2QCGCjmSz04m/szjUVjJ643cVu00IkL+
EDopq16KOQIzJlOqk7oU0tv5YaVnDD51trGOAqAMjmriDvVRK5aVMCPPaDpSDddv
sK4lhJyv4CBv5kFDfwlBRbceEhYPO5tuXj+Udk9UFvGMaV19VrJSsOwJ1SxrdAYU
ZZF71yIfX9CxqpeSalTKz7cY8IqQmV94pvMrth9EuJucFlDPZlBLNSkLXfpXSUZu
Bmi+RRvqkiDdKW4OIAzwgyVfzpFsKp0hvErbrA8BNi90MrzYtbQ=
=7ZqW
-----END PGP SIGNATURE-----

==> SHASUM file signed by key id 806bb28aed779869
==> Uploader SHASUM verified (080b43eaec3434326bb0f61653a82d27aba15c311ddde9d3f68cb364314f7aae  codecov)
==> Running version latest
==> Running version v0.3.5
/home/runner/work/_actions/codecov/codecov-action/v3/dist/codecov -n  -Q github-action-3.1.1 -Z -C 2344664aa35ace78c877dee90895da128a448ac1 -v
[2023-02-21T15:38:12.944Z] ['verbose'] Start of uploader: 1676993892944...
[2023-02-21T15:38:12.945Z] ['info'] 
     _____          _
    / ____|        | |
   | |     ___   __| | ___  ___ _____   __
   | |    / _ \ / _` |/ _ \/ __/ _ \ \ / /
   | |___| (_) | (_| |  __/ (_| (_) \ V /
    \_____\___/ \__,_|\___|\___\___/ \_/

  Codecov report uploader 0.3.5
[2023-02-21T15:38:12.951Z] ['info'] => Project root located at: /home/runner/work/plugin-helloworld/plugin-helloworld
[2023-02-21T15:38:12.953Z] ['info'] ->  Token found by environment variables
[2023-02-21T15:38:12.954Z] ['verbose'] Start of network processing...
[2023-02-21T15:38:12.954Z] ['verbose'] Searching for files in /home/runner/work/plugin-helloworld/plugin-helloworld
[2023-02-21T15:38:12.966Z] ['verbose'] coveragepy is not installed
[2023-02-21T15:38:12.966Z] ['info'] Searching for coverage files...
[2023-02-21T15:38:13.308Z] ['info'] Warning: Some files located via search were excluded from upload.
[2023-02-21T15:38:13.308Z] ['info'] If Codecov did not locate your files, please review https://docs.codecov.com/docs/supported-report-formats
[2023-02-21T15:38:13.308Z] ['verbose'] Preparing to clean the following coverage paths: plugins/HelloWorldBundle/coverage.xml
[2023-02-21T15:38:13.308Z] ['info'] => Found 1 possible coverage files:
  plugins/HelloWorldBundle/coverage.xml
[2023-02-21T15:38:13.308Z] ['verbose'] End of network processing
[2023-02-21T15:38:13.308Z] ['info'] Processing /home/runner/work/plugin-helloworld/plugin-helloworld/plugins/HelloWorldBundle/coverage.xml...
[2023-02-21T15:38:13.319Z] ['info'] Using manual override from args.
[2023-02-21T15:38:13.319Z] ['info'] Detected GitHub Actions as the CI provider.
[2023-02-21T15:38:13.319Z] ['verbose'] -> Using the following env variables:
[2023-02-21T15:38:13.319Z] ['verbose']      GITHUB_ACTION: __codecov_codecov-action
[2023-02-21T15:38:13.319Z] ['verbose']      GITHUB_HEAD_REF: ci
[2023-02-21T15:38:13.319Z] ['verbose']      GITHUB_REF: refs/pull/11/merge
[2023-02-21T15:38:13.319Z] ['verbose']      GITHUB_REPOSITORY: mautic/plugin-helloworld
[2023-02-21T15:38:13.319Z] ['verbose']      GITHUB_RUN_ID: 4234313765
[2023-02-21T15:38:13.319Z] ['verbose']      GITHUB_SERVER_URL: https://github.com
[2023-02-21T15:38:13.320Z] ['verbose']      GITHUB_SHA: c59c190a888d0a2b366938ebb1075a9c447a4be2
[2023-02-21T15:38:13.320Z] ['verbose']      GITHUB_WORKFLOW: Mautic Plugin tests
[2023-02-21T15:38:13.320Z] ['verbose'] Using the following upload parameters:
[2023-02-21T15:38:13.320Z] ['verbose'] branch
[2023-02-21T15:38:13.320Z] ['verbose'] build
[2023-02-21T15:38:13.320Z] ['verbose'] buildURL
[2023-02-21T15:38:13.320Z] ['verbose'] commit
[2023-02-21T15:38:13.320Z] ['verbose'] job
[2023-02-21T15:38:13.320Z] ['verbose'] pr
[2023-02-21T15:38:13.320Z] ['verbose'] service
[2023-02-21T15:38:13.321Z] ['verbose'] slug
[2023-02-21T15:38:13.321Z] ['verbose'] name
[2023-02-21T15:38:13.321Z] ['verbose'] tag
[2023-02-21T15:38:13.321Z] ['verbose'] flags
[2023-02-21T15:38:13.321Z] ['verbose'] parent
[2023-02-21T15:38:13.322Z] ['info'] Pinging Codecov: https://codecov.io/upload/v4?package=github-action-3.1.1-uploader-0.3.5&token=*******&branch=ci&build=4234313765&build_url=https%3A%2F%2Fgithub.com%2Fmautic%2Fplugin-helloworld%2Factions%2Fruns%2F4234313765&commit=2344664aa35ace78c877dee90895da128a448ac1&job=Mautic+Plugin+tests&pr=11&service=github-actions&slug=mautic%2Fplugin-helloworld&name=&tag=&flags=&parent=
[2023-02-21T15:38:13.322Z] ['verbose'] Passed token was 36 characters long
[2023-02-21T15:38:13.322Z] ['verbose'] https://codecov.io/upload/v4?package=github-action-3.1.1-uploader-0.3.5&branch=ci&build=4234313765&build_url=https%3A%2F%2Fgithub.com%2Fmautic%2Fplugin-helloworld%2Factions%2Fruns%2F4234313765&commit=2344664aa35ace78c877dee90895da128a448ac1&job=Mautic+Plugin+tests&pr=11&service=github-actions&slug=mautic%2Fplugin-helloworld&name=&tag=&flags=&parent=
        Content-Type: 'text/plain'
        Content-Encoding: 'gzip'
        X-Reduced-Redundancy: 'false'
[2023-02-21T15:38:13.589Z] ['info'] https://app.codecov.io/github/mautic/plugin-helloworld/commit/2344664aa35ace78c877dee90895da128a448ac1
https://storage.googleapis.com/codecov/v4/raw/2023-02-21/2F4DF2FC09F5935CDB217A398343ECF9/2344664aa35ace78c877dee90895da128a448ac1/bff8e1c2-6652-4076-a8c0-d544d597cc5a.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=GOOG1EJOGFN2JQ4OCTGA2MU5AEIT7OT5Z7HTFOAN2SPG4NWSN2UJYOY5U6LZQ%2F20230221%2FUS%2Fs3%2Faws4_request&X-Amz-Date=20230221T153813Z&X-Amz-Expires=10&X-Amz-SignedHeaders=host&X-Amz-Signature=fbc01aaf9255884a1fec0c4858e1dba466fc0b5beaabb36f4b2c2bca238eed31
[2023-02-21T15:38:13.590Z] ['verbose'] Returned upload url: https://storage.googleapis.com/codecov/v4/raw/2023-02-21/2F4DF2FC09F5935CDB217A398343ECF9/2344664aa35ace78c877dee90895da128a448ac1/bff8e1c2-6652-4076-a8c0-d544d597cc5a.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=GOOG1EJOGFN2JQ4OCTGA2MU5AEIT7OT5Z7HTFOAN2SPG4NWSN2UJYOY5U6LZQ%2F20230221%2FUS%2Fs3%2Faws4_request&X-Amz-Date=20230221T153813Z&X-Amz-Expires=10&X-Amz-SignedHeaders=host&X-Amz-Signature=fbc01aaf9255884a1fec0c4858e1dba466fc0b5beaabb36f4b2c2bca238eed31
[2023-02-21T15:38:13.590Z] ['info'] Uploading...
[2023-02-21T15:38:13.814Z] ['info'] {"status":"success","resultURL":"https://app.codecov.io/github/mautic/plugin-helloworld/commit/2344664aa35ace78c877dee90895da128a448ac1"}
[2023-02-21T15:38:13.814Z] ['verbose'] End of uploader: 870 milliseconds

Expected Results

CodeCov shows the files from the GitHub - mautic/plugin-helloworld: Hello World plugin built on the Integration framework repository and not random files from other repositories

Actual Results

I made it work but after removing a comment it got back into wrong output and there was no reason for that. So I expect there is some cache that shows the wrong output no matter what changes I make.

Additional Information

I’m also outputing the XML file that is being sent to CodeCov to make sure it contains the right files with some coverage:

The codecov action is configured like so:

- name: Upload coverage report
      uses: codecov/codecov-action@v3
      with:
        token: ${{ secrets.CODECOV_TOKEN }}
        fail_ci_if_error: true
        working-directory: plugins/HelloWorldBundle
        verbose: true

Notice that I’ve configured the working-directory option. That’s because Mautic is installed first and then the plugin is installed into it like so:

    - name: Checkout Mautic 4
      uses: actions/checkout@v3
      with:
        repository: mautic/mautic
        ref: 4.4

    - name: Checkout this plugin
      uses: actions/checkout@v3
      with:
        path: plugins/HelloWorldBundle

Full workflow:

github.com

mautic/plugin-helloworld/blob/ci/.github/workflows/tests.yml

name: Mautic Plugin tests

on:
  push:
    branches:
      - mautic-4 # Main branch
      - '[0-9]+\.[0-9]+'
  pull_request:

env:
  PLUGIN_DIR: plugins/HelloWorldBundle # Same as extra.install-directory-name in composer.json

jobs:
  phpunit:
    runs-on: ubuntu-latest
    
    strategy:
      matrix:
        php-versions: ['7.4', '8.0'] # The supported PHP versions
        db-types: ['mysql'] # can be: ['mysql', 'mariadb'] but not necessary for this plugin that does not add any DB schema

This file has been truncated. show original

I also mapped the path from the coverage report xml in codecov.yml like this:

codecov:
  disable_default_path_fixes: true
fixes:
  - "/home/runner/work/plugin-helloworld/plugin-helloworld/plugins/HelloWorldBundle/::"

tom · February 22, 2023, 5:49pm

@escopecz I’ll take a look at this today

tom · February 23, 2023, 5:34pm

Ok, it looks like this is a problem with the GitHub app connection. Can you try re-installing the GitHub app?

escopecz · February 23, 2023, 7:34pm

I noticed that the repository was not “selected” in the CodeCov app. So I selected it and re-run the last GH Actions job but it ended up with the same result.

So you suggest that I click on the “Uninstall” button where it says:

Uninstall “Codecov”

This will remove the app and revoke access to all resources.

Is that what you mean by re-installing? I worry to do that because it affects also our main repository. I wouldn’t want to break it there as it would affect many people during active refactoring sprint. Is there anything else to try?

tom · February 24, 2023, 5:04am

Huh, that is a little weird. Do you mind sharing the commit SHA that it’s having issues on?

escopecz · February 24, 2023, 6:20am

It’s the last commit of CI init by escopecz · Pull Request #11 · mautic/plugin-helloworld · GitHub, so

2344664aa35ace78c877dee90895da128a448ac1

escopecz · February 24, 2023, 12:25pm

I don’t want to jinx it but when I re-run the same GH Action for the 3rd time this morning then it generated this report which looks correct: Codecov

tom · February 24, 2023, 5:56pm

@escopecz looks like the integration is working now at least on our side. I’ll keep this open for a few more days, but let me know if things are working for you now.

escopecz · February 26, 2023, 5:03pm

I added some more commits today and it looks stable. Thank you for your help! Pointing me to the Codecov GH app showed me the issue. This can be closed.

Topic		Replies	Views
Update to codecov/codecov-action@v2 causes coverage change Bug Fixes	1	935	October 13, 2021
Some files in coverage report not appearing in Codecov Support python , github	1	213	November 15, 2022
Coverage is comparing with incorrect base branch Support golang , github	6	220	February 28, 2023
Reports with git submodules Support github	0	645	July 29, 2019
"There is an error processing the coverage reports " Support	2	366	September 17, 2022