PR commits showing up in default (main) branch coverage

Support Bug Fixes
1

Please search for similar issues before creating a ticket.
All fields below are required.

Description

Until about April 15, we had monotonically increasing coverage (other than a little dip that I think was due to some temporary CI failures in late Jan/early feb):

Around April 15 you can see that our coverage started going a bit crazy. It’s because coverage from PRs (with some CIs failing and thus not uploading coverage) is leaking into the coverage reports for the main branch. I looked at our commit list and don’t see anything related to CIs near April 15th, so I suspect some bug with the Python uploader might have been introduced around this time (?).

Commit SHAs

Looking at the commit according to codecov for our main branch:

  1. Correct 7b428b96c2a719902ae72a498618062f879d674c
  2. Incorrect 8ad135c1fdb5026bde1c80a6e5c4b69c59d4c702

And in the actual commit list you can see that the order should be:

  1. Correct: MRG ENH/BUG Add exclude parameter to plot_evoked_topo (#9278) · mne-tools/mne-python@7b428b9 · GitHub
  2. Then: FIX: flat map plot in notebook (#9305) · mne-tools/mne-python@ead743b · GitHub

Repository

CI/CD or Build URL

Uploader

Python on Azure, codecov-action (which I think uses Bash?) on GitHub Actions. Will try switching to the Bash uploader on Azure to see if it helps.

Codecov Output

For this, let’s work with the most recent incorrect upload in the GUI so that all CI reports are accessible. That would be for this report, corresponding to this PR. Azure builds the merged version and gives the message:

HEAD is now at e502a1f2e Merge 27c545be4209f3ca416f113679df2b673bcbde40 into 0d5068dbbce76598c9c6433e1ace1265e1955564

And looking farther down we see the report message:

Script contents:
codecov --root %BUILD_REPOSITORY_LOCALPATH% -t %CODECOV_TOKEN%
========================== Starting Command Output ===========================
"C:\Windows\system32\cmd.exe" /D /E:ON /V:OFF /S /C "CALL "D:\a\_temp\92ace1a5-17e0-4295-bead-1fe8cda7a25f.cmd""

      _____          _
     / ____|        | |
    | |     ___   __| | ___  ___ _____   __
    | |    / _ \ / _  |/ _ \/ __/ _ \ \ / /
    | |___| (_) | (_| |  __/ (_| (_) \ V /
     \_____\___/ \____|\___|\___\___/ \_/
                                    v2.1.11

==> Detecting CI provider
  -> Got branch from git/hg
  -> Got sha from git/hg
==> Preparing upload
==> Processing gcov (disable by -X gcov)
==> Collecting reports
    Generating coverage xml reports for Python
    + D:\a\1\s\coverage.xml bytes=4852039
==> Uploading
    .url https://codecov.io
    .query commit=e502a1f2e332a861d52eebc41f458560322dec4d&branch=HEAD&token=<secret>&yaml=codecov.yml&package=py2.1.11
    Gzipping contents..
    Compressed contents to 318424 bytes
    Pinging Codecov...
    Uploading to S3...
    Uploading to S3 took 0:00:00.303807
    https://codecov.io/github/mne-tools/mne-python/commit/27c545be4209f3ca416f113679df2b673bcbde40
Finishing: Codecov

Expected Results

Reports for main shouldn’t come from un-merged PRs.

Actual Results

Un-merged PR coverage reports end up on main.

Additional Information

N/A

2

Looks like the link generated by the bash uploader works to show that it’s part of a PR:

and it no longer shows up as part of the main branch:

https://app.codecov.io/gh/mne-tools/mne-python/branch/main

It says “Missing base report” on the PR page, but I think this is just because the latest commit to main had all CIs skipped:

I’m happy with using the bash uploader instead, so feel free to mark this as “solved”, though in principle it seems like the Python uploader has some bug / was doing something wrong here.