Coverage dropped by > 60% after merging a commit

Before submitting a topic, please confirm the following

I have searched for similar issues before creating this topic.
I have verified that my repository is using the Codecov GitHub app, if using GitHub
I have validated my codecov.yaml configuration file.
I have filled out the below sections to the best of my ability.

Description

Noticing a drastic drop in coverage after merging a commit for the repository GitHub - ballerina-platform/module-ballerina-graphql: This is the Ballerina GraphQL module, which is a part of Ballerina Language Standard Library after merging [AUTO MERGE] Update Dependencies (Ballerina Lang : 2201.4.0-20230213-151200-6d175fd1) by ballerina-bot · Pull Request #1220 · ballerina-platform/module-ballerina-graphql · GitHub

CI/CD URL

Codecov Output

Run codecov/codecov-action@v2
==> linux OS detected
https://uploader.codecov.io/latest/linux/codecov.SHA256SUM
==> SHASUM file signed by key id 806bb28aed779869
==> Uploader SHASUM verified (20f9c9d78483fce977b6cc39e231a734a23bcd36f4d536bb7355222fb88d02bc  codecov)
==> Running version latest
==> Running version v0.3.2
/home/runner/work/_actions/codecov/codecov-action/v2/dist/codecov -n  -Q github-action-2.1.0
[2023-02-14T00:58:39.669Z] ['info'] 
     _____          _
    / ____|        | |
   | |     ___   __| | ___  ___ _____   __
   | |    / _ \ / _` |/ _ \/ __/ _ \ \ / /
   | |___| (_) | (_| |  __/ (_| (_) \ V /
    \_____\___/ \__,_|\___|\___\___/ \_/

  Codecov report uploader 0.3.2
[2023-02-14T00:58:39.677Z] ['info'] => Project root located at: /home/runner/work/module-ballerina-graphql/module-ballerina-graphql
[2023-02-14T00:58:39.681Z] ['info'] -> No token specified or token is empty
[2023-02-14T00:58:39.690Z] ['info'] Searching for coverage files...
[2023-02-14T00:58:40.054Z] ['info'] Warning: Some files located via search were excluded from upload.
[2023-02-14T00:58:40.054Z] ['info'] If Codecov did not locate your files, please review https://docs.codecov.com/docs/supported-report-formats
[2023-02-14T00:58:40.055Z] ['info'] => Found 2 possible coverage files:
  ballerina/target/report/graphql/coverage-report.xml
  ballerina-tests/target/report/graphql_tests/coverage-report.xml
[2023-02-14T00:58:40.055Z] ['info'] Processing /home/runner/work/module-ballerina-graphql/module-ballerina-graphql/ballerina/target/report/graphql/coverage-report.xml...
[2023-02-14T00:58:40.065Z] ['info'] Processing /home/runner/work/module-ballerina-graphql/module-ballerina-graphql/ballerina-tests/target/report/graphql_tests/coverage-report.xml...
[2023-02-14T00:58:40.154Z] ['info'] Detected GitHub Actions as the CI provider.
[2023-02-14T00:58:40.156Z] ['info'] Pinging Codecov: https://codecov.io/upload/v4?package=github-action-2.1.0-uploader-0.3.2&token=*******&branch=master&build=4169324782&build_url=https%3A%2F%2Fgithub.com%2Fballerina-platform%2Fmodule-ballerina-graphql%2Factions%2Fruns%2F4169324782&commit=25577c282d6b5ed08066f352b0de68231b20e6a4&job=Build&pr=&service=github-actions&slug=ballerina-platform%2Fmodule-ballerina-graphql&name=&tag=&flags=&parent=
[2023-02-14T00:58:40.638Z] ['info'] https://app.codecov.io/github/ballerina-platform/module-ballerina-graphql/commit/25577c282d6b5ed08066f352b0de68231b20e6a4
https://storage.googleapis.com/codecov/v4/raw/2023-02-14/E634D17C5687D8A565E626788477D264/25577c282d6b5ed08066f352b0de68231b20e6a4/8e19bb54-d93f-4325-bf31-cad8e89ffc08.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=GOOG1EJOGFN2JQ4OCTGA2MU5AEIT7OT5Z7HTFOAN2SPG4NWSN2UJYOY5U6LZQ%2F20230214%2FUS%2Fs3%2Faws4_request&X-Amz-Date=20230214T005840Z&X-Amz-Expires=10&X-Amz-SignedHeaders=host&X-Amz-Signature=7419fccdd8b59590a6563007151ff7f6bcd714a724163fe040f9117dd4b4e727
[2023-02-14T00:58:40.638Z] ['info'] Uploading...
[2023-02-14T00:58:40.864Z] ['info'] {"status":"success","resultURL":"https://app.codecov.io/github/ballerina-platform/module-ballerina-graphql/commit/25577c282d6b5ed08066f352b0de68231b20e6a4"}

Expected Results

Code coverage not changed.

Actual Results

Code coverage drastically dropped.

Additional Information

  1. Noticing the below error when checking the commit in CI/CD pipeline. Tried triggering the build, but the error status doesn’t seem to change. Didn’t notice any issue in the CI flow.

  1. There is a coverage drop reported for the following repos as well.

According to our observation, the coverage report excludes the test coverage from the ballerina-tests submodule. All the affected modules have a separate Gradle submodule for tests separate from where the source files are located. But, when checked the uploaded files list to Codecov there’s no difference.

They also have multiple jacoco coverage XML files uploaded to Codecov from same repository.

1 Like

@tom Appreciate your thoughts

@Dilhasha, there’s a lot going on here, and I’ve spent a few hours on this. I’ll come back to it later tonight.

Can you see if reverting the change in CI fixes the issue?

2 Likes

Hi @tom,
Thank you for looking into this. Yes, we did try reverting the changes. But the coverage drop did not recover.

1 Like

@ThisaruGuruge looking at the commit stack, the commit that brought down your coverage so much was this one(49d7316).

That commit refers to this PR. What happens if you revert that change?

Hi @tom,

We opened this PR [TEST] Revert "[Automated] Update dependencies" by ThisaruGuruge · Pull Request #1225 · ballerina-platform/module-ballerina-graphql · GitHub to revert it. If you look at the coverage report for the PR [TEST] Revert "[Automated] Update dependencies" by ThisaruGuruge · Pull Request #1225 · ballerina-platform/module-ballerina-graphql · GitHub it still says no change to coverage.

Apart from that, we created a new branch with the master branch(where coverage dropped) and merged a PR with the same changes [TEST] Revert lang version by Thevakumar-Luheerathan · Pull Request #1226 · ballerina-platform/module-ballerina-graphql · GitHub
If you look at Codecov coverage details for this new branch, it still shows the dropped coverage Codecov.

We sent similar revert PRs for the other repositories with the same issue as well and noticed the same behavior i.e. the coverage drop didn’t recover.

Apart from the above, we compared the coverage XML files uploaded to Codecov for the problematic commit and the previous commit and could not see any difference to line coverages, that could have resulted in such a drop in coverage.

Appreciate your efforts to resolve this issue. This coverage drop has impacted our development and review processes as well.

Hi @tom

I sent a PR reverting the PR you mentioned:

As per this, the code coverage is not reduced.

In our repositories, we have two code coverage xml files generated. In the GitHub action logs, it shows that the two files are identified, but it seems the coverage report generation ignores one file, and only calculates the coverage using one file.

Is there a way to check whether the two coverage reports are merged or not before calculating the code coverage?

1 Like

@tom Sorry for bothering, but can you please let us know if there’s any solution for this? Our development process is highly affected by this.

I did some playing around on this issue.

As I mentioned earlier, we have two coverage XML files. One from ballerina submodule, and another from ballerina-tests submodule. Even though these are two submodules, the src files are in the ballerina submodule, i.e. the ballerina-tests submodule contains tests files that are testing the source code inside the ballerina submodule. After some investigation, it indeed looks like the reason for coverage drop is an issue with merging these two xml files.

Here’s what I did.

  • I added files field to the CodeCov action configs in GitHub workflow, where I specifically mentioned the xml file that was ignored in the first place. Then the missing coverage appeared back (but reduced the coverage that has been reported in the other xml file). In the PR I have linked below shows that the coverage is 77% after this.
  • Then I tried to add both files using the files field (by adding both files separated by a comma), and then it reverted back to the erroneous stage, where one file is ignored. The coverage is again reported as 26%.
  • No matter the order, it always rejects/ignores the coverage reported by ballerina-tests submodule, and only calculates the coverage using ballerina submodule. If we specifically mention the XML file from the ballerina-tests submodule, it does use it, but ignores the ballerina submodule.

Here’s the branch where I specifically added the XML file from the ballerina-tests submodule:

Conclusion:

Whenever there are two coverage XML files, it only consider the first submodule and the others are ignored.

Is there a solution for this? Please let us know if we can fix this.

Sorry @ThisaruGuruge I was trying to debug this a bit this week but was having some issues.

I still haven’t gotten to root cause, give me a few hours to try to get this working. The above mentioned with the rejection of coverage is extremely helpful.

1 Like
  • I added files field to the CodeCov action configs in GitHub workflow, where I specifically mentioned the xml file that was ignored in the first place. Then the missing coverage appeared back (but reduced the coverage that has been reported in the other xml file). In the PR I have linked below shows that the coverage is 77% after this.

Ok, I will refute this because it is missing the ballerina/ folder altogether

I think the two files is a red herring. This is the coverage with just the ballerina/.../.xml coverage file.


This is the coverage with just the ballerina-tests/.../.xml coverage file.


Are you sure there hasn’t been a change in the way you collect coverage?

Hi @tom,

We revisited the code coverage collection and identified that the issue is in the path fixes.

When we have 2 submodules(ballerina and ballerina-tests) and ballerina-tests that cover the source in the ballerina module, it requires path fixes to map the coverage information to the source files.

We have this path fix(module-ballerina-graphql/codecov.yml at master · ballerina-platform/module-ballerina-graphql · GitHub) added in the codecov.yml to address these and it was working until 2 weeks ago. There hasn’t been any change to the path fixes from the PR that dropped the coverage.

But, when looking at the coverage that hasn’t been considered, it comes from the source files which require the path fixes. So, we opened a PR to check if we can explicitly add a path fix Add explicit path fix by Dilhasha · Pull Request #1252 · ballerina-platform/module-ballerina-graphql · GitHub

This adds path fix only for ballerina/annotation_processor.bal and you can notice that the coverage information is picked up for that file with this PR.

But, when we use the wildcard to do the path fix, it fails to do the path fixes correctly. Please refer to the PR Change path fixes to update coverage by Dilhasha · Pull Request #1253 · ballerina-platform/module-ballerina-graphql · GitHub which has no coverage change.

So, we would highly appreciate it if you can look into this matter and help us to resolve this issue.

@Dilhasha I’m not sure the wildcard is needed here since it’s more of a path replacement. What happens if you just remove the /*?

Following is a sample path that we need to fix.

  • ballerina/graphql/1/annotation_processor needs to be corrected as ballerina/annotation_processor

  • But in the above path 1 can be varying. it could be 0 or any other value. That’s why we need the wildcard.
    According to my understanding, The mapping is ideally ballerina/graphql/*/ mapped to ballerina/ like the one we’ve attempted here. But it isn’t working. Can you please suggest a mapping that might work?

I even tried removing the wild card, but that doesn’t fix the issue. Add path fix without wildcard by Dilhasha · Pull Request #1255 · ballerina-platform/module-ballerina-graphql · GitHub

Please note that we haven’t changed this codecov.yaml since 2022 October.

1 Like

@Dilhasha a ha! I think I got it.

I made a PR here to show the 80% coverage, and I’ll make the change in a PR to the upstream repo.

2 Likes

Thanks @tom. You’d have to accept the contributor license in the upstream PR in order for us to merge it. I’ll do the same fix for other repositories and see.

1 Like

@Dilhasha, it may be faster if someone on the ballerina team makes the PR as I will need to go through some hoops before I can sign the CLA. But the turnover for that could be EOD today.

2 Likes

Thank you @tom for the help provided. We will send a separate PR and merge it. Really appreciate your help resolving this.

2 Likes