Before submitting a topic, please confirm the following
I have searched for similar issues before creating this topic.
I have verified that my repository is using the Codecov GitHub app, if using GitHub
I have validated my codecov.yaml configuration file.
I have filled out the below sections to the best of my ability.
Run codecov/codecov-action@v2
==> linux OS detected
https://uploader.codecov.io/latest/linux/codecov.SHA256SUM
==> SHASUM file signed by key id 806bb28aed779869
==> Uploader SHASUM verified (20f9c9d78483fce977b6cc39e231a734a23bcd36f4d536bb7355222fb88d02bc codecov)
==> Running version latest
==> Running version v0.3.2
/home/runner/work/_actions/codecov/codecov-action/v2/dist/codecov -n -Q github-action-2.1.0
[2023-02-14T00:58:39.669Z] ['info']
_____ _
/ ____| | |
| | ___ __| | ___ ___ _____ __
| | / _ \ / _` |/ _ \/ __/ _ \ \ / /
| |___| (_) | (_| | __/ (_| (_) \ V /
\_____\___/ \__,_|\___|\___\___/ \_/
Codecov report uploader 0.3.2
[2023-02-14T00:58:39.677Z] ['info'] => Project root located at: /home/runner/work/module-ballerina-graphql/module-ballerina-graphql
[2023-02-14T00:58:39.681Z] ['info'] -> No token specified or token is empty
[2023-02-14T00:58:39.690Z] ['info'] Searching for coverage files...
[2023-02-14T00:58:40.054Z] ['info'] Warning: Some files located via search were excluded from upload.
[2023-02-14T00:58:40.054Z] ['info'] If Codecov did not locate your files, please review https://docs.codecov.com/docs/supported-report-formats
[2023-02-14T00:58:40.055Z] ['info'] => Found 2 possible coverage files:
ballerina/target/report/graphql/coverage-report.xml
ballerina-tests/target/report/graphql_tests/coverage-report.xml
[2023-02-14T00:58:40.055Z] ['info'] Processing /home/runner/work/module-ballerina-graphql/module-ballerina-graphql/ballerina/target/report/graphql/coverage-report.xml...
[2023-02-14T00:58:40.065Z] ['info'] Processing /home/runner/work/module-ballerina-graphql/module-ballerina-graphql/ballerina-tests/target/report/graphql_tests/coverage-report.xml...
[2023-02-14T00:58:40.154Z] ['info'] Detected GitHub Actions as the CI provider.
[2023-02-14T00:58:40.156Z] ['info'] Pinging Codecov: https://codecov.io/upload/v4?package=github-action-2.1.0-uploader-0.3.2&token=*******&branch=master&build=4169324782&build_url=https%3A%2F%2Fgithub.com%2Fballerina-platform%2Fmodule-ballerina-graphql%2Factions%2Fruns%2F4169324782&commit=25577c282d6b5ed08066f352b0de68231b20e6a4&job=Build&pr=&service=github-actions&slug=ballerina-platform%2Fmodule-ballerina-graphql&name=&tag=&flags=&parent=
[2023-02-14T00:58:40.638Z] ['info'] https://app.codecov.io/github/ballerina-platform/module-ballerina-graphql/commit/25577c282d6b5ed08066f352b0de68231b20e6a4
https://storage.googleapis.com/codecov/v4/raw/2023-02-14/E634D17C5687D8A565E626788477D264/25577c282d6b5ed08066f352b0de68231b20e6a4/8e19bb54-d93f-4325-bf31-cad8e89ffc08.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=GOOG1EJOGFN2JQ4OCTGA2MU5AEIT7OT5Z7HTFOAN2SPG4NWSN2UJYOY5U6LZQ%2F20230214%2FUS%2Fs3%2Faws4_request&X-Amz-Date=20230214T005840Z&X-Amz-Expires=10&X-Amz-SignedHeaders=host&X-Amz-Signature=7419fccdd8b59590a6563007151ff7f6bcd714a724163fe040f9117dd4b4e727
[2023-02-14T00:58:40.638Z] ['info'] Uploading...
[2023-02-14T00:58:40.864Z] ['info'] {"status":"success","resultURL":"https://app.codecov.io/github/ballerina-platform/module-ballerina-graphql/commit/25577c282d6b5ed08066f352b0de68231b20e6a4"}
Expected Results
Code coverage not changed.
Actual Results
Code coverage drastically dropped.
Additional Information
Noticing the below error when checking the commit in CI/CD pipeline. Tried triggering the build, but the error status doesnāt seem to change. Didnāt notice any issue in the CI flow.
There is a coverage drop reported for the following repos as well.
According to our observation, the coverage report excludes the test coverage from the ballerina-tests submodule. All the affected modules have a separate Gradle submodule for tests separate from where the source files are located. But, when checked the uploaded files list to Codecov thereās no difference.
They also have multiple jacoco coverage XML files uploaded to Codecov from same repository.
We sent similar revert PRs for the other repositories with the same issue as well and noticed the same behavior i.e. the coverage drop didnāt recover.
Apart from the above, we compared the coverage XML files uploaded to Codecov for the problematic commit and the previous commit and could not see any difference to line coverages, that could have resulted in such a drop in coverage.
Appreciate your efforts to resolve this issue. This coverage drop has impacted our development and review processes as well.
In our repositories, we have two code coverage xml files generated. In the GitHub action logs, it shows that the two files are identified, but it seems the coverage report generation ignores one file, and only calculates the coverage using one file.
Is there a way to check whether the two coverage reports are merged or not before calculating the code coverage?
As I mentioned earlier, we have two coverage XML files. One from ballerina submodule, and another from ballerina-tests submodule. Even though these are two submodules, the src files are in the ballerina submodule, i.e. the ballerina-tests submodule contains tests files that are testing the source code inside the ballerina submodule. After some investigation, it indeed looks like the reason for coverage drop is an issue with merging these two xml files.
Hereās what I did.
I added files field to the CodeCov action configs in GitHub workflow, where I specifically mentioned the xml file that was ignored in the first place. Then the missing coverage appeared back (but reduced the coverage that has been reported in the other xml file). In the PR I have linked below shows that the coverage is 77% after this.
Then I tried to add both files using the files field (by adding both files separated by a comma), and then it reverted back to the erroneous stage, where one file is ignored. The coverage is again reported as 26%.
No matter the order, it always rejects/ignores the coverage reported by ballerina-tests submodule, and only calculates the coverage using ballerina submodule. If we specifically mention the XML file from the ballerina-tests submodule, it does use it, but ignores the ballerina submodule.
Hereās the branch where I specifically added the XML file from the ballerina-tests submodule:
Conclusion:
Whenever there are two coverage XML files, it only consider the first submodule and the others are ignored.
Is there a solution for this? Please let us know if we can fix this.
Sorry @ThisaruGuruge I was trying to debug this a bit this week but was having some issues.
I still havenāt gotten to root cause, give me a few hours to try to get this working. The above mentioned with the rejection of coverage is extremely helpful.
I added files field to the CodeCov action configs in GitHub workflow, where I specifically mentioned the xml file that was ignored in the first place. Then the missing coverage appeared back (but reduced the coverage that has been reported in the other xml file). In the PR I have linked below shows that the coverage is 77% after this.
Ok, I will refute this because it is missing the ballerina/ folder altogether
We revisited the code coverage collection and identified that the issue is in the path fixes.
When we have 2 submodules(ballerina and ballerina-tests) and ballerina-tests that cover the source in the ballerina module, it requires path fixes to map the coverage information to the source files.
This adds path fix only for ballerina/annotation_processor.bal and you can notice that the coverage information is picked up for that file with this PR.
ballerina/graphql/1/annotation_processor needs to be corrected as ballerina/annotation_processor
But in the above path 1 can be varying. it could be 0 or any other value. Thatās why we need the wildcard.
According to my understanding, The mapping is ideally ballerina/graphql/*/ mapped to ballerina/ like the one weāve attempted here. But it isnāt working. Can you please suggest a mapping that might work?
Thanks @tom. Youād have to accept the contributor license in the upstream PR in order for us to merge it. Iāll do the same fix for other repositories and see.
@Dilhasha, it may be faster if someone on the ballerina team makes the PR as I will need to go through some hoops before I can sign the CLA. But the turnover for that could be EOD today.
