Gaps in functionality

@rafmudaf ,

First, I want to say, Thank you for this very detailed feedback. It has been shared with our product team.

As the lead support engineer (not Tom’s boss :smiley: ) I would like to walk though the issues you raise here one by one for my own understanding and thoughts on how we can improve these process in the future .

Please note that I am not expecting you to try any of them, so please don’t think this is an attempt to get you to try us again, even though I would love if you did. I completely understand your frustration.

  • The comments included in the pull requests sometimes list incorrect files as “diffs”
    ** I think this happens when there’s a force-push and the parent-child commit relationship changes
    ** Regenerate or update GitHub pull request comment

This is a frequent source of confusion to users and we are looking into ways to make this clearer. (I will probably say this a lot in this response, though I hope less then I expect.

Assuming we are talking about the Coverage Diff section, this is generated directly from the raw coverage reports that are uploaded. It is not connected to the GitHub diff, even though the same terms are used. There are a number of reasons why the two coverage reports may differ, and we have a troubleshooting guide in our docs (that we need to surface better. Some of the unexpected reasons are tests not running, or parts of the overage not uploading if there are multiple reports.

Regarding the feature request to restart / regenerate the comment: We have a feature request in the backlog for this. Given the way we require CI / Repo information to be available at processing time, tjis is not as straightforward as I would expect it to be. In cases where the commit gets “deleted” due to squash, rebase, etc, it becomes impossible.

The code coverage treemap is initially completely incorrect in the GitHub comment.

Yes. 100% agree. The best solution here is for you to remove that section completely but you absolutely should not have to. We are most likely going to remove it ourselves. This is being discussed as part of the project to overhaul and provide better value to the PR comment overall.

Some of the links in the GH comment are not accessible unless the viewer is logged in to Codecov

I have a bug report for this. I do not know why we require a login for some pages and not others. I will say that we are redoing the frontend and since it’s a deep underlying change it was perhaps tabled in the hopes it would clear up, I will resurface this bug report to product.

The uploader fails from GitHub Actions if the Codecov app integration is not installed in the user’s GitHub account

This should NOT be the case. GitHub supports tokenless uploads for public repos and Codecov via an internal API integration we created. In order for this to happen, we need to have the slug and the build number sent to is correctly so we can ask the GitHub API about it. I will dig into this issue with Tom and review your linked topic to see what is happening here.

Codecov only supports XML reports from gcov.

Text reports migh be able to be parsed, HTML reports can not. In order to read your reports they need to be in a machine friendly format. Most test reporter allow multiple formats for the reports, I myself use both, because yes, HTML is best for humans.

It looks like you were very close with your command of gcovr -g -k -r . --xml regressioncov.xml # --html Is there a reason you commented out the HTML part? Was this not working correctly?

The lack of visibility into issues on the CodeCov dashboard is crippling to me.

If you believe me, we are slowly making progress into better surfacing errors such as “can’t read report type”. This is actively in progress and is improving with the new UI. It still can be much more helpful.


I think I covered all the issues you raised. If you have any questions, or if I can assist further , please let us know. I hope that you will give us another try in the future.

– Joe

1 Like