Gaps in functionality

Feature Requests
1

Summary
I’ve put a lot of time into trying to configure Codecov to work for my projects, but unfortunately I have to pull the plug. Instead of silently removing the integrations, I’d like to let Codecov know the issues that led me to this decision in case it helps guide future decisions.

Description
I’m looking for a basic, stable method for reporting code coverage and inspecting coverage reports. The GitHub integrations that Codecov offer are great but nice-to-haves for me. At the end of the day, I need a product that performs a base level of functionality correctly 99.9% of the time. Currently, Codecov provides some good functionality, but it generally adds to the noise around our development cycle and funnels more questions in my direction from users who can’t access Codecov resources when they expect.
I think I’m using Codecov in a typical manner, but I also understand I may be doing something abnormal that’s causing me to run into these issues.

These are the problems that I can’t get past:

Examples
All of the failing jobs here are due to the user not having the GitHub Codecov app installed.
This diff comment is way off - the files listed are not changed at all in the pull request.
The links in this comment aren’t accessible if the user isn’t logged in to Codecov. Try opening them in a private browser window.

Versions

All of these cases use GitHub Actions and the Codecov uploader (codecov/codecov-action@v2). There’s a combination of C++ and Fortran, and all use gcov and gcovr to generate the coverage reports.

@tom DESERVES A RAISE
Finally, I want to point out that @tom makes life much easier with the support provided in the forum. I also attended one of the webinars and it was very helpful to see in real time how to use the Codecov dashboard and hear about some of the things on the development roadmap. So cheers to you @tom. Thanks for the help.

2

Hey @rafmudaf – Jerrod here from Codecov, just wanted to say deep thanks for such a detailed explanation of your product friction points. It is super helpful for us, this is in front of the entire team internally.

We hope to make multiple aspects of this user experience better based on your feedback.

Also, we agree Tom is amazing :slight_smile:

Jerrod

1 Like
3

@rafmudaf ,

First, I want to say, Thank you for this very detailed feedback. It has been shared with our product team.

As the lead support engineer (not Tom’s boss :smiley: ) I would like to walk though the issues you raise here one by one for my own understanding and thoughts on how we can improve these process in the future .

Please note that I am not expecting you to try any of them, so please don’t think this is an attempt to get you to try us again, even though I would love if you did. I completely understand your frustration.

  • The comments included in the pull requests sometimes list incorrect files as “diffs”
    ** I think this happens when there’s a force-push and the parent-child commit relationship changes
    ** Regenerate or update GitHub pull request comment

This is a frequent source of confusion to users and we are looking into ways to make this clearer. (I will probably say this a lot in this response, though I hope less then I expect.

Assuming we are talking about the Coverage Diff section, this is generated directly from the raw coverage reports that are uploaded. It is not connected to the GitHub diff, even though the same terms are used. There are a number of reasons why the two coverage reports may differ, and we have a troubleshooting guide in our docs (that we need to surface better. Some of the unexpected reasons are tests not running, or parts of the overage not uploading if there are multiple reports.

Regarding the feature request to restart / regenerate the comment: We have a feature request in the backlog for this. Given the way we require CI / Repo information to be available at processing time, tjis is not as straightforward as I would expect it to be. In cases where the commit gets “deleted” due to squash, rebase, etc, it becomes impossible.

The code coverage treemap is initially completely incorrect in the GitHub comment.

Yes. 100% agree. The best solution here is for you to remove that section completely but you absolutely should not have to. We are most likely going to remove it ourselves. This is being discussed as part of the project to overhaul and provide better value to the PR comment overall.

Some of the links in the GH comment are not accessible unless the viewer is logged in to Codecov

I have a bug report for this. I do not know why we require a login for some pages and not others. I will say that we are redoing the frontend and since it’s a deep underlying change it was perhaps tabled in the hopes it would clear up, I will resurface this bug report to product.

The uploader fails from GitHub Actions if the Codecov app integration is not installed in the user’s GitHub account

This should NOT be the case. GitHub supports tokenless uploads for public repos and Codecov via an internal API integration we created. In order for this to happen, we need to have the slug and the build number sent to is correctly so we can ask the GitHub API about it. I will dig into this issue with Tom and review your linked topic to see what is happening here.

Codecov only supports XML reports from gcov.

Text reports migh be able to be parsed, HTML reports can not. In order to read your reports they need to be in a machine friendly format. Most test reporter allow multiple formats for the reports, I myself use both, because yes, HTML is best for humans.

It looks like you were very close with your command of gcovr -g -k -r . --xml regressioncov.xml # --html Is there a reason you commented out the HTML part? Was this not working correctly?

The lack of visibility into issues on the CodeCov dashboard is crippling to me.

If you believe me, we are slowly making progress into better surfacing errors such as “can’t read report type”. This is actively in progress and is improving with the new UI. It still can be much more helpful.

I think I covered all the issues you raised. If you have any questions, or if I can assist further , please let us know. I hope that you will give us another try in the future.

– Joe

1 Like
4

Thanks for the responses @drazisil and @jerrod. I hope this post wasn’t critical, but instead helpful in identifying pain points for some users. I’m also a software engineer on projects that are used by a diverse community, so I completely understand that there are infinite challenges and finite resources. Codecov is doing a great job in a lot of areas, and I’ll keep an eye out for future progress.

@drazisil thanks for the explanations and suggestions. You brought up some good points about how I’m using Codecov, and it’s helpful to clarify my misunderstanding and hear whats on your radar for future development. I appreciate you taking the time to write such a detailed response.

2 Likes
5

@rafmudaf just wanted to jump here and thank you for the kind words. I am really proud to be here and help our users, and hearing your praise is deeply touching.

Of course, I’m disappointed that Codecov wasn’t able to fit your needs, and more importantly, that you ran into so many issues using core functionality. This is clearly not how we want your experience to be. Thank you for writing up such a detailed report. We will strive to do better.

6

thanks for the awesome information.

1 Like