I’ve put a lot of time into trying to configure Codecov to work for my projects, but unfortunately I have to pull the plug. Instead of silently removing the integrations, I’d like to let Codecov know the issues that led me to this decision in case it helps guide future decisions.
I’m looking for a basic, stable method for reporting code coverage and inspecting coverage reports. The GitHub integrations that Codecov offer are great but nice-to-haves for me. At the end of the day, I need a product that performs a base level of functionality correctly 99.9% of the time. Currently, Codecov provides some good functionality, but it generally adds to the noise around our development cycle and funnels more questions in my direction from users who can’t access Codecov resources when they expect.
I think I’m using Codecov in a typical manner, but I also understand I may be doing something abnormal that’s causing me to run into these issues.
These are the problems that I can’t get past:
- The comments included in the pull requests sometimes list incorrect files as “diffs”
- I think this happens when there’s a force-push and the parent-child commit relationship changes
- Regenerate or update GitHub pull request comment
- The code coverage treemap is initially completely incorrect in the GitHub comment. It does eventually update presumably after the report is processed on Codecov, but it is misleading. For example, I made a pull request that only changes the README of my project, but the diff image from Codecov initially looks like this:
It would be much more clear if no image, a blank graph, or a spinning wheel was loaded initially instead of this.
- Some of the links in the GH comment are not accessible unless the viewer is logged in to Codecov
- Logging in requires creating an account. Though this is very simple, it’s a barrier that I do not want to impose on every developer of my project
- GitHub API: Forbidden, when viewing coverage report for PR - #5 by rafmudaf
- The uploader fails from GitHub Actions if the Codecov app integration is not installed in the user’s GitHub account
- Codecov only supports XML reports from gcov. This means that our coverage reports are really only useful with Codecov.io or another report visualization tool. Instead, it would be helpful to be able to upload html or txt reports from gcov since those are also easily human-readable.
- Also, this cost me a ton of time while setting up the system and it really should be documented better here: Supported Coverage Report Formats
- Coverage report processing error - help! - #6 by tom
- The lack of visibility into issues on the CodeCov dashboard is crippling to me. The development cycle is very slow. Some sort of automated feedback about issues is a must.
All of the failing jobs here are due to the user not having the GitHub Codecov app installed.
This diff comment is way off - the files listed are not changed at all in the pull request.
The links in this comment aren’t accessible if the user isn’t logged in to Codecov. Try opening them in a private browser window.
All of these cases use GitHub Actions and the Codecov uploader (codecov/codecov-action@v2). There’s a combination of C++ and Fortran, and all use gcov and gcovr to generate the coverage reports.
@tom DESERVES A RAISE
Finally, I want to point out that @tom makes life much easier with the support provided in the forum. I also attended one of the webinars and it was very helpful to see in real time how to use the Codecov dashboard and hear about some of the things on the development roadmap. So cheers to you @tom. Thanks for the help.