Codecov cannot analyze Cobertura reports from AppVeyor

Description

Codecov stopped analyzing reports successfully uploaded from AppVeyor (according to build log) in Cobertura format.

Around ~4 months ago analysis worked correctly and builds uploaded from AppVeyor were visible under “Build” tab of particular commit as well as Lcov reports uploaded from Travis CI. Lines covered by tests executed during AppVeyor build only (i.e. lines which are covered by Cobertura reports uploaded from AppVeyor only) were shown as covered and were included into total coverage.

Now Codecov shows “Reports are queued for processing…” message infinitely (for more than a week) and builds successfully (!) uploaded from AppVeyor are not shown on “Build” tab of particular commit (while Travis CI builds are shown). Coverage reported by missing reports only is not taken into account - lines are shown as not covered and total coverage has reduced.

Repository

GitHub repository: https://github.com/mabrarov/asio_samples

Links to Travis CI and AppVeyor can be found in README (I cannot put them here as new user)

Versions

Language: C++
Format: Cobertura
Tool: OpenCppCoverage
Platform / OS: Windows Server x64 on AppVeyor

Last successful Codecov analysis: https://codecov.io/gh/mabrarov/asio_samples/commit/e8997afe0ed393b9c10300fd1ef2069e66e6a011/build

Last failed Codecov analysis: https://codecov.io/gh/mabrarov/asio_samples/commit/c6cfb877e428026247acf4d4662191f6c61dc8b8/build and https://ci.appveyor.com/project/mabrarov/asio-samples/builds/30054615/job/twr0joscmwtgg75i

Additional Information

The same issue happens with yatest repository (same mabrarov GitHub user) for which I even recreated Codecov project but this didn’t help.

Hi @mabrarov

I fixed your ability to post additional links and images.

Can you share a recent commit SHA for an Appveyor build that does not show? I’d like to check our logs and see what happened to it.

Hi @drazisil,

Here are the links for the latest (just completed) builds and analysis.

Commit: https://github.com/mabrarov/asio_samples/commit/3852d780ba46d7d50affd63615c4e109d0b457d9

AppVeyor build: https://ci.appveyor.com/project/mabrarov/asio-samples/builds/30321782 (consists of multiple jobs, some of which send report to Codecov, like https://ci.appveyor.com/project/mabrarov/asio-samples/builds/30321782/job/h8c6lcmwl1r6mu01)

Codecov analysis of commit: https://codecov.io/gh/mabrarov/asio_samples/commit/3852d780ba46d7d50affd63615c4e109d0b457d9

It looks like we failed to process the Appveyor upload. It’s not clear from the logs if the upload timed out, or something else, still investigating.

Couple of questions:

  • Do you know how large the Appveyor upload is?
  • Do you know roughly when it stopped working?

Size of Cobertura report generated by AppVeyor is 200-250 KB. You can find it in “Console” view of particular AppVeyor build job (at the end of log) - like https://ci.appveyor.com/project/mabrarov/asio-samples/builds/30321782/job/h8c6lcmwl1r6mu01

==> Detecting CI provider
    AppVeyor Detected
==> Preparing upload
XX> Skip processing gcov
==> Collecting reports
XX> Searching for reports disabled
    Targeting specific files
    + C:/projects/asio-samples/build/coverage/coverage.xml bytes=213422
==> Uploading
    .url https://codecov.io
    .query service=appveyor&build=h8c6lcmwl1r6mu01&commit=3852d780ba46d7d50affd63615c4e109d0b457d9&token=<secret>&slug=mabrarov%2Fasio_samples&flags=windows_6_3_9600__x64__msvc_14_0__boost_1_72_0__qt_5_11_3&job=mabrarov%2Fasio-samples%2F1.0.1007&package=py2.0.15&yaml=codecov.yml&branch=feature%2Fcodecov
    Pinging Codecov...
    Uploading to S3...
    https://codecov.io/github/mabrarov/asio_samples/commit/3852d780ba46d7d50affd63615c4e109d0b457d9

Cobertura report (coverage.xml file) can be found among build artifacts (“Artifacts” view / tab of job) of particular AppVeyor build job, like at https://ci.appveyor.com/project/mabrarov/asio-samples/builds/30321782/job/h8c6lcmwl1r6mu01/artifacts

It looks like issues started ~2 months ago

Thanks for that, I’ll keep digging here.

I see you are using the python uploader, do you use the same uploader across CI providers? Would it be possible to try the bash uploader to rule that out? The python uploader isn’t as maintained, and I know there have been a couple Appvayor-related fixes to the bash uploader in that time that might help.

I’d prefer to use python uploader if it’s not the root cause (I’ll test bash uploader soon).

I use python uploader for both Travis CI (refer to https://travis-ci.org/mabrarov/asio_samples/jobs/641131620) and AppVeyor. According to logs:

      _____          _
     / ____|        | |
    | |     ___   __| | ___  ___ _____   __
    | |    / _ \ / _  |/ _ \/ __/ _ \ \ / /
    | |___| (_) | (_| |  __/ (_| (_) \ V /
     \_____\___/ \____|\___|\___\___/ \_/
                                    v2.0.15

version of python uploader used by both CI systems is the same

I tried Codecov bash uploader in another project facing the same issue - https://github.com/mabrarov/yatest.

Here is Travis CI build which uses python uploader and is visible in Codecov: https://travis-ci.org/mabrarov/yatest/jobs/641593992

Here is AppVeyor build which uses bash uploader, completed successfully but is not visible in Codecov: https://ci.appveyor.com/project/mabrarov/yatest/builds/30346304/job/hvx0pfw2fhthjgpq

Here is Codecov view for 19e71b10 commit: https://codecov.io/gh/mabrarov/yatest/commit/19e71b106937b1a90dce277650e9064ac924e47a

I think I am running into the same issue.

I have an AppVeyor job that creates a bunch of partial cobertura reports, merges them and then uploads them to codecov. According to the log, the upload succeeds, but I cannot see it on codecov.

Job log: https://ci.appveyor.com/project/CatchOrg/catch2/builds/30419141/job/nw6k4trce09cpt7o (sorry it is noisy, but I enabled all the prints to help me debug this… the relevant part is at the end)

Codecov for that commit: https://codecov.io/github/catchorg/Catch2/commit/f118a63a5ca11a137d771bd4efb0c0cf075a38b4

Hi @mabrarov and @horenmar

Besides you two, I have a couple attitudinal reports of this. I’m not sure what’s going on, but I’m investigating, wanted to let you know I wasn’t ignoring this.

@mabrarov, @horenmar, @yitam

It looks like the reports you are uploading have an empty timestamp, which is completly breaking our parsing library. Can you check if this is the case?

Here is an example of what I’m seeing

<coverage line-rate="0.77029569892473115" branch-rate="0" complexity="0" branches-covered="0" branches-valid="0" timestamp="0" lines-covered="5731" lines-valid="7440" version="0">

Hi @drazisil,

According to your answer timestamp is zero and not empty. You are right - the latest failed build (6b4dabd2 commit) has

<coverage line-rate="0.9755576649264357" branch-rate="0" complexity="0" branches-covered="0" branches-valid="0" timestamp="0" lines-covered="4111" lines-valid="4214" version="0">

but timestamp had the same value

<coverage line-rate="0.9755576649264357" branch-rate="0" complexity="0" branches-covered="0" branches-valid="0" timestamp="0" lines-covered="4111" lines-valid="4214" version="0">

previously - when Codecov worked fine. Refer to artifacts of https://ci.appveyor.com/project/mabrarov/asio-samples/builds/27250158/job/71id95s4ypdu59vb/artifacts build job (to coverage.xml artifact) for e8997afe commit and to related Codecov report - https://codecov.io/gh/mabrarov/asio_samples/commit/e8997afe0ed393b9c10300fd1ef2069e66e6a011/build - which contains reports uploaded from AppVeyor

@mabrarov

Well, drat! I though we had this solved. The code that’s throwing the error hasn’t been touched in years: https://github.com/codecov/timestring/blame/master/timestring/Date.py#L51

Will continue digging.

Can I ask one of you 3 to try adding this to your codecov.yml and see if it helps?

codecov:
  max_report_age: off

Hi @drazisil,

I added

max_report_age: off

in yatest project which has the same issue. Refer to bae0fd69ce5bdb976658ba9b0f3a245846083cd8 commit and to related AppVeyor build job which completed successfully (as well as related Travis CI build job).

Codecov (refer to https://codecov.io/gh/mabrarov/yatest/branch/feature%2Fcodecov/commits) looks being hanged for this commit - it shows “Processing…” on commits view of feature/codecov branch

and shows " Unable to find report content in the storage archive" when trying to look at the report for this commit:

It looks like I added new line into wrong section of codecov.yml… Let me fix and retest.

@drazisil,

It looks like your suggestion implemented in 01f1f87cccbf3efdc0b072e54b4ef39bbbfd334d commit fixed this issue for yatest project - refer to https://codecov.io/gh/mabrarov/yatest/commit/01f1f87cccbf3efdc0b072e54b4ef39bbbfd334d/build where both Linux (from Travis CI) and Windows (from AppVeyor) builds are visible.

I’m going to test the same solution for asio_samples project now.

Is it permanent solution I can keep or is it needed just for your investigation?

1 Like

Nope, you can totally keep it. I’ll mark as the solution for now, since tracking down what changed will take us a while.

The yaml change has also fixed the problem for me, with a small exception that for some reason codecov does not add its own “tickmark” to the commit the way it used to.

That shouldn’t have changed anything, it’s possible there’s a delay. Can you share a SHA?