Help debugging 'There was an error processing coverage reports'

Support
#1

Please check the troubleshooting guide before creating a ticket.

Description

I’m using bazel 3.5.0 to generate code coverage.dat files for my C++ build. The coverage.dat files seem to be in lcov format.

I’m using the bash uploader to upload these files, and it appears to work. However, when I try to view the results in the codecov.io UI, I see a generic message like “There was an error processing coverage reports.”, but it doesn’t tell me what the error was, nor provide any way to debug.

Repository

CI/CD

An internal Google thing called “kokoro”.

Uploader

/bin/bash <(curl -s https://codecov.io/bash) -s /h

Commit SHAs

82daaace24c8bf3fa890b2c222a4c4f360555beb is an example

Codecov YAML

# Disable commit statuses
coverage:
  range: 90..100
  status:
    project: off
    patch: off

ignore:
  # Ignore code generated by protobuf, we do not need to test that.
  - "*.pb.cc"
  - "*.pb.h"
  # Also ignore the unit tests.
  - "**/*_test.cc$"

Codecov Output

There’s too much output, so here’s a snippet of the beginning and end:

  _____          _
 / ____|        | |
| |     ___   __| | ___  ___ _____   __
| |    / _ \ / _` |/ _ \/ __/ _ \ \ / /
| |___| (_) | (_| |  __/ (_| (_) \ V /
 \_____\___/ \__,_|\___|\___\___/ \_/
                              Bash-20200910-5a68dc2


x> No CI provider detected.
    Testing inside Docker? http://docs.codecov.io/docs/testing-with-docker
    Testing with Tox? https://docs.codecov.io/docs/python#section-testing-with-tox
    project root: .
    Fixing merge commit SHA
--> token set from env
    Yaml found at: .codecov.yml
==> Running gcov in . (disable via -X gcov)
==> Python coveragepy not found
==> Searching for coverage reports in:
    + /h
    -> Found 869 reports
==> Detecting git/mercurial file structure
==> Reading reports
    + /h/.cache/bazel/_bazel_kbuilder/a7ecd8237744645c5d189c197108d6d2/execroot/com_github_googleapis_google_cloud_cpp/bazel-out/k8-fastbuild/testlogs/pubsub_client/baseline_coverage.dat bytes=9821
    + /h/.cache/bazel/_bazel_kbuilder/a7ecd8237744645c5d189c197108d6d2/execroot/com_github_googleapis_google_cloud_cpp/bazel-out/k8-fastbuild/testlogs/spanner_client/baseline_coverage.dat bytes=10846
    + /h/.cache/bazel/_bazel_kbuilder/a7ecd8237744645c5d189c197108d6d2/execroot/com_github_googleapis_google_cloud_cpp/bazel-out/k8-fastbuild/testlogs/storage_client/baseline_coverage.dat bytes=13637
    + /h/.cache/bazel/_bazel_kbuilder/a7ecd8237744645c5d189c197108d6d2/execroot/com_github_googleapis_google_cloud_cpp/bazel-out/k8-fastbuild/testlogs/bigtable_client/baseline_coverage.dat bytes=9809
    + /h/.cache/bazel/_bazel_kbuilder/a7ecd8237744645c5d189c197108d6d2/execroot/com_github_googleapis_google_cloud_cpp/bazel-out/k8-fastbuild/testlogs/google/cloud/future_void_test/coverage.dat bytes=71643
    + /h/.cache/bazel/_bazel_kbuilder/a7ecd8237744645c5d189c197108d6d2/execroot/com_github_googleapis_google_cloud_cpp/bazel-out/k8-fastbuild/testlogs/google/cloud/future_void_test/baseline_coverage.dat bytes=4416
    + /h/.cache/bazel/_bazel_kbuilder/a7ecd8237744645c5d189c197108d6d2/execroot/com_github_googleapis_google_cloud_cpp/bazel-out/k8-fastbuild/testlogs/google/cloud/future_generic_test/coverage.dat bytes=71261
    + /h/.cache/bazel/_bazel_kbuilder/a7ecd8237744645c5d189c197108d6d2/execroot/com_github_googleapis_google_cloud_cpp/bazel-out/k8-fastbuild/testlogs/google/cloud/future_generic_test/baseline_coverage.dat bytes=4416
    + /h/.cache/bazel/_bazel_kbuilder/a7ecd8237744645c5d189c197108d6d2/execroot/com_github_googleapis_google_cloud_cpp/bazel-out/k8-fastbuild/testlogs/google/cloud/internal_async_retry_unary_rpc_test/coverage.dat bytes=251749
...
[snip]
...
    + /h/.cache/bazel/_bazel_kbuilder/a7ecd8237744645c5d189c197108d6d2/execroot/com_github_googleapis_google_cloud_cpp/bazel-out/k8-fastbuild/testlogs/generator/google_cloud_cpp_generator/baseline_coverage.dat bytes=5283
    + /h/.cache/bazel/_bazel_kbuilder/a7ecd8237744645c5d189c197108d6d2/external/io_bazel_rules_go/tests/core/nogo/coverage/coverage_target_test.go bytes=29
    + /h/.cache/bazel/_bazel_kbuilder/a7ecd8237744645c5d189c197108d6d2/external/io_bazel_rules_go/tests/core/coverage/coverage_test.go bytes=70
    + /h/.cache/bazel/_bazel_kbuilder/a7ecd8237744645c5d189c197108d6d2/external/@remote_coverage_tools.marker bytes=106
==> Appending adjustments
    https://docs.codecov.io/docs/fixing-reports
    + Found adjustments
==> Gzipping contents
==> Uploading reports
    url: https://codecov.io
    query: branch=pull_branch&commit=82daaace24c8bf3fa890b2c222a4c4f360555beb&build=2115&build_url=&name=&tag=&slug=googleapis%2Fgoogle-cloud-cpp&service=&flags=&pr=5070&job=2115&cmd_args=s
->  Pinging Codecov
https://codecov.io/upload/v4?package=bash-20200910-5a68dc2&token=secret&branch=pull_branch&commit=82daaace24c8bf3fa890b2c222a4c4f360555beb&build=2115&build_url=&name=&tag=&slug=googleapis%2Fgoogle-cloud-cpp&service=&flags=&pr=5070&job=2115&cmd_args=s
->  Uploading to
https://storage.googleapis.com/codecov/v4/raw/2020-09-17/A6A6358C08BED374DB3FEE6AD1B038DF/82daaace24c8bf3fa890b2c222a4c4f360555beb/8a2295c6-46cd-4e26-b941-87fff0c60b09.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=GOOG1EQX6OZVJGHKK3633AAFGLBUCOOATRACRQRQF6HMSMLYUP6EAD6XSWAAY%2F20200917%2FUS%2Fs3%2Faws4_request&X-Amz-Date=20200917T015228Z&X-Amz-Expires=10&X-Amz-SignedHeaders=host&X-Amz-Signature=7d01afcc92d892b7217e4c793b3226916fd1a63ad7420b27897de8d2ce2e260b
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 39.2M    0     0  100 39.2M      0  69.3M --:--:-- --:--:-- --:--:-- 69.2M
100 39.2M    0     0  100 39.2M      0  69.3M --:--:-- --:--:-- --:--:-- 69.2M
    -> View reports at https://codecov.io/github/googleapis/google-cloud-cpp/commit/82daaace24c8bf3fa890b2c222a4c4f360555beb

The full output is available at the bottom of this log file: https://source.cloud.google.com/results/invocations/283e0ca6-41c2-41cb-aa67-4f5d4941ff84/targets/cloud-cpp%2Fgithub%2Fgoogle-cloud-cpp%2Fmaster%2Fdocker%2Fcoverage-presubmit/log

Additional Information

This appears to work sometimes. As you can from the commits in this PR https://codecov.io/gh/googleapis/google-cloud-cpp/pull/5070, some of them do process the code coverage info. I’ve no idea why it fails sometimes, and I have no idea how to debug this or where to look for errors.

#2

Note that the coverage report that gets uploaded is about 40MB compressed, and about 900MB uncompressed. Is that exceeding some size limit that’s causing this error?

#3

@devjgm I am 99% sure that is the case (that the upload is massive). Do you have any way of breaking this up into separate reports?

#4

I solved this issue by merging all the coverage.dat files into a single merged lcov file. The filesize dropped from 900MB to ~16 MB, and everything seems to be working now.

So, even though I think codecov.io officially has no file size limit, it appears that there actually are some limits you’ll bump into with very large files. My wild guess is that their backend processing pipeline (or whatever) was hitting some timeout while trying to munge a 900MB coverage file.

1 Like