Sudden significant coverage drop

Support Bug Fixes
,
1

Description

Recently the coverage has significantly dropped on my repo (cpp code), even though the last changes we’ve merged was related to docs (which are excluded from the coverage scan :wink: ). You can see the drop in here: https://codecov.io/gh/pmem/libpmemobj-cpp/branch/master

We didn’t change anything in our testing methods, in our coverage approach (same CMake flag set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -coverage"), same bash uploader commands, same amount of tests executed, etc.). The drop appeared few days after our release, we prepared some docs’ changes as Pull Requests and they suddenly showed us ~23% coverage drop on the PR, we’ve merged it and it shows the same on the master branch.

Repository

Link to repo (and more precisely to the PR which caused the drop): add document describing release steps by lukaszstolarczuk · Pull Request #775 · pmem/libpmemobj-cpp · GitHub

CI/CD

Coverage is uploaded from Travis CI (we also use GitHub Actions, simultaneously).

Uploader

We’re using bash uploader, I’ve checked different versions of that script - e.g. tested one dated before the drop - I thought this might be the reason for sudden drop, but nope.

Codecov YAML

coverage:
  status:
    project:
      default:
        threshold: 0.2

  ignore:
    - benchmarks/
    - doc/
    - examples/
    - tests/
    - utils/

Additional Information

I’ve tried to check verbose output from bash-uploader and is still the same as it was before.

The same happens on our other repos (in pmem organization) in c only and c + cpp libraries.

Any hints, what could go wrong? Did you guys change anything e.g. in default settings for c/cpp analyses that may affect us?

2

Hi @lukaszstolarczuk, sorry that you’re getting this massive coverage drop. From what I’m seeing, it looks like an issue with the coverage report uploaded under tests_gcc_debug. The base commit shows this file has coverage under that flag, but the head commit shows that coverage is lacking.

Looking at the raw reports uploaded, I’m seeing

image
image1418×372 43.4 KB

image
image1380×444 44.4 KB

So I’m not totally sure what might be happening here, but there seems to have been a change in the way coverage is being generated before they are uploaded.

3

A big thanks for your quick response and analysis. As I said, I believe we didn’t change anything on our side that I believe may affect the coverage report, but I’ll double-check everything, once more.

On the other hand, it happens also (also at the beginning of this month) on other repos in our organization:

  • pmdk (C code):
    https://codecov.io/gh/pmem/pmdk/branch/master - they are merging some code changes (not only docs) and they also had some significant (even worst then ours) coverage drop; and somehow they (without any specific fixes) building their coverage up… which seems weird… :wink:

  • pmemkv (C + Cpp code):
    Codecov
    we can see here just ~0,66% drop, but also on docs’ changes…

Nonetheless, I’ll double-check it and make sure some dependencies of ours didn’t got upgraded and influenced us, perhaps, or something…

1 Like
4

Hi @lukaszstolarczuk, I just wanted to circle back on this and see if you are able to resolve.

5

Thanks for taking an interest in our case. For now I didn’t manage to find the root cause, but I didn’t have much time (I had to switch my focus to another project for a while). I’ll certainly get back to this topic and update you.

1 Like
6

@lukaszstolarczuk sounds good. I’ll mark that as the solution, but feel free to message on this thread when you get a chance to switch back, and I’ll be happy to work with you on it again.

7

@tom, I’ve finally managed to find the reason of the drop.

short answer :wink:
I blame this change:

I fixed it like this:

longer answer:
I realized our *.gcov files were just overridden, and the coverage was just reported for the last test that touched selected source file. Before “the drop” your bash script in our CI Found 1126 reports and in June there were merely 189 reports. Because of the find param -execdir changed to -exec paths like:
/pmemkv/build/CMakeFiles/pmemkv.dir/src/engines/#pmemkv#src#engines#cmap.h.gcov e[0;90mbytes=4363e[0m
were changed and was just
/pmemkv/build/#pmemkv#src#engines#cmap.h.gcov e[0;90mbytes=4266e[0m
compare builds:
Travis CI - Test and Deploy Your Code with Confidence vs Travis CI - Test and Deploy Your Code with Confidence

I tried to fix it using -a param for gcov_args, but I couldn’t find a way, hence my patch in our CI.

Also, I had to manually disable the output of “reading reports”, it’s spamming our logs, a lot :wink: We have over 22k reports to read and upload, our CI log didn’t survive that.
Did you perhaps plan an option to disable it…?

Anyway, I hope you’ll fix that part of gcov’s reports’ finding, so we may come back to use the upstream codecov script soon.

8

Hi @lukaszstolarczuk, thanks for the detailed write-up here! I’ll take some time to parse through it, but I was wondering what would happen if you included -X gcov in the bash script, since you already run coverage reports.

9

I’m not sure if I get what you mean. I have to run gcov command, since compiler + tests execution provides me gcda and gcno data files. As I believe it’s required to generate gcov files for the report to upload and your script does the job.

Side note, I’ve pasted in my prev. post logs from “pmemkv” repository and fix from “libpmemobj-cpp”, but they have pretty much the same way of execution and compilation. pmemkv’s logs are just shorter and easier to read.

1 Like