Reported change in coverage when no code changes made

Support Bug Fixes
,
1

Description

While debugging an unexplained coverage change in another PR (#803 of the same repo, if curious) I created Create codecov.yml slightly relaxing patch coverage rule by obi1kenobi · Pull Request #806 · kensho-technologies/graphql-compiler · GitHub which adds a Codecov config file but makes no code changes at all. I branched off of the master branch and a few seconds later opened the PR, triggering Travis and Codecov runs.

I changed no code files, and nothing was merged into master between when I branched off of it and when I opened the PR, so I expected no coverage changes. However, Codecov still reports a change in coverage, and the default codecov/project check fails as a result, with default settings.

I found this behavior unexpected and surprising, and couldn’t explain it based on anything I found in the docs or in this forum — apologies if I missed something well-known and obvious.

Repository

Example affected PR: Create codecov.yml slightly relaxing patch coverage rule by obi1kenobi · Pull Request #806 · kensho-technologies/graphql-compiler · GitHub

I think my coauthors and I on that project have observed this behavior in the past as well, but this is the most clear-cut example we’ve had.

Steps to Reproduce

Not sure about what the prior conditions necessary to reproduce this issue are, but for the repo above they are:

  1. open a new PR via the web UI, making an unrelated, non-code change (e.g. add a blank line to a text file)
  2. wait for tests and coverage to finish
  3. observe change in coverage reported by Codecov, despite no code changes

Expected behavior: No coverage changes when no code was modified.

Actual behavior: In this particular example, new “missed lines” reported as a change in coverage.

Flakiness? Definitely has happened in the past, but difficult to pin down between “sometimes” / “often” / “always”.

Versions

We get this issue while running tests on Travis CI and submitting coverage via its codecov script.

2

I also have encountered same kind of problem in my PR for other repo here.
I didn’t change the code but the coverage here is changed from master to my PR.

3

Hi @obi1kenobi, I dug into that pull request, and it seems like it has to do with the coverage reports you are uploading.

´The change in the PR points to this line being different Codecov

image
image1540×966 137 KB

For the base commit Code coverage done right. (the commit that we are comparing to), we see that for build 4273.3 on python 3.7, that this line is covered
Build report: https://codecov.io/codecov/v4/raw/2020-04-28/8781BA0FDA3C43564FE94343C04916E8/0cf541487d8d22776a2c525a5bfa34f113695e9f/c3601297-b62a-4a27-89a6-20f4b0a8f898.txt

image
image2366×1286 286 KB

However for the commit in this PR, this line is not covered (https://codecov.io/gh/kensho-technologies/graphql-compiler/commit/1ebd37f9f1aeebe97816a0055ddfe8a160df311f/build).

This is why you are seeing the coverage drop. Let me know if that answers your question.

4

Hi @tom, thanks for looking into this.

Here is the diff between the PR’s commit and the base commit it is comparing to:

$ git checkout codecov_config 
Branch 'codecov_config' set up to track remote branch 'codecov_config' from 'origin'.
Switched to a new branch 'codecov_config'

$ git diff 0cf5414
diff --git a/codecov.yml b/codecov.yml
new file mode 100644
index 0000000..dfc0998
--- /dev/null
+++ b/codecov.yml
@@ -0,0 +1,7 @@
+coverage:
+  status:
+    patch:
+      default:
+        target: auto
+        threshold: 0.03%
+        base: auto

If I understood your message correctly, it would mean that the best explanation given the available evidence (change in coverage but no change in code) is that the test suite is in some way non-deterministic. Would you agree?

If my interpretation above is correct, I am happy to take a look at this with my team — while our tests are do not have any unintentional non-determinism, it’s always possible some has snuck in anyway.

Thanks again for your time, and thank you for working on this awesome tool :slight_smile:

5

We were able to confirm nondeterminism in our tests. We are still investigating where that nondeterminism comes from. However, we were able to confirm that codecov is not the source of the problem.

Thank you for your help!

1 Like
6

Amazing! Glad we were able to help!

7

Hey @obi1kenobi , Jerrod from Codecov here, did you all ever find the root cause of the non-determinism in the test results? I’d be really curious to hear.

Thanks!

8

I understand that checking for non-determinism may still be a good idea but perhaps Codecov could filter-out unchanged files from the report and print a warning instead saying the test suite looks non-deterministic? It’d make it easier for reviewers to see that the contributor is not lazy. :slight_smile:

You can look at this PR if you want an example of such code change: A few more standard conversions for DeserBorrowStr by Kixunil · Pull Request #43 · rust-amplify/rust-amplify · GitHub

9

Sorry for reviving an old discussion but it appears this issue is still present. Since the codecov/project and codecov/patch actions suddenly appeared today we have had two failures with the same -0.03% change in coverage with no file changes being shown in the report.

See

10

@firewave do you mind opening a new topic for this?