Codecov reporting impacted files for unchanged and completely unrelated file

Support Bug Fixes
1

Description

CodeCov is reporting an unchanged file as an impacted file in PR comment
See fix Docker Hub login step by akphi · Pull Request #96 · finos/legend-studio · GitHub

Commit SHAs

e9216f7c16c8b2309103291efc7cee786149fe4d

Repository

CI/CD or Build URL

Under “Upload test coverage report”

Uploader

GitHub Actions

Codecov Output

Run codecov/codecov-action@v1.2.1
/bin/bash codecov.sh -n  -F  -Q github-action -s ./build/coverage

  _____          _
 / ____|        | |
| |     ___   __| | ___  ___ _____   __
| |    / _ \ / _` |/ _ \/ __/ _ \ \ / /
| |___| (_) | (_| |  __/ (_| (_) \ V /
 \_____\___/ \__,_|\___|\___\___/ \_/
                              Bash-20210309-2b87ace


==> git version 2.30.1 found
==> curl 7.58.0 (x86_64-pc-linux-gnu) libcurl/7.58.0 OpenSSL/1.1.1j zlib/1.2.11 libidn2/2.0.4 libpsl/0.19.1 (+libidn2/2.0.4) nghttp2/1.30.0 librtmp/2.3
Release-Date: 2018-01-24
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp smb smbs smtp smtps telnet tftp 
Features: AsynchDNS IDN IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz TLS-SRP HTTP2 UnixSockets HTTPS-proxy PSL 
==> GitHub Actions detected.
    Env vars used:
      -> GITHUB_ACTIONS:    true
      -> GITHUB_HEAD_REF:   docker-pls
      -> GITHUB_REF:        refs/pull/96/merge
      -> GITHUB_REPOSITORY: finos/legend-studio
      -> GITHUB_RUN_ID:     636943168
      -> GITHUB_SHA:        909cd1c87ff0209df2a6e53cf3272db3648ed7e9
      -> GITHUB_WORKFLOW:   Node CI
    Fixing merge commit SHA 909cd1c87ff0209df2a6e53cf3272db3648ed7e9 -> e9216f7c16c8b2309103291efc7cee786149fe4d
    project root: .
    Yaml found at: .github/codecov.yml
==> Running gcov in . (disable via -X gcov)
==> Python coveragepy not found
==> Searching for coverage reports in:
    + ./build/coverage
    -> Found 3 reports
==> Detecting git/mercurial file structure
==> Reading reports
    + ./build/coverage/clover.xml bytes=1445645
    + ./build/coverage/coverage-final.json bytes=4628383
    + ./build/coverage/lcov.info bytes=769688
==> Appending adjustments
    https://docs.codecov.io/docs/fixing-reports
    -> No adjustments found
==> Gzipping contents
        792K	/tmp/codecov.5eQPbv.gz
==> Uploading reports
    url: https://codecov.io
    query: branch=docker-pls&commit=e9216f7c16c8b2309103291efc7cee786149fe4d&build=636943168&build_url=http%3A%2F%2Fgithub.com%2Ffinos%2Flegend-studio%2Factions%2Fruns%2F636943168&name=&tag=&slug=finos%2Flegend-studio&service=github-actions&flags=&pr=96&job=Node%20CI&cmd_args=n,F,Q,s
->  Pinging Codecov
https://codecov.io/upload/v4?package=github-action-20210309-2b87ace&token=secret&branch=docker-pls&commit=e9216f7c16c8b2309103291efc7cee786149fe4d&build=636943168&build_url=http%3A%2F%2Fgithub.com%2Ffinos%2Flegend-studio%2Factions%2Fruns%2F636943168&name=&tag=&slug=finos%2Flegend-studio&service=github-actions&flags=&pr=96&job=Node%20CI&cmd_args=n,F,Q,s
->  Uploading to
https://storage.googleapis.com/codecov/v4/raw/2021-03-09/DAB0CDE5E55B3EC3317486B5D802380D/e9216f7c16c8b2309103291efc7cee786149fe4d/82a71e6b-6c05-42eb-b520-fe9a1164f1c5.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=GOOG1EQX6OZVJGHKK3633AAFGLBUCOOATRACRQRQF6HMSMLYUP6EAD6XSWAAY%2F20210309%2FUS%2Fs3%2Faws4_request&X-Amz-Date=20210309T194024Z&X-Amz-Expires=10&X-Amz-SignedHeaders=host&X-Amz-Signature=41b488ece3c39675c507936107012e5d0ec2f024c1eed2600ced70ec6c927fc3
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  789k    0     0  100  789k      0  2935k --:--:-- --:--:-- --:--:-- 2935k
    -> Reports have been successfully queued for processing at https://codecov.io/github/finos/legend-studio/commit/e9216f7c16c8b2309103291efc7cee786149fe4d

Expected Results

I expect to not see any change in coverage

Actual Results

I see this unrelated file listed as an impacted file

Impacted Files Coverage Δ
…s/legend-studio/src/stores/ChangeDetectionState.ts 67.74% <0.00%> (-1.08%) :arrow_down:
2

@akphi, I took a look at the report on Codecov and it’s showing no changes. My guess is that there might have been a timing issue, and we did not send an updated notification in time.

Are you still seeing this issue on other open PRs?

3

So perhaps I could have done some test PR to show you. I have these 3 PRs with coverage report:

And you can sorta see the change in the file ChangeDetectionState.ts jumping from negative to positive. I suspect it’s a timing issue which I would try to investigate a bit more. I’ll let you know what I find, though I just want to confirm the problem does not come from some sort of caching problem.

4

@akphi, this is strange. Unfortunately, I don’t know the codebase well enough, but it’s always happening on these lines. Is there any suspicion you might have that your tests are not necessarily idempotent in calling those lines?

5

That’s fair. We can close this issue if you want to. Do we have a re-open feature here? I can re-open this if I have more solid evidence to believe the problem comes from codecov. What do you think @tom?

1 Like
6

@akphi, sounds good to me. I’ll mark the previous response as a solution. If you find that there is a problem with Codecov, feel free to just reply back on this thread.

7

@tom So looks like it’s a timing issue on our part, we use mobx reaction with throttling, so I have reduced throttling duration to 0 for test. I will continue monitoring this, but this is great. Thank you so much and sorry for the false alarm!

1 Like