Codecov reporting impacted files for unchanged and completely unrelated file

Description

CodeCov is reporting an unchanged file as an impacted file in PR comment
See fix Docker Hub login step by akphi · Pull Request #96 · finos/legend-studio · GitHub

Commit SHAs

e9216f7c16c8b2309103291efc7cee786149fe4d

Repository

CI/CD or Build URL

Under “Upload test coverage report”

Uploader

GitHub Actions

Codecov Output

Run codecov/codecov-action@v1.2.1
/bin/bash codecov.sh -n  -F  -Q github-action -s ./build/coverage

  _____          _
 / ____|        | |
| |     ___   __| | ___  ___ _____   __
| |    / _ \ / _` |/ _ \/ __/ _ \ \ / /
| |___| (_) | (_| |  __/ (_| (_) \ V /
 \_____\___/ \__,_|\___|\___\___/ \_/
                              Bash-20210309-2b87ace


==> git version 2.30.1 found
==> curl 7.58.0 (x86_64-pc-linux-gnu) libcurl/7.58.0 OpenSSL/1.1.1j zlib/1.2.11 libidn2/2.0.4 libpsl/0.19.1 (+libidn2/2.0.4) nghttp2/1.30.0 librtmp/2.3
Release-Date: 2018-01-24
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp smb smbs smtp smtps telnet tftp 
Features: AsynchDNS IDN IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz TLS-SRP HTTP2 UnixSockets HTTPS-proxy PSL 
==> GitHub Actions detected.
    Env vars used:
      -> GITHUB_ACTIONS:    true
      -> GITHUB_HEAD_REF:   docker-pls
      -> GITHUB_REF:        refs/pull/96/merge
      -> GITHUB_REPOSITORY: finos/legend-studio
      -> GITHUB_RUN_ID:     636943168
      -> GITHUB_SHA:        909cd1c87ff0209df2a6e53cf3272db3648ed7e9
      -> GITHUB_WORKFLOW:   Node CI
    Fixing merge commit SHA 909cd1c87ff0209df2a6e53cf3272db3648ed7e9 -> e9216f7c16c8b2309103291efc7cee786149fe4d
    project root: .
    Yaml found at: .github/codecov.yml
==> Running gcov in . (disable via -X gcov)
==> Python coveragepy not found
==> Searching for coverage reports in:
    + ./build/coverage
    -> Found 3 reports
==> Detecting git/mercurial file structure
==> Reading reports
    + ./build/coverage/clover.xml bytes=1445645
    + ./build/coverage/coverage-final.json bytes=4628383
    + ./build/coverage/lcov.info bytes=769688
==> Appending adjustments
    https://docs.codecov.io/docs/fixing-reports
    -> No adjustments found
==> Gzipping contents
        792K	/tmp/codecov.5eQPbv.gz
==> Uploading reports
    url: https://codecov.io
    query: branch=docker-pls&commit=e9216f7c16c8b2309103291efc7cee786149fe4d&build=636943168&build_url=http%3A%2F%2Fgithub.com%2Ffinos%2Flegend-studio%2Factions%2Fruns%2F636943168&name=&tag=&slug=finos%2Flegend-studio&service=github-actions&flags=&pr=96&job=Node%20CI&cmd_args=n,F,Q,s
->  Pinging Codecov
https://codecov.io/upload/v4?package=github-action-20210309-2b87ace&token=secret&branch=docker-pls&commit=e9216f7c16c8b2309103291efc7cee786149fe4d&build=636943168&build_url=http%3A%2F%2Fgithub.com%2Ffinos%2Flegend-studio%2Factions%2Fruns%2F636943168&name=&tag=&slug=finos%2Flegend-studio&service=github-actions&flags=&pr=96&job=Node%20CI&cmd_args=n,F,Q,s
->  Uploading to
https://storage.googleapis.com/codecov/v4/raw/2021-03-09/DAB0CDE5E55B3EC3317486B5D802380D/e9216f7c16c8b2309103291efc7cee786149fe4d/82a71e6b-6c05-42eb-b520-fe9a1164f1c5.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=GOOG1EQX6OZVJGHKK3633AAFGLBUCOOATRACRQRQF6HMSMLYUP6EAD6XSWAAY%2F20210309%2FUS%2Fs3%2Faws4_request&X-Amz-Date=20210309T194024Z&X-Amz-Expires=10&X-Amz-SignedHeaders=host&X-Amz-Signature=41b488ece3c39675c507936107012e5d0ec2f024c1eed2600ced70ec6c927fc3
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  789k    0     0  100  789k      0  2935k --:--:-- --:--:-- --:--:-- 2935k
    -> Reports have been successfully queued for processing at https://codecov.io/github/finos/legend-studio/commit/e9216f7c16c8b2309103291efc7cee786149fe4d

Expected Results

I expect to not see any change in coverage

Actual Results

I see this unrelated file listed as an impacted file

Impacted Files Coverage Δ
…s/legend-studio/src/stores/ChangeDetectionState.ts 67.74% <0.00%> (-1.08%) :arrow_down:

@akphi, I took a look at the report on Codecov and it’s showing no changes. My guess is that there might have been a timing issue, and we did not send an updated notification in time.

Are you still seeing this issue on other open PRs?

So perhaps I could have done some test PR to show you. I have these 3 PRs with coverage report:

And you can sorta see the change in the file ChangeDetectionState.ts jumping from negative to positive. I suspect it’s a timing issue which I would try to investigate a bit more. I’ll let you know what I find, though I just want to confirm the problem does not come from some sort of caching problem.

@akphi, this is strange. Unfortunately, I don’t know the codebase well enough, but it’s always happening on these lines. Is there any suspicion you might have that your tests are not necessarily idempotent in calling those lines?

That’s fair. We can close this issue if you want to. Do we have a re-open feature here? I can re-open this if I have more solid evidence to believe the problem comes from codecov. What do you think @tom?

1 Like

@akphi, sounds good to me. I’ll mark the previous response as a solution. If you find that there is a problem with Codecov, feel free to just reply back on this thread.

@tom So looks like it’s a timing issue on our part, we use mobx reaction with throttling, so I have reduced throttling duration to 0 for test. I will continue monitoring this, but this is great. Thank you so much and sorry for the false alarm!

1 Like