Description
Recently we’ve been seeing Codecov report incorrect coverage deltas for pull requests. Upon inspection, I noticed that all our PRs are using the same random ~1 month old commit as the pull request base, and not one of the more recent commits with code coverage information.
I inspected the Codecov docmentation for signs of any recent changes and noticed this comment on Status Checks
base [Deprecated as of July 2020 – Codecov’s product roadmap includes a novel way to manually pick base of pull requests]
I wonder if this could have anything to do with the issue we’re seeing.
Repository
private
CI/CD
Buildkite
Uploader
Commit SHAs
Commit always used as the PR base: 9ccb0f8a3562fc5fd3cb7a3af05d3bbc131b5aed
One of the many commits on PRs affected by this: fd809f3c116ca56478a0189922585f9f2d3e56c2
Codecov YAML
coverage:
status:
project:
default:
target: auto
threshold: 0.5%
base: auto
flags:
- model
- ui
patch:
default:
target: 100
threshold: 10%
base: auto
if_not_found: success
informational: true
flags:
- model
- ui
comment: false
flags:
model:
carryforward: true
ui:
carryforward: true
Codecov Output
We upload a couple of reports for every run. There’s nothing special to see here, but here’s one:
curl -s https://codecov.io/bash | bash -s - -f iOS/slather-report/cobertura.xml -t $CODECOV_TOKEN -F model'
2020-07-22 09:47:40 INFO Searching for artifacts: "iOS/slather-report/cobertura.xml" within step: ":xcode: Model Tests iPhone - iOS 14.0, iPad - iOS 14.0"
2020-07-22 09:47:40 INFO Found 1 artifacts. Starting to download to: /source
2020-07-22 09:47:42 INFO Successfully downloaded "iOS/slather-report/cobertura.xml" 3948838 bytes
_____ _
/ ____| | |
| | ___ __| | ___ ___ _____ __
| | / _ \ / _` |/ _ \/ __/ _ \ \ / /
| |___| (_) | (_| | __/ (_| (_) \ V /
\_____\___/ \__,_|\___|\___\___/ \_/
Bash-20200707-353aa93
==> Buildkite CI detected.
project root: .
Yaml found at: .codecov.yml
-> Found 1 reports
==> Detecting git/mercurial file structure
==> Reading reports
+ iOS/slather-report/cobertura.xml bytes=3948838
==> Appending adjustments
https://docs.codecov.io/docs/fixing-reports
+ Found adjustments
==> Gzipping contents
==> Uploading reports
url: https://codecov.io
query: branch=peter%2Fusingswift&commit=fd809f3c116ca56478a0189922585f9f2d3e56c2&build=&build_url=&name=&tag=&slug=PSPDFKit%2FPSPDFKit&service=buildkite&flags=model&pr=24946&job=806fe299-2ed0-47d1-97cc-2bad9c32877b&cmd_args=f,t,F
-> Pinging Codecov
https://codecov.io/upload/v4?package=bash-20200707-353aa93&token=secret&branch=peter%2Fusingswift&commit=fd809f3c116ca56478a0189922585f9f2d3e56c2&build=&build_url=&name=&tag=&slug=PSPDFKit%2FPSPDFKit&service=buildkite&flags=model&pr=24946&job=806fe299-2ed0-47d1-97cc-2bad9c32877b&cmd_args=f,t,F
-> Uploading to
https://storage.googleapis.com/codecov/v4/raw/2020-07-22/1C0A472DEF7E9691AAB76D8D1BFF298E/fd809f3c116ca56478a0189922585f9f2d3e56c2/e7b4ca56-9c4a-4a05-9b4d-e3c42d9d43d0.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=GOOG1EQX6OZVJGHKK3633AAFGLBUCOOATRACRQRQF6HMSMLYUP6EAD6XSWAAY%2F20200722%2FUS%2Fs3%2Faws4_request&X-Amz-Date=20200722T094916Z&X-Amz-Expires=10&X-Amz-SignedHeaders=host&X-Amz-Signature=ae4f50459046dc4414f71ccc7d79a4f04024c32ca0e12db8805f702db5630745
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 3220k 0 0 100 3220k 0 4112k --:--:-- --:--:-- --:--:-- 4107k
-> View reports at https://codecov.io/github/PSPDFKit/PSPDFKit/commit/fd809f3c116ca56478a0189922585f9f2d3e56c2
Additional Information
Unknown.
