Incorrect base commit of pull request

Description

Recently we’ve been seeing Codecov report incorrect coverage deltas for pull requests. Upon inspection, I noticed that all our PRs are using the same random ~1 month old commit as the pull request base, and not one of the more recent commits with code coverage information.

I inspected the Codecov docmentation for signs of any recent changes and noticed this comment on https://docs.codecov.io/docs/commit-status#base:

base [Deprecated as of July 2020 – Codecov’s product roadmap includes a novel way to manually pick base of pull requests]

I wonder if this could have anything to do with the issue we’re seeing.

Repository

private

CI/CD

Buildkite

Uploader

bash uploader

Commit SHAs

Commit always used as the PR base: 9ccb0f8a3562fc5fd3cb7a3af05d3bbc131b5aed
One of the many commits on PRs affected by this: fd809f3c116ca56478a0189922585f9f2d3e56c2

Codecov YAML

coverage:
  status:
    project:
      default:
        target: auto
        threshold: 0.5%
        base: auto
        flags:
          - model
          - ui
    patch:
      default:
        target: 100
        threshold: 10%
        base: auto
        if_not_found: success
        informational: true
        flags:
          - model
          - ui
comment: false
flags:
  model:
    carryforward: true
  ui:
    carryforward: true

Codecov Output

We upload a couple of reports for every run. There’s nothing special to see here, but here’s one:

curl -s https://codecov.io/bash | bash -s - -f iOS/slather-report/cobertura.xml -t $CODECOV_TOKEN -F model'
2020-07-22 09:47:40 INFO   Searching for artifacts: "iOS/slather-report/cobertura.xml" within step: ":xcode: Model Tests iPhone - iOS 14.0, iPad - iOS 14.0"
2020-07-22 09:47:40 INFO   Found 1 artifacts. Starting to download to: /source
2020-07-22 09:47:42 INFO   Successfully downloaded "iOS/slather-report/cobertura.xml" 3948838 bytes
  _____          _
 / ____|        | |
| |     ___   __| | ___  ___ _____   __
| |    / _ \ / _` |/ _ \/ __/ _ \ \ / /
| |___| (_) | (_| |  __/ (_| (_) \ V /
 \_____\___/ \__,_|\___|\___\___/ \_/
                              Bash-20200707-353aa93
==> Buildkite CI detected.
    project root: .
    Yaml found at: .codecov.yml
    -> Found 1 reports
==> Detecting git/mercurial file structure
==> Reading reports
    + iOS/slather-report/cobertura.xml bytes=3948838
==> Appending adjustments
    https://docs.codecov.io/docs/fixing-reports
    + Found adjustments
==> Gzipping contents
==> Uploading reports
    url: https://codecov.io
    query: branch=peter%2Fusingswift&commit=fd809f3c116ca56478a0189922585f9f2d3e56c2&build=&build_url=&name=&tag=&slug=PSPDFKit%2FPSPDFKit&service=buildkite&flags=model&pr=24946&job=806fe299-2ed0-47d1-97cc-2bad9c32877b&cmd_args=f,t,F
->  Pinging Codecov
https://codecov.io/upload/v4?package=bash-20200707-353aa93&token=secret&branch=peter%2Fusingswift&commit=fd809f3c116ca56478a0189922585f9f2d3e56c2&build=&build_url=&name=&tag=&slug=PSPDFKit%2FPSPDFKit&service=buildkite&flags=model&pr=24946&job=806fe299-2ed0-47d1-97cc-2bad9c32877b&cmd_args=f,t,F
->  Uploading to
https://storage.googleapis.com/codecov/v4/raw/2020-07-22/1C0A472DEF7E9691AAB76D8D1BFF298E/fd809f3c116ca56478a0189922585f9f2d3e56c2/e7b4ca56-9c4a-4a05-9b4d-e3c42d9d43d0.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=GOOG1EQX6OZVJGHKK3633AAFGLBUCOOATRACRQRQF6HMSMLYUP6EAD6XSWAAY%2F20200722%2FUS%2Fs3%2Faws4_request&X-Amz-Date=20200722T094916Z&X-Amz-Expires=10&X-Amz-SignedHeaders=host&X-Amz-Signature=ae4f50459046dc4414f71ccc7d79a4f04024c32ca0e12db8805f702db5630745
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 3220k    0     0  100 3220k      0  4112k --:--:-- --:--:-- --:--:-- 4107k
    -> View reports at https://codecov.io/github/PSPDFKit/PSPDFKit/commit/fd809f3c116ca56478a0189922585f9f2d3e56c2

Additional Information

Unknown.

Hi @matej, would you be able to screenshot the commit log for that pull request? I’d like to compare it to the commits here: https://codecov.io/gh/:owner:/:repo:/pull/:pull_number:/commits

Hi @tom. Thanks for looking into this!

Yes, I can. There’s nothing sensitive in this one. Will this do?

Hi @matej, I’m not seeing any logs for 2db4b47, which might mean that coverage reports weren’t uploaded for that commit. Would you be able to double-check to see if Buildkite attempted to run the bash script?

That job was automatically canceled, likely because the next commit came quickly after it. So there was no coverage data uploaded for that specific commit.

Marking this as the solution as this is being taken offline.