Infer coverage for partial builds from branch master

It would be nice to infer coverage for partial builds from branch master.

We run a mono repo and we don’t always run coverage for every sub repo on every PR to save CI time.

We upload reports with -F <topic> to specify a topic. It would be nice if codecov would infer coverage for the other -F <topic> as the coverage of the branch commit if nothing is reported before the timeout.

This would make the coverage accurate with respect to a merge to master.

1 Like

Great idea @nickpalmer!! We are calling this “carryforward” flags where we carryforward any coverage not updated from wherever it was most recently run. Looking at building this in the next several sprint cycles.

Awesome! Looking forward to seeing it roll out.

If you need a beta tester, let me know. I would be happy to turn it on and let you know how it is working on our repository! :slight_smile:

Thanks @nickpalmer – still building this. I’ll let you know when we get to beta!!

Any updates to this feature request? We tried using codecov in a monorepo and not having this made it unreliable.

1 Like

I’m working on a project using dynamic build pipelines (via Buildkite), and we are tagging all of our coverage submissions using -F <flags>. This would be incredibly helpful to us, as currently have had to turn off coverage diff messages since they don’t contain accurate data, and it would be a very useful thing to be able to alert the team on coverage regressions!

Latest update is “coming very soon!” Jerrod’s last update stands, but work is coming along nicely. Really looking forward to sharing when done.

1 Like

@ccarpita – getting very close! Definitely by end of March. Here’s a small sample for y’all

4 Likes

Hey @nickpalmer @JCMais @ccarpita – this feature for partial test runs “Carryforward Flags” is now in production in beta, would any of you like to test it out?

Invite open to rest of community as well :slight_smile:

2 Likes

I would be interested in trying it out this week, please let me know how :slight_smile:

I would love to be able to test it. Thanks for your work so far!

I would love to! I am just about to setup another mono repo project that could use this feature.

Hi everyone, thanks for all your patience. We released Carryforward Flags last week )https://blog.codecov.io/2020/04/21/new-test-only-what-you-change-with-carryforward-flags/) and you can find docs here (https://docs.codecov.io/docs/carryforward-flags). Closing this out, but would love to get feedback and bug reports.

1 Like

Thanks @tom.

We are not getting what I would expect with carry forward turned on in the repository I am attempting to use it on.

On a recent change that swapped 3 ‘let’ statements for ‘const’ statements across two flags in areas which are uncovered, we do see that it correctly identified the delta of zero in those three files. However, the coverage diff does not seem to be carrying forward, and shows -12.19% as an overall coverage drop.

I believe we have setup carry forward correctly and are certainly uploading with flags set. Our codecov.yml looks like this:

codecov:
  notify:
    require_ci_to_pass: yes

coverage:
  status:
    project:
      defaut:
        target: auto
      common:
        target: auto
        flags:
          - common
      agent:
        target: auto
        flags:
          - agent
      client:
        target: auto
        flags:
          - client
      messages:
        target: auto
        flags:
          - messages
      api:
        target: auto
        flags:
          - api
    patch:
      default:
        only_pulls: true
    changes: no
  precision: 2
  if_not_found: success
  round: down
  range: '0...100'

comment:
  layout: 'reach, diff, flags, files'
  behavior: default
  require_changes: false
  require_base: no
  require_head: yes
  branches: !staging

flags:
  common:
    paths: packages/common
    carryforward: true
  agent:
    paths: packages/agent
    carryforward: true
  client:
    paths: packages/client
    carryforward: true
  messages:
    paths: packages/lambda/messages
    carryforward: true
  api:
    paths: packages/api
    carryforward: true

parsers:
  javascript:
    enable_partials: yes

We would love to get the correct coverage numbers on the status report to make them mandatory passing checks on Github.

As an example of another branch with this, when we go look at this in the UI, it shows no changes in overview, but “Coverage Changes” shows 99.

Hi @nickpalmer, looks like there are some issues with your yaml

I have removed comment -> branches: !staging because there is currently a known issue with negative expressions.

if_not_found is also misplaced in the yaml. It should go under coverage -> status -> project -> <project-name>

Finally, I have pasted a valid YAML without the above two issues. Let me know if this works for you!

codecov:
  notify:
    require_ci_to_pass: yes

coverage:
  status:
    project:
      defaut:
        target: auto
      common:
        target: auto
        flags:
          - common
      agent:
        target: auto
        flags:
          - agent
      client:
        target: auto
        flags:
          - client
      messages:
        target: auto
        flags:
          - messages
      api:
        target: auto
        flags:
          - api
    patch:
      default:
        only_pulls: true
    changes: no
  precision: 2
  round: down
  range: '0...100'

comment:
  layout: 'reach, diff, flags, files'
  behavior: default
  require_changes: false
  require_base: no
  require_head: yes

flags:
  common:
    paths: 
      - packages/common
    carryforward: true
  agent:
    paths:
      - packages/agent
    carryforward: true
  client:
    paths:
      - packages/client
    carryforward: true
  messages:
    paths:
      - packages/lambda/messages
    carryforward: true
  api:
    paths:
      - packages/api
    carryforward: true

parsers:
  javascript:
    enable_partials: yes

Thanks you! That fixed it for us. Is there a linter for codecov.yml that would reveal these issues, or a place to see parse errors in the UI? I searched docs for that but didn’t find it.

Hi @nickpalmer, we are working on getting more of these issues user-facing, but as of now there isn’t. We are planning to get a better yaml validator out soon.

@tom Awesome thanks. I did find the validator we can post to, but it just seems to check if the yaml is valid and return it as json, not validate it against a schema.

I posted a new support issue around multiple comments on our PRs which I am not sure if that is related to carry forward, but it might be.