Threshold confusion

Support
1

Description

I only want a failing GitHub status check when a PR drops project coverage by more than 1%.

I’ve set a threshold of “1” and also tried with “1%”, but neither seems to actually work, in this PR for example: Changes to psort to show event counters as status by joachimmetz · Pull Request #2442 · log2timeline/plaso · GitHub there’s a failing status even though coverage dropped by 0.02%.

Any suggestions for how to get this behavior?
Secondarily, is “1” or “1%” the correct way to specify a threshold?

Repository

Versions

Chrome

Additional Information

Full team YAML:

comment:
  require_changes: true  # Only post a comment if coverage changes.
coverage:
  status:
    project:
      default:
        threshold: 1  # Allow coverage to drop by up to 1% in a PR before marking it failed
    patch: off
2

Hey, thanks for the reach out.

A couple thoughts:

1.) For the %, please do “1%” instead of “1” (1 would = 100%) docs here: Notifications

2.) We are running into an intermittent bug of cached YAML’s not updating, so if you update this and still not working please post again.

Jerrod

3

Thanks Jerrod!

I’ve changed this, and I’m still getting failures - see Added additional checks for first utmp record validity #2381 by Onager · Pull Request #2459 · log2timeline/plaso · GitHub where a drop of 0.01% is causing the check to fail.

4

Hey Onager, yes I checked your YAML and you made the right change. Investigating why your are still failing. I think either:

  1. Your YAML change was invalidated and reverted back to old YAML

  2. There is a bug in failing coverage

5

Hey @Onager, just an update here that we are working on better error messaging around invalidated YAML changes --> it’s like your YAML failed silently. In future, we’d like to have this YAML failure announce an error to you.

6

Thanks @jerrod - let me know if there’s anything else I can do to help debug this.

7

We are seeing this as well. CodeCov often reports a failure with 0.1%. I think it’s related to transformations with babble and what not, but I could be wrong.

Regardless, since I am not familiar with your service, do we need a yam file to configure this, or is this a default you can “increase”? Or we increase somewhere else? I’d like to set it globally for our org if possible.

Thanks! :slight_smile:

8

Hey @till, thanks for the question. Separate from the improvements we are making for error messaging of YAML invalidation → Yes, you need a .yml file to configure Codecov. You can check out the global YAML settings doc here: About the Codecov YAML

9

@jerrod Thanks. I guess one more question, can I just add a yaml with the threshold piece? Or should I copy the complete default yaml and customise it?

10

No need to copy the whole, just add the threshold setting. (You do need the tree structure to the setting though.)

coverage:
  status:
    project:
      default:
        threshold: 1%

For organisation wide (all repositories in your account) settings, you can use the " Team yaml" in your account settings.