Codecov reporting unexpected status update for deleted project

Description

On Feb 26 we committed a change to our repo to delete the config for a specific project in Codecov (see Remove codecov for protocol until coverage working again (#2947) · celo-org/celo-monorepo@1fbe7f8 · GitHub).

We did this because Codecov was reporting a failing status check for the protocol project seemingly randomly.
Specifically, the protocol test coverage is only uploaded daily. That’s why we initially added if_not_found: success to its config.
However we saw that - randomly - the status check for the protocol project would report an error with 0.00% (target 90.00%) instead of the expected success status with No files found while filtering the report.

Today, Feb 27, we see Codecov is still reporting the failing status check for the removed protocol project (also seemingly randomly).

It looks like Codecov isn’t fully taking our updated config file into account.

Repository

Steps to Reproduce

  1. Have multiple projects in your Codecov config
  2. See statuses being reported to GitHub
  3. Delete one project from the config
  4. See Codecov reporting status for the deleted project even when no coverage was uploaded for it

Expected behavior:

Codecov should not report failing status for a deleted project.
Codecov should consistently report a success status with No files found while filtering the report. when not uploading coverage and if_not_found: success is set for a project.

Actual behavior:

Codecov is reporting failing status for a deleted project even when not uploading any coverage for it.
Codecov is reporting an error status with 0.00% (target 90.00%) instead of success status with No files found while filtering the report. when not uploading coverage and if_not_found: success is set for a project.

Flakiness?

Seemingly randomly.

Versions

Safari 13.0.5
macOS Catalina 10.15.3

Additional Information

Configuration:

Hi @jeanregisser

Commenting out a status will disable that status, it won’t tell Codecov to stop processing those files. I recommend adding that path to your ignore section if you want us to not handle it.

Hi,

I’m confused about your answer.
The screenshot above shows Codecov still reporting the status for the project that has been commented out in the config. And you’re saying it shouldn’t, right?
Also, as far as I saw, we did not upload anything related to that path when that status in the screenshot was reported.

What’s happening?

Hi @jeanregisser

I’m sorry, I must have been tired when I responded.

Can you please rename the file from .codecov.yml to codecov.yml (no leading period) and share an updated SHA if that doesn’t fix it?

Hi @drazisil,

Thank you for your help.

I’ve renamed the Codecov config file in Fix codecov config by jeanregisser · Pull Request #3049 · celo-org/celo-monorepo · GitHub
I did receive different statuses, but it looks like the custom projects were ignored, and got a generic codecov/project instead of the expected codecov/project/mobile and codecov/project/web:

Also I now see an unexpected codecov/patch status which is disabled in the config.

Another thing I noticed is Codecov web UI seems to indicate it didn’t find any config now that it’s been renamed:

Screenshot 2020-03-10 at 21.44.48|690x294

See: https://codecov.io/gh/celo-org/celo-monorepo/commit/78097c75c93094fff2a29c6c1d33dc8e60024e26/build

Before the rename, the web UI was correctly showing the config there.

What do you think is wrong?

Based on the errors I’m seeing in the logs, I think you need to change the formatting of the paths: key to match the way it in the flags: section.

So,

    project:
      default: off
      mobile:
        paths: 
          - 'packages/mobile/'

For example.

Thanks I tried in Fix codecov config by jeanregisser · Pull Request #3049 · celo-org/celo-monorepo · GitHub but the status reported were the same:

Related to this, both configs validated correctly and had the same output using your online validator https://codecov.io/validate:

❯ cat codecov.yml
coverage:
  status:
    project:
      default: off
      mobile:
        paths: 'packages/mobile/'
        flags: mobile
        threshold: 5%
        target: 70%
      web:
        paths: 'packages/web/'
        flags: web
        threshold: 2%
        target: 75%
      # Protocol codecov disabled until coverage tests are fixed.
      #protocol:
        #paths: 'packages/protocol/'
        #flags: protocol
        #threshold: 5%
        #target: 90%
        #if_no_uploads: error
        #if_not_found: success
        #if_ci_failed: error
    patch:
      default: off

flags:
  mobile:
    paths:
      - packages/mobile/
  #protocol:
    #paths:
      #- packages/protocol/

❯ curl --data-binary @codecov.yml https://codecov.io/validate
Valid!

{
  "flags": {
    "mobile": {
      "paths": [
        "^packages/mobile/.*"
      ]
    }
  },
  "coverage": {
    "status": {
      "project": {
        "default": false,
        "mobile": {
          "threshold": 5.0,
          "paths": [
            "^packages/mobile/.*"
          ],
          "flags": [
            "mobile"
          ],
          "target": 70.0
        },
        "web": {
          "threshold": 2.0,
          "paths": [
            "^packages/web/.*"
          ],
          "flags": [
            "web"
          ],
          "target": 75.0
        }
      },
      "patch": {
        "default": false
      }
    }
  }
}
❯ cat codecov.yml
coverage:
  status:
    project:
      default: off
      mobile:
        paths:
          - packages/mobile/
        flags: mobile
        threshold: 5%
        target: 70%
      web:
        paths:
          - packages/web/
        flags: web
        threshold: 2%
        target: 75%
      # Protocol codecov disabled until coverage tests are fixed.
      #protocol:
        #paths: 'packages/protocol/'
        #flags: protocol
        #threshold: 5%
        #target: 90%
        #if_no_uploads: error
        #if_not_found: success
        #if_ci_failed: error
    patch:
      default: off

flags:
  mobile:
    paths:
      - packages/mobile/
  #protocol:
    #paths:
      #- packages/protocol/

❯ curl --data-binary @codecov.yml https://codecov.io/validate
Valid!

{
  "flags": {
    "mobile": {
      "paths": [
        "^packages/mobile/.*"
      ]
    }
  },
  "coverage": {
    "status": {
      "project": {
        "default": false,
        "mobile": {
          "threshold": 5.0,
          "paths": [
            "^packages/mobile/.*"
          ],
          "flags": [
            "mobile"
          ],
          "target": 70.0
        },
        "web": {
          "threshold": 2.0,
          "paths": [
            "^packages/web/.*"
          ],
          "flags": [
            "web"
          ],
          "target": 75.0
        }
      },
      "patch": {
        "default": false
      }
    }
  }
}

What else can we try?

I think the flags need to be made into lists as well

flags:
  - mobile

Etc. Can you try that, and pass me a new SHA if still seeing issues?

Thanks, getting some progress with Fix codecov config by jeanregisser · Pull Request #3049 · celo-org/celo-monorepo · GitHub, I now also see codecov/project/mobile and codecov/project/web statuses as expected:

Next step, disabling statuses for codecov/patch and codecov/project.

From what I see, Codecov online validator still reports the same valid output though (curl --data-binary @codecov.yml https://codecov.io/validate).
Could you explain why those changes are necessary given the validator indicates they are the same?

Yes, the validate endpoint currently points to the old validator, not the stricter, schema-driven one the processor uses. This make it only 97% accurate currently, but this is on track to be fixed soon.


I think patch: off (drop the default part) should work. Still thinking on why the default project status is showing, let me check the logs.

Ok now I understand better :wink:

I tried disabling the codecov/patch status in Fix codecov config by jeanregisser · Pull Request #3049 · celo-org/celo-monorepo · GitHub but it broke the custom projects statuses, so I assume it’s because of this stricter internal check.

Is there a way for me to check my config against this stricter version? I feel I’m flying quite blind here.

What’s the proper way of disabling codecov/patch and codecov/project?

I really appreciate your help here.

There is not, and trust me, I know how you feel. The only/best way right now is for me to check the logs to see what the parser sees.

It’s odd that b41b9210efd4a4853db244624ebd01692485b71c broke, because it looks valid.

What happens if you drop default: false from the project section?

Dropped default: false in Fix codecov config by jeanregisser · Pull Request #3049 · celo-org/celo-monorepo · GitHub and it’s still broken:

What else can we try? Thanks.

I made a change on our side that I think will fix this. Can you put the default key back and try again?

Thanks for the patience.

Custom projects are back but the codecov/patch and codecov/default are still there in https://github.com/celo-org/celo-monorepo/pull/3049/commits/c29644730a7136776f12d9239e2de9cd22b8378f:

Let me know what you want me to try next. Thanks!

That is super annoying. Checking with an engineer, I’ll be in touch.

@drazisil anything new about this?

Thanks!

Hi @jeanregisser, I’m a bit out of the loop on this issue, and I just wanted to see if I could give you an update here, but I’m not sure what state your projects are currently in. Are you still experiencing an issue with this, and if so, how can I help?