Codecov.yml Ignore not working

Support Bug Fixes
1

Description

As requested here:

I hereby report the following bug:
Adding “Ignore” to the codecov.yml file in following project, does not work:

Other settings from the codecov.yml work fine, making changes to those also can be previewed fine in a Pull Request. As can be seen here:
[CANT POST LINK 1]
(ignore the wrong ignore syntax in that PR, this has been fixed later)

We tested multiple writingstyles, paths and moving codecov.yml to root in 2 PR’s, which did not work out:
[CANT POST LINK 2]
[CANT POST LINK 3]

Goal is to make sure the following path is fully excluded:
/tests/**/*
(/tests, recursively very file and folder under it)

Repository

See above

Steps to Reproduce

See above, no known way to reproduce and as it is very much connected to a whole test suite not feasable to repeat outside this repo.

Expected behavior:
Those tests folders being completely ignored

Actual behavior:
No ignore happening

Flakiness?
All the time

Versions

Irrelevant

Additional Information

codecov validate results:

   {
      "comment": {
        "require_changes": true,
        "layout": "reach, diff, flags, footer",
        "behavior": "once"
      },
      "ignore": [
        "^tests/.*/.*"
      ],
      "coverage": {
        "status": {
          "project": {
            "default": {
              "threshold": 1.0
            }
          },
          "patch": {
            "default": {
              "threshold": 1.0
            }
          }
        },
        "range": [
          50.0,
          90.0
        ],
        "precision": 0,
        "round": "nearest"
      },
      "codecov": {
        "notify": {
          "after_n_builds": 2
        },
        "require_ci_to_pass": false
      }
    }

Taking above results into account this might be related to this:
[CANT POST LINK 4]

2

[CANT POST LINK 1]

[CANT POST LINK 2]

3

[CANT POST LINK 3]

Mod, please add these links to the opening posts for readability and remove these additional posts)
(an serieusly, if it’s this easy to circumvent that link limit, please just remove the link limit >.< )

4

(post withdrawn by author, will be automatically deleted in 24 hours unless flagged)

5

Taking the following into account:

  • There are multiple references to this issue/bug
  • They are being mostly ignored
  • Support is acting like they don’t know those reports exist
  • When confronted support starts: Lying, Putting up smokescreens, Ignoring, putting blame on unrelated “issues” and so forth…

I hereby conclude Codecov is unwilling and/or unable to fix this bug.

I’m just going the way of ignoring before content even gets send to Codecov, because we surely cant rely on codecov.

Maybe it would help if their developers where putting more time in development and less time in enforcing wrong english pronoun use? (no you are still not multiple people, otherwise you might actually have manpower to… you know… fix these bugs you keep ignoring :wink: )

7

I’m investigating this.

8

Crossposting our response on this matter here: Coverage report ignoring the configuration file on main branch - #12 by jerrod

9

(Edit: it looks like @jerrod has crossposted another response as well. Both posts lead to the same solution. I’m leaving this one here, though, since it specifically addresses your issue using your own project’s commit history, @Ornias1993 .)

Hi @Ornias1993,

My name is Eli, and I’m the CTO of Codecov. Your posts here were brought to my attention by the support team, and I felt that I needed to step in a bit and address your issue directly.

First, a dive into your issue, what’s causing it, and how to remedy it…

Your core issue stems from the fact that your codecov.yml file is located in a subdirectory of your project. While our docs state that this behavior is supported, a deep dive of our codebase makes me believe that this functionality was never actually working fully as expected. The fast remedy to your solution is one of the following:

  1. move the codecov.yml to the project’s root directory
  2. Uploading using the -y flag to specify the path to your codecov.yml in your project.

For the curious, a bit of a forensic investigation into how we came to this conclusion…

There was a commit in your repository, shown here:

https://codecov.io/gh/zfsonlinux/zfs/tree/f12999148d2f3620c39bfebbb6ffd3c95009bc23

that properly ignored the tests directory. Viewing that commit on github at:

we see your codecov.yml, with the ignore section included, located in the root of the repository. Our internal logs also revealed that ignore was being properly applied for this commit.

Additional commits, such as the current HEAD of master:

show that the codecov.yml was moved to a .github directory at some point. It was at this point I’m assuming the ignore functionality began to fail, as codecov’s recursive folder search for the codecov.yml appears to not be functioning as intended in this case. Therefore, I believe adopting one of the two solutions above, moving the codecov.yml to root or specifying its location at upload with the -y flag, should solve your problem.

Upon additional internal discussion with the team, we will very likely deprecate the recursive codecov.yml search functionality for a few reasons:

  1. It appears to be flaky at best
  2. A vast, vast majority of codecov users place their codecov.yml in the root
  3. Codecov provides a straightforward workaround, the -y flag at upload time, if an alternative path is desired.

I believe this will solve your issue, but if it does not, please let us know.

10

Crossposting our general response on this topic here: Coverage report ignoring the configuration file on main branch - #12 by jerrod

Separately, @Ornias1993 – some of the things you said here were pretty hurtful to us – we are indeed humans on the receiving end of your words :cry:. I hope our discourse can be more productive and forward-looking as we continue to work together on this. Code coverage is our absolute passion and we support our open source community for free always and forever.

Jerrod

11

The reason the above commit worked is because i’ve stopped trying to get codecov ignore to work and just changed LCOV to removed files before sending things to codecov.

So no: Moving it to root never solved any issue, the 4 times I tried.

Like it says:
“Try lcov ignore” and that worked out great, in stark contrast with codecov.

I’m not really interested in spending time on the issue, it is still broken in root or not. I fixed it for us, but not using codecov, because everything I tried (including moving it to root) didn’t work.

12

@Ornias1993 – ok, sorry to hear to this didn’t get to work for you. We will continue to investigate and hope to earn back your trust and usage in the future.

Jerrod