It appears that our master branch over at Authelia has gone out of sync, with a little investigation it seems that when someone made a PR from their forked master branch this somehow has assumed itself as the main repos master branch.
Hi @nightah, thanks for the detailed writeup here. I think the master branch is actually a red herring, because what’s more interesting is that we stopped getting coverage reports uploaded for commits to master for the past 15 days (here). Is there any reason that you might be running into this since then?
Hi @tom I honestly have no idea why this would be the case.
We never run/upload coverage to master directly. All of our commits are PR’d from branches or external forks (where coverage is run and uploaded) and eventually squash merged into master.
This has been the case since the introduction of Codecov to our repo. Nothing else has changed since 15 days ago, you should be able to see this in our commit history too.
Hi @nightah, if I had to guess, it might have something to do with this line here. I’ll be honest, my bash isn’t great so I can’t be sure, but it looks like you might no longer by uploading coverage on master branch runs?
I think you may have missed what I mentioned in my previous comment @tom. We have never uploaded coverage for the master branch. That conditional exists because the only time a full set of coverage is run is outside of a deployment (any push that isn’t master or a tagged release).
So all of the runs that are appearing in the master branch right now are PR’d from other branches, never direct runs on master.
Here’s the PR in question where we enabled coverage collection/codecov: [CI] Add Codecov support by nightah · Pull Request #1065 · authelia/authelia · GitHub, you can see that the conditional has existing from 21 days ago and has not changed, so we have never actually run coverage collection/uploading on master and have not once since the inception of codecov to our repo.
Right now it says master hasn’t been updated in several days. I’m 99% certain that if I go into that PR I mentioned initially and update it, our master branch will say it has been updated, as this is what happened last time. I can elaborate further with some screenshots if that might help?
Actually to further illustrate the problem you can see another commit/PR that was made just today by another user on a forked repos master branch https://github.com/authelia/authelia/pull/1157, 11 hours prior to this post.
You can see our “default” master branch Codecov also now says it was last updated 11 hours ago and has that respective user tagged the last person perform said update Updated 11 hours ago by BankaiNoJutsu.
I suspect this behaviour is not correct, as a forked repo should never be assumed as the default branch set in the codecov settings page. My understanding is that this branch will then be used as the pseudo-commit to compare all future coverage changes.
This is now the second example of what appears to be consistant behaviour where a forked repo with a master branch somehow gets assumed as codecov’s default branch assigned to our repo.
@tom is there any other information we can provide here?
I’m fairly certain that we can wipe our codecov data and start from scratch (while not ideal) and prevent users from submitting PR’s from forked master branches in attempts to prevent this issue.
Should we just go down this path or is there anything else can can be done to understand the root cause of this problem?
Hi @nightah, apologies for the delay here. I’m still digging into why this is happening. If this is blocking you, you are welcome to do the wipe, but if this can wait, I’d like to be able to fix it for you. Let me know what you prefer.
Hi @tom no problems, I understand there’s a significant time delay. I did end up performing the wipe and put some safety mesaures in place which meant coverage will not be run and uplodaed if a PR is submitted from a forked master branch.
The only thing is that now we’re not seeing any commits that are merged showing up in the codecov master branch.
If you have a look at this screenshot you can see that with the same codecov settings, it previously did result in commits being recognised from master and it in turn posted github checks under the master branch.
Could this issue still be on the back of the symptoms we were experiencing before, do you have any advice on how we might resolve this? Should I disconnect it from github and reconnect?
Hi @nightah, my gut instinct is that commits aren’t showing up in Codecov because coverage is not being generated for a commit to the master branch from a fork. When you merge, do you run your CI/CD process again, and if so, do you happen to have a link to a build?
@tom the CI/CD process does run, however, coverage is not run again when merged in to master it’s a simplified deployment process since we don’t need to run tests all over again for the same code.
Essentially tests and coverage uploads are only run in branches and PR’s.
All of our builds have followed this same process, so I’m at a little bit of a loss as to why and how we managed to get commits show up in master before but not now haha.
Hi @nightah, I have escalated this to the product team to work on a fix for this issue. I believe that we will need to fix the master branch issue first.
In the interim I have setup our CI/CD process to run tests and upload coverage for the master branch.
When/if there is a fix I would love to remove those steps cut down our CI/CD process for master deployments back to what they originally were, so I’ll hang tight in the meantime.
@tom so now that we are actually running coverage on every single upload to master it still appears that if a branch is updated within a PR it still causes some odd issues.
Hi @tom thanks for the update, since I’m not sure if it’s entirely clear I have worked around the issue when a PR falls out of sync with master and presents the following message:
This branch is out-of-date with the base branch
Merge the latest changes from master into this branch.
By avoiding the “Update branch” button on the GitHub WebUI and instead rebasing all PRs.
I believe you should be able to replicate this issue by updating PR’s with that button and observing the behaviour it will illicit with codecov.