Codecov yanked from PyPI (all versions)

This morning our builds started failing, and it appears that all versions of the codecov package have been yanked from PyPI: https://pypi.org/project/codecov/. There is only version “0.0.0a2”, published as of 2023-04-12 13:05:21 UTC, and a note that the package has been removed.

I assume that either 1) there’s been some kind of automation error, 2) an attacker has gained control of the PyPI account and removed the package, or 3) perhaps there was malware in some recent versions and everything has been removed as a precaution.

Please advise – is this a “sit back and wait an hour” sort of situation, or do we need to stop using any recent version of codecov immediately?

EDIT #1 09:56 ET: It sounds like this package was deprecated a while back and was supposed to stop working on 2022-02-01 (see https://github.com/codecov/codecov-python) so maybe this was an intentional action? Pretty scary way to go about it, if so! The package has actually been completely deleted now (a 404, not just a yanked-versions page) but I’ve confirmed with PyPI Security that this package can no longer be registered, so at least there should be no risk of squatters.

EDIT #2: In our own situation, we’ve actually been using the codecov-action GHA uploader for a while, but still had a lingering installation of the codecov Python package – it’s the latter that is breaking the builds. If others are in a similar situation, just removing any codecov package requirements should work.

2 Likes

I’m hearing that based on https://about.codecov.io/blog/codecov-uploader-deprecation-plan/ the pypi package should’ve been unnecessary for users of the codecov uploader github action

So this may be a “we already deprecated that thing, what we have to notify you about when we shut it off too??” situation

Hi, PyPI administrator here. Just wanted to confirm that the entire codecov project was removed by the project maintainers at 2023-04-12 12:41:12 UTC. The 0.0.0a2 release was later uploaded by an unrelated third party that was helpfully trying to prevent the released project name from being acquired by attackers and distributing malware. That version has since been removed and the name prohibited from re-registration.

5 Likes

Thank you, @dustin! <3

@drazisil, maybe you could add some context about what you expect your users to do here? The PyPI team is getting a lot of queries about this, and we’ve been directing them to this thread for clarification.

Hi @dustin, we have written up some context here.

Thanks for the followup. I think one of the critical points for us is that we had actually done the migration to the new uploader, but were still installing the old package even though we weren’t using it. I suspect other organizations were in the same boat.

The suggested approach of restoring the package, even if all it does is print out an error, seems like it should work to unblock CI – since we weren’t actually using the package, just installing it. I hope that you will consider uploading multiple versions, too. All 70-80 affected repos on our side were installing codecov==2.1.12 but other people may be on other versions.

1 Like

@timmc-edx thanks for the heads up here. We have messaged with the PyPI team to reinstate the package with an error message and migration steps. I"ll keep your use case in mind.

1 Like

Thanks for the writeup. About this (emphasis added):

Our intent was to remove a deprecated, rarely-used package from active support.

As this package was unmaintained and limited in use, there are no plans for the codecov package in PyPi to be made functional again.

The codecov package wasn’t “rarely-used package” or “limited in use”, it was downloaded 1.3 million times according to Codecov download stats, which places it in the top 1% (maybe top 0.5%) and I expect enough for PyPI to designate it as a “critical project” requiring 2FA for maintainers.


Anyway, in our case, we were still using it on AppVeyor, so are switching that to the uploader. We were also still installing it on GitHub Actions but not using it, as we’d already switched to the action.


For an example of how another project is dealing with this sort of thing, see how the scikit-learn team is deprecating the sklearn PyPI package in favour of scikit-learn via an increasingly frequent brownout schedule over 12 months:

2 Likes

Are there any updates on getting codecov back on pypi with an error message?

We had a similar issue as the rest of the posters here within GitHub - pyvista/pyvista: 3D plotting and mesh analysis through a streamlined interface for the Visualization Toolkit (VTK), despite having migrated to codecov/codecov-action@v3 months ago. A helpful error message would have made tracking this down much easier.

Since many on GitHub are using dependabot, the best approach would have been to:

  • Push a new version that raises an error on import. Those that pin testing or build dependencies would have still have their workflows pass, but fail when dependabot suggests bumping codecov
  • Users would then be able to upgrade on their own to the GitHub action.
  • To get those that use older versions to migrate, you could incrementally yank older versions allowing them to slowly “get with the program.”

Something that I don’t see addressed here is how we are supposed to migrate if codecov is required by other dependencies that haven’t updated. We could migrate our own usage of codecov to something else, but probably half of our dependencies are also requiring codecov.

@tylerpuleo22 still working with the team. I expect to get something up with an error message and a failed install ASAP.

@akaszynski thanks for the tips here. We will push a new version with the above. As for the 2.1.12 and prior versions, they will likely not be re-pushed back up. Since we are looking to pull down other uploaders, we will take this strategy, and I thank you for your write-up.

@BrianRamsay I am not actually aware of other packages that use Codecov as a dependency. Would you be able to share which ones you are running into?

1 Like

I would greatly appreciate it if you could re-upload 2.1.12. We still have about 70 repositories that will need to have codecov manually removed from them in order to unbreak development. That’s a lot of unnecessary toil that will interrupt feature work, bugfixes, and possibly emergency maintenance. All of those repos are specifically expecting 2.1.12 to exist. Having that version reappear even if it is just an otherwise empty package would allow us to treat this as a maintenance task rather than an interruption.

2 Likes

When I dig deeper, it looks like all of the ones that are a problem stem from a single package, config2 · PyPI, which hasn’t been updated in a while. We can’t just swap it out easily though, because our application configuration for different environments depends on it. We can fork it and modify the package hopefully, but it would be a lot easier if there was a version of codecov still available with a deprecation warning or something.

@timmc-edx we are unable to re-upload 2.1.12, but we have pushed 2.1.13 which contains 2.1.12 and a deprecation message.

@BrianRamsay I’ll dig a little deeper into that and see what we can do from our side.

Thanks for putting the Python uploader back for now. I’m still using this in some places.

I’ve described why in this issue, but briefly: I can use pip to install a 20 KB package which works on all platforms and all architectures. The new uploader is 40 MB, needs a separate download step for each platform, and most importantly, doesn’t have binaries for all the architectures I’m testing on!

I get that you don’t want to maintain a separate uploader for each language, but why not let the language’s community do it? Your upload API seems to be pretty stable, so it’s not much work to keep an uploader tool working. At least one third-party project (pytest-codecov) is doing this anyway. All you really need to do is say that third-party uploaders are OK and you won’t try to prevent them. :smiley: (Even better, give us some notice if the upload API is changing)

1 Like

Can you give more detail on why you’re unable to upload a 2.1.12? (It doesn’t have to do anything, just be downloadable.)

@timmc-edx this is a limitation from the PyPI side. We are blocked from uploading a version that exists or already existed. In this case, codecov==2.1.12 was once a package that existed.

@takluyver that’s a really good point (uploader size). I will take a look this quarter to see how we can possibly shrink the size of it. If you have any ideas, they are extremely welcome.

As far as the upload API, I’ve let the product team know. I personally would like to see more transparency around it, but I know that we are working on revamping it currently.

Thanks, but that’s actually the least important of my reasons for preferring the Python uploader. This is the order of importance to me:

  1. Availability on different architectures: at least one project I work on has CI jobs on Linux arm64 and ppc64le. It’s all but impossible to use the new uploader there (I could in theory set up the toolchain needed to build it from the typescript source, but I can’t see myself bothering. :wink: )
  2. Ease of installation: it needs a download URL per platform, and a separate set of instructions for Windows. Not a big deal, but it’s extra complexity in the CI definitions for something that could be simple.
  3. Size: seems kind of silly to download a 40 MB binary for a pretty simple task, but it’s not that big, and it can probably be cached.

So long as the official uploader is open source (I hope that’s not changing!), it’s kind of transparent anyway. We can see what the typescript code does and translate it to any language we choose. :smiley: Documentation would be great, but if the product team just don’t try to block third-party uploaders, that will work for us.