400 "too many uploads to this commit"

Support
1

Description

My CI is failing is a 400 error being reported by the uploader, “too many uploads to this commit”. I’ve never seen this before and nothing special is going on in the repository. This is on the daily scheduled CI run on the main branch. I’m using the new/binary uploader, latest (1.0.5 on this run).

Commit SHAs

61ba6ea7e69e11a1616287d5996a025da8482fd3

Repository

CI/CD or Build URL

Uploader

binary uploader 1.0.5

Codecov Output

Full output 
       _____          _
      / ____|        | |
     | |     ___   __| | ___  ___ _____   __
     | |    / _ \ / _` |/ _ \/ __/ _ \ \ / /
     | |___| (_) | (_| |  __/ (_| (_) \ V /
      \_____\___/ \__,_|\___|\___\___/ \_/
  
    Codecov report uploader 0.1.5
  [2021-10-08T15:10:58.442Z] ['info'] => Project root located at: /home/runner/work/community.hashi_vault/community.hashi_vault
  [2021-10-08T15:10:58.444Z] ['info'] -> No token specified or token is empty
  [2021-10-08T15:10:58.448Z] ['info'] Searching for coverage files...
  [2021-10-08T15:10:58.482Z] ['info'] => Found 20 possible coverage files:
    /home/runner/work/community.hashi_vault/community.hashi_vault/cov-units-pre2.12/coverage=units=ansible_stable-2.9=data/coverage=units=units=docker-default=python-3.5.xml
    /home/runner/work/community.hashi_vault/community.hashi_vault/cov-units-pre2.12/coverage=units=ansible_stable-2.9=data/coverage=units=units=docker-default=python-3.6.xml
    /home/runner/work/community.hashi_vault/community.hashi_vault/cov-units-pre2.12/coverage=units=ansible_stable-2.9=data/coverage=units=units=docker-default=python-3.8.xml
    /home/runner/work/community.hashi_vault/community.hashi_vault/cov-units-pre2.12/coverage=units=ansible_stable-2.9=data/coverage=units=units=docker-default=python-2.6.xml
    /home/runner/work/community.hashi_vault/community.hashi_vault/cov-units-pre2.12/coverage=units=ansible_stable-2.9=data/coverage=units=units=docker-default=python-2.7.xml
    /home/runner/work/community.hashi_vault/community.hashi_vault/cov-units-pre2.12/coverage=units=ansible_stable-2.9=data/coverage=units=units=docker-default=python-3.7.xml
    /home/runner/work/community.hashi_vault/community.hashi_vault/cov-units-pre2.12/coverage=units=ansible_stable-2.10=data/coverage=units=units=docker-default=python-3.5.xml
    /home/runner/work/community.hashi_vault/community.hashi_vault/cov-units-pre2.12/coverage=units=ansible_stable-2.10=data/coverage=units=units=docker-default=python-3.6.xml
    /home/runner/work/community.hashi_vault/community.hashi_vault/cov-units-pre2.12/coverage=units=ansible_stable-2.10=data/coverage=units=units=docker-default=python-3.8.xml
    /home/runner/work/community.hashi_vault/community.hashi_vault/cov-units-pre2.12/coverage=units=ansible_stable-2.10=data/coverage=units=units=docker-default=python-2.6.xml
    /home/runner/work/community.hashi_vault/community.hashi_vault/cov-units-pre2.12/coverage=units=ansible_stable-2.10=data/coverage=units=units=docker-default=python-2.7.xml
    /home/runner/work/community.hashi_vault/community.hashi_vault/cov-units-pre2.12/coverage=units=ansible_stable-2.10=data/coverage=units=units=docker-default=python-3.9.xml
    /home/runner/work/community.hashi_vault/community.hashi_vault/cov-units-pre2.12/coverage=units=ansible_stable-2.10=data/coverage=units=units=docker-default=python-3.7.xml
    /home/runner/work/community.hashi_vault/community.hashi_vault/cov-units-pre2.12/coverage=units=ansible_stable-2.11=data/coverage=units=units=docker-default=python-3.5.xml
    /home/runner/work/community.hashi_vault/community.hashi_vault/cov-units-pre2.12/coverage=units=ansible_stable-2.11=data/coverage=units=units=docker-default=python-3.6.xml
    /home/runner/work/community.hashi_vault/community.hashi_vault/cov-units-pre2.12/coverage=units=ansible_stable-2.11=data/coverage=units=units=docker-default=python-3.8.xml
    /home/runner/work/community.hashi_vault/community.hashi_vault/cov-units-pre2.12/coverage=units=ansible_stable-2.11=data/coverage=units=units=docker-default=python-2.6.xml
    /home/runner/work/community.hashi_vault/community.hashi_vault/cov-units-pre2.12/coverage=units=ansible_stable-2.11=data/coverage=units=units=docker-default=python-2.7.xml
    /home/runner/work/community.hashi_vault/community.hashi_vault/cov-units-pre2.12/coverage=units=ansible_stable-2.11=data/coverage=units=units=docker-default=python-3.9.xml
    /home/runner/work/community.hashi_vault/community.hashi_vault/cov-units-pre2.12/coverage=units=ansible_stable-2.11=data/coverage=units=units=docker-default=python-3.7.xml
  [2021-10-08T15:10:58.482Z] ['info'] Processing /home/runner/work/community.hashi_vault/community.hashi_vault/cov-units-pre2.12/coverage=units=ansible_stable-2.9=data/coverage=units=units=docker-default=python-3.5.xml...
  [2021-10-08T15:10:58.487Z] ['info'] Processing /home/runner/work/community.hashi_vault/community.hashi_vault/cov-units-pre2.12/coverage=units=ansible_stable-2.9=data/coverage=units=units=docker-default=python-3.6.xml...
  [2021-10-08T15:10:58.488Z] ['info'] Processing /home/runner/work/community.hashi_vault/community.hashi_vault/cov-units-pre2.12/coverage=units=ansible_stable-2.9=data/coverage=units=units=docker-default=python-3.8.xml...
  [2021-10-08T15:10:58.489Z] ['info'] Processing /home/runner/work/community.hashi_vault/community.hashi_vault/cov-units-pre2.12/coverage=units=ansible_stable-2.9=data/coverage=units=units=docker-default=python-2.6.xml...
  [2021-10-08T15:10:58.490Z] ['info'] Processing /home/runner/work/community.hashi_vault/community.hashi_vault/cov-units-pre2.12/coverage=units=ansible_stable-2.9=data/coverage=units=units=docker-default=python-2.7.xml...
  [2021-10-08T15:10:58.491Z] ['info'] Processing /home/runner/work/community.hashi_vault/community.hashi_vault/cov-units-pre2.12/coverage=units=ansible_stable-2.9=data/coverage=units=units=docker-default=python-3.7.xml...
  [2021-10-08T15:10:58.492Z] ['info'] Processing /home/runner/work/community.hashi_vault/community.hashi_vault/cov-units-pre2.12/coverage=units=ansible_stable-2.10=data/coverage=units=units=docker-default=python-3.5.xml...
  [2021-10-08T15:10:58.493Z] ['info'] Processing /home/runner/work/community.hashi_vault/community.hashi_vault/cov-units-pre2.12/coverage=units=ansible_stable-2.10=data/coverage=units=units=docker-default=python-3.6.xml...
  [2021-10-08T15:10:58.499Z] ['info'] Processing /home/runner/work/community.hashi_vault/community.hashi_vault/cov-units-pre2.12/coverage=units=ansible_stable-2.10=data/coverage=units=units=docker-default=python-3.8.xml...
  [2021-10-08T15:10:58.502Z] ['info'] Processing /home/runner/work/community.hashi_vault/community.hashi_vault/cov-units-pre2.12/coverage=units=ansible_stable-2.10=data/coverage=units=units=docker-default=python-2.6.xml...
  [2021-10-08T15:10:58.503Z] ['info'] Processing /home/runner/work/community.hashi_vault/community.hashi_vault/cov-units-pre2.12/coverage=units=ansible_stable-2.10=data/coverage=units=units=docker-default=python-2.7.xml...
  [2021-10-08T15:10:58.503Z] ['info'] Processing /home/runner/work/community.hashi_vault/community.hashi_vault/cov-units-pre2.12/coverage=units=ansible_stable-2.10=data/coverage=units=units=docker-default=python-3.9.xml...
  [2021-10-08T15:10:58.504Z] ['info'] Processing /home/runner/work/community.hashi_vault/community.hashi_vault/cov-units-pre2.12/coverage=units=ansible_stable-2.10=data/coverage=units=units=docker-default=python-3.7.xml...
  [2021-10-08T15:10:58.504Z] ['info'] Processing /home/runner/work/community.hashi_vault/community.hashi_vault/cov-units-pre2.12/coverage=units=ansible_stable-2.11=data/coverage=units=units=docker-default=python-3.5.xml...
  [2021-10-08T15:10:58.505Z] ['info'] Processing /home/runner/work/community.hashi_vault/community.hashi_vault/cov-units-pre2.12/coverage=units=ansible_stable-2.11=data/coverage=units=units=docker-default=python-3.6.xml...
  [2021-10-08T15:10:58.505Z] ['info'] Processing /home/runner/work/community.hashi_vault/community.hashi_vault/cov-units-pre2.12/coverage=units=ansible_stable-2.11=data/coverage=units=units=docker-default=python-3.8.xml...
  [2021-10-08T15:10:58.506Z] ['info'] Processing /home/runner/work/community.hashi_vault/community.hashi_vault/cov-units-pre2.12/coverage=units=ansible_stable-2.11=data/coverage=units=units=docker-default=python-2.6.xml...
  [2021-10-08T15:10:58.506Z] ['info'] Processing /home/runner/work/community.hashi_vault/community.hashi_vault/cov-units-pre2.12/coverage=units=ansible_stable-2.11=data/coverage=units=units=docker-default=python-2.7.xml...
  [2021-10-08T15:10:58.507Z] ['info'] Processing /home/runner/work/community.hashi_vault/community.hashi_vault/cov-units-pre2.12/coverage=units=ansible_stable-2.11=data/coverage=units=units=docker-default=python-3.9.xml...
  [2021-10-08T15:10:58.508Z] ['info'] Processing /home/runner/work/community.hashi_vault/community.hashi_vault/cov-units-pre2.12/coverage=units=ansible_stable-2.11=data/coverage=units=units=docker-default=python-3.7.xml...
  [2021-10-08T15:10:58.525Z] ['info'] Detected GitHub Actions as the CI provider.
  [2021-10-08T15:10:58.526Z] ['info'] Pinging Codecov: https://codecov.io/upload/v4?package=uploader-0.1.5&token=*******&branch=main&build=1320652759&build_url=https%3A%2F%2Fgithub.com%2Fansible-collections%2Fcommunity.hashi_vault%2Factions%2Fruns%2F1320652759&commit=61ba6ea7e69e11a1616287d5996a025da8482fd3&job=CI&pr=&service=github-actions&slug=ansible-collections%2Fcommunity.hashi_vault&name=&tag=&flags=env_docker-default%2Cunits&parent=
  [2021-10-08T15:10:58.837Z] ['error'] Error POSTing to https://codecov.io: 400 [ErrorDetail(string='Too many uploads to this commit.', code='invalid')]
  [2021-10-08T15:10:58.838Z] ['error'] There was an error running the uploader: Error uploading to https://codecov.io: Error: Bad Request

Expected Results

Success

Actual Results

400 error from codecov servers

Additional Information

Re-running CI did not help. There is no additional or unusual CI activity happening on this commit.

2

@tom are you able to help with this? My CI is still failing because of the issue and I have no insight into why.

3

Hi @briantist, the main issue is that you are, in fact, uploading too many builds for the same commit. This isn’t strictly documented as it rarely occurs.

Has your daily scheduled build run on the same commit for the past few days? Do you know about how many uploads you do per CI run?

4

There are a lot of uploads per run, because of the matrix of all the testing scenarios, plus using the flags feature, so that results in lots more reports. I don’t have a specific number but you could take a look at the codecov job in CI on a successful run.

It had definitely run on the same commit for several days; possibly longer. But that’s not new; when there are no new commits in the repo, the daily will keep running against the same commit. That’s never caused a problem before, even on commits that had more active development, and remained the HEAD of main for a much longer time. So this really came as a surprise.

As the repo gets more and more stable, I expect that will happen more over time…

  • What is the upper limit on number of uploads per commit?
  • Why have a strict maximum number of uploads vs. some kind of rate limiting (max uploads per Y time period)?
  • Please document this!
5

This is happening again: CI · ansible-collections/community.hashi_vault@a7c2a19 · GitHub

This commit is only 5 days old. Something has changed on the codecov side… how can I expect to work around this??

Reiterating these points:

  • What is the upper limit on number of uploads per commit?
  • Why have a strict maximum number of uploads vs. some kind of rate limiting (max uploads per Y time period)?
  • Please document this!
6

I’m still having this issue, would really appreciate some help!
Latest time this happened: CI · ansible-collections/community.hashi_vault@8d5d5e2 · GitHub

7

@briantist the commit above has 165 uploads while the previous one has 161 uploads. The upper limit is within 100-200 but is also dependent on upload size, and it’s in place due to our ability to process that number of uploads for a commit.

Unfortunately, we just aren’t able to support that many uploads per commit. I recommend not uploading coverage for your daily CI runs if your code hasn’t changed, because there shouldn’t be a change in coverage either.

1 Like
8

I’ve done two things on my end to address this:

  • reduced the number of flags (which reduces the number of uploads)
  • stopped doing coverage reports on daily runs, as suggested

Thanks @tom

1 Like
9

Hi @tom, I am having the same issue.

We are uploading 200+ reports per commit when we have a full rebuild of our monorepo.

Is there any way to increase this limit for us? Does the paid version support this increase?

1 Like
10

@dabbeg sorry for the delay here, can you open a new request for this? We don’t have any way to up the limit, free or paid. We might have to see if there’s a way to get under 200 reports.

11

Sorry to “crash” into the topic.

We started seeing this recently, and I’m in the process of adding more flags to the builds, so even more uploads.

However, I believe we are nowhere near 100 uploads, Codecov but I can’t be sure since that doesn’t load.

This is not welcoming to big projects, let alone, mono repo projects.

What are you planning to do to support these kinds of projects?

12

@s0undt3ch, do you mind opening a new topic on this? I’d also like to get some commits where you are seeing this

13

Two years later, we’re in kind of the same spot.

[2023-09-23T06:55:30.453Z] ['info'] Detected GitHub Actions as the CI provider.
[2023-09-23T06:55:30.664Z] ['info'] Pinging Codecov: https://codecov.io/upload/v4?package=uploader-0.6.2&token=*******&branch=hotfix%2Fcodecov&build=6275610333&build_url=https%3A%2F%2Fgithub.com%2Fsaltstack%2Fsalt%2Factions%2Fruns%2F6275610333&commit=4ae2351f0b8cb9498bccc7db37ed46364b6fdebc&job=CI&pr=65216&service=github-actions&slug=saltstack%2Fsalt&name=windows-2022.ci-test-onedir.integration&tag=&flags=salt%2Cwindows-2022%2Czeromq%2Cintegration&parent=55c55bc4b3ebc815f65f38f24bf31b1ba3bf22da
[2023-09-23T06:55:31.019Z] ['error'] There was an error running the uploader: Error uploading to https://codecov.io: Error: There was an error fetching the storage URL during POST: 400 - [ErrorDetail(string='Too many uploads to this commit.', code='invalid')]

image
image1747×286 78 KB

What is the limit?
Can we get a limit bump?