Coverage upload fails with 408

Support
1

Description

Coverage file upload fails with a 408 response. It’s also causing about 7 minutes to be added to our build times on CircleCI.

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  157k    0     0  100  157k      0    446  0:06:00  0:06:00 --:--:--     0
curl: (22) The requested URL returned error: 408 
    X> Failed to upload
==> Uploading to Codecov
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  157k  100   279  100  157k      9   5346  0:00:31  0:00:30  0:00:01    57
Warning: Transient problem: HTTP error Will retry in 2 seconds. 5 retries 
Warning: left.
100  157k  100   107  100  157k    233   342k --:--:-- --:--:-- --:--:--  341k
    View reports at <html><head>
CircleCI received exit code 0
2

Hi @robmaceachern, just need some more details here

  • What repo is this for?
  • What commit did this fail on?
  • What CI/CD is this for?
  • Can you post the entire output?
  • Can you run with the -v flag?
3

Repo: a private repo.
Commit: It’s been happening for many different commits for at least 10 days.
CI: CircleCI
Output: I’ll DM it to you.
-v flag: I can try soon.

4

Unfortunately the verbose output is too long to send via DM (I DM’d you the non-verbose output). Looks like an upload request is timing out.

<html><head>
<meta http-equiv="content-type" content="text/html;charset=utf-8">
<title>408 Request Timeout</title>
</head>
<body text=#000000 bgcolor=#ffffff>
<h1>Error: Request Timeout</h1>
<h2>Your client has taken too long to issue its request.</h2>
<h2></h2>
</body></html>
5

Hi Tom. Any advice on what we can do to resolve this 408 response issue?

6

@robmaceachern, I took a quick look, how large of a report are you uploading? You can use the -q argument to save down the file.

7

Based on the curl output it doesn’t look big (157k). If I understand the output, it looks like it’s 100% uploaded by the connection isn’t closing.

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  157k    0     0  100  157k      0   405k --:--:-- --:--:-- --:--:--  405k
100  157k    0     0  100  157k      0   113k  0:00:01  0:00:01 --:--:--  113k
100  157k    0     0  100  157k      0  67393  0:00:02  0:00:02 --:--:-- 67393
100  157k    0     0  100  157k      0  47496  0:00:03  0:00:03 --:--:-- 47496
100  157k    0     0  100  157k      0  36669  0:00:04  0:00:04 --:--:-- 36669
100  157k    0     0  100  157k      0  29862  0:00:05  0:00:05 --:--:--     0
100  157k    0     0  100  157k      0  25186  0:00:06  0:00:06 --:--:--     0
100  157k    0     0  100  157k      0  21777  0:00:07  0:00:07 --:--:--     0
100  157k    0     0  100  157k      0  19180  0:00:08  0:00:08 --:--:--     0

Seems to be the same problem as this user (Oct 6, 2020) Uploads fail intermittently · Issue #154 · codecov/codecov-bash · GitHub

9

@robmaceachern, would you be able to run the bash command with -U "-v" -A "-v"? I’d like to see the curl logs.

10

I’m seeing the same issue. Here is a build that has the -U “-v” -A “-v” flags.

and here is the relevant output

> Appending adjustments

    https://docs.codecov.io/docs/fixing-reports

    + Found adjustments

==> Gzipping contents

        144K	/tmp/codecov.YuWm5A.gz

==> Uploading reports

    url: https://codecov.io

    query: branch=mac-ci&commit=f9c68dd8f3acb90582ba3d9bd84dfc2897c74c94&build=837.1&build_url=&name=&tag=&slug=openkim%2Fkim-api&service=travis&flags=&pr=false&job=454979709&cmd_args=x,U,A

->  Pinging Codecov

https://codecov.io/upload/v4?package=bash-20201130-cc6d3fe&token=secret&branch=mac-ci&commit=f9c68dd8f3acb90582ba3d9bd84dfc2897c74c94&build=837.1&build_url=&name=&tag=&slug=openkim%2Fkim-api&service=travis&flags=&pr=false&job=454979709&cmd_args=x,U,A

*   Trying 35.201.106.201...

* TCP_NODELAY set

* Connected to codecov.io (35.201.106.201) port 443 (#0)

* ALPN, offering h2

* ALPN, offering http/1.1

* successfully set certificate verify locations:

*   CAfile: /etc/ssl/cert.pem

  CApath: none

* TLSv1.2 (OUT), TLS handshake, Client hello (1):

} [224 bytes data]

* TLSv1.2 (IN), TLS handshake, Server hello (2):

{ [96 bytes data]

* TLSv1.2 (IN), TLS handshake, Certificate (11):

{ [2466 bytes data]

* TLSv1.2 (IN), TLS handshake, Server key exchange (12):

{ [300 bytes data]

* TLSv1.2 (IN), TLS handshake, Server finished (14):

{ [4 bytes data]

* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):

} [37 bytes data]

* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):

} [1 bytes data]

* TLSv1.2 (OUT), TLS handshake, Finished (20):

} [16 bytes data]

* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):

{ [1 bytes data]

* TLSv1.2 (IN), TLS handshake, Finished (20):

{ [16 bytes data]

* SSL connection using TLSv1.2 / ECDHE-RSA-CHACHA20-POLY1305

* ALPN, server accepted to use h2

* Server certificate:

*  subject: CN=codecov.io

*  start date: Nov  4 03:00:58 2020 GMT

*  expire date: Feb  2 03:00:58 2021 GMT

*  subjectAltName: host "codecov.io" matched cert's "codecov.io"

*  issuer: C=US; O=Google Trust Services; CN=GTS CA 1D2

*  SSL certificate verify ok.

* Using HTTP2, server supports multi-use

* Connection state changed (HTTP/2 confirmed)

* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0

* Using Stream ID: 1 (easy handle 0x7fef8b00f600)

> POST /upload/v4?package=bash-20201130-cc6d3fe&token=&branch=mac-ci&commit=f9c68dd8f3acb90582ba3d9bd84dfc2897c74c94&build=837.1&build_url=&name=&tag=&slug=openkim%2Fkim-api&service=travis&flags=&pr=false&job=454979709&cmd_args=x,U,A HTTP/2

> Host: codecov.io

> User-Agent: curl/7.64.1

> Accept: */*

> X-Reduced-Redundancy: false

> X-Content-Type: application/x-gzip

> Content-Length: 0

> 

* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!

< HTTP/2 200 

< content-length: 560

< x-xss-protection: 1; mode=block

< content-security-policy: default-src 'self' https://widget.driftqa.com https://codecov-cdn.storage.googleapis.com;img-src 'self'  https://googletagmanager.com https://googleads.g.doubleclick.net https://www.google.com https://www.google-analytics.com http://www.google-analytics.com https://bitbucket-assetroot.s3.amazonaws.com https://avatar-management--avatars.us-west-2.prod.public.atl-paas.net https://codecov-cdn.storage.googleapis.com/ https://avatar-cdn.atlassian.com https://avatars.io https://bitbucket.org http://www.gravatar.com https://secure.gravatar.com https://avatars0.githubusercontent.com https://assets.gitlab-static.net https://gitlab.com *.stripe.com http://t.co http://p.adsymptotic.com https://*.linkedin.com data:; frame-src https://widget.drift.com https://www.googletagmanager.com *.stripe.com;connect-src 'self' https://cancel.baremetrics.com https://sentry.baremetrics.com https://sentry.baremetrics.io https://api.segment.io https://rs.fullstory.com *.stripe.com https://wdzsn5dlywj9.statuspage.io https://stats.g.doubleclick.net https://api.codecov.io; font-src 'self' https://codecov-cdn.storage.googleapis.com data: https://fonts.gstatic.com; style-src 'self' 'unsafe-inline' https://baremetrics-barecancel.baremetrics.com https://codecov-cdn.storage.googleapis.com https://fonts.googleapis.com data:; script-src 'self' 'unsafe-eval' https://js.driftt.com https://cdn.mxpnl.com https://baremetrics-barecancel.baremetrics.com https://www.googletagmanager.com https://www.googleadservices.com https://www.google-analytics.com http://www.google-analytics.com https://cdn.segment.com https://edge.fullstory.com https://ajax.googleapis.com *.stripe.com https://fullstory.com https://codecov-cdn.storage.googleapis.com https://ws.zoominfo.com https://snap.licdn.com http://static.ads-twitter.com https://analytics.twitter.com 'unsafe-inline';

< x-content-type-options: nosniff

< strict-transport-security: max-age=31536000; includeSubdomains; preload

< vary: X-PJAX, Accept-Encoding

< server: envoy

< x-ua-compatible: IE=Edge,chrome=1

< cache-control: no-cache

< date: Sat, 05 Dec 2020 17:45:38 GMT

< x-frame-options: deny

< access-control-allow-headers: Origin, Content-Type, Accept, X-User-Agent

< content-type: text/plain

< access-control-allow-origin: *

< x-envoy-upstream-service-time: 1039

< via: 1.1 google

< alt-svc: clear

< 

{ [560 bytes data]

* Connection #0 to host codecov.io left intact

* Closing connection 0

->  Uploading to

https://storage.googleapis.com/codecov/v4/raw/2020-12-05/256A1D69DFA7ACD5898DD6460ADFAF82/f9c68dd8f3acb90582ba3d9bd84dfc2897c74c94/5a99119a-f2b4-477e-8539-9bfd64a46d3c.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=GOOG1EQX6OZVJGHKK3633AAFGLBUCOOATRACRQRQF6HMSMLYUP6EAD6XSWAAY%2F20201205%2FUS%2Fs3%2Faws4_request&X-Amz-Date=20201205T174539Z&X-Amz-Expires=10&X-Amz-SignedHeaders=host&X-Amz-Signature=56d9340ba993444e8108b20b54ea5cc0f2f7aa60a992f2f8c57a47137fd18918

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current

                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 172.217.9.208...

* TCP_NODELAY set

* Connected to storage.googleapis.com (172.217.9.208) port 443 (#0)

* ALPN, offering h2

* ALPN, offering http/1.1

* successfully set certificate verify locations:

*   CAfile: /etc/ssl/cert.pem

  CApath: none

* TLSv1.2 (OUT), TLS handshake, Client hello (1):

} [236 bytes data]

* TLSv1.2 (IN), TLS handshake, Server hello (2):

{ [96 bytes data]

* TLSv1.2 (IN), TLS handshake, Certificate (11):

{ [2759 bytes data]

* TLSv1.2 (IN), TLS handshake, Server key exchange (12):

{ [116 bytes data]

* TLSv1.2 (IN), TLS handshake, Server finished (14):

{ [4 bytes data]

* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):

} [37 bytes data]

* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):

} [1 bytes data]

* TLSv1.2 (OUT), TLS handshake, Finished (20):

} [16 bytes data]

* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):

{ [1 bytes data]

* TLSv1.2 (IN), TLS handshake, Finished (20):

{ [16 bytes data]

* SSL connection using TLSv1.2 / ECDHE-ECDSA-CHACHA20-POLY1305

* ALPN, server accepted to use h2

* Server certificate:

*  subject: C=US; ST=California; L=Mountain View; O=Google LLC; CN=*.storage.googleapis.com

*  start date: Nov  3 07:37:41 2020 GMT

*  expire date: Jan 26 07:37:41 2021 GMT

*  subjectAltName: host "storage.googleapis.com" matched cert's "*.googleapis.com"

*  issuer: C=US; O=Google Trust Services; CN=GTS CA 1O1

*  SSL certificate verify ok.

* Using HTTP2, server supports multi-use

* Connection state changed (HTTP/2 confirmed)

* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0

* Using Stream ID: 1 (easy handle 0x7fb7e680f600)

> PUT /codecov/v4/raw/2020-12-05/256A1D69DFA7ACD5898DD6460ADFAF82/f9c68dd8f3acb90582ba3d9bd84dfc2897c74c94/5a99119a-f2b4-477e-8539-9bfd64a46d3c.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=GOOG1EQX6OZVJGHKK3633AAFGLBUCOOATRACRQRQF6HMSMLYUP6EAD6XSWAAY%2F20201205%2FUS%2Fs3%2Faws4_request&X-Amz-Date=20201205T174539Z&X-Amz-Expires=10&X-Amz-SignedHeaders=host&X-Amz-Signature=56d9340ba993444e8108b20b54ea5cc0f2f7aa60a992f2f8c57a47137fd18918 HTTP/2

> Host: storage.googleapis.com

> User-Agent: curl/7.64.1

> Accept: */*

> Content-Type: application/x-gzip

> Content-Encoding: gzip

> Content-Length: 144133

> 

* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!

} [65536 bytes data]

* We are completely uploaded and fine

100  140k    0     0  100  140k      0    400  0:06:00  0:05:59  0:00:01     0* The requested URL returned error: 408 

* stopped the pause stream!

100  140k    0     0  100  140k      0    400  0:06:00  0:06:00 --:--:--     0

* Connection #0 to host storage.googleapis.com left intact

curl: (22) The requested URL returned error: 408 

* Closing connection 0

    X> Failed to upload

==> Uploading to Codecov

Note: Unnecessary use of -X or --request, POST is already inferred.

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current

                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 35.201.106.201...

* TCP_NODELAY set

* Connected to codecov.io (35.201.106.201) port 443 (#0)

* ALPN, offering h2

* ALPN, offering http/1.1

* successfully set certificate verify locations:

*   CAfile: /etc/ssl/cert.pem

  CApath: none

* TLSv1.2 (OUT), TLS handshake, Client hello (1):

} [224 bytes data]

* TLSv1.2 (IN), TLS handshake, Server hello (2):

{ [96 bytes data]

* TLSv1.2 (IN), TLS handshake, Certificate (11):

{ [2466 bytes data]

* TLSv1.2 (IN), TLS handshake, Server key exchange (12):

{ [300 bytes data]

* TLSv1.2 (IN), TLS handshake, Server finished (14):

{ [4 bytes data]

* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):

} [37 bytes data]

* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):

} [1 bytes data]

* TLSv1.2 (OUT), TLS handshake, Finished (20):

} [16 bytes data]

* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):

{ [1 bytes data]

* TLSv1.2 (IN), TLS handshake, Finished (20):

{ [16 bytes data]

* SSL connection using TLSv1.2 / ECDHE-RSA-CHACHA20-POLY1305

* ALPN, server accepted to use h2

* Server certificate:

*  subject: CN=codecov.io

*  start date: Nov  4 03:00:58 2020 GMT

*  expire date: Feb  2 03:00:58 2021 GMT

*  subjectAltName: host "codecov.io" matched cert's "codecov.io"

*  issuer: C=US; O=Google Trust Services; CN=GTS CA 1D2

*  SSL certificate verify ok.

* Using HTTP2, server supports multi-use

* Connection state changed (HTTP/2 confirmed)

* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0

* Using Stream ID: 1 (easy handle 0x7fe10400d800)

> POST /upload/v2?package=bash-20201130-cc6d3fe&token=&branch=mac-ci&commit=f9c68dd8f3acb90582ba3d9bd84dfc2897c74c94&build=837.1&build_url=&name=&tag=&slug=openkim%2Fkim-api&service=travis&flags=&pr=false&job=454979709&cmd_args=x,U,A&attempt=woff|eot|otf|gif|png|jpg|jpeg|psd|ptt|pptx|numbers|pages|md|txt|xlsx|docx|doc|pdf|csv|yml|yaml|.gitignore|html HTTP/2

> Host: codecov.io

> User-Agent: curl/7.64.1

> Content-Type: text/plain

> Content-Encoding: gzip

> X-Content-Encoding: gzip

> Accept: text/plain

> Content-Length: 144133

> 

* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!

} [65536 bytes data]

* We are completely uploaded and fine

100  140k    0     0  100  140k      0   4861  0:00:29  0:00:29 --:--:--     0< HTTP/2 408 

< content-type: text/html; charset=UTF-8

< referrer-policy: no-referrer

< content-length: 279

< date: Sat, 05 Dec 2020 17:52:09 GMT

< alt-svc: clear

< 

{ [279 bytes data]

100  141k  100   279  100  140k      9   4777  0:00:31  0:00:30  0:00:01    61

* Connection #0 to host codecov.io left intact

Warning: Transient problem: HTTP error Will retry in 2 seconds. 5 retries 

Warning: left.

* 30 bytes stray data read before trying h2 connection

* Found bundle for host codecov.io: 0x7fe10351ba40 [can multiplex]

* Re-using existing connection! (#0) with host codecov.io

* Connected to codecov.io (35.201.106.201) port 443 (#0)

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Using Stream ID: 3 (easy handle 0x7fe10400d800)

> POST /upload/v2?package=bash-20201130-cc6d3fe&token=&branch=mac-ci&commit=f9c68dd8f3acb90582ba3d9bd84dfc2897c74c94&build=837.1&build_url=&name=&tag=&slug=openkim%2Fkim-api&service=travis&flags=&pr=false&job=454979709&cmd_args=x,U,A&attempt=woff|eot|otf|gif|png|jpg|jpeg|psd|ptt|pptx|numbers|pages|md|txt|xlsx|docx|doc|pdf|csv|yml|yaml|.gitignore|html HTTP/2

> Host: codecov.io

> User-Agent: curl/7.64.1

> Content-Type: text/plain

> Content-Encoding: gzip

> X-Content-Encoding: gzip

> Accept: text/plain

> Content-Length: 144133

> 

} [65536 bytes data]

* We are completely uploaded and fine

100  140k    0     0  100  140k      0   223k --:--:-- --:--:-- --:--:--  223k< HTTP/2 200 

< content-length: 99

< x-xss-protection: 1; mode=block

< content-security-policy: default-src 'self' https://widget.driftqa.com https://codecov-cdn.storage.googleapis.com;img-src 'self'  https://googletagmanager.com https://googleads.g.doubleclick.net https://www.google.com https://www.google-analytics.com http://www.google-analytics.com https://bitbucket-assetroot.s3.amazonaws.com https://avatar-management--avatars.us-west-2.prod.public.atl-paas.net https://codecov-cdn.storage.googleapis.com/ https://avatar-cdn.atlassian.com https://avatars.io https://bitbucket.org http://www.gravatar.com https://secure.gravatar.com https://avatars0.githubusercontent.com https://assets.gitlab-static.net https://gitlab.com *.stripe.com http://t.co http://p.adsymptotic.com https://*.linkedin.com data:; frame-src https://widget.drift.com https://www.googletagmanager.com *.stripe.com;connect-src 'self' https://cancel.baremetrics.com https://sentry.baremetrics.com https://sentry.baremetrics.io https://api.segment.io https://rs.fullstory.com *.stripe.com https://wdzsn5dlywj9.statuspage.io https://stats.g.doubleclick.net https://api.codecov.io; font-src 'self' https://codecov-cdn.storage.googleapis.com data: https://fonts.gstatic.com; style-src 'self' 'unsafe-inline' https://baremetrics-barecancel.baremetrics.com https://codecov-cdn.storage.googleapis.com https://fonts.googleapis.com data:; script-src 'self' 'unsafe-eval' https://js.driftt.com https://cdn.mxpnl.com https://baremetrics-barecancel.baremetrics.com https://www.googletagmanager.com https://www.googleadservices.com https://www.google-analytics.com http://www.google-analytics.com https://cdn.segment.com https://edge.fullstory.com https://ajax.googleapis.com *.stripe.com https://fullstory.com https://codecov-cdn.storage.googleapis.com https://ws.zoominfo.com https://snap.licdn.com http://static.ads-twitter.com https://analytics.twitter.com 'unsafe-inline';

< x-content-type-options: nosniff

< strict-transport-security: max-age=31536000; includeSubdomains; preload

< vary: X-PJAX, Accept-Encoding

< server: envoy

< x-ua-compatible: IE=Edge,chrome=1

< cache-control: no-cache

< date: Sat, 05 Dec 2020 17:52:12 GMT

< x-frame-options: deny

< access-control-allow-headers: Origin, Content-Type, Accept, X-User-Agent

< content-type: text/plain

< access-control-allow-origin: *

< x-envoy-upstream-service-time: 539

< via: 1.1 google

< alt-svc: clear

< 

{ [99 bytes data]

100  140k  100    99  100  140k    156   223k --:--:-- --:--:-- --:--:--  222k

* Connection #0 to host codecov.io left intact

* Closing connection 0

    View reports at <html><head>
11

Hi @ellio167 and @robmaceachern, apologies this got lost here. Are you still experiencing this issue?

12

Hi @tom,

Thanks for the reply. I think I resolved the issue. The cmake specification of the coverage build flags were not right. I’ve also moved the coverage testing from a macOS job to a centos job. Things seem to be working now (although with the recent travis-ci changes, we’re currently out of credits, so our ci is stalled and I’m not positive everything is back in action.) If there is any continuing problem, I’ll post again.

Cheers,
Ryan

1 Like