Intermittent timeout contacting codecov.io from GitHub Actions

[jsiirola]

Description

We have been seeing intermittent timeouts contacting codecov.io from GitHub Actions. This is likely related to other problems (e.g., Sporadic timeouts when uploading reports with Ruby uploader). Our build harness is set up to download the codecov bash uploader and then run it. We have logic to retry both the download (after a delay) and the upload (again, after a delay) in the event of a failure.

The behavior we see is that on some builds, curl is unable to connect to codecov.io to download the bash uploader. Every attempt to download the bash uploader fails on connection timeout (we generally retry 3-5 times). It is not correlated with a specific build and appears to randomly occur for Windows and Linux (but rarely - if ever - for OSX) build hosts. Of out ~20 jobs in a build, generally only 1-2 of them exhibit this problem. For examples:

• https://github.com/Pyomo/pyomo/runs/823826598?check_suite_focus=true

• https://github.com/Pyomo/pyomo/runs/828370294?check_suite_focus=true

• https://github.com/Pyomo/pyomo/runs/827741680?check_suite_focus=true

• Implement Github Actions by mrmundt · Pull Request #109 · PyUtilib/pyutilib · GitHub

We added logic to cache an older copy of the bash uploader in the event that downloading a fresh copy fails. For that scenario, when the download fails (after all retries) and we fall back on the cached uploader, we see the same problem uploading: that is, all attempts to connect with codecov.io result in connection timeout.

I do not believe that it is due to network connectivity of the build host, as the job is able to successfully download source code from Github, and our build harness uses curl to explicitly download additional data from non-github.com network resources.

Overall, it feels like codecov.io is rejecting / ignoring connections from a subset of GHA build hosts.

Repository

GitHub - Pyomo/pyomo: An object-oriented algebraic modeling language in...

GitHub - PyUtilib/pyutilib: A collection of general Python utilities,...

CI/CD

GitHub Actions

Uploader

We are using the current BASH uploader.

Commit SHAs

N/A (all)

Example connection failures:

• https://github.com/Pyomo/pyomo/runs/823826598?check_suite_focus=true

• https://github.com/Pyomo/pyomo/runs/828370294?check_suite_focus=true

• https://github.com/Pyomo/pyomo/runs/827741680?check_suite_focus=true

• Implement Github Actions by mrmundt · Pull Request #109 · PyUtilib/pyutilib · GitHub

Codecov YAML

N/A

Codecov Output

2020-07-01T18:37:56.0524155Z Downloading current codecov script (attempt 1)
2020-07-01T18:37:56.2152958Z   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
2020-07-01T18:37:56.2154129Z                                  Dload  Upload   Total   Spent    Left  Speed
2020-07-01T18:37:56.2154524Z 
2020-07-01T18:37:57.4239385Z   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
2020-07-01T18:37:58.4242231Z   0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
2020-07-01T18:37:59.4244127Z   0     0    0     0    0     0      0      0 --:--:--  0:00:02 --:--:--     0
2020-07-01T18:38:00.4247088Z   0     0    0     0    0     0      0      0 --:--:--  0:00:03 --:--:--     0
2020-07-01T18:38:01.4250595Z   0     0    0     0    0     0      0      0 --:--:--  0:00:04 --:--:--     0
2020-07-01T18:38:02.4258930Z   0     0    0     0    0     0      0      0 --:--:--  0:00:05 --:--:--     0
2020-07-01T18:38:03.4264443Z   0     0    0     0    0     0      0      0 --:--:--  0:00:06 --:--:--     0
2020-07-01T18:38:04.4274383Z   0     0    0     0    0     0      0      0 --:--:--  0:00:07 --:--:--     0
2020-07-01T18:38:05.5089839Z   0     0    0     0    0     0      0      0 --:--:--  0:00:08 --:--:--     0
2020-07-01T18:38:06.4289977Z   0     0    0     0    0     0      0      0 --:--:--  0:00:09 --:--:--     0
2020-07-01T18:38:07.4294708Z   0     0    0     0    0     0      0      0 --:--:--  0:00:10 --:--:--     0
2020-07-01T18:38:08.4302406Z   0     0    0     0    0     0      0      0 --:--:--  0:00:11 --:--:--     0
2020-07-01T18:38:09.4304072Z   0     0    0     0    0     0      0      0 --:--:--  0:00:12 --:--:--     0
2020-07-01T18:38:10.4304596Z   0     0    0     0    0     0      0      0 --:--:--  0:00:13 --:--:--     0
2020-07-01T18:38:11.4315597Z   0     0    0     0    0     0      0      0 --:--:--  0:00:14 --:--:--     0
2020-07-01T18:38:12.4320458Z   0     0    0     0    0     0      0      0 --:--:--  0:00:15 --:--:--     0
2020-07-01T18:38:13.4322234Z   0     0    0     0    0     0      0      0 --:--:--  0:00:16 --:--:--     0
2020-07-01T18:38:14.4323697Z   0     0    0     0    0     0      0      0 --:--:--  0:00:17 --:--:--     0
2020-07-01T18:38:15.4325464Z   0     0    0     0    0     0      0      0 --:--:--  0:00:18 --:--:--     0
2020-07-01T18:38:16.4326701Z   0     0    0     0    0     0      0      0 --:--:--  0:00:19 --:--:--     0
2020-07-01T18:38:17.4331263Z   0     0    0     0    0     0      0      0 --:--:--  0:00:20 --:--:--     0
2020-07-01T18:38:17.4365210Z   0     0    0     0    0     0      0      0 --:--:--  0:00:21 --:--:--     0
2020-07-01T18:38:17.4365873Z curl: (28) Failed to connect to codecov.io port 443: Timed out
2020-07-01T18:38:17.4416634Z Pausing 49 seconds before re-attempting download
2020-07-01T18:39:06.4742599Z Downloading current codecov script (attempt 2)
2020-07-01T18:39:06.5071047Z   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
2020-07-01T18:39:06.5072146Z                                  Dload  Upload   Total   Spent    Left  Speed
2020-07-01T18:39:06.5072568Z 
2020-07-01T18:39:07.7124465Z   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
2020-07-01T18:39:08.7125537Z   0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
2020-07-01T18:39:09.7134815Z   0     0    0     0    0     0      0      0 --:--:--  0:00:02 --:--:--     0
2020-07-01T18:39:10.7134530Z   0     0    0     0    0     0      0      0 --:--:--  0:00:03 --:--:--     0
2020-07-01T18:39:11.7141903Z   0     0    0     0    0     0      0      0 --:--:--  0:00:04 --:--:--     0
2020-07-01T18:39:12.7150662Z   0     0    0     0    0     0      0      0 --:--:--  0:00:05 --:--:--     0
2020-07-01T18:39:14.5214359Z   0     0    0     0    0     0      0      0 --:--:--  0:00:06 --:--:--     0
2020-07-01T18:39:14.7157123Z   0     0    0     0    0     0      0      0 --:--:--  0:00:07 --:--:--     0
2020-07-01T18:39:15.7158174Z   0     0    0     0    0     0      0      0 --:--:--  0:00:08 --:--:--     0
2020-07-01T18:39:16.7159233Z   0     0    0     0    0     0      0      0 --:--:--  0:00:09 --:--:--     0
2020-07-01T18:39:17.7163543Z   0     0    0     0    0     0      0      0 --:--:--  0:00:10 --:--:--     0
2020-07-01T18:39:18.7170551Z   0     0    0     0    0     0      0      0 --:--:--  0:00:11 --:--:--     0
2020-07-01T18:39:19.7174222Z   0     0    0     0    0     0      0      0 --:--:--  0:00:12 --:--:--     0
2020-07-01T18:39:20.7178899Z   0     0    0     0    0     0      0      0 --:--:--  0:00:13 --:--:--     0
2020-07-01T18:39:21.7182347Z   0     0    0     0    0     0      0      0 --:--:--  0:00:14 --:--:--     0
2020-07-01T18:39:22.7183572Z   0     0    0     0    0     0      0      0 --:--:--  0:00:15 --:--:--     0
2020-07-01T18:39:23.7191897Z   0     0    0     0    0     0      0      0 --:--:--  0:00:16 --:--:--     0
2020-07-01T18:39:24.7192301Z   0     0    0     0    0     0      0      0 --:--:--  0:00:17 --:--:--     0
2020-07-01T18:39:25.7196263Z   0     0    0     0    0     0      0      0 --:--:--  0:00:18 --:--:--     0
2020-07-01T18:39:26.7198046Z   0     0    0     0    0     0      0      0 --:--:--  0:00:19 --:--:--     0
2020-07-01T18:39:27.7205186Z   0     0    0     0    0     0      0      0 --:--:--  0:00:20 --:--:--     0
2020-07-01T18:39:27.7232757Z   0     0    0     0    0     0      0      0 --:--:--  0:00:21 --:--:--     0
2020-07-01T18:39:27.7233260Z curl: (28) Failed to connect to codecov.io port 443: Timed out
2020-07-01T18:39:27.7267694Z Pausing 30 seconds before re-attempting download
2020-07-01T18:39:57.7465325Z Downloading current codecov script (attempt 3)
2020-07-01T18:39:57.7783831Z   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
2020-07-01T18:39:57.7785046Z                                  Dload  Upload   Total   Spent    Left  Speed
2020-07-01T18:39:57.7785398Z 
2020-07-01T18:39:57.9842589Z   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
2020-07-01T18:39:58.9846660Z   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
2020-07-01T18:39:59.9852642Z   0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
2020-07-01T18:40:00.9855405Z   0     0    0     0    0     0      0      0 --:--:--  0:00:02 --:--:--     0
2020-07-01T18:40:01.9857322Z   0     0    0     0    0     0      0      0 --:--:--  0:00:03 --:--:--     0
2020-07-01T18:40:02.9858903Z   0     0    0     0    0     0      0      0 --:--:--  0:00:04 --:--:--     0
2020-07-01T18:40:03.9862212Z   0     0    0     0    0     0      0      0 --:--:--  0:00:05 --:--:--     0
2020-07-01T18:40:04.9872815Z   0     0    0     0    0     0      0      0 --:--:--  0:00:06 --:--:--     0
2020-07-01T18:40:05.9874815Z   0     0    0     0    0     0      0      0 --:--:--  0:00:07 --:--:--     0
2020-07-01T18:40:06.9874713Z   0     0    0     0    0     0      0      0 --:--:--  0:00:08 --:--:--     0
2020-07-01T18:40:07.9880761Z   0     0    0     0    0     0      0      0 --:--:--  0:00:09 --:--:--     0
2020-07-01T18:40:08.9881703Z   0     0    0     0    0     0      0      0 --:--:--  0:00:10 --:--:--     0
2020-07-01T18:40:09.9882181Z   0     0    0     0    0     0      0      0 --:--:--  0:00:11 --:--:--     0
2020-07-01T18:40:10.9892427Z   0     0    0     0    0     0      0      0 --:--:--  0:00:12 --:--:--     0
2020-07-01T18:40:11.9884401Z   0     0    0     0    0     0      0      0 --:--:--  0:00:13 --:--:--     0
2020-07-01T18:40:12.9887310Z   0     0    0     0    0     0      0      0 --:--:--  0:00:14 --:--:--     0
2020-07-01T18:40:13.9889572Z   0     0    0     0    0     0      0      0 --:--:--  0:00:15 --:--:--     0
2020-07-01T18:40:14.9900547Z   0     0    0     0    0     0      0      0 --:--:--  0:00:16 --:--:--     0
2020-07-01T18:40:15.9902933Z   0     0    0     0    0     0      0      0 --:--:--  0:00:17 --:--:--     0
2020-07-01T18:40:17.5268703Z   0     0    0     0    0     0      0      0 --:--:--  0:00:18 --:--:--     0
2020-07-01T18:40:17.9914249Z   0     0    0     0    0     0      0      0 --:--:--  0:00:19 --:--:--     0
2020-07-01T18:40:18.9922777Z   0     0    0     0    0     0      0      0 --:--:--  0:00:20 --:--:--     0
2020-07-01T18:40:18.9957592Z   0     0    0     0    0     0      0      0 --:--:--  0:00:21 --:--:--     0
2020-07-01T18:40:18.9958281Z curl: (28) Failed to connect to codecov.io port 443: Timed out
2020-07-01T18:40:18.9996961Z Pausing 52 seconds before re-attempting download
2020-07-01T18:41:11.0200036Z Uploading coverage to codecov (attempt 1)
2020-07-01T18:41:12.1232598Z 
2020-07-01T18:41:12.1233249Z   _____          _
2020-07-01T18:41:12.1233553Z  / ____|        | |
2020-07-01T18:41:12.1233802Z | |     ___   __| | ___  ___ _____   __
2020-07-01T18:41:12.1234051Z | |    / _ \ / _` |/ _ \/ __/ _ \ \ / /
2020-07-01T18:41:12.1234314Z | |___| (_) | (_| |  __/ (_| (_) \ V /
2020-07-01T18:41:12.1234555Z  \_____\___/ \__,_|\___|\___\___/ \_/
2020-07-01T18:41:12.1234799Z                               Bash-20200629-ffaf297
2020-07-01T18:41:12.1234992Z 
2020-07-01T18:41:12.1235135Z 
2020-07-01T18:41:12.1255669Z e[0;90m==>e[0m GitHub Actions detected.
2020-07-01T18:41:12.2078234Z     e[0;90mproject root:e[0m D:/a/pyomo/pyomo
2020-07-01T18:41:12.5511017Z     e[0;90mYaml found at:e[0m .codecov.yml
2020-07-01T18:41:13.4273386Z     e[0;90m->e[0m Found 1 reports
2020-07-01T18:41:13.4274213Z e[0;90m==>e[0m Detecting git/mercurial file structure
2020-07-01T18:41:13.6072968Z e[0;90m==>e[0m Reading reports
2020-07-01T18:41:13.7173275Z     e[0;32m+e[0m coverage.xml e[0;90mbytes=3291415e[0m
2020-07-01T18:41:13.7559798Z e[0;90m==>e[0m Appending adjustments
2020-07-01T18:41:13.7560495Z     e[0;36mhttps://docs.codecov.io/docs/fixing-reportse[0m
2020-07-01T18:45:13.0914151Z     e[0;32m+e[0m Found adjustments
2020-07-01T18:45:13.1957244Z e[0;90m==>e[0m Gzipping contents
2020-07-01T18:45:13.9069056Z e[0;90m==>e[0m Uploading reports
2020-07-01T18:45:13.9069978Z     e[0;90murl:e[0m https://codecov.io
2020-07-01T18:45:13.9070368Z     e[0;90mquery:e[0m branch=move-termination-methods-to-environ&commit=11c0a0a90c14e146e1da7be314a4023253bfa981&build=154425195&build_url=http%3A%2F%2Fgithub.com%2FPyomo%2Fpyomo%2Factions%2Fruns%2F154425195&name=win%2F3.8&tag=&slug=Pyomo%2Fpyomo&service=github-actions&flags=&pr=1540&job=
2020-07-01T18:45:14.0481422Z e[0;90m==>e[0m Uploading to Codecov
2020-07-01T18:45:15.1395668Z curl: (3) URL using bad/illegal format or missing URL
2020-07-01T18:45:15.1396219Z curl: (3) URL using bad/illegal format or missing URL
2020-07-01T18:45:15.1396779Z   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
2020-07-01T18:45:15.1397335Z                                  Dload  Upload   Total   Spent    Left  Speed
2020-07-01T18:45:15.1397707Z 
2020-07-01T18:45:16.3448840Z   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
2020-07-01T18:45:17.3451869Z   0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
2020-07-01T18:45:18.3457391Z   0     0    0     0    0     0      0      0 --:--:--  0:00:02 --:--:--     0
2020-07-01T18:45:19.3466123Z   0     0    0     0    0     0      0      0 --:--:--  0:00:03 --:--:--     0
2020-07-01T18:45:20.3467281Z   0     0    0     0    0     0      0      0 --:--:--  0:00:04 --:--:--     0
2020-07-01T18:45:21.3467706Z   0     0    0     0    0     0      0      0 --:--:--  0:00:05 --:--:--     0
2020-07-01T18:45:22.3469411Z   0     0    0     0    0     0      0      0 --:--:--  0:00:06 --:--:--     0
2020-07-01T18:45:23.3473983Z   0     0    0     0    0     0      0      0 --:--:--  0:00:07 --:--:--     0
2020-07-01T18:45:24.3475834Z   0     0    0     0    0     0      0      0 --:--:--  0:00:08 --:--:--     0
2020-07-01T18:45:25.3478140Z   0     0    0     0    0     0      0      0 --:--:--  0:00:09 --:--:--     0
2020-07-01T18:45:26.3478813Z   0     0    0     0    0     0      0      0 --:--:--  0:00:10 --:--:--     0
2020-07-01T18:45:27.3482906Z   0     0    0     0    0     0      0      0 --:--:--  0:00:11 --:--:--     0
2020-07-01T18:45:28.3490416Z   0     0    0     0    0     0      0      0 --:--:--  0:00:12 --:--:--     0
2020-07-01T18:45:29.3494505Z   0     0    0     0    0     0      0      0 --:--:--  0:00:13 --:--:--     0
2020-07-01T18:45:30.3504830Z   0     0    0     0    0     0      0      0 --:--:--  0:00:14 --:--:--     0
2020-07-01T18:45:31.3506999Z   0     0    0     0    0     0      0      0 --:--:--  0:00:15 --:--:--     0
2020-07-01T18:45:32.3509036Z   0     0    0     0    0     0      0      0 --:--:--  0:00:16 --:--:--     0
2020-07-01T18:45:33.3511159Z   0     0    0     0    0     0      0      0 --:--:--  0:00:17 --:--:--     0
2020-07-01T18:45:34.3521132Z   0     0    0     0    0     0      0      0 --:--:--  0:00:18 --:--:--     0
2020-07-01T18:45:35.3523022Z   0     0    0     0    0     0      0      0 --:--:--  0:00:19 --:--:--     0
2020-07-01T18:45:36.3529567Z   0     0    0     0    0     0      0      0 --:--:--  0:00:20 --:--:--     0
2020-07-01T18:45:36.3561720Z   0     0    0     0    0     0      0      0 --:--:--  0:00:21 --:--:--     0
2020-07-01T18:45:36.3562304Z curl: (28) Failed to connect to codecov.io port 443: Timed out
2020-07-01T18:45:36.4692582Z     e[0;90m->e[0m Sleeping for 30s and trying again...
2020-07-01T18:46:06.5023271Z d:\a\pyomo\pyomo/cache/download/codecov.sh: line 1721: [: 0+1: integer expression expected
2020-07-01T18:46:06.5023702Z     e[0;31mX> Failed to upload coverage reportse[0m
2020-07-01T18:46:06.5035662Z Pausing 43 seconds before re-attempting upload
2020-07-01T18:46:49.5244306Z Uploading coverage to codecov (attempt 2)
2020-07-01T18:46:50.6236924Z 
2020-07-01T18:46:50.6237657Z   _____          _
2020-07-01T18:46:50.6237955Z  / ____|        | |
2020-07-01T18:46:50.6238375Z | |     ___   __| | ___  ___ _____   __
2020-07-01T18:46:50.6238661Z | |    / _ \ / _` |/ _ \/ __/ _ \ \ / /
2020-07-01T18:46:50.6238907Z | |___| (_) | (_| |  __/ (_| (_) \ V /
2020-07-01T18:46:50.6239145Z  \_____\___/ \__,_|\___|\___\___/ \_/
2020-07-01T18:46:50.6239391Z                               Bash-20200629-ffaf297
2020-07-01T18:46:50.6239587Z 
2020-07-01T18:46:50.6239752Z 
2020-07-01T18:46:50.6254752Z e[0;90m==>e[0m GitHub Actions detected.
2020-07-01T18:46:50.7067273Z     e[0;90mproject root:e[0m D:/a/pyomo/pyomo
2020-07-01T18:46:50.9337151Z     e[0;90mYaml found at:e[0m .codecov.yml
2020-07-01T18:46:51.4138405Z     e[0;90m->e[0m Found 1 reports
2020-07-01T18:46:51.4139128Z e[0;90m==>e[0m Detecting git/mercurial file structure
2020-07-01T18:46:51.5737450Z e[0;90m==>e[0m Reading reports
2020-07-01T18:46:51.6753889Z     e[0;32m+e[0m coverage.xml e[0;90mbytes=3291415e[0m
2020-07-01T18:46:51.7115755Z e[0;90m==>e[0m Appending adjustments
2020-07-01T18:46:51.7116454Z     e[0;36mhttps://docs.codecov.io/docs/fixing-reportse[0m
2020-07-01T18:50:50.7260693Z     e[0;32m+e[0m Found adjustments
2020-07-01T18:50:50.8200802Z e[0;90m==>e[0m Gzipping contents
2020-07-01T18:50:51.5542251Z e[0;90m==>e[0m Uploading reports
2020-07-01T18:50:51.5553725Z     e[0;90murl:e[0m https://codecov.io
2020-07-01T18:50:51.5554624Z     e[0;90mquery:e[0m branch=move-termination-methods-to-environ&commit=11c0a0a90c14e146e1da7be314a4023253bfa981&build=154425195&build_url=http%3A%2F%2Fgithub.com%2FPyomo%2Fpyomo%2Factions%2Fruns%2F154425195&name=win%2F3.8&tag=&slug=Pyomo%2Fpyomo&service=github-actions&flags=&pr=1540&job=
2020-07-01T18:50:51.7095003Z e[0;90m==>e[0m Uploading to Codecov
2020-07-01T18:50:52.8055054Z curl: (3) URL using bad/illegal format or missing URL
2020-07-01T18:50:52.8056236Z curl: (3) URL using bad/illegal format or missing URL
2020-07-01T18:50:52.8056846Z   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
2020-07-01T18:50:52.8057677Z                                  Dload  Upload   Total   Spent    Left  Speed
2020-07-01T18:50:52.8058052Z 
2020-07-01T18:50:53.0096461Z   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
2020-07-01T18:50:54.0096975Z   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
2020-07-01T18:50:55.0107215Z   0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
2020-07-01T18:50:56.0112098Z   0     0    0     0    0     0      0      0 --:--:--  0:00:02 --:--:--     0
2020-07-01T18:50:57.0120518Z   0     0    0     0    0     0      0      0 --:--:--  0:00:03 --:--:--     0
2020-07-01T18:50:58.0124125Z   0     0    0     0    0     0      0      0 --:--:--  0:00:04 --:--:--     0
2020-07-01T18:50:59.0125907Z   0     0    0     0    0     0      0      0 --:--:--  0:00:05 --:--:--     0
2020-07-01T18:51:00.0128843Z   0     0    0     0    0     0      0      0 --:--:--  0:00:06 --:--:--     0
2020-07-01T18:51:01.0131343Z   0     0    0     0    0     0      0      0 --:--:--  0:00:07 --:--:--     0
2020-07-01T18:51:02.0133893Z   0     0    0     0    0     0      0      0 --:--:--  0:00:08 --:--:--     0
2020-07-01T18:51:03.0134218Z   0     0    0     0    0     0      0      0 --:--:--  0:00:09 --:--:--     0
2020-07-01T18:51:04.0136948Z   0     0    0     0    0     0      0      0 --:--:--  0:00:10 --:--:--     0
2020-07-01T18:51:05.0139787Z   0     0    0     0    0     0      0      0 --:--:--  0:00:11 --:--:--     0
2020-07-01T18:51:06.0142445Z   0     0    0     0    0     0      0      0 --:--:--  0:00:12 --:--:--     0
2020-07-01T18:51:07.0150818Z   0     0    0     0    0     0      0      0 --:--:--  0:00:13 --:--:--     0
2020-07-01T18:51:08.0159619Z   0     0    0     0    0     0      0      0 --:--:--  0:00:14 --:--:--     0
2020-07-01T18:51:09.0161746Z   0     0    0     0    0     0      0      0 --:--:--  0:00:15 --:--:--     0
2020-07-01T18:51:10.0167304Z   0     0    0     0    0     0      0      0 --:--:--  0:00:16 --:--:--     0
2020-07-01T18:51:11.0172021Z   0     0    0     0    0     0      0      0 --:--:--  0:00:17 --:--:--     0
2020-07-01T18:51:12.0179082Z   0     0    0     0    0     0      0      0 --:--:--  0:00:18 --:--:--     0
2020-07-01T18:51:13.0182334Z   0     0    0     0    0     0      0      0 --:--:--  0:00:19 --:--:--     0
2020-07-01T18:51:14.0183815Z   0     0    0     0    0     0      0      0 --:--:--  0:00:20 --:--:--     0
2020-07-01T18:51:14.0206364Z   0     0    0     0    0     0      0      0 --:--:--  0:00:21 --:--:--     0
2020-07-01T18:51:14.0207148Z curl: (28) Failed to connect to codecov.io port 443: Timed out
2020-07-01T18:51:14.1319550Z     e[0;90m->e[0m Sleeping for 30s and trying again...
2020-07-01T18:51:44.1621533Z d:\a\pyomo\pyomo/cache/download/codecov.sh: line 1721: [: 0+1: integer expression expected
2020-07-01T18:51:44.1621798Z     e[0;31mX> Failed to upload coverage reportse[0m
2020-07-01T18:51:44.1638592Z Pausing 43 seconds before re-attempting upload
2020-07-01T18:52:27.1937635Z Uploading coverage to codecov (attempt 3)
2020-07-01T18:52:28.2928162Z 
2020-07-01T18:52:28.2929166Z   _____          _
2020-07-01T18:52:28.2929453Z  / ____|        | |
2020-07-01T18:52:28.2929727Z | |     ___   __| | ___  ___ _____   __
2020-07-01T18:52:28.2929981Z | |    / _ \ / _` |/ _ \/ __/ _ \ \ / /
2020-07-01T18:52:28.2930241Z | |___| (_) | (_| |  __/ (_| (_) \ V /
2020-07-01T18:52:28.2930493Z  \_____\___/ \__,_|\___|\___\___/ \_/
2020-07-01T18:52:28.2930943Z                               Bash-20200629-ffaf297
2020-07-01T18:52:28.2931144Z 
2020-07-01T18:52:28.2931287Z 
2020-07-01T18:52:28.2944379Z e[0;90m==>e[0m GitHub Actions detected.
2020-07-01T18:52:28.3749689Z     e[0;90mproject root:e[0m D:/a/pyomo/pyomo
2020-07-01T18:52:28.6064164Z     e[0;90mYaml found at:e[0m .codecov.yml
2020-07-01T18:52:29.0921384Z     e[0;90m->e[0m Found 1 reports
2020-07-01T18:52:29.0928997Z e[0;90m==>e[0m Detecting git/mercurial file structure
2020-07-01T18:52:29.2502330Z e[0;90m==>e[0m Reading reports
2020-07-01T18:52:29.3547423Z     e[0;32m+e[0m coverage.xml e[0;90mbytes=3291415e[0m
2020-07-01T18:52:29.3917125Z e[0;90m==>e[0m Appending adjustments
2020-07-01T18:52:29.3917740Z     e[0;36mhttps://docs.codecov.io/docs/fixing-reportse[0m
2020-07-01T18:56:26.0509663Z     e[0;32m+e[0m Found adjustments
2020-07-01T18:56:26.1432102Z e[0;90m==>e[0m Gzipping contents
2020-07-01T18:56:26.8772579Z e[0;90m==>e[0m Uploading reports
2020-07-01T18:56:26.8782518Z     e[0;90murl:e[0m https://codecov.io
2020-07-01T18:56:26.8790547Z     e[0;90mquery:e[0m branch=move-termination-methods-to-environ&commit=11c0a0a90c14e146e1da7be314a4023253bfa981&build=154425195&build_url=http%3A%2F%2Fgithub.com%2FPyomo%2Fpyomo%2Factions%2Fruns%2F154425195&name=win%2F3.8&tag=&slug=Pyomo%2Fpyomo&service=github-actions&flags=&pr=1540&job=
2020-07-01T18:56:27.0235922Z e[0;90m==>e[0m Uploading to Codecov
2020-07-01T18:56:28.1292346Z curl: (3) URL using bad/illegal format or missing URL
2020-07-01T18:56:28.1294617Z curl: (3) URL using bad/illegal format or missing URL
2020-07-01T18:56:28.1295205Z   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
2020-07-01T18:56:28.1295874Z                                  Dload  Upload   Total   Spent    Left  Speed
2020-07-01T18:56:28.1296241Z 
2020-07-01T18:56:29.3351178Z   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
2020-07-01T18:56:30.3351397Z   0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
2020-07-01T18:56:31.3355137Z   0     0    0     0    0     0      0      0 --:--:--  0:00:02 --:--:--     0
2020-07-01T18:56:32.3361016Z   0     0    0     0    0     0      0      0 --:--:--  0:00:03 --:--:--     0
2020-07-01T18:56:33.3362824Z   0     0    0     0    0     0      0      0 --:--:--  0:00:04 --:--:--     0
2020-07-01T18:56:34.3368704Z   0     0    0     0    0     0      0      0 --:--:--  0:00:05 --:--:--     0
2020-07-01T18:56:35.3369755Z   0     0    0     0    0     0      0      0 --:--:--  0:00:06 --:--:--     0
2020-07-01T18:56:36.3372780Z   0     0    0     0    0     0      0      0 --:--:--  0:00:07 --:--:--     0
2020-07-01T18:56:37.3373245Z   0     0    0     0    0     0      0      0 --:--:--  0:00:08 --:--:--     0
2020-07-01T18:56:38.3376114Z   0     0    0     0    0     0      0      0 --:--:--  0:00:09 --:--:--     0
2020-07-01T18:56:39.3384434Z   0     0    0     0    0     0      0      0 --:--:--  0:00:10 --:--:--     0
2020-07-01T18:56:40.3387259Z   0     0    0     0    0     0      0      0 --:--:--  0:00:11 --:--:--     0
2020-07-01T18:56:41.3390821Z   0     0    0     0    0     0      0      0 --:--:--  0:00:12 --:--:--     0
2020-07-01T18:56:42.3393038Z   0     0    0     0    0     0      0      0 --:--:--  0:00:13 --:--:--     0
2020-07-01T18:56:43.3403103Z   0     0    0     0    0     0      0      0 --:--:--  0:00:14 --:--:--     0
2020-07-01T18:56:44.3406825Z   0     0    0     0    0     0      0      0 --:--:--  0:00:15 --:--:--     0
2020-07-01T18:56:45.3416621Z   0     0    0     0    0     0      0      0 --:--:--  0:00:16 --:--:--     0
2020-07-01T18:56:46.3420924Z   0     0    0     0    0     0      0      0 --:--:--  0:00:17 --:--:--     0
2020-07-01T18:56:47.3422757Z   0     0    0     0    0     0      0      0 --:--:--  0:00:18 --:--:--     0
2020-07-01T18:56:48.3424176Z   0     0    0     0    0     0      0      0 --:--:--  0:00:19 --:--:--     0
2020-07-01T18:56:49.3434066Z   0     0    0     0    0     0      0      0 --:--:--  0:00:20 --:--:--     0
2020-07-01T18:56:49.3470473Z   0     0    0     0    0     0      0      0 --:--:--  0:00:21 --:--:--     0
2020-07-01T18:56:49.3471555Z curl: (28) Failed to connect to codecov.io port 443: Timed out
2020-07-01T18:56:49.4549675Z     e[0;90m->e[0m Sleeping for 30s and trying again...
2020-07-01T18:57:19.4857583Z d:\a\pyomo\pyomo/cache/download/codecov.sh: line 1721: [: 0+1: integer expression expected
2020-07-01T18:57:19.4858081Z     e[0;31mX> Failed to upload coverage reportse[0m
2020-07-01T19:09:06.7156291Z e[He[2Je[3J
2020-07-01T19:09:06.7197528Z ##[error]Process completed with exit code 1.

Steps to Reproduce

The following is the excerpt from our build harness that manages the coverage uploads:

    # Always attempt to update the codecov script, but fall back on
    # the previously cached script if it fails
    CODECOV="${GITHUB_WORKSPACE}/cache/download/codecov.sh"
    for i in `seq 3`; do
        echo "Downloading current codecov script (attempt ${i})"
        curl -L https://codecov.io/bash -o $CODECOV
        if test $? == 0; then
            break
        fi
        DELAY=$(( RANDOM % 30 + 30))
        echo "Pausing $DELAY seconds before re-attempting download"
        sleep $DELAY
    done
    i=0
    while : ; do
        ((i+=1))
        echo "Uploading coverage to codecov (attempt ${i})"
        bash $CODECOV -Z -X gcov -X s3 -f coverage.xml
        if test $? == 0; then
            break
        elif test $i -ge 3; then
            exit 1
        fi
        DELAY=$(( RANDOM % 30 + 30))
        echo "Pausing $DELAY seconds before re-attempting upload"
        sleep $DELAY
    done

Expected behavior: Connection succeeds, script is downloaded and results are uploaded

Actual behavior: Connection timeout contacting codecov.io

Flakiness? Occurs on 10% of our Linux / Windows GitHub Actions jobs

Additional Information

None

[tom]

Hi @jsiirola, is there any reason in particular you are using -X s3? That pathway uses our v2 upload endpoint, which works, but is not as stable as using the v4 upload endpoint. If you remove -X s3 it will attempt to use the v4 endpoint and fallback to v2 in case of failures.

[jsiirola]

We test on a number of platforms, one of which is behind a proxy. In my experience, the s3 endpoint has been significantly more fragile (especially for the build systems behind a proxy). To simplify things, I have been coding everything against the “lowest common denominator” that seems to work everywhere.

[tom]

@jsiirola, there were a few issues that I saw in the Codecov output that we pushed fixes for. I’d love to know if you are still getting as much instability.

[jsiirola]

@tom, I just fired off a build this morning (testing issues we are seeing with Python 3.7 and 3.8 under OSX on GHA), and hit a codecov upload timeout on the linux/2.7 build (see Update GHA to use setup-python@v2; resolve OSX python environment failure by jsiirola · Pull Request #1555 · Pyomo/pyomo · GitHub)

[tom]

@jsiirola, looks like you still might be using a cached version. The latest is 20200707-353aa93 while yours is showing from last month

[jsiirola]

@tom, there was a cached version, however it is only used after the attempt to download the current version fails. I did go in and clear the cache, and we are still seeing fairly regular connection failures, even with 20200707-353aa93. For example, a build tonight, 2 of the 13 Linux/Windows builds for a recent commit failed (win/3.5 and win/3.6; see, e.g. https://github.com/Pyomo/pyomo/runs/871714927)

[tom]

@jsiirola, there are two things that I would like to try here. First, would you be able to run this and save down the report using the -q parameter? I’m wondering if the file you are trying to upload is too big. I would also strongly advise not using the -X s3 argument if at all possible. I’m guessing that Codecov is timing out on this side. We have some tasks on our end to accept larger upload sizes, but I’d rather get you unblocked first.

[rowanseymour]

We’re also seeing this on several projects. I would estimate as many as 1 in 10 uploads fail with

[error]Codecov failed with the following error: connect ETIMEDOUT 35.199.43.247:443

The entry in our workflow is just…

- name: Upload coverage
  if: success()
  uses: codecov/codecov-action@v1
  with:
    token: ${{ secrets.CODECOV_TOKEN }}
    fail_ci_if_error: true

[tom]

Thanks here @rowanseymour, still working on some fixes to help alleviate this issue. I appreciate all of your patience.

[Drowze]

I am also seeing the same issue on some of my builds (I am depending on the codecov github action to upload the report).

https://github.com/Drowze/aspirebudgeting_ruby/runs/922644096

Run codecov/codecov-action@v1.0.10
##[warning]Codecov warning: connect ETIMEDOUT 35.199.43.247:443
bash codecov.sh -f coverage/.resultset.json -n  -F 
codecov.sh: line 1: undefined: command not found
##[warning]Codecov warning: The process 'bash' failed with exit code 127

[tom]

Hi all, we made some changes to the underlying bash script that should help with timeouts. Let me know if you are still seeing issues.

[rowanseymour]

Just seen again here 📇 Allow URN inequality in elastic searches by rowanseymour · Pull Request #951 · nyaruka/goflow · GitHub

[tom]

Thanks @rowanseymour, we’ll continue to work on this.

[rowanseymour]

This one timed out after 11 minutes https://github.com/nyaruka/mailroom/runs/954776939

Codecov warning: connect ETIMEDOUT 35.199.43.247:443
bash codecov.sh -n -F
codecov.sh: line 1: undefined: command not found
Codecov warning: The process 'bash' failed with exit code 127

[rowanseymour]

It does seem all the timeouts now take 11 minutes so I’m guessing someone “fixed” this by bumping a timeout setting up to at least 10 minutes… which would fix things if the issue here was large payloads or slow upload speeds, but the coverage files for this particular project are ~460KB uncompressed.

[tom]

Hi @rowanseymour, I’ve released 1.0.13 which should really help long timeouts. Would you mind trying it out and seeing if you still run into long timeouts?

[arnaudh]

We are also seeing timeouts for our julia library. We’ve bumped the timeout to 60 seconds but seems it’s not enough.

This is the curl command that produces a timeout sporadically:

curl -sI --proto =http,https,ftp,ftps https://codecov.io/gh/invenia/LibPQ.jl --max-time 60 -o /dev/null --write-out '%{http_code} %{url_effective} %{redirect_url}'

Note if I try to run it from my machine now it returns a 405 Method Not Allowed (I’m assuming that the HTTP library we use works around the 405 when it doesn’t timeout).

Maybe this gives a hint as to the potential source of the issue?

curl failure when building docs

[tom]

Hi @arnaudh, we are working on our infrastructure to prevent intermittent timeouts. Until then, I would recommend adding --retry 5 --retry-delay 2 --connect-timeout 2 to the curl command.

[arnaudh]

Thanks @tom, appreciate the tip.