Intermittent timeout contacting codecov.io from GitHub Actions

Hi @jsiirola, @rowanseymour, @Drowze, @arnaudh, we’ve made significant updates to our infrastructure to prevent timeouts. Please let me know if you are still experiencing this issue, but it should now be solved.

Awesome, we’d started using fail_ci_if_error: false but if you’re feeling confident I’ll re-enable CI failures!

1 Like

I’m trying this but I am not sure how to pass this to the curl correctly. Should this be passed via the ‘-U’ option, as per the BASH script docs?

When I do

bash <(curl -s https://codecov.io/bash) -Z -U "--retry 5 --connect-timeout 5"

the upload fails with:
Could not resolve host: "--retry

Also, in that case the Codecov upload does not return an error, even though I used the -Z option.

@danielhollas, are you running into issues with the -U? It should not be necessary anymore.

It was some issue with how Github Actions expand environmental variables. I am now able to pass the -U parameters, though looking at the currect BASH uploader script, it doesn’t seem neccessary as the options are already included in the script.

However, I am still having issues with intermittently missing builds and I am not sure how to debug this since the BASH uploader script exits successfully (and I enabled the -Z option).

For example, here for this commit I see six builds sent to Codecov:

https://codecov.io/gh/PHOTOX/ABIN/commit/6a1c13bf723672998b9bb7587fca145c2e29a07f/build

obrazek

The corresponding Github CI build is here:
https://github.com/PHOTOX/ABIN/actions/runs/513161893
There are 11 successful CI jobs, and 8 of these should be sending data to Codecov. So somehow 2 builds are missing, but when I look at the outputs of the Codecov upload steps, all seems to be fine.
And I am unable to see which of the two builds are missing, since the builds in the Codecov UI look identical (see screenshot above). Is there a way to distinguish these jobs somehow?

Okay, I figured out how to pass the build name via the -n parameter or via CODECOV_NAME shell variable. However, I am still missing random builds and unable to debug because the BASH uploader does not report any error. :frowning:

For example, this commit
https://codecov.io/gh/PHOTOX/ABIN/commit/3f52650c28e5eb66b831a5b9caf1299a75aa2a83/build

is missing the basic_build GCC9 build. However, Github Actions output show that the report has been successfully uploaded to https://storage.googleapis.com/codecov/v4/raw/2021-01-27/F9CD1506F93C1687DF0FD91D04B9924B/3f52650c28e5eb66b831a5b9caf1299a75aa2a83/a4db3510-37ac-4057-bf2e-93edbbab5908.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=GOOG1EQX6OZVJGHKK3633AAFGLBUCOOATRACRQRQF6HMSMLYUP6EAD6XSWAAY%2F20210127%2FUS%2Fs3%2Faws4_request&X-Amz-Date=20210127T020216Z&X-Amz-Expires=10&X-Amz-SignedHeaders=host&X-Amz-Signature=500d18158d0b1b53fd208454bdb722da21a1d1b11a03ed0c08d5cd5dad4fa8fe

https://github.com/PHOTOX/ABIN/runs/1773802523?check_suite_focus=true

I am out of ideas, any help to debug this would be highly appreciated.

@danielhollas incredible solution. So the reason you don’t see failures from the bash uploader, is that it only adds the report to the processing queue. It doesn’t actually get feedback on whether the report can be successfully parsed.

In the case of basic_build GCC9, the coverage report is basically empty.

# path=estimators.F90.gcov.reduced
        -:    0:Source:estimators.F90
<<<<<< EOF
# path=analysis.F90.gcov.reduced
        -:    0:Source:analysis.F90
<<<<<< EOF
# path=force_bound.F90.gcov.reduced
        -:    0:Source:force_bound.F90
<<<<<< EOF
# path=read_cmdline.F90.gcov.reduced
        -:    0:Source:read_cmdline.F90
<<<<<< EOF
# path=gle.F90.gcov.reduced
        -:    0:Source:gle.F90
<<<<<< EOF
# path=force_tera.F90.gcov.reduced
        -:    0:Source:force_tera.F90
<<<<<< EOF
# path=arrays.F90.gcov.reduced
        -:    0:Source:arrays.F90
<<<<<< EOF
# path=force_terash.F90.gcov.reduced
        -:    0:Source:force_terash.F90
<<<<<< EOF
# path=forces.F90.gcov.reduced
        -:    0:Source:forces.F90
<<<<<< EOF
# path=surfacehop.F90.gcov.reduced
        -:    0:Source:surfacehop.F90
<<<<<< EOF
# path=potentials.F90.gcov.reduced
        -:    0:Source:potentials.F90
<<<<<< EOF
# path=fftw_interface.F90.gcov.reduced
        -:    0:Source:fftw_interface.F90
<<<<<< EOF
# path=abin.F90.gcov.reduced
        -:    0:Source:abin.F90
<<<<<< EOF
# path=force_cp2k.F90.gcov.reduced
        -:    0:Source:force_cp2k.F90
<<<<<< EOF
# path=analyze_ext_template.F90.gcov.reduced
        -:    0:Source:analyze_ext_template.F90
<<<<<< EOF
# path=sh_integ.F90.gcov.reduced
        -:    0:Source:sh_integ.F90
<<<<<< EOF
# path=plumed.F90.gcov.reduced
        -:    0:Source:plumed.F90
<<<<<< EOF
# path=modules.F90.gcov.reduced
        -:    0:Source:modules.F90
<<<<<< EOF
# path=minimizer.F90.gcov.reduced
        -:    0:Source:minimizer.F90
<<<<<< EOF
# path=water.F90.gcov.reduced
        -:    0:Source:water.F90
<<<<<< EOF
# path=compile_info.F90.gcov.reduced
        -:    0:Source:compile_info.F90
<<<<<< EOF
# path=nosehoover.F90.gcov.reduced
        -:    0:Source:nosehoover.F90
<<<<<< EOF
# path=io.F90.gcov.reduced
        -:    0:Source:io.F90
<<<<<< EOF
# path=ekin.F90.gcov.reduced
        -:    0:Source:ekin.F90
<<<<<< EOF
# path=mdstep.F90.gcov.reduced
        -:    0:Source:mdstep.F90
<<<<<< EOF
# path=force_mm.F90.gcov.reduced
        -:    0:Source:force_mm.F90
<<<<<< EOF
# path=init.F90.gcov.reduced
        -:    0:Source:init.F90
<<<<<< EOF
# path=force_abin.F90.gcov.reduced
        -:    0:Source:force_abin.F90
<<<<<< EOF
# path=transform.F90.gcov.reduced
        -:    0:Source:transform.F90
<<<<<< EOF
# path=geom_analysis.F90.gcov.reduced
        -:    0:Source:geom_analysis.F90
<<<<<< EOF
# path=utils.F90.gcov.reduced
        -:    0:Source:utils.F90
<<<<<< EOF
# path=en_restraint.F90.gcov.reduced
        -:    0:Source:en_restraint.F90
<<<<<< EOF
# path=shake.F90.gcov.reduced
        -:    0:Source:shake.F90
<<<<<< EOF
# path=vinit.F90.gcov.reduced
        -:    0:Source:vinit.F90
<<<<<< EOF
# path=landau_zener.F90.gcov.reduced
        -:    0:Source:landau_zener.F90
<<<<<< EOF
# path=random.F90.gcov.reduced
        -:    0:Source:random.F90

Because of the report is empty, we do not show the build in the builds tab. I will make a request to the product team to see if we can get that behavior changed.

For the future, you can always use the -q argument in the bash uploader to write the report down so that you can view the actual report.

1 Like

Thanks! I have just today figured it out. :slight_smile: I am a little bit unclear why the report is empty when compiling with GCC 8 or GCC 9, while it seems to work fine with GCC 7. (my code is in Fortran).

What seems to fix the issue is if I run the uploader from the src/ directory where the source code lives.
I wonder whether the BASH uploader could be improved so that it runs gcov from within the directory of the source file?

Here’s what I ended up doing:

run: cd src && bash <(curl -s https://codecov.io/bash) $CODECOV_OPTIONS -U “$CURL_OPTS”

@danielhollas would the -p argument be helpful here?

1 Like

Ah, thanks! Seems like that would do the trick. Though I wonder how would that work in more complex codebases where source code is distributed over multiple directories?

@danielhollas, do you have an example? We can work through that!