We use Travis to test our Julia project in parallel. That is, different parts of the code are tested in separate jobs to reduce the build time. Merging of the reports generally works fine. However, since we changed from one job per build to multiple parallel jobs per build, the Codecov.io coverage reports indicate a significantly lower coverage than before. We tracked down the issue to the fact that Codecov erroneously reports method declarations as executable lines (which it didn’t before when we did not have parallel builds enabled). To make sure that it’s not just a simple snafu, we compared it to the report by Coveralls, which does not seem to have this issue. Below is an example for a file that shows the weird behavior in action (line 9).
Hi @sloede, thanks for this. Could you provide a build before you switched to multiple parallel jobs?
As for why you are seeing that line being uncovered, builds 133.3 and 133.7 show explicitly that that line is uncovered. How are you collecting coverage metrics?
Hi all, this one was tricky, and I still don’t have a great answer.
I’m focusing on this build and the file src/amr/amr.jl. What you’ll notice is that the Codecov step happens before the Coveralls step. But in between those two lines, something else runs. You can see this line in between the two that states
[ Info: CoverageTools.process_file: Detecting coverage for src/amr/amr.jl
┌ Info: CoverageTools.process_cov: Coverage file(s) for src/amr/amr.jl do not exist.
└ Assuming file has no coverage.
My guess is for this build, Coveralls is not receiving line coverage for that file, but we still are. I can’t figure out how to view the raw report sent to Coveralls, but you can find the build here.
We, however, do get coverage for that file here. which shows those two lines being uncovered:
My guess as to how to fix this would be to run the two commands at the same time and you should expect the same coverage (dependent on partial coverage). I was unaware of being able to do this in Travis, but we always recommend the bash uploader. In this case, I’m not sure how you would control for timing.
Sorry for not replying earlier to you. First of all, thank you very much for taking the time to investigate. We have since switched to GitHub Actions and the problem persists. We also found at least one other Julia project with similar issues, so we don’t believe it is something specific to our CI setup.
To recap, we still get the same wrong coverage analysis that the first line of a Julia function definition (starting with function ...) is counted as “executable” with zero executions, thus artificially lowering the overall coverage percentage. Examples are in
However, I am not 100% convinced that it is related to the parallel builds (or at least not in a simply reproducible way). For example, in these projects, the function definition line function ... is correctly counted as “not executable” and thus does not affect the overall coverage, even though they are tested in parallel:
As you can see with the example for OrdinaryDiffEq, even within the same repository sometimes the lines are correctly counted, sometimes they are not. Any idea what might be causing this and how to fix it?
Hi @sloede, unfortunately, this one got away from me last month as I was recovering from COVID. I’m getting is that the function lines are still being shown as green (covered) when they shouldn’t be. Is that accurate? If so, are the examples you pasted above still valid?
