Partial results when one of the parallel builds fail

Support
#1

Description

When using codecov merge report feature with 3 parallel builds if 1 build fails then partial results are still updated to the dashboard. If you have some flaky tests this results in constantly fluctuating coverage reports. (like +8%, -8% etc)

CI/CD URL

Example: fix: Implicit ignore pricing rule check on returns (backport #30662) by mergify · Pull Request #30664 · frappe/erpnext · GitHub (2 builds failed so coverage dropped by 14% and it was reported)

You can look at this graph for long term effect of it: Codecov

Codecov Output

N/A

Expected Results

There should be a minimum_builds_required similar to after_n_builds to reject partial coverage submissions.

Actual Results

Partial results are stored

Additional Information

I am considering collecting coverage after success and then submitting all collected reports if all builds pass, that would resolve this issue but felt like this is something that can be done out of the box with config hence first confirming if this exists already or if it can be considered in future.

#2

@ankush I think you might be looking for something like Carryforward Flags. If you add a Carryforward Flag to each upload, those uploads that aren’t received will have their coverage pulled from the most recent commit (with coverage).