When using codecov merge report feature with 3 parallel builds if 1 build fails then partial results are still updated to the dashboard. If you have some flaky tests this results in constantly fluctuating coverage reports. (like +8%, -8% etc)
Example: fix: Implicit ignore pricing rule check on returns (backport #30662) by mergify · Pull Request #30664 · frappe/erpnext · GitHub (2 builds failed so coverage dropped by 14% and it was reported)
You can look at this graph for long term effect of it: Codecov
There should be a
minimum_builds_required similar to
after_n_builds to reject partial coverage submissions.
Partial results are stored
I am considering collecting coverage after success and then submitting all collected reports if all builds pass, that would resolve this issue but felt like this is something that can be done out of the box with config hence first confirming if this exists already or if it can be considered in future.