Missing parts in CircleCI parallelized tests

Support Bug Fixes
1

Description

When we run our backend tests on circleCI we use its parallel features. In particular, we set parallelism to 4. Therefore the tests will be partitioned into 4 part and executed. At the end of each run, results will be sent to CodeCov.
We notice that sometimes although all 4 partitions run successfully and show that report has been uploaded to CodeCov, not all shows up on the CodeCov dashboard. One such example is this one: https://codecov.io/gh/scaleapi/scaleapi/commit/572d53641ebda3c334e9098dad75500a35a48e97/build. Only 2 out of the 4 partitions showed up. This results in incorrect test coverage report.
This type of issue happens continuously.

Repository

Private

Steps to Reproduce

N/A

Expected behavior: 4 CircleCI build parts.

Actual behavior: Sometimes less than 4 parts.

Flakiness? Sometimes

Versions

Please include the browser and OS and what versions you’re running.

Additional Information

Any additional information, configuration or data that might be necessary to reproduce the issue.

2

Hi @yaoyuan0801

I can confirm we only have two uploads for that commit, but it’s too old to check the logs (I was on vacation)

Do you have a newer example within 2-3 days that you can provide the commit SHA for and I can see what happened during the upload of the missing reports?

4

Here is a good example: https://codecov.io/gh/scaleapi/scaleapi/commit/d163c115226b20976f60b144faf31bcde89b12d1/build

5

Hi @yaoyuan0801, I went over the last couple of recent commits and am seeing 4 builds. Is this still a problem you’re having?

6

Hi,

Sorry to barge in, but I think we are having the same problem here…

Out of our 40 parallel builds we sometimes are missing some of them… for example on this commit we only got 38 builds https://codecov.io/gh/wantedly/wantedly/commit/ce7507e27507de78dd8e454d127cc3cbbf1e0c62/build

Not sure if this is related, but sometimes the spec failed to post the code-coverage results to codecov, so we started using a fork that retried posting a couple of times when that happens.

Hope this helps towards some resolution :bowing_woman:

7

Hi @CamilleDrapier, Is this happening frequently for you? I see this commit: https://codecov.io/gh/wantedly/wantedly/commit/1d6153a1a04579516f811b97d9e61c5b812eaebc/build and I thought it might be the 38 builds issue, but I’m seeing 40.

I’m, unfortunately, not seeing anything strange on my end here. Do you have text from your CI/CD run that shows an improper upload or an upload that should have happened but didn’t make it?

8

Hi @tom,

Thanks for you reply.

It was happening quite often until the end of last week, but these last days it seems to have become much better.

Considering I posted my last message about ~2 hours after the last commit, something might have been stuck (on CodeCov?) somewhere after the specs ran (and at that time it was still showing 38 builds…), but now it looks good indeed :+1:

I checked the recently merged PRs and everything seems fine these day :slight_smile:

Also I have been fixing some non-deterministic code paths in coverage of our tests recently and that might have some positive side effects ?

Anyway, if no other concerns, please consider this fixed/working on our end :+1:

Thank you for your time on this concern! :heart:

Camille,

1 Like
9

Amazing @CamilleDrapier ! I’m wondering if this has to do with some issues we were having with GitHub recently. If you haven’t already, I would recommend using our GitHub marketplace app

10

I think we have been using it for at least a few months now :thinking:, so I’m not sure if it is related tot his :slight_smile:

Thanks anyway for pointing it out!

1 Like