How far back do carry forward flags look?

Before submitting a topic, please confirm the following

I have searched for similar issues before creating this topic.
I have verified that my repository is using the Codecov GitHub app, if using GitHub
I have validated my codecov.yaml configuration file.
I have filled out the below sections to the best of my ability.

Description

I am looking to find out how far back the carry forward flags look with regards to what it should carry forward. We have this setup and I can see it working but sometimes it does not carry forward flags and my assumption is that if they weren’t part of the last CI run, they can’t be carried forward.

An example I have is as follows:

PR 0

  • All five actions runs and upload coverage succesfully.

PR 1

  • Only three actions run and the other two are carried forward successfully.

RD 2

  • The same as PR 1 but this time, the two that were previously carried forward are not. Codecov thinks only three actions were run and so the diff reporting is wrong.

CI/CD URL

Private repo.

Expected Results

As per my example, tests from PR 0 are carried forward all the way through to PR 2 and beyond until these CI actions are actually run again.

Actual Results

PR 2 does not carry forward from PR 0 or PR 1.

@MeanBoyCousin Carryforward flags follow through from the git tree, so it depends also on the branch that they are on. Are you able to diagram something out, or does that make sense?

Hi Tom. I think that makes sense but I have tried to lay out what seems to be happening regarding my latest deployments. I hope this is helpful!

- 0fd246 is opened to merge from Staging into Master - All of the CI is run and all five reports are uploaded with full and correct reporting.

- e0c34a is showing as the last commit to Master on Codecov. This is the commit before 0fd246 (HEAD~1) but is the last commit that had reports run against it.

- 0a1ace is the current head of Staging and is three commits ahead of master.

- 0fd246 is the current head of Master.

- A PR is opened to merge staging into master. Only 3/5 reports are run. Coverage data is based on HEAD 0a1ace3 compared to PARENT e0c34a6. This means that the coverage is based on the head of Staging and HEAD~1 of Master. I assume this is because it was the last SHA that recieved reports?

- Codecov says that is has carried forward the necessary missing 2 reports, but the PR coverage for these shows "?".

For now, I am just manually dispatching the missing workflows against Staging to force new reports into Codecov but this isn’t ideal.

Another point that might be worth noting is that we have a workflow action that will automatically back merge Master into Staging when anything is merged into Master.

@MeanBoyCousin ok this is a little tough to follow without digging in a bit more. Could you give the full SHA for 0fd246?

Hi Tom. The full SHA is 0fd246fe543361cee0da0174b29ca907c946c052.

@MeanBoyCousin oh, how old is that commit? I was looking for something in the past week or two.

That SHA is only three days old on our Staging and Master branches.

@MeanBoyCousin I’m not seeing any uploads from that commit to Codecov at all. Is this not what’s expected? Or am I digging into a hole that might not be related to your problem?

This is interesting. I’ve just had a look at the PR in question and I see this.

You can see the CI and reporting was run against e0c34a6 but the merge committed 0fd246f into Master. I guess this makes sense as e0c34a6 was the PARENT in the example above. Still not sure why it never carried forward though, feeling very confused now! :rofl:

Hmmm, what’s the full SHA for e0c34a6?

That would be e0c34a6d8db92dbaba1baf5a2ab38828888d4ae9

Thanks @MeanBoyCousin

Ok, so my suspicion is that you are running any uploads consistently on the staging branch. 0fd246 and 0a1ace do not have any uploads, and the last known commit on this branch is 831ef804 (except for the commit that was made earlier).

So I’m a little confused when you say that coverage data is based on HEAD 0a1ace as that doesn’t exist in our system. Is this making sense, is that not what you were asking for?

Hi Tom. I’ve shot you a message, figured we might need to take this further to find a solution as it feels quite specific to my scenario. Hope you can help.

As it turns out, the solution was to remove flags that were being shared. Now that each test has one unique flag, carry forward works a treat. Thanks to @tom for the help on this.

1 Like