Carryforward flag problems with delta testing

Support
1

Description

We have a monorepo which we are trying to use carryforward flags with delta tests. Our test suite only runs tests on files which were changed or depend on changed files. We are trying to carryforward the test coverage for files that were not tested. Since the tests are all part of 1 big suite and run only on changed files, we have just 1 flag: unit

CI/CD URL

Start commit w/ full report uploaded
Commit w/ no changes and no report uploaded
Commit w/ changes and report only for changed files uploaded

Codecov Output

Expected Results

Expected results to be carried forward since no tests changed.

Actual Results

No test results carried forward. All coverage removed.

Additional Information

codecov.yml file

flag_management:
  default_rules:
    carryforward: true
2

@mattrunyon let me make sure I understand what’s going on. So you have one test suite, where only a certain subset of those tests will run on a commit. You are looking for a way to maintain coverage for entire suite?

3

@tom Basically. We only run tests which are affected by the changes in a pull request using Jest’s --changedSince to detect which tests should run

I can see where this would be a difficult problem though as it seems coverage reports don’t include which test covered which lines, so it wouldn’t really be possible to merge the reports in this way I’m guessing. Also if a test is deleted you’d have to do something in git to see that I’m guessing.

We might be able to at least split tests per package (monorepo w/ 10-15 packages tested), but I’m not sure our setup will do that cleanly. The reason all the tests are in a single suite is because there are big performance gains in GitHub Actions if you run Jest from the root of a monorepo instead of spawning many different Jest processes to run each project separately.

4

@mattrunyon, sorry it took me so long to get back to you here. I’m not sure that this is possible with Flags or Carryforward Flags. The reason is that each flag is tied to a coverage report uploaded as opposed to by test. I’m sorry I don’t have a cleaner answer for you here.