Region based codecoverage (llvm-cov export)

Feature Requests
,
1

Summary

It would be great if codecov would support the upload of llvm region based coverage json files that gets generated by llvm-cov export ( llvm-cov - emit coverage information — LLVM 16.0.0git documentation ). At the same time it would be nice if the codecov web gui displays the regions instead of just the lines.

Currently the coverage files are accepted by the upload script but the web gui sais that there was an error processing the report.

Description

Rust has now support for region based coverage which allows a more fine grained coverage instead of just using line based coverage (see https://doc.rust-lang.org/nightly/unstable-book/compiler-flags/source-based-code-coverage.html & Source-based code coverage in nightly | Inside Rust Blog ).

So far I was only able to export these coverage reports as lcov files, which loose the region information and falls back to line based coverage. It would be greate if it was possible to really report the source code regions to codecov instead of the lines (& branches).

Examples

An example for a case where the line based coverage misses cases is for example this line of code:

the ? part of the code is not covered by the tests, but still marked as coverage in the line based coverage (I wanted to include some screenshot but it would not let me).

You can use the following project & script to generate an example json:

git clone git@github.com:JulianSchmid/actions-rs-llvm-coverage.git
cd actions-rs-llvm-coverage
./scripts/local_coverage.bash
# results/export.txt has a llvm json report

Note that you will have to install Rust nightly as well as the llvm tools

rustup component add llvm-tools-preview

and some cargo tools

cargo install cargo-binutils rustfilt

Versions

Rust latest nightly

2

@JulianSchmid, could you post a Codecov link where we are showing the error processing report?

3

from the following CI run

https://github.com/JulianSchmid/actions-rs-llvm-coverage/runs/1617482381?check_suite_focus=true

4

@JulianSchmid, apologies for not getting to this due to the holidays. Would you be able to push up a new commit?

5

@tom Hi Tom, I pushed another commit ( Code coverage done right. ). But again using the llvm native codecoverage json format.

I currently can not find the format as supported in the supported coverage formats list. So the error is not unexpected.

Additionally here is the json file that was uploaded to codecov:

{"data":[{"files":[{"expansions":[],"filename":"rust_test_project/src/lib.rs","segments":[[1,0,1,true,true,false],[1,1,0,false,false,false],[2,1,1,true,true,false],[4,2,0,false,false,false],[6,1,1,true,true,false],[7,17,0,false,false,false],[8,9,0,true,true,false],[8,27,0,false,false,false],[10,9,1,true,true,false],[10,17,0,false,false,false],[12,1,1,true,true,false],[12,2,0,false,false,false],[14,1,1,true,true,false],[14,74,0,false,false,false],[15,9,1,true,true,false],[15,11,0,false,false,false],[15,14,1,true,true,false],[15,31,0,true,true,false],[15,32,0,false,false,false],[16,5,1,true,true,false],[16,15,0,false,false,false],[17,1,1,true,true,false],[17,2,0,false,false,false],[22,5,1,true,true,false],[22,6,1,true,false,false],[22,19,0,false,false,false],[23,20,1,true,true,false],[23,25,0,false,false,false],[24,5,1,true,true,false],[24,6,0,false,false,false],[27,5,1,true,true,false],[27,6,1,true,false,false],[27,22,0,false,false,false],[29,27,1,true,true,false],[29,79,0,false,false,false],[30,5,1,true,true,false],[30,6,0,false,false,false]],"summary":{"functions":{"count":6,"covered":5,"percent":83.333333333333343},"instantiations":{"count":10,"covered":7,"percent":70},"lines":{"count":24,"covered":15,"percent":62.5},"regions":{"count":22,"covered":15,"notcovered":7,"percent":68.181818181818173}}},{"expansions":[],"filename":"rust_test_project/tests/integration_tests.rs","segments":[[1,0,1,true,true,false],[1,1,0,false,false,false],[4,1,1,true,true,false],[4,2,1,true,false,false],[4,12,0,false,false,false],[5,19,1,true,true,false],[5,30,0,false,false,false],[6,1,1,true,true,false],[6,2,0,false,false,false]],"summary":{"functions":{"count":2,"covered":2,"percent":100},"instantiations":{"count":3,"covered":3,"percent":100},"lines":{"count":4,"covered":4,"percent":100},"regions":{"count":4,"covered":4,"notcovered":0,"percent":100}}}],"functions":[{"count":1,"filenames":["rust_test_project/src/lib.rs"],"name":"_RNvNtCs7aXSJLL72kE_17rust_test_project5testss_8it_works","regions":[[22,5,22,19,1,0,0,0],[23,20,23,25,1,0,0,0],[24,5,24,6,1,0,0,0]]},{"count":1,"filenames":["rust_test_project/src/lib.rs"],"name":"_RNvNtCs7aXSJLL72kE_17rust_test_project5testss_11some_result","regions":[[27,5,27,22,1,0,0,0],[29,27,29,79,1,0,0,0],[30,5,30,6,1,0,0,0]]},{"count":1,"filenames":["rust_test_project/src/lib.rs"],"name":"_RNCNvNtCs7aXSJLL72kE_17rust_test_project5tests11some_result0B5_","regions":[[27,5,27,6,1,0,0,0]]},{"count":1,"filenames":["rust_test_project/src/lib.rs"],"name":"_RNCNvNtCs7aXSJLL72kE_17rust_test_project5tests8it_works0B5_","regions":[[22,5,22,6,1,0,0,0]]},{"count":0,"filenames":["rust_test_project/src/lib.rs"],"name":"_RNvCs7aXSJLL72kE_17rust_test_project11some_result","regions":[[2,1,4,2,0,0,0,0],[6,1,7,17,1,0,0,0],[8,9,8,27,0,0,0,0],[10,9,10,17,1,0,0,0],[12,1,12,2,1,0,0,0]]},{"count":1,"filenames":["rust_test_project/src/lib.rs"],"name":"_RNvCs7aXSJLL72kE_17rust_test_project4main","regions":[[1,0,1,1,1,0,0,0]]},{"count":1,"filenames":["rust_test_project/src/lib.rs"],"name":"_RNvCs7aXSJLL72kE_17rust_test_project33some_result_return_using_function","regions":[[14,1,14,74,1,0,0,0],[15,9,15,11,1,0,0,0],[15,14,15,31,1,0,0,0],[15,31,15,32,0,0,0,0],[16,5,16,15,1,0,0,0],[17,1,17,2,1,0,0,0]]},{"count":1,"filenames":["rust_test_project/tests/integration_tests.rs"],"name":"_RNvCscMYet2iMjtG_17integration_testss_5dummy","regions":[[4,1,4,12,1,0,0,0],[5,19,5,30,1,0,0,0],[6,1,6,2,1,0,0,0]]},{"count":1,"filenames":["rust_test_project/tests/integration_tests.rs"],"name":"_RNvCscMYet2iMjtG_17integration_tests4main","regions":[[1,0,1,1,1,0,0,0]]},{"count":1,"filenames":["rust_test_project/tests/integration_tests.rs"],"name":"_RNCNvCscMYet2iMjtG_17integration_tests5dummy0B3_","regions":[[4,1,4,2,1,0,0,0]]},{"count":0,"filenames":["rust_test_project/src/lib.rs"],"name":"_RNvCsfAnIH3UfvFK_17rust_test_project11some_result","regions":[[6,1,7,17,0,0,0,0],[8,9,8,27,0,0,0,0],[10,9,10,17,0,0,0,0],[12,1,12,2,0,0,0,0]]},{"count":1,"filenames":["rust_test_project/src/lib.rs"],"name":"_RNvCsfAnIH3UfvFK_17rust_test_project9some_func","regions":[[2,1,4,2,1,0,0,0]]},{"count":0,"filenames":["rust_test_project/src/lib.rs"],"name":"_RNvCsfAnIH3UfvFK_17rust_test_project33some_result_return_using_function","regions":[[14,1,14,74,0,0,0,0],[15,9,15,11,0,0,0,0],[15,14,15,31,0,0,0,0],[15,31,15,32,0,0,0,0],[16,5,16,15,0,0,0,0],[17,1,17,2,0,0,0,0]]}],"totals":{"functions":{"count":8,"covered":7,"percent":87.5},"instantiations":{"count":13,"covered":10,"percent":76.923076923076934},"lines":{"count":28,"covered":19,"percent":67.857142857142861},"regions":{"count":26,"covered":19,"notcovered":7,"percent":73.076923076923066}}}],"type":"llvm.coverage.json.export","version":"2.0.1"}

7

@JulianSchmid, just letting you know that this hasn’t gotten lost, and thanks for the report here.

9

Any movement on this? We would also love this support. :orange_heart:

10

@dconnolly, we are unlikely to do so in the near future, but if this FR gains more traction, I’ll bump the priority.