Feature request: filterset using Cargo features

[BatmanAoD]

Motivation

We have integration tests that need exclusive access to a specific hardware resource. Because we only have one of this resource, we want to run those integration tests separately from building the tests, running other tests, or generating a code-coverage report.

We are currently using a specific Cargo feature to track whether or not each test requires the hardware resource (and the feature flag enables an optional dependency that actually provides access to the resource). So we are building two separate nextest archives, one with the feature enabled and one with the feature disabled. For the build with the feature enabled, we use --test '*', because currently all integration tests require the hardware. When we add integration tests that don't require the hardware, we will use #[cfg(no(<feature>))] on those tests.

We then generate coverage metrics with llvm-cov, and we want them to include coverage from running the tests that require the hardware.

Proposal

Add a filterset function to include or exclude tests based on whether they are enabled by a specific Cargo feature. This would enable using one nextest-archive built with all features enabled, but using the feature to select which tests are run in which CI jobs.

In its simplest form, I think this would effectively just be features(foo), and would mean "all tests built by Cargo when --features foo is provided on the command line." If possible, features() should take a variable number of comma-delimited arguments. Limiting this to only such tests, in contrast to all other possible feature-set combinations, would presumably be much trickier to implement and, in my opinion, wouldn't be necessary, since that could equally well be accomplished (though perhaps less performantly) with features(foo) and not features().

Along with features(), it would make sense to also provide all_features().

Alternatives

Solutions with one nextest-archive:

• Instead of (or in addition to) the Cargo feature, we could pull all the tests that require the hardware resource into a separate standalone crate, and use that crate as the relevant nextest filterset.
• We could follow a test-naming convention. This seems somewhat more onerous and error-prone, though.

Solutions with a separate nextest-archive for each --features build configuration:

• It may be possible to "merge" nextest archives in preparation for running cargo llvm-cov report, rather than making cargo-llvm-cov read directly from a nextest archive.
• I've opened a separate feature request for cargo-llvm-cov to add support for reading from multiple nextest-archive files.

Additional context

No response

[sunshowers]

Thanks for the report. I don't think this is possible at the moment, Rust doesn't provide any information around this and any kind of scanning nextest implements will likely be quite fragile.