pytest_generate_tests based on params and marks from previous parametrize

[Anton3]

Issue

In userver, we add indirect params to all tests based on marks using pytest_generate_tests.
(See the full story in #13217.)

The issue arises when we want to apply our implicit param only to some of the test items spawned from an already parametrized test:

@pytest.mark.parametrize(
    "foo, bar",
    [
        ("a", "x"),
        pytest.param("b", "y", marks=pytest.mark.my_mark("hmm")),
        pytest.param("c", "z", marks=pytest.mark.my_mark("what")),
    ])
def my_test(foo, bar, my_fixture, ...):
    # ...


def pytest_parametrize_tests(metafunc):
    # metafunc represents a test function before the split into items.
    # How do we set indirect param "my_fixture" based on marks
    # from one of the cases?

The issue has been previously explored: #4050

Proposed syntax

What I want to discuss here is an implementation of that idea, starting with a possible syntax.

# pytest
@dataclasses.dataclass(frozen=True)
class ProtoItem(pytest.Node):
    # essentially CallSpec2 with a rich API from Node
    params: dict[str, object]
    own_markers: list[Mark]

# usage
def pytest_parametrize_tests(metafunc):
    def param_generator(proto_item: pytest.ProtoItem) -> ParameterSet | Sequence[object] | object:
        if (mark := proto_item.get_closest_marker("my_mark")) is not None:
            return (compute_param_value_from_mark(mark),)
        else:
            return (get_some_fallback_value(),)

    metafunc.parametrize(("my_fixture",), param_generator)

[RonnyPfannschmidt]

The idea is something under discussion since 2014 as covariant parameters

It's necessary for any parameterisation where you want/need to choose parameter sets dependent on others

It's also a prerequisite for correct lazy fixtures as currently it's not possible to use a parameterized fixture via lazy fixtures

[Anton3]

@RonnyPfannschmidt what about the proposed syntax? I need multi-parametrize that is not just a Cartesian product, and this would be enough for my use cases. Also looks unintrusive enough in terms of API.

[RonnyPfannschmidt]

The meaning of the proposed object is very unclear to me, and I'd prefer to be able to work in terms of callspec

So more like a conditional parameterize that chooses based on individual callspecs instead of expanding all calls as product

[Anton3]

@RonnyPfannschmidt I've mostly implemented the feature, but now there's the issue that I want to use params and marks from pytest.mark.parametrize in my pytest_generate_tests. But I can't get my pytest_generate_tests to be executed after pytest.mark.parametrize, pytest.mark.parametrize always seems to be applied last. Any ideas how to change that?

[RonnyPfannschmidt]

Either a trylast hook or a hookwrapper

[Anton3]

@RonnyPfannschmidt Done, PR is mostly ready for review, but I've left a comment