Enable @parametrize for test classes even if it uses a fixture

[Gutifarra]

I'm currently using fixtures as a base to generate cases with, in order to perform a bunch of tests which are grouped into various classes. An simple example which illustrates something similar to what I'm doing:

class A:
    pass

@fixture
def create_a():
    return A()

def case_a_1(create_a):
    return create_a, 1

def case_a_2(create_a):
    return create_a, 2


@parametrize_with_cases("a", cases=".")
class tests:
    def test_check_a_exists(self, a):
        assert a

If I attempt to parametrize the actual tests class, pytest-cases raises a NotImplementedError:

E NotImplementedError: @parametrize can not be used to decorate a Test class when the argvalues contain at least one reference to a fixture.

Is there any particular reason for this not being supported? I'm not sure if this is just me designing my tests in a way which is incompatible with some fundamental design behind pytest and pytest-cases, or if it's something which should be in a future implementation.

I would expect decorating this class with @parametrize to do exactly the same as if I decorated each of the test functions.

[smarie]

Thanks @Gutifarra for pushing pytest-cases to the limits !

pytest has foundamental limitations in its design that prevent most of the use cases in pytest-cases to be achievable without hacking the engine. One of these limitations is that the pytest parameter values of pytest.mark.parametrize are not allowed to reference fixtures. Indeed that creates a "fixture union", that is, pytest would have to activate first fixture a, then potentially fixture b (deactivating fixture a) for the same test. This mechanism of is required, at the core of pytest-cases: cases are basically parameter values, and they can require a fixture.

So pytest-cases is doing a few hacks, indeed - but they are thoroughly tested against a very large test base, and reported compliance issues with the rest of the pytest ecosystem are fixed quickly (we used to have several such reports in the early days, but there have not been much recently - it seems to have well stabilized now)

One of the hacks is the hack that you found. It is about how @parametrize works:

from pytest_cases import fixture, parametrize

@fixture
def create_a():
    return 1

@parametrize("a", [create_a])
class tests:
    def test_check_a_exists(self, a):
        assert a

This yields the same error as the one you found. This is because @parametrize first generates a fixture that will replace the parametrization of a (e.g. generated_fixture_for_a), and then it replaces the decorated function with a wrapper, so test_foo(a) would become test_foo(generated_fixture_for_a).

When the decorator is applied on a class, currently I just raise an error because it would not make sense to apply the above procedure. However, we could easily implement the following

• extract all test functions inside the class
• apply the @parametrize decorator to all of them
• add them to a new wrapper class
• return the wrapper class

So this is quite easily feasible, but needs to be properly tested especially against the edge cases: does pytest class mechanism support inheritance ? If so we would have to list inherited test functions too. etc.

If you feel like having a try in a PR, do not hesitate to jump in ! Otherwise, I'll have a look when I have some bandwidth

[Gutifarra]

On the contrary, thanks for taking the time to make something which makes pytest feel so much more satisfying to use! I've never been particularly fond of the syntax used for some of their parametrizations and was particularly peeved by not being able to parametrize using fixtures, so pytest-cases was a great find, and is really allowing for a lot of flexibility and building natural test hierarchies. Also, thanks a lot for the in-depth explanation, this is not a big stopper for me at the moment since I can just work around it by parametrizing all of the test functions individually, but it would be helpful to make the test files more concise and readable.

I am certainly willing (though maybe not entirely able) to put in a PR for this, there is a significant amount of reading I would like to do before I feel comfortable doing so, as I still consider myself an intermediate user at most for unit testing and more advanced Python topics. Will be glad to have a closer look when I have some time though, your approach definitely sounds like a good idea. Thanks again!

[smarie]

Thanks for the feedback and kind words @Gutifarra ! I'll leave the ticket open so that we can process it later then.