You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The tests/load_models.py/tests/unload_models.py scripts that handle loading and unloading the models used by the test suite currently need to be invoked separately from the test suite execution.
While this approach works, it does mean that you can easily get spurious failures if you forget to run tox -e load-test-models before a local testing session.
To make it even more efficient, the lmstudio marker could potentially be parameterised with the model IDs that a given test expects to already have loaded. For example:
When scanning the selected test nodes, the fixture would keep track of the named models, and ensure those models were loaded before running any tests. To avoid problems with unexpected JIT model unloading during the test suite execution, the fixture would still mark models to remain loaded indefinitely.
The text was updated successfully, but these errors were encountered:
ncoghlan
changed the title
Testing: switch setup/teardown to session scoped fixture
Testing: session scoped fixture for required model loading
Feb 18, 2025
The
tests/load_models.py
/tests/unload_models.py
scripts that handle loading and unloading the models used by the test suite currently need to be invoked separately from the test suite execution.While this approach works, it does mean that you can easily get spurious failures if you forget to run
tox -e load-test-models
before a local testing session.We could potentially add a session scoped fixture in
conftest.py
to getpytest
to natively handle ensuring that the required models have been loaded rather than needing to manage it externally: https://stackoverflow.com/questions/76608690/pytest-best-practice-to-setup-and-teardown-before-and-after-all-testsTo avoid pointlessly setting up the models when
-m "not lmstudio"
has been passed, the fixture should scan the loaded modules and ensure the models are only loaded if that mark is present on at least one selected test (see https://docs.pytest.org/en/stable/example/markers.html and https://docs.pytest.org/en/stable/example/special.html)To make it even more efficient, the
lmstudio
marker could potentially be parameterised with the model IDs that a given test expects to already have loaded. For example:When scanning the selected test nodes, the fixture would keep track of the named models, and ensure those models were loaded before running any tests. To avoid problems with unexpected JIT model unloading during the test suite execution, the fixture would still mark models to remain loaded indefinitely.
The text was updated successfully, but these errors were encountered: