Annotate test cases which are timing sensitive

[rossburton]

Feature or enhancement

Proposal:

Many tests in the cpython test suite are sensitive to timing and assume an entirely unloaded system. For example, the TimerfdTests check for 1ms accuracy in expected durations. Some people (eg people building and testing python) may be running the test suite on build machines which are potentially heavily loaded so this sort of expectation isn't feasible.

It seems like a good solution to this would be to annotate the tests which are timing sensitive in some way, so they can be skipped easily.

For reference, we're manually patching python to skip tests which are failing under load but we're now having to rebase these patches on upgrades:

• https://git.yoctoproject.org/poky/tree/meta/recipes-devtools/python/python3/0001-Skip-failing-tests-due-to-load-variability-on-YP-AB.patch
• https://git.yoctoproject.org/poky/tree/meta/recipes-devtools/python/python3/0001-skip-no_stdout_fileno-test-due-to-load-variability.patch
• https://git.yoctoproject.org/poky/tree/meta/recipes-devtools/python/python3/0001-test_active_children-skip-problematic-test.patch
• https://git.yoctoproject.org/poky/tree/meta/recipes-devtools/python/python3/0001-test_deadlock-skip-problematic-test.patch
• https://git.yoctoproject.org/poky/tree/meta/recipes-devtools/python/python3/0001-test_shutdown-skip-problematic-test.patch
• https://git.yoctoproject.org/poky/tree/meta/recipes-devtools/python/python3/0001-test_storlines-skip-due-to-load-variability.patch

Has this already been discussed elsewhere?

No response given

Links to previous discussion of this feature:

No response

[encukou]

Sounds like it could be a new resource, see python -m test --help and test.support.requires.