Fixed parallelism is ignored with ProcessBuilder.wait

[jeanbza]

Describe the bug

I'd like concurrent testing, but only 4 tests to run at a time. If I set runner.parallel.enabled I get the former. Using .fixed does not fix the latter.

To Reproduce

My SpockConfig.groovy has,

runner {
    parallel {
        enabled true
        fixed(4)
    }
}

This works when I have basic tests like this.

This does not work when my tests do,

Process process = processBuilder.start();
process.waitFor();

When my tests hit that point, the test framework or the executor (not sure) admits more tests than the fixed(4) limit I tried to specify.

FWIW I'd guess what's happening is that waitFor is doing a yield and allowing the executor to run something else until the OS comes back with some I/O.

Expected behavior

4 tests are run.

Fixed parallelism of tests should always limit the number of inflight tests.

It seems to currently be too tied to executors and not enough to the test framework actually limiting inflight tests.

Actual behavior

9 tests (all of them) are run.

Java version

$ java --version
openjdk 21.0.5 2024-10-15 LTS
OpenJDK Runtime Environment Zulu21.38+21-CA (build 21.0.5+11-LTS)
OpenJDK 64-Bit Server VM Zulu21.38+21-CA (build 21.0.5+11-LTS, mixed mode, sharing)

Buildtool version

$ gw --version

------------------------------------------------------------
Gradle 8.12.1
------------------------------------------------------------

Build time:    2025-01-24 12:55:12 UTC
Revision:      0b1ee1ff81d1f4a26574ff4a362ac9180852b140

Kotlin:        2.0.21
Groovy:        3.0.22
Ant:           Apache Ant(TM) version 1.10.15 compiled on August 25 2024
Launcher JVM:  21.0.5 (Azul Systems, Inc. 21.0.5+11-LTS)
Daemon JVM:    Compatible with Java 21, vendor matching('azul') (from gradle/gradle-daemon-jvm.properties)
OS:            Mac OS X 15.3.1 aarch64

What operating system are you using

Mac

Dependencies

I'd prefer not to share this, let me know if you need it though.

Additional context

No response

[Vampire]

Listen to your own words, as you said the important thing already yourself. :-)
You are not configuring a "fixed amount of tests", you are configuring a "fixed parallelity" which by far is not the same.
Also this is nothing that Spock influences, this is managed by JUnit platform for which we are just an engine.

Besides that, here even when using a Java 21 Azul Zulu JDK to run the tests I indeed get 4 Tests run in parallel only.
But as the waitFor method is implemented in native code, who knows that happens there and whether a macOS implementation greatly differs from other OS.

Maybe your issue - resp. the native code - is doing something like junit-team/junit5#3108 and thus suffering from the same bug that is there. Maybe the work-around I described at junit-team/junit5#3108 (comment) can also work for you if it is a similar problem. 🤷‍♂️

[jeanbza]

You are not configuring a "fixed amount of tests", you are configuring a "fixed parallelity" which by far is not the same.

Ack. Is there a way to specify max number of tests running concurrently? I think not but thought I'd check.

[Vampire]

As I said, if it is similar to that issue, you can try my work-around, or also the other options mentioned in that issue's comments

[Vampire]

As it is most likely a JUnit Platform issue if at all, I'm going to close this here.
Feel free to leave a comment if you think this should be re-opened.