Skip to content

Commit 95193a5

Browse files
marcphilippmarcphilipp
committed
Introduce EngineTestKit support for test discovery
The goal is to ease testing for discovery issues encountered by an engine (see #242).
1 parent d25a530 commit 95193a5

File tree

8 files changed

+343
-19
lines changed

8 files changed

+343
-19
lines changed

‎documentation/src/docs/asciidoc/release-notes/release-notes-5.13.0-M1.adoc

+3
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,9 @@ repository on GitHub.
3131
redirecting `stdout` and `stderr` output streams to files.
3232
* Add `TestDescriptor.Visitor.composite(List)` factory method for creating a composite
3333
visitor that delegates to the given visitors in order.
34+
* Introduce test _discovery_ support in `EngineTestKit` to ease testing for discovery
35+
issues produced by a `TestEngine`. Please refer to the
36+
<<../user-guide/index.adoc#testkit-engine, User Guide>> for details.
3437

3538

3639
[[release-notes-5.13.0-M1-junit-jupiter]]

‎documentation/src/docs/asciidoc/user-guide/advanced-topics/testkit.adoc

+28-9
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,5 @@
1+
:testDir: ../../../../../src/test/java
2+
13
[[testkit]]
24
=== JUnit Platform Test Kit
35

@@ -9,16 +11,17 @@ JUnit Platform and then verifying the expected results. As of JUnit Platform
911
[[testkit-engine]]
1012
==== Engine Test Kit
1113

12-
The `{testkit-engine-package}` package provides support for executing a `{TestPlan}` for a
13-
given `{TestEngine}` running on the JUnit Platform and then accessing the results via a
14-
fluent API to verify the expected results. The key entry point into this API is the
15-
`{EngineTestKit}` which provides static factory methods named `engine()` and `execute()`.
16-
It is recommended that you select one of the `engine()` variants to benefit from the
17-
fluent API for building a `LauncherDiscoveryRequest`.
14+
The `{testkit-engine-package}` package provides support for discovering and executing a
15+
`{TestPlan}` for a given `{TestEngine}` running on the JUnit Platform and then accessing
16+
the results via convenient result objects. For execution, a fluent API may be used to
17+
verify the expected execution events were received. The key entry point into this API is
18+
the `{EngineTestKit}` which provides static factory methods named `engine()`,
19+
`discover()`, and `execute()`. It is recommended that you select one of the `engine()`
20+
variants to benefit from the fluent API for building a `LauncherDiscoveryRequest`.
1821

1922
NOTE: If you prefer to use the `LauncherDiscoveryRequestBuilder` from the `Launcher` API
20-
to build your `LauncherDiscoveryRequest`, you must use one of the `execute()` variants in
21-
`EngineTestKit`.
23+
to build your `LauncherDiscoveryRequest`, you must use one of the `discover()` or
24+
`execute()` variants in `EngineTestKit`.
2225

2326
The following test class written using JUnit Jupiter will be used in subsequent examples.
2427

@@ -34,8 +37,24 @@ own `TestEngine` implementation, you need to use its unique engine ID. Alternati
3437
may test your own `TestEngine` by supplying an instance of it to the
3538
`EngineTestKit.engine(TestEngine)` static factory method.
3639

40+
[[testkit-engine-discovery]]
41+
==== Verifying Test Discovery
42+
43+
The following test demonstrates how to verify that a `TestPlan` was discovered as expected
44+
by the JUnit Jupiter `TestEngine`.
45+
46+
[source,java,indent=0]
47+
----
48+
include::{testDir}/example/testkit/EngineTestKitDiscoveryDemo.java[tags=user_guide]
49+
----
50+
<1> Select the JUnit Jupiter `TestEngine`.
51+
<2> Select the <<testkit-engine-ExampleTestCase, `ExampleTestCase`>> test class.
52+
<3> Discover the `TestPlan`.
53+
<4> Assert engine root descriptor has expected display name.
54+
<5> Assert no discovery issues were encountered.
55+
3756
[[testkit-engine-statistics]]
38-
==== Asserting Statistics
57+
==== Asserting Execution Statistics
3958

4059
One of the most common features of the Test Kit is the ability to assert statistics
4160
against events fired during the execution of a `TestPlan`. The following tests demonstrate

‎documentation/src/test/java/example/testkit/EngineTestKitDiscoveryDemo.java

+37
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
/*
2+
* Copyright 2015-2025 the original author or authors.
3+
*
4+
* All rights reserved. This program and the accompanying materials are
5+
* made available under the terms of the Eclipse Public License v2.0 which
6+
* accompanies this distribution and is available at
7+
*
8+
* https://www.eclipse.org/legal/epl-v20.html
9+
*/
10+
11+
package example.testkit;
12+
13+
// tag::user_guide[]
14+
import static java.util.Collections.emptyList;
15+
import static org.junit.jupiter.api.Assertions.assertEquals;
16+
import static org.junit.platform.engine.discovery.DiscoverySelectors.selectClass;
17+
18+
import example.ExampleTestCase;
19+
20+
import org.junit.jupiter.api.Test;
21+
import org.junit.platform.testkit.engine.EngineDiscoveryResults;
22+
import org.junit.platform.testkit.engine.EngineTestKit;
23+
24+
class EngineTestKitDiscoveryDemo {
25+
26+
@Test
27+
void verifyJupiterDiscovery() {
28+
EngineDiscoveryResults results = EngineTestKit.engine("junit-jupiter") // <1>
29+
.selectors(selectClass(ExampleTestCase.class)) // <2>
30+
.discover(); // <3>
31+
32+
assertEquals("JUnit Jupiter", results.getEngineDescriptor().getDisplayName()); // <4>
33+
assertEquals(emptyList(), results.getDiscoveryIssues()); // <5>
34+
}
35+
36+
}
37+
// end::user_guide[]

‎junit-platform-launcher/src/main/java/org/junit/platform/launcher/core/DiscoveryIssueNotifier.java

+11-3
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@
1616

1717
import java.io.PrintWriter;
1818
import java.io.StringWriter;
19+
import java.util.ArrayList;
1920
import java.util.List;
2021
import java.util.Map;
2122
import java.util.function.Consumer;
@@ -35,9 +36,10 @@
3536
*/
3637
class DiscoveryIssueNotifier {
3738

38-
static final DiscoveryIssueNotifier NO_ISSUES = new DiscoveryIssueNotifier(emptyList(), emptyList());
39+
static final DiscoveryIssueNotifier NO_ISSUES = new DiscoveryIssueNotifier(emptyList(), emptyList(), emptyList());
3940
private static final Logger logger = LoggerFactory.getLogger(DiscoveryIssueNotifier.class);
4041

42+
private final List<DiscoveryIssue> allIssues;
4143
private final List<DiscoveryIssue> criticalIssues;
4244
private final List<DiscoveryIssue> nonCriticalIssues;
4345

@@ -47,14 +49,20 @@ static DiscoveryIssueNotifier from(Severity criticalSeverity, List<DiscoveryIssu
4749
.collect(partitioningBy(issue -> issue.severity().compareTo(criticalSeverity) >= 0));
4850
List<DiscoveryIssue> criticalIssues = issuesByCriticality.get(true);
4951
List<DiscoveryIssue> nonCriticalIssues = issuesByCriticality.get(false);
50-
return new DiscoveryIssueNotifier(criticalIssues, nonCriticalIssues);
52+
return new DiscoveryIssueNotifier(new ArrayList<>(issues), criticalIssues, nonCriticalIssues);
5153
}
5254

53-
private DiscoveryIssueNotifier(List<DiscoveryIssue> criticalIssues, List<DiscoveryIssue> nonCriticalIssues) {
55+
private DiscoveryIssueNotifier(List<DiscoveryIssue> allIssues, List<DiscoveryIssue> criticalIssues,
56+
List<DiscoveryIssue> nonCriticalIssues) {
57+
this.allIssues = allIssues;
5458
this.criticalIssues = criticalIssues;
5559
this.nonCriticalIssues = nonCriticalIssues;
5660
}
5761

62+
List<DiscoveryIssue> getAllIssues() {
63+
return allIssues;
64+
}
65+
5866
boolean hasCriticalIssues() {
5967
return !criticalIssues.isEmpty();
6068
}

‎junit-platform-launcher/src/main/java/org/junit/platform/launcher/core/LauncherDiscoveryResult.java

+7
Original file line numberDiff line numberDiff line change
@@ -16,13 +16,15 @@
1616

1717
import java.util.Collection;
1818
import java.util.LinkedHashMap;
19+
import java.util.List;
1920
import java.util.Map;
2021
import java.util.Optional;
2122
import java.util.function.Predicate;
2223
import java.util.stream.Collectors;
2324

2425
import org.apiguardian.api.API;
2526
import org.junit.platform.engine.ConfigurationParameters;
27+
import org.junit.platform.engine.DiscoveryIssue;
2628
import org.junit.platform.engine.TestDescriptor;
2729
import org.junit.platform.engine.TestEngine;
2830
import org.junit.platform.engine.reporting.OutputDirectoryProvider;
@@ -51,6 +53,11 @@ public TestDescriptor getEngineTestDescriptor(TestEngine testEngine) {
5153
return getEngineResult(testEngine).getRootDescriptor();
5254
}
5355

56+
@API(status = INTERNAL, since = "1.13")
57+
public List<DiscoveryIssue> getDiscoveryIssues(TestEngine testEngine) {
58+
return getEngineResult(testEngine).getDiscoveryIssueNotifier().getAllIssues();
59+
}
60+
5461
EngineResultInfo getEngineResult(TestEngine testEngine) {
5562
return this.testEngineResults.get(testEngine);
5663
}

‎junit-platform-testkit/src/main/java/org/junit/platform/testkit/engine/EngineDiscoveryResults.java

+58
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,58 @@
1+
/*
2+
* Copyright 2015-2025 the original author or authors.
3+
*
4+
* All rights reserved. This program and the accompanying materials are
5+
* made available under the terms of the Eclipse Public License v2.0 which
6+
* accompanies this distribution and is available at
7+
*
8+
* https://www.eclipse.org/legal/epl-v20.html
9+
*/
10+
11+
package org.junit.platform.testkit.engine;
12+
13+
import static java.util.Collections.unmodifiableList;
14+
import static org.apiguardian.api.API.Status.EXPERIMENTAL;
15+
16+
import java.util.List;
17+
18+
import org.apiguardian.api.API;
19+
import org.junit.platform.commons.util.Preconditions;
20+
import org.junit.platform.engine.DiscoveryIssue;
21+
import org.junit.platform.engine.TestDescriptor;
22+
23+
/**
24+
* {@code EngineDiscoveryResults} represents the results of test discovery
25+
* by a {@link org.junit.platform.engine.TestEngine TestEngine} on the JUnit
26+
* Platform and provides access to the {@link TestDescriptor} of the engine
27+
* and any {@link DiscoveryIssue DiscoveryIssues} that were encountered.
28+
*
29+
* @since 1.13
30+
*/
31+
@API(status = EXPERIMENTAL, since = "1.13")
32+
public class EngineDiscoveryResults {
33+
34+
private final TestDescriptor engineDescriptor;
35+
private final List<DiscoveryIssue> discoveryIssues;
36+
37+
EngineDiscoveryResults(TestDescriptor engineDescriptor, List<DiscoveryIssue> discoveryIssues) {
38+
this.engineDescriptor = Preconditions.notNull(engineDescriptor, "Engine descriptor must not be null");
39+
this.discoveryIssues = unmodifiableList(
40+
Preconditions.notNull(discoveryIssues, "Discovery issues list must not be null"));
41+
Preconditions.containsNoNullElements(discoveryIssues, "Discovery issues list must not contain null elements");
42+
}
43+
44+
/**
45+
* {@return the root {@link TestDescriptor} of the engine}
46+
*/
47+
public TestDescriptor getEngineDescriptor() {
48+
return engineDescriptor;
49+
}
50+
51+
/**
52+
* {@return the issues that were encountered during discovery}
53+
*/
54+
public List<DiscoveryIssue> getDiscoveryIssues() {
55+
return discoveryIssues;
56+
}
57+
58+
}

‎junit-platform-testkit/src/main/java/org/junit/platform/testkit/engine/EngineTestKit.java

+100-7
Original file line numberDiff line numberDiff line change
@@ -16,9 +16,11 @@
1616
import static org.apiguardian.api.API.Status.EXPERIMENTAL;
1717
import static org.apiguardian.api.API.Status.MAINTAINED;
1818
import static org.apiguardian.api.API.Status.STABLE;
19+
import static org.junit.platform.launcher.core.EngineDiscoveryOrchestrator.Phase.DISCOVERY;
1920
import static org.junit.platform.launcher.core.EngineDiscoveryOrchestrator.Phase.EXECUTION;
2021

2122
import java.nio.file.Path;
23+
import java.util.List;
2224
import java.util.Map;
2325
import java.util.ServiceLoader;
2426
import java.util.stream.Stream;
@@ -29,6 +31,7 @@
2931
import org.junit.platform.commons.util.CollectionUtils;
3032
import org.junit.platform.commons.util.Preconditions;
3133
import org.junit.platform.engine.DiscoveryFilter;
34+
import org.junit.platform.engine.DiscoveryIssue;
3235
import org.junit.platform.engine.DiscoverySelector;
3336
import org.junit.platform.engine.EngineDiscoveryRequest;
3437
import org.junit.platform.engine.EngineExecutionListener;
@@ -46,15 +49,24 @@
4649
import org.junit.platform.launcher.core.ServiceLoaderTestEngineRegistry;
4750

4851
/**
49-
* {@code EngineTestKit} provides support for executing a test plan for a given
50-
* {@link TestEngine} and then accessing the results via
51-
* {@linkplain EngineExecutionResults a fluent API} to verify the expected results.
52+
* {@code EngineTestKit} provides support for discovering and executing tests
53+
* for a given {@link TestEngine} and provides convenient access to the results.
54+
*
55+
* <p>For <em>discovery</em>, {@link EngineDiscoveryResults} provides access to
56+
* the {@link TestDescriptor} of the engine and any {@link DiscoveryIssue
57+
* DiscoveryIssues} that were encountered.
58+
*
59+
* <p>For <em>execution</em>, {@link EngineExecutionResults} provides a fluent
60+
* API to verify the expected results.
5261
*
5362
* @since 1.4
5463
* @see #engine(String)
5564
* @see #engine(TestEngine)
65+
* @see #discover(String, LauncherDiscoveryRequest)
66+
* @see #discover(TestEngine, LauncherDiscoveryRequest)
5667
* @see #execute(String, LauncherDiscoveryRequest)
5768
* @see #execute(TestEngine, LauncherDiscoveryRequest)
69+
* @see EngineDiscoveryResults
5870
* @see EngineExecutionResults
5971
*/
6072
@API(status = MAINTAINED, since = "1.7")
@@ -121,6 +133,65 @@ public static Builder engine(TestEngine testEngine) {
121133
return new Builder(testEngine);
122134
}
123135

136+
/**
137+
* Discover tests for the given {@link LauncherDiscoveryRequest} using the
138+
* {@link TestEngine} with the supplied ID.
139+
*
140+
* <p>The {@code TestEngine} will be loaded via Java's {@link ServiceLoader}
141+
* mechanism, analogous to the manner in which test engines are loaded in
142+
* the JUnit Platform Launcher API.
143+
*
144+
* <p>{@link org.junit.platform.launcher.core.LauncherDiscoveryRequestBuilder}
145+
* provides a convenient way to build an appropriate discovery request to
146+
* supply to this method. As an alternative, consider using
147+
* {@link #engine(TestEngine)} for a more fluent API.
148+
*
149+
* @param engineId the ID of the {@code TestEngine} to use; must not be
150+
* {@code null} or <em>blank</em>
151+
* @param discoveryRequest the {@code LauncherDiscoveryRequest} to use
152+
* @return the results of the discovery
153+
* @throws PreconditionViolationException for invalid arguments or if the
154+
* {@code TestEngine} with the supplied ID cannot be loaded
155+
* @since 1.13
156+
* @see #discover(TestEngine, LauncherDiscoveryRequest)
157+
* @see #engine(String)
158+
* @see #engine(TestEngine)
159+
*/
160+
@API(status = EXPERIMENTAL, since = "1.13")
161+
public static EngineDiscoveryResults discover(String engineId, LauncherDiscoveryRequest discoveryRequest) {
162+
Preconditions.notBlank(engineId, "TestEngine ID must not be null or blank");
163+
return discover(loadTestEngine(engineId.trim()), discoveryRequest);
164+
}
165+
166+
/**
167+
* Discover tests for the given {@link LauncherDiscoveryRequest} using the
168+
* supplied {@link TestEngine}.
169+
*
170+
* <p>{@link org.junit.platform.launcher.core.LauncherDiscoveryRequestBuilder}
171+
* provides a convenient way to build an appropriate discovery request to
172+
* supply to this method. As an alternative, consider using
173+
* {@link #engine(TestEngine)} for a more fluent API.
174+
*
175+
* @param testEngine the {@code TestEngine} to use; must not be {@code null}
176+
* @param discoveryRequest the {@code EngineDiscoveryResults} to use; must
177+
* not be {@code null}
178+
* @return the recorded {@code EngineExecutionResults}
179+
* @throws PreconditionViolationException for invalid arguments
180+
* @since 1.13
181+
* @see #discover(String, LauncherDiscoveryRequest)
182+
* @see #engine(String)
183+
* @see #engine(TestEngine)
184+
*/
185+
@API(status = EXPERIMENTAL, since = "1.13")
186+
public static EngineDiscoveryResults discover(TestEngine testEngine, LauncherDiscoveryRequest discoveryRequest) {
187+
Preconditions.notNull(testEngine, "TestEngine must not be null");
188+
Preconditions.notNull(discoveryRequest, "EngineDiscoveryRequest must not be null");
189+
LauncherDiscoveryResult discoveryResult = discover(testEngine, discoveryRequest, DISCOVERY);
190+
TestDescriptor engineDescriptor = discoveryResult.getEngineTestDescriptor(testEngine);
191+
List<DiscoveryIssue> discoveryIssues = discoveryResult.getDiscoveryIssues(testEngine);
192+
return new EngineDiscoveryResults(engineDescriptor, discoveryIssues);
193+
}
194+
124195
/**
125196
* Execute tests for the given {@link EngineDiscoveryRequest} using the
126197
* {@link TestEngine} with the supplied ID.
@@ -260,13 +331,16 @@ private static void executeDirectly(TestEngine testEngine, EngineDiscoveryReques
260331

261332
private static void executeUsingLauncherOrchestration(TestEngine testEngine,
262333
LauncherDiscoveryRequest discoveryRequest, EngineExecutionListener listener) {
263-
LauncherDiscoveryResult discoveryResult = new EngineDiscoveryOrchestrator(singleton(testEngine),
264-
emptySet()).discover(discoveryRequest, EXECUTION);
265-
TestDescriptor engineTestDescriptor = discoveryResult.getEngineTestDescriptor(testEngine);
266-
Preconditions.notNull(engineTestDescriptor, "TestEngine did not yield a TestDescriptor");
334+
LauncherDiscoveryResult discoveryResult = discover(testEngine, discoveryRequest, EXECUTION);
267335
new EngineExecutionOrchestrator().execute(discoveryResult, listener);
268336
}
269337

338+
private static LauncherDiscoveryResult discover(TestEngine testEngine, LauncherDiscoveryRequest discoveryRequest,
339+
EngineDiscoveryOrchestrator.Phase phase) {
340+
return new EngineDiscoveryOrchestrator(singleton(testEngine), emptySet()) //
341+
.discover(discoveryRequest, phase);
342+
}
343+
270344
@SuppressWarnings("unchecked")
271345
private static TestEngine loadTestEngine(String engineId) {
272346
Iterable<TestEngine> testEngines = new ServiceLoaderTestEngineRegistry().loadTestEngines();
@@ -446,6 +520,25 @@ public Builder outputDirectoryProvider(OutputDirectoryProvider outputDirectoryPr
446520
return this;
447521
}
448522

523+
/**
524+
* Discover tests for the configured {@link TestEngine},
525+
* {@linkplain DiscoverySelector discovery selectors},
526+
* {@linkplain DiscoveryFilter discovery filters}, and
527+
* <em>configuration parameters</em>.
528+
*
529+
* @return the recorded {@code EngineDiscoveryResults}
530+
* @since 1.13
531+
* @see #selectors(DiscoverySelector...)
532+
* @see #filters(Filter...)
533+
* @see #configurationParameter(String, String)
534+
* @see #configurationParameters(Map)
535+
*/
536+
@API(status = EXPERIMENTAL, since = "1.13")
537+
public EngineDiscoveryResults discover() {
538+
LauncherDiscoveryRequest request = this.requestBuilder.build();
539+
return EngineTestKit.discover(this.testEngine, request);
540+
}
541+
449542
/**
450543
* Execute tests for the configured {@link TestEngine},
451544
* {@linkplain DiscoverySelector discovery selectors},

0 commit comments

Comments
 (0)