Introduce error handling mechanism for validation and discovery errors

[marcphilipp]

Overview

Validation errors (e.g., for invalid @BeforeEach method declarations) should not abort the entire discovery phase. Instead the discovery phase should continue, with the error tracked and reported during the execution phase.

Areas of Applicability

• unresolvable unique IDs, as discussed in Handle unparseable UniqueID consistently #210
  • note, however, that a TestEngine should not report a unique ID as unresolvable if the unique ID does not apply to the particular TestEngine (see JupiterTestEngine attempts to resolve unique IDs that belong to other engines #1026 ).
• invalid @BeforeAll, @AfterAll, @BeforeEach, and @AfterEach method declarations
• Competing method-level annotations (e.g. @Test and @TestTemplate on the same method)
• invalid @Test, @TestFactory, @RepeatedTest, and @ParameterizedTest method declarations (see IsTestableMethod silently ignores @Test annotated methods that return a value #2244)
• invalid @Nested test class declarations (see Improve diagnostics when Vintage test engine is missing #1223, Abstract classes with @Nested are not ignored / not handled properly #2717, and Feature suggestion: Detect missing @Nested annotations #1736)
• non-static test class declaration (see Better diagnostics for orphaned selectClass() on a nested class #2311)
• invalid @Suite class declarations
• blank display names supplied by the user, as discussed in Display names must not be allowed to be blank #743
• exceptions thrown by test engines, as discussed in DefaultLauncher should catch exceptions thrown by test engines #750

Proposals

1. Allow engines to track errors by creating a special type of TestDescriptor such as an AlwaysFailingTestDescriptor, DeadOnArrivalTestDescriptor, or ErrorReportingTestDescriptor.
   • Tracked errors for such a corresponding TestDescriptor could then be thrown as an exception during the execution phase instead of executing the corresponding container or test.
2. Introduce a new property in TestDescriptor that signals an error that was encountered during the discovery phase.
3. Pass a reporting object from the Launcher into each TestEngine to report errors.
   • See Introduce error handling mechanism for validation and discovery errors #242 (comment)

Related Issues

• Introduce validation rules for tests #121
• Handle unparseable UniqueID consistently #210
• Display names must not be allowed to be blank #743
• DefaultLauncher should catch exceptions thrown by test engines #750
• Forbid return values from test and lifecycle methods #835
• Improve error message if multiple "test" annotations are present on a method #876
• Move @TestFactory return type validation to IsTestFactoryMethod #949
• Static @BeforeEach method causes entire test plan to abort #971
• JupiterTestEngine attempts to resolve unique IDs that belong to other engines #1026
• Complain when @Test is applied to an inappropriate method #1074
• Improve diagnostics when Vintage test engine is missing #1223
• Unsupported or missing version of JUnit 4 should fail the build #1944
• IsTestableMethod silently ignores @Test annotated methods that return a value #2244
• Better diagnostics for orphaned selectClass() on a nested class #2311
• Abstract classes with @Nested are not ignored / not handled properly #2717
• Tests within inner classes that are children of an abstract class are not executed despite @Nested #4125

Deliverables

• Add initial Platform support for reporting discovery issues #4385
• Report problematic selector resolution results as discovery issues #4389
• Introduce EngineTestKit support for test discovery #4393
• Report discovery issues for Jupiter lifecycle methods #4398
• Report discovery issue for competing test method annotations #4401
• Report discovery issues for invalid suite lifecycle methods #4405
• Report discovery issues for invalid params invocation lifecycle methods #4408
• Report discovery errors for ineffectual lifecycle methods #4415
• Use the new mechanism to replace the current ad hoc use of logging and transparent defaulting as a work-around.
  • Search for TODO [#242] within the code base.
  • See DefaultLauncher should catch exceptions thrown by test engines #750 (comment)
• Determine where else the new mechanism can be used and apply it.
• Revisit the results of Forbid return values from test and lifecycle methods #835 and update the code accordingly by tracking such errors instead of just ignoring such incorrect usage.
• Revisit the changes made in Static @BeforeEach method causes entire test plan to abort #971 and determine if it makes sense to move the look-up of lifecycle methods back to the constructor of ClassTestDescriptor.
• Add JFR events for reporting validation and configuration errors

[sbrannen]

The same is true for invalid custom display names supplied by the user, as described in #743.

I will therefore update the description of this issue to reflect the broader scope.

[sbrannen]

FYI: I have introduced a minimum set of Deliverables.

[gaganis]

I am thinking this reminds me the approach we took for vintage to handle multiple assertion errors using a MultipleFailuresError.

We could be possible that we could store all validation errors in a Throwable - using a MultipleFailuresError when there is more than one, that would be carried over to the execution phase and when this is nonEmpty we would throw the exception instead of executing the test.

---

It might also be worth considering for this mechanism to track all exceptions that occur while processing a single discovered test and throw it in the execution phase.

This way instead of the whole process stopping, we could still process the rest of the tests that encountered no problems and only fail on single tests.

PS While I was writing, this @sbrannen changed the title and description and now it sound more like what I am writing here! 😃

[sbrannen]

@gaganis, thanks for sharing your ideas.

A few notes...

No need to use MultipleFailuresError.

We're on Java 8 and can use suppressed exceptions.

Plus, we already have a ThrowableCollector within the JUnit Jupiter engine that could potentially be used for this purpose as well.

[sbrannen]

The scope of this issue has been broadened in order to encompass the Platform as well.

[jlink]

Rather than using exceptions of any kind, having some kind of notification collector as a return or in/out parameter seems more flexible since any exception will stop an engine's discovery phase whereas many notifications are just warnings or pieces of information .

[sbrannen]

Rather than using exceptions of any kind, having some kind of notification collector as a return or in/out parameter seems more flexible since any exception will stop an engine's discovery phase whereas many notifications are just warnings or pieces of information .

Good point!

I'll change the title of this issue accordingly.

[sbrannen]

FYI: title and description have updated.