#[Retry] attribute to support retrying flaky test

[santysisi]

✨ Add Retry Attribute for Retrying Flaky Tests

This PR introduces a new Retry attribute that can be applied to individual test methods in PHPUnit. Its goal is to help deal with flaky or unstable tests that occasionally fail due to non-deterministic reasons (e.g. network issues, race conditions, external dependencies).

🔧 How it works

The Retry attribute allows a test to be automatically retried a specified number of times before being marked as failed. You can also control the delay between retries, and optionally restrict retries to specific exception types.

🧪 Usage

Here’s how you can use it:

#[Retry(3)] // Retry up to 3 times on any exception
public function testFlakyApi(): void
{
    // test logic
}

You can also specify a delay (in seconds) between retries:

#[Retry(2, delay: 2)] // Retry twice, with 2 seconds delay between attempts

And restrict retries to certain exception types only:

#[Retry(4, retryOn: TimeoutException::class)]

All parameters:

• maxRetries (int) — how many times the test can be retried.
• delay (?int) — seconds to wait before retrying. Default: 0
• retryOn (?string) — exception type to retry on. If omitted, retries on any throwable.

✅ Example

#[Retry(2)]
public function testRetriesUntilMaxAttemptsThenSucceeds(): void
{
    static $attempt = 0;

    if ($attempt++ < 2) {
        throw new RuntimeException('Transient failure');
    }

    $this->assertSame(3, $attempt);
}

📈 Benefits

• Stabilize CI pipelines by automatically retrying flaky tests
• Fine-grained control over retry behavior per test
• Non-intrusive: no need to change existing test logic
• Optional support for retrying only on specific exceptions

🔄 Prior art

Other popular testing frameworks already provide similar functionality:

• NUnit: Retry Attribute
• JUnit Pioneer: RetryingTest

[santysisi]

This PR could also offer a complementary approach to the discussion in #5718: Bring back --repeat CLI option.

While --repeat focuses on re-running tests via the CLI, the #[Retry] attribute is designed for more fine-grained control, enabling retries at the test method level, optionally with delay or exception-based filtering. It might be a useful alternative for handling flaky tests directly in code, without requiring changes to CI configs or CLI commands.

Thanks again, looking forward to your feedback! 😊

[sebastianbergmann]

Only had a very cursory look at the proposed changes yet, but this caught my eye: how are tests retried that use expectException(), for example?

[santysisi]

Thanks for pointing that out!

Just to clarify how this works: in PHPUnit, if an exception is thrown after calling expectException(), it's treated as a failure, not an error and failures are not retried. On the other hand, if an exception is thrown before expectException() is set, PHPUnit treats it as an error, which will trigger a retry.

For example:

#[Retry(2)]
public function testFoo(): void
{
    throw new Exception;
    $this->expectException(Exception::class);
}

This will be retried, since the exception occurs before the expected exception is declared.

Whereas:

#[Retry(2)]
public function testFoo(): void
{
    $this->expectException(Exception::class);
    throw new Exception;
}

This will not be retried, because the exception is expected and treated as a failure if something goes wrong, not an unexpected error.

So i believe (maybe I'm wrong) that the the retry behavior aligns with how PHPUnit distinguishes between errors and failures.

Thanks again for your comment
I really appreciate the feedback 😄 .

[sebastianbergmann]

The manual call to getIterator() is superfluous as MetadataCollection implements IteratorAggregate.

[santysisi]

Thank you for the comment 😄
good catch. I agree with you, the manual call to getIterator() is unnecessary given that MetadataCollection implements IteratorAggregate. I'll make that change.