[MM’s] Boot Notes — JUnit Retries: The Good, The Bad and The Flaky

Recognizing and managing non-deterministic test failures.

Every developer knows the steps.

A pull request fails CI.
You open the logs.
Nothing obvious.
You click “Re-run failed jobs”.
Everything passes.

Welcome to flaky tests — failures that appear and disappear without code changes. They’re not just annoying; they’re expensive. In a suite of 1,000 tests, even a 1% flakiness rate almost guarantees daily CI failures.

The uncomfortable truth? Some flaky tests are here to stay for a while. The challenge isn’t just fixing them — it’s maintaining a stable pipeline without pretending they don’t exist.

⚡ TL;DR (Quick Recap)

Flaky tests don’t fail once — they fail your process.
JUnit 5 extensions allow retries without restarting the JVM or hiding failures.
Retries should be limited, logged and visible — a safety net, not a solution.
Use retry logic to stabilize CI while actively fixing the root causes.

Why Tests Become Flaky

Most non-deterministic failures fall into familiar categories:

Asynchronous timing Reactive pipelines, message brokers or background jobs tested with sleep() instead of explicit waits.
Infrastructure noise Network hiccups, shared CI runners or transient cloud service issues.
State leakage Dirty databases, static fields or parallel tests stepping on each other.

Retries don’t fix any of these — but they can keep your pipeline usable while you work on them.

Retry Strategies (and Their Limits)

Before adding retry logic, it’s worth knowing your options:

Fix the root cause Always the goal. Often not immediate.
Quarantine flaky tests Move them into a non-blocking suite.
Build-tool retries Maven Surefire or Gradle retries work — but they restart the JVM.
External libraries: JUnit Pioneer’s @RetryingTest
JUnit extensions Fine-grained, explicit and local to the test lifecycle.

A retries with JUnit

JUnit exposes the InvocationInterceptor interface, which allows interception of test failures inside the engine.

No JVM restarts
No build-tool magic
Full access to test context and state

Let’s build a retry extension that is small, visible and transparent.

The Retry Extension

@Target(ElementType.METHOD)
@Retention(RetentionPolicy.RUNTIME)
@ExtendWith(RetryableExtension.class)
public @interface Retryable {
    int maxAttempts() default 3;
}

public class RetryableExtension implements InvocationInterceptor {
    private static final Logger log = LoggerFactory.getLogger(RetryableExtension.class);
    private static final String RETRY_COUNT_KEY = "retry_count";

    @Override
    public void interceptTestMethod(@NonNull Invocation<Void> invocation,
                                    @NonNull ReflectiveInvocationContext<Method> invocationContext,
                                    ExtensionContext extensionContext) throws Throwable {

        var retryable = extensionContext.getRequiredTestMethod()
                .getAnnotation(Retryable.class);

        if (retryable == null) {
            invocation.proceed();
            return;
        }

        var maxAttempts = retryable.maxAttempts();
        var store = getStore(extensionContext);

        for (var attempt = 0; attempt < maxAttempts; attempt++) {
            store.put(RETRY_COUNT_KEY, attempt);

            try {
                if (attempt == 0) {
                    // First attempt - use normal JUnit invocation (includes full lifecycle)
                    invocation.proceed();
                } else {
                    // Retry - re-invoke test method only
                    // Note: @BeforeEach/@AfterEach won't run again
                    extensionContext.getRequiredTestMethod()
                            .invoke(extensionContext.getRequiredTestInstance());
                }

                // Test passed
                if (attempt > 0) {
                    log.info("✓ Test '{}' passed on attempt {}/{}",
                            extensionContext.getDisplayName(),
                            attempt,
                            maxAttempts
                    );
                }
                return; // Exit on success

            } catch (Throwable throwable) {
                // Unwrap reflection exceptions to get the real cause
                var actualException = unwrapException(throwable);

                if (attempt < (maxAttempts - 1)) {
                    log.warn("✗ Test '{}' failed on attempt {}/{}. Error: {} - retrying...",
                            extensionContext.getDisplayName(),
                            attempt,
                            maxAttempts,
                            actualException.getMessage()
                    );
                } else {
                    log.error("✗ Test '{}' failed after {} attempts. Giving up.",
                            extensionContext.getDisplayName(),
                            maxAttempts
                    );
                    throw actualException;
                }
            }
        }
    }

    /**
     * Unwrap InvocationTargetException to get the actual test failure.
     */
    private Throwable unwrapException(Throwable throwable) {
        if (throwable instanceof java.lang.reflect.InvocationTargetException) {
            var cause = throwable.getCause();
            return cause != null ? cause : throwable;
        }
        return throwable;
    }

    private ExtensionContext.Store getStore(ExtensionContext context) {
        return context.getStore(
                ExtensionContext.Namespace.create(getClass(), context.getRequiredTestMethod())
        );
    }
}

How This Works

Namespaced state Each test method gets its own retry counter — no leakage between tests.
Same test instance The retry runs on the existing test object, preserving context.
Hard stop Once retries are exhausted, the original failure is rethrown.

Limitations

Bypasses Other Extensions: Won’t work with other lifecycle extensions
No Parallel Support: May fail with @Execution(CONCURRENT)
Nested Tests: Doesn't handle @Nested test classes
More Complex: More code paths, more potential bugs

For production code requiring full lifecycle re-execution, consider:

JUnit Pioneer’s @RetryingTest

Battle-tested in production
Handles all edge cases correctly
Supports parallel execution
Works with nested tests

Using the Extension

Apply it where flakiness is expected — typically integration tests.


class CloudServiceIntegrationTest {
    @Test
    @Retryable
    void unstableExternalDependency() {
        assertTrue(remoteService.call().isSuccessful());
    }
}

Unit tests should remain deterministic.

Engineering Trade-offs

Retries are a loan against technical debt. Treat them accordingly.

Longer CI times Acceptable if scoped to a small test set.
Stateful side effects Tests must clean up after themselves.
False confidence A “flaky pass” is still a failure signal.

A simple rule that works well in practice:

If more than 5% of builds rely on retries, stop feature work and fix the system.

Retries should buy time, not erase accountability.

Final Takeaways

JUnit extension model gives us a precise tool for handling flaky tests without polluting build logic or restarting the JVM. Used carefully, a retry extension can stabilize CI pipelines while teams focus on fixing real problems.

Use retries to keep your pipeline green. Use logs and metrics to keep your architecture honest.

You can find example of code on GitHub.

Originally posted on marconak-matej.medium.com.

[MM’s] Boot Notes — JUnit Retries: The Good, The Bad and The Flaky