Run test cases in order

Leave a reply

My project uses CI/CD pipeline. This builds the projects, run test cases and deploy to Stage/Dev/QA environments.

For CI/CD pipeline, my project uses GitHub actions.

Most of the times, when test cases are running, they run in random order and the order is not deterministic.

Maven Surefire is basically Maven’s test runner engine for your build lifecycle.

Primary Purpose

  • Discover your test classes (based on naming patterns like *Test, *Tests, Test*, etc.).

  • Run them using a supported testing framework (JUnit, TestNG, etc.).

  • Report the results in the console and generate HTML/XML reports.

Without Surefire, Maven would build your project but wouldn’t automatically execute your unit tests during the test phase.

Where it fits in Maven lifecycle

  • Maven has phases: validate → compile → test → package → verify → install → deploy.

  • Surefire is tied to the test phase.

  • So when you run:

mvn test

Maven:

  1. Compiles your main code (src/main/java).

  2. Compiles your test code (src/test/java).

  3. Hands over execution to Surefire to actually run the tests.

What it can do

  • Framework support: Runs JUnit (3, 4, 5), TestNG, Spock, Cucumber, etc.

  • Parallel execution: Run tests in parallel threads.

  • Filtering: Include/exclude specific test classes or methods.

  • Ordering: Control test run order (runOrder).

  • Reporting: Creates target/surefire-reports with plain text, XML, and sometimes HTML results.

  • Fail fast: Stop at the first failing test.

  • System property injection: Pass environment/config values to tests via Maven.

Why it matters

  • Ensures your unit tests are automatically executed every build.

  • Integrates seamlessly with CI/CD tools (Jenkins, GitHub Actions, GitLab CI, etc.).

  • Provides consistent test behavior across machines — no “works on my machine” excuses.

In a Maven/Gradle + JUnit + Mockito + Spring Boot project, there’s no built-in guarantee that one test class runs before another—JUnit treats tests as independent units and (by design) doesn’t define a default execution order.

Configure Maven Surefire Plugin

You can explicitly specify test run order in pom.xml:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>3.2.5</version>
    <configuration>
        <runOrder>
            alphabetical
        </runOrder>
    </configuration>
</plugin>

For the Maven Surefire Plugin, the default runOrder is:

filesystem

Meaning of filesystem

  • Tests are run in the order the operating system’s filesystem returns them.

  • This is not guaranteed to be alphabetical — it depends on the filesystem and OS.

  • On some machines it looks alphabetical, but on others (especially Linux with ext4) it can be seemingly random.

Other runOrder options you can set in Surefire:

  • alphabetical – strictly by class name in ascending order.

  • reversealphabetical – reverse order by class name.

  • random – shuffles the test class order.

  • hourly – changes the order every hour to help catch order-dependent tests.

  • balanced – orders tests based on previous execution time (slowest first).

  • filesystem – OS filesystem order (default).

 

Leave a comment