Starting with a new programming language is always exciting. However, it can be overwhelming as we have to get comfortable with the language, the tools, conventions, and the general development workflow. This holds true for both developing and testing our applications. When testing Java applications with Maven, there are several concepts and conventions to understand: Maven lifecycles, build phases, plugins, etc. With this blog post, we'll cover the basic concepts for you to understand how testing Java applications with Maven works.
What Do We Need Maven For?
When writing applications with Java, we can't just pass our .java files to the JVM (Java Virtual Machine) to run our program. We first have to compile our Java source code to bytecode (.class files) using the Java Compiler (javac). Next, we pass this bytecode to the JVM (java binary on our machines) which then interprets our program and/or compiles parts of it even further to native machine code.
Given this two-step process, someone has to compile our Java classes and package our application accordingly. Manually calling javac and passing the correct classpath is a cumbersome task. A build tool automates this process. As developers, we then only have to execute one command, and everything get's build automatically.
The two most adopted build tools for the Java ecosystem are Maven and Gradle. Ancient devs might still prefer Ant, while latest-greatest devs might advocate for Bazel as a build tool for their Java applications.
We're going to focus on Maven with this article.
To build and test our Java applications, we need a JDK (Java Development Kit) installed on our machine and Maven. We can either install Maven as a command-line tool (i.e., place the Maven binary on our system's PATH) or use the portable Maven Wrapper.
The Maven Wrapper is a convenient way to work with Maven without having to install it locally. It allows us to conveniently build Java projects with Maven without having to install and configure Maven as a CLI tool on our machine
When creating a new Spring Boot project, for example, you might have already wondered what the mvnw and mvnw.cmd files inside the root of the project are used for. That's the Maven Wrapper (the idea is borrowed from Gradle).
Creating a New Maven Project
There are several ways to bootstrap a new Maven project. Most of the popular Java application frameworks offer a project bootstrapping wizard-like interface. Good examples are the Spring Initializr for new Spring Boot applications, Quarkus, MicroProfile.
If we want to create a new Maven project without any framework support, we can use a Maven Archetype to create new projects. These archetypes are a project templating toolkit to generate a new Maven project conveniently.
Maven provides a set of default Archetypes artifacts for several purposes like a new web app, a new Maven plugin project, or a simple quickstart project.
We bootstrap a new Java project from one of these Archetypes using the mvn command-line tool:
| 1 2 3 4 5 6 | mvn archetype:generate \ -DarchetypeGroupId=org.apache.maven.archetypes \ -DarchetypeArtifactId=maven-archetype-quickstart \ -DarchetypeVersion=1.4 \ -DgroupId=com.mycompany \ -DartifactId=order-service |
The skeleton projects we create with the official Maven Archetypes are a good place to start.
However, some of these archetypes generate projects with outdated dependency versions like JUnit 4.11. While it's not a big effort to manually bump the dependency version after the project initialization, having an up-to-date Maven Archetype in the first place is even better.
Minimal Maven Project For Testing Java Applications
As part of my Custom Maven Archetype open-source project on GitHub, I've published a collection of useful Maven Archetypes. One of them is the java-testing-toolkit to create a Java Maven project with basic testing capabilities. Creating our own Maven Archetype is almost no effort.
We can create a new testing playground project using this custom Maven Archetype with the following Maven command (for Linux & Mac):
| 1 2 3 4 5 6 | mvn archetype:generate \ -DarchetypeGroupId=de.rieckpil.archetypes \ -DarchetypeArtifactId=testing-toolkit \ -DarchetypeVersion=1.0.0 \ -DgroupId=com.mycompany \ -DartifactId=order-service |
For Windows (both PowerShell and CMD), we can use the following command to bootstrap a new project from this template:
| 1 | mvn archetype:generate "-DarchetypeGroupId=de.rieckpil.archetypes" "-DarchetypeArtifactId=testing-toolkit" "-DarchetypeVersion=1.0.0" "-DgroupId=com.mycompany" "-DartifactId=order-service" "-DinteractiveMode=false" |
We can adjust both -DgroupId and -DartifactId to our project's or company's preference.
The generated project comes with a basic set of the most central Java testing libraries. We can use it as a blueprint for our next project or explore testing Java applications with this playground.
In summary, the following default configuration and libraries are part of this project shell:
- Java 11
- JUnit Jupiter, Mockito, and Testcontainers dependencies
- A basic unit test
- Maven Surefire and Failsafe Plugin configuration
- A basic
.gitignore - Maven Wrapper
Next, we have to ensure a JDK 11 (or higher) is on our PATH and also JAVA_HOME points to the installation folder:
| 1 2 3 4 5 6 7 8 9 10 11 12 | $ java -version openjdk version "11.0.10" 2021-01-19 OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.10+9) OpenJDK 64-Bit Server VM AdoptOpenJDK (build 11.0.10+9, mixed mode) # Windows $ echo %JAVA_HOME% C:\Program Files\AdoptOpenJDK\jdk-11.0.10.9-hotspot # Mac and Linux $ echo $JAVA_HOME /usr/lib/jvm/adoptopenjdk-11.0.10.9-hotspot |
As a final verification step, we can now build and test this project with Maven:
| 1 2 3 4 5 6 7 8 9 10 11 | $ mvn archetype:generate ... // generate the project $ cd order-service // navigate into the folder $ ./mvnw package // mvnw.cmd package for Windows .... [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 3.326 s [INFO] Finished at: 2021-06-03T08:31:11+02:00 [INFO] ------------------------------------------------------------------------ |
We can now open and import the project to our editor or IDE (IntelliJ IDEA, Eclipse, NetBeans, Visual Code, etc.) to inspect the generated project in more detail.
Important Testing Folders and Files
First, let's take a look at the folders and files that are relevant for testing Java applications with Maven:
src/test/java
This folder is the main place to add our Java test classes (.java files). As a general recommendation, we should try to mirror the package structure of our production code (src/main/java). Especially if there's a direct relationship between the test and a source class.
The corresponding CustomerServiceTest for a CustomerService class inside the package com.company.customer should be placed in the same package within src/test/java. This improves the likelihood that our colleagues (and our future us) locate the corresponding test for a particular Java class without too many facepalms.
Most of the IDEs and editors provide further support to jump to a test class. IntelliJ IDEA, for example, provides a shortcut (Ctrl+ Shift + T) to navigate from a source file to its test classes(s) and vice-versa.
src/test/resources
As part of this folder, we store static files that are only relevant for our test. This might be a CSV file to import test customers for an integration test, a dummy JSON response for testing our HTTP clients, or a configuration file.
target/test-classes
At this location, Maven places our compiled test classes (.class files) and test resources whenever the Maven compiler compiles our test sources. We can explicitly trigger this with mvn test-compile and add a clean if we want to remove the existing content of the entire target folder first.
Usually, there's no need to perform any manual operations inside this folder as it contains build artifacts. Nevertheless, it's helpful to investigate the content for this folder whenever we face test failures because we, e.g., can't read a file from the classpath. Taking a look at this folder (after the Maven compiler did its work), can help understanding where a resources file ended up on the classpath.
pom.xml
This is the heart of our Maven project. The abbreviation stands for Project Object Model. Within this file, we define metadata about our project (e.g., description, artifactId, developers, etc.), which dependencies we require, and the configuration of our plugins.
Maven and Java Testing Naming Conventions
Next, let's take a look at the naming conventions for our test classes. We can separate our tests into two (or even more) basic categories: unit and integration test. To distinguish the tests for both of these two categories, we use different naming conventions with Maven.
The Maven Surefire Plugin, more about this plugin later, is designed to run our unit tests. The following patterns are the defaults so that the plugin will detect a class as a test:
**/Test*.java**/*Test.java**/*Tests.java**/*TestCase.java
So what's actually a unit test?
Several smart people came up with a definition for this term. One of such smart people is Michael Feathers. He's turning the definition around and defines what a unit test is not:
A test is not a unit test if it …
- talks to the database
- communicates across the network
- touches the file system
- can't run at the same time as any of your other unit tests
- or you have to do special things to your environment (such as editing config files) to run it.
Kevlin Henney is also a great source of inspiration for a definition of the term unit test.
Nevertheless, our own definition or the definition of our coworkers might be entirely different. In the end, the actual definition is secondary as long as we're sharing the same definition within our team and talking about the same thing when referring to the term unit test.
The Maven Failsafe Plugin, designed to run our integration tests, detects our integration tests by the following default patterns:
**/IT*.java**/*IT.java**/*ITCase.java
We can also override the default patterns for both plugins and come up with a different naming convention. However, sticking to the defaults is recommended.
When Are Our Java Tests Executed?
Maven is built around the concept of build lifecycles. There are three built-in lifecycles:
default: handling project building and deploymentclean: project cleaningsite: the creation of our project's (documentation) site
Each of the three built-in lifecycles has a list of build phases. For our testing example, the default lifecycle is important.
The default lifecycle compromises a set of build phases to handle building, testing, and deploying our Java project. Each phase represents a stage in the build lifecycle with a central responsibility:
In short, the several phases have the following responsibilities:
validate: validate that our project setup is correct (e.g., we have the correct Maven folder structure)compile: compile our source code withjavactest: run our unit testspackage: build our project in its distributable format (e.g., JAR or WAR)verify: run our integration tests and further checks (e.g., the OWASP dependency check)install: install the distributable format into our local repository (~/.m2folder)deploy: deploy the project to a remote repository (e.g., Maven Central or a company hosted Nexus Repository/Artifactory)
These build phases represent the central phases of the default lifecycle. There are actually more phases. For a complete list, please refer to the Lifecycle Reference of the official Maven documentation.
Whenever we execute a build phase, our project will go through all build phases and sequentially until the build phase we specified. To phrase it differently, when we run mvn package, for example, Maven will execute the default lifecycle phases up to package in order:
| 1 | validate -> compile -> test -> package |
If one of the build phases in the chain fails, the entire build process will terminate. Imagine our Java source code has a missing semicolon, the compile phase would detect this and terminate the process. As with a corrupt source file, there'll be no compiled .class file to test.
When it comes to testing our Java project, both the test and verify build phases are of importance. As part of the test phase, we're running our unit tests with the Maven Surefire Plugin, and with verify our integration tests are executed by the Maven Failsafe Plugin.
Let's take a look at these two plugins.
Running Unit Tests With the Maven Surefire Plugin
The Maven Surefire is responsible for running our unit tests. We must either follow the default naming convention of our test classes, as discussed above, or configure a different pattern that matches our custom naming convention. In both cases, we have to place our tests inside src/test/java folder for the plugin to pick them up.
For the upcoming examples, we're using a basic format method
| 1 2 3 4 5 6 | public class Main { public String format(String input) { return input.toUpperCase(); } } |
… and its corresponding test as a unit test blueprint:
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 | import org.junit.jupiter.api.BeforeEach; import org.junit.jupiter.api.Test; import static org.junit.jupiter.api.Assertions.assertEquals; class MainTest { private Main cut; @BeforeEach void setUp() { this.cut = new Main(); } @Test void shouldReturnFormattedUppercase() { String input = "duke"; String result = cut.format(input); assertEquals("DUKE", result); } } |
Depending on the Maven version and distribution format of our application (e.g., JAR or WAR), Maven defines default versions for the core plugins. Besides the Maven Compiler Plugin, the Maven Resource Plugin, and other plugins, the Maven Surefire Plugin is such a core plugin.
When packaging our application as a JAR file and using Maven 3.8.1, for example, Maven picks the Maven Surefire Plugin with version 2.12.4 by default unless we override it. As the default versions are sometimes a little bit behind the latest plugin versions, it's worth updating the plugin versions and manually specifying the plugin version inside our pom.xml:
| 1 2 3 4 5 6 7 8 9 10 11 12 | <project> <!-- dependencies --> <build> <plugins> <plugin> <artifactId>maven-surefire-plugin</artifactId> <version>3.0.0-M5</version> </plugin> </plugins> </build> </project> |
As part of the test phase of the default lifecycle, we'll now see the Maven Surefire Plugin executing our tests:
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 | $ mvn test [INFO] --- maven-surefire-plugin:3.0.0-M5:test (default-test) @ testing-example --- [INFO] [INFO] ------------------------------------------------------- [INFO] T E S T S [INFO] ------------------------------------------------------- [INFO] Running de.rieckpil.blog.MainTest [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.04 s - in de.rieckpil.blog.MainTest [INFO] [INFO] Results: [INFO] [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0 [INFO] [INFO] ------------------------------------------------------------------------ |
For the example above, we're running one unit test with JUnit 5 (testing provider). There's no need to configure the testing provider anywhere, as with recent Surefire versions, the plugin will pick up the correct test provider by itself. The Maven Surefire Plugin integrates both JUnit and TestNG as testing providers out-of-the-box.
If we don't want to execute all build phases before running our tests, we can also explicitly execute the test goal of the Surefire plugin:
| 1 | mvn surefire:test |
But keep in mind that we have to ensure that the test classes have been compiled first (e.g., by a previous build).
We can further tweak and configure the Maven Surefire Plugin to, e.g., parallelize the execution of our unit tests. This is only relevant for JUnit 4, as JUnit 5 (JUnit Jupiter to be precise) supports parallelization on the test framework level.
Whenever we want to skip our unit tests when building our project, we can use an additional parameter:
| 1 | mvn package -DskipTests |
We can also explicitly run only one or multiple tests:
| 1 2 3 4 5 | mvn test -Dtest=MainTest mvn test -Dtest=MainTest#testMethod mvn surefire:test -Dtest=MainTest |
Running Integration Tests With the Maven Failsafe Plugin
Unlike the Maven Surefire Plugin, the Maven Failsafe Plugin is not a core plugin and hence won't be part of our project unless we manually include it. As already outlined, the Maven Failsafe plugin is used to run our integration test.
In contrast to our unit tests, the integration tests usually take more time, more setup effort (e.g., start Docker containers for external infrastructure with Testcontainers), and test multiple components of our application together.
We integrate the Maven Failsafe Plugin by adding it to the build section of our pom.xml:
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 | <project> <!-- other dependencies --> <build> <!-- further plugins --> <plugin> <artifactId>maven-failsafe-plugin</artifactId> <version>3.0.0-M5</version> <executions> <execution> <goals> <goal>integration-test</goal> <goal>verify</goal> </goals> </execution> </executions> </plugin> </plugins> </build> </project> |
As part of the executions configuration, we specify the goals of the Maven Failsafe plugin we want to execute as part of our build process. A common pitfall is to only execute the integration-test goal. Without the verify goal, the plugin will run our integration tests but won't fail the build if there are test failures.
The Maven Failsafe Plugin is invoked as part of the verify build phase of the default lifecycle. That's right after the package build phase where we build our distributable artifact (e.g., JAR):
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 | [INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ testing-example --- [INFO] Building jar: C:\Users\phili\Desktop\junk\testing-example\target\testing-example.jar [INFO] [INFO] --- maven-failsafe-plugin:3.0.0-M5:integration-test (default) @ testing-example --- [INFO] [INFO] ------------------------------------------------------- [INFO] T E S T S [INFO] ------------------------------------------------------- [INFO] Running de.rieckpil.blog.MainIT [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.039 s - in de.rieckpil.blog.MainIT [INFO] [INFO] Results: [INFO] [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0 [INFO] [INFO] [INFO] --- maven-failsafe-plugin:3.0.0-M5:verify (default) @ testing-example --- [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ |
If we want to run our integration tests manually, we can do so with the following command:
| 1 | mvn failsafe:integration-test failsafe:verify |
For scenarios where we don't want to run our integration test (but still our unit tests), we can add -DskipITs to our Maven execution:
| 1 | mvn verify -DskipITs |
Similar to the Maven Surefire Plugin, we can also run a subset of our integration tests:
| 1 2 3 | mvn -Dit.test=MainIT failsafe:integration-test failsafe:verify mvn -Dit.test=MainIT#firstTest failsafe:integration-test failsafe:verify |
When using the command above, make sure the test classes have been compiled previously, as otherwise, there won't be any test execution.
There's also a property available to entirely skip the compilation of test classes and avoid running any tests when building our project (not recommended):
| 1 | mvn verify -Dmaven.test.skip=true |
Summary of Testing Java Applications With Maven
Maven is a powerful, mature, and well-adopted build tool for Java projects. As a newcomer or when coming from a different programming language, the basics of the Maven build lifecycle and how and when different Maven Plugins interact is something to understand first.
With the help of Maven Archetypes or using a framework initializer, we can easily bootstrap new Maven projects. There's no need to install Maven as a CLI tool for our machine as we can instead use the portable Maven Wrapper.
Furthermore, keep this in mind when testing your Java applications and use Maven as the build tool:
- With Maven, we can separate the unit and integration test execution
- The Maven Surefire Plugin runs our unit tests
- The Maven Failsafe Plugin runs our integration tests
- By following the default naming conventions for both plugins, we can easily separate our tests
- The Maven default lifecycle consists of several build phases that are executed in order and sequentially
- Use the Java Testing Toolkit Maven archetype for your next testing adventure
The source code for this article about testing Java applications with Maven is available on GitHub.
Joyful testing,
Philip
