Run Java Tests With Maven Silently (Only Log on Failure)

When running our Java tests with Maven they usually produce a lot of noise in the console. While this log output can help understand test failures, it's typically superfluous when our test suite is passing. Nobody will take a look at the entire output if the tests are green. It's only making the build logs more bloated. A better solution would be to run our tests with Maven silent with no log output and only dump the log output once the tests fail. This blog post demonstrates how to achieve this simple yet convenient technique to run Java tests with Maven silently for a compact and followable build log.

The Status Quo: Noisy Java Tests

Based on our logging configuration, our tests produce quite some log output. When running our entire test suite locally or on a CI server (e.g., GitHub Actions or Jenkins), analyzing a test failure is tedious when there's a lot of noise in the logs. We first have to find our way to the correct position by scrolling or using the search functionality of, e.g., our browser or the integrated terminal of our IDE.

A demo output for a test that verifies email functionality using GreenMail looks like the following:

There's usually a lot of default noise of frameworks and test libraries that add up to quite some log output when running an entire test suite:

While we could tweak our logger configuration and set the log level to ERROR for the framework and libraries logs, their INFO can still be quite relevant when analyzing a test failure.

When scrolling through the log output of passing tests, we might also see stack traces and exceptions that are intended but might confuse newcomers as they wonder if something went wrong there.

Having a clean build log without much noise would better help us follow the current build. The bigger our test suite, the more we have to scroll.

If all tests pass, why pollute the console with log output from the tests? Our Maven build might also fail for different reasons than test failures, e.g., a failing OWASP dependency check or a dependency convergence issue. Getting fast to the root cause of the build failure is much simpler with a compact build log.

The Goal: Run Tests with Maven Silently

Our goal for this optimization is to have a compact Maven build log and only log the test output if it's really necessary (aka. tests are failing).

Gradle is doing this already by default. When running tests with Gradle, we'll only see a test summary after running our tests. There's no intermediate noise inside our console.

The goal is to achieve a somehow similar behavior as Gradle and run our tests silently. If they're passing, we're fine, and there's (usually) no need to investigate the log outcome of our tests. If one of our tests fails, report the build log to the console to analyze the test failure.

In short, with our target solution, we have two scenarios:

• No log output for tests in the console when all tests pass
• Print the log output of our tests when a test fails

The second scenario should be (hopefully) less likely. Hence most of our Maven builds should result in a compact and clean build log. We're fine with the log noise if there's a failure, as it helps us understand what went wrong.

Let's see how we can achieve this with the least amount of configuration.

The Solution: Customized Maven Setup

As a first step, we configure the desired log level for testing:

We're using Logback (any logger works) and log any INFO (and above) statement to the console for the example above. We don't differentiate between our application's log and framework or test libraries.

Next comes the important configuration that'll make our test silent. The Maven Surefire (unit tests) and the Failsafe (integration tests) plugin allow redirecting the console output to a file. We won't see any test log output in the console with this configuration as it's stored within a file:

When activating this functionality (redirectTestOutputToFile), both plugins create an output file inside the target folder for each test class with the naming scheme TestClassName-output.txt.

We can override the location of the output files using the reportsDirectory configuration option. Overriding this location helps us store the output of the Surefire and Failsafe plugin at the same place:

This configuration for bot the Surefire and Failsafe plugin will mute our test runs, and Maven will only display a test execution summary for each test class:

This compact build log makes it even fun to watch the test execution (assuming there are no flaky tests).

After running our tests, we can take a look at the content of the test-reports folder:

For each test class, we'll find (at least) one text file that contains the test summary as we saw it in the build log. If the test prints output to the console, there'll be a -output.txt file with the content:

• de.rieckpil.blog.greenmail.MailServiceTest-output.txt: All console output of the test
• de.rieckpil.blog.greenmail.MailServiceTest.txt: The test summary, as seen in the build log

What's left is to extract the content of all our *-output.txt files if our build is failing. As long as our tests are all green, we can ignore the content of the output files. In case of a test failure, we must become active and dump the file contents to the console.

For this purpose, we're using a combination of find and tail.For demonstration purposes, we'll use GitHub Actions. However, the present solution is portable to any other build server that provides functionality to detect a build failure and execute shell commands:

As part of the last step of our build workflow, we find all *-output.txt files and print their content. We only print the content of the test output files in case of a failure.

With GitHub Actions, we can conditionally execute a step using a boolean expression: if: failure() || cancelled(). Both failure() and cancelled() are built-in functions of GitHub Actions. Every other CI server provides some similar functionality. We include cancelled() to the expression to cover the scenario when our test suite is stuck and we manually stop (aka. cancel) the build.

If the build is passing, this last logging step is skipped, and no test log output is logged.

By using tail -n +1 {} we print the file name before dumping its content to the console. This helps search for the failed test class to start the investigation:

Summary: Silent and Compact Build Logs

We'll get compact Maven build logs with this small tweak to the Maven Surefire and Failsafe plugin and the additional step inside our build server. No more noisy test runs. We won't lose any test log output as we temporarily park it inside files and inspect the files if a test fails.

This configuration will only affect the way our tests are run with Maven. We can still see the console output when executing tests within our IDE.

We'll capture any test console output with this mechanism, both from logging libraries and plain System.out.println calls.

This technique also works when running our tests in parallel. However, if we parallelize the test methods, the console statements may be out of order inside the test output file.

If you want to see this technique in action for a public repository, take a look at the Java Testing Toolbox repository on GitHub. As part of the main GitHub Actions workflow that builds the project(s) with Maven, you'll see the Java tests being run silently. If there's a build failure, you'll see the content of the test output files as one of the last jobs.

Joyful testing,

Philip