Java Benchmarking With JMH (Java Microbenchmark Harness)

Last Updated:  December 26, 2020 | Published: May 12, 2019

In one of my previous blog posts, I showed you a simple way to load-test your application with Apache Benchmark. As this solution is suitable for e.g. testing a REST interface, this approach is not the best for benchmarking Java methods directly. Luckily we have a tool for this within the JVM ecosystem: JMH (Java Microbenchmark Harness).

The official page says the following about JMH:

JMH is a Java harness for building, running, and analysing nano/micro/milli/macro benchmarks written in Java and other languages targetting the JVM.

To show you how easy you can write benchmarks with JMH, I'll provide an example with this blog post and will benchmark the serialization of JSON strings to Java objects with Jackson and Gson on Java 8 (the provided example is just for demonstration purposes, please don't take this as a valid and bulletproof benchmark comparison of these libraries).

JMH Project setup

Bootstrapping a new benchmark project is done with the help of a Maven archetype:

This archetype will create a simple Maven project with a MyBenchmark class at default and the required dependencies and build configurations already in place.

For this showcase, I just added the Jackson and Gson dependency on top of the generated pom.xml:

Writing benchmarks with JMH

Benchmarks are written with plain Java methods and are marked with the @Benchmark annotation. In addition, you can configure a bunch of parameters for the benchmark execution: the number of iterations for the benchmark, the warmup phase, the time unit, the benchmark mode, additional JVM arguments, and much more.

The benchmark method for Jackson has some of these configurations applied:

Later on, the benchmark won't be executed against a cold JVM as previous to the actual measurements, JMH will start with some warmup iterations.

Most of the benchmarks will require some test/sample data for the code execution and JMH offers the concept of a state class for this.

In this example, I'm using a simple static inner class which contains the raw JSON string for the serialization:

You can use this state classes to e.g. set up and populate random data for your benchmark methods. These state classes can then be passed to your benchmark as a method argument as you might have already recognized it in the Jackson benchmark.

The whole benchmark code looks like this:

Running the benchmarks

For executing the benchmark, you just have to build the Maven project with:

And then run:

During the execution, you'll get statistics for both the warmup and the actual benchmark phase:

After the benchmark has completed, you'll see a final output for all of the executed benchmark methods:

Final thoughts

With the example above you should be able to write your own benchmarks. In addition, here are my key takeaways after writing some benchmarks:

  • try to use a (near) prod identical environment for your benchmarks (e.g. VM options, memory, CPU, OS, other settings …)
  • always return data within your benchmark method, so the JVM won't optimize it
  • use the @Setup annotation to prepare data for your benchmark (use the exact sample data if you want to compare two algorithms, methods …)
  • have a close look at the setup if you want to compare to benchmark results

You can find the sample benchmark on GitHub and more information about JMH here. If you are looking for a simple way to load-test e.g. a REST API, consider using Apache Benchmark.

Have fun benchmarking your methods,

Phil

>