In one of my previous blog posts, I showed you a simple way to load-test your application with Apache Benchmark. As this solution is suitable for e.g. testing a REST interface, this approach is not the best for benchmarking Java methods directly. Luckily we have a tool for this within the JVM ecosystem: JMH (Java Microbenchmark Harness).
The official page says the following about JMH:
JMH is a Java harness for building, running, and analysing nano/micro/milli/macro benchmarks written in Java and other languages targetting the JVM.
To show you how easy you can write benchmarks with JMH, I'll provide an example with this blog post and will benchmark the serialization of JSON strings to Java objects with Jackson and Gson on Java 8 (the provided example is just for demonstration purposes, please don't take this as a valid and bulletproof benchmark comparison of these libraries).
JMH Project setup
Bootstrapping a new benchmark project is done with the help of a Maven archetype:
| 1 2 3 4 5 6 7 | mvn archetype:generate \ -DinteractiveMode=false \ -DarchetypeGroupId=org.openjdk.jmh \ -DarchetypeArtifactId=jmh-java-benchmark-archetype \ -DgroupId=org.sample \ -DartifactId=your-benchmark \ -Dversion=1.0 |
This archetype will create a simple Maven project with a MyBenchmark class at default and the required dependencies and build configurations already in place.
For this showcase, I just added the Jackson and Gson dependency on top of the generated pom.xml:
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 | <project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://maven.apache.org/POM/4.0.0" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>de.rieckpil.blog</groupId> <artifactId>java-benchmarking-with-jmh</artifactId> <version>1.0</version> <packaging>jar</packaging> <name>Serialization benchmark (Jackson and Gson) with JMH</name> <dependencies> <dependency> <groupId>org.openjdk.jmh</groupId> <artifactId>jmh-core</artifactId> <version>${jmh.version}</version> </dependency> <dependency> <groupId>org.openjdk.jmh</groupId> <artifactId>jmh-generator-annprocess</artifactId> <version>${jmh.version}</version> <scope>provided</scope> </dependency> <dependency> <groupId>com.google.code.gson</groupId> <artifactId>gson</artifactId> <version>2.8.6</version> </dependency> <dependency> <groupId>com.fasterxml.jackson.core</groupId> <artifactId>jackson-databind</artifactId> <version>2.11.0</version> </dependency> </dependencies> <build> <!-- auto-generated section --> </build> </project> |
Writing benchmarks with JMH
Benchmarks are written with plain Java methods and are marked with the @Benchmark annotation. In addition, you can configure a bunch of parameters for the benchmark execution: the number of iterations for the benchmark, the warmup phase, the time unit, the benchmark mode, additional JVM arguments, and much more.
The benchmark method for Jackson has some of these configurations applied:
| 1 2 3 4 5 6 7 8 | @Benchmark @OutputTimeUnit(TimeUnit.MILLISECONDS) @Warmup(iterations = 5) @Measurement(iterations = 10) @BenchmarkMode(Mode.AverageTime) public User benchmarkSerializationWithJackson(SerializationDataProvider serializationDataProvider) throws IOException { return objectMapper.readValue(serializationDataProvider.jsonString, User.class); } |
Later on, the benchmark won't be executed against a cold JVM as previous to the actual measurements, JMH will start with some warmup iterations.
Most of the benchmarks will require some test/sample data for the code execution and JMH offers the concept of a state class for this.
In this example, I'm using a simple static inner class which contains the raw JSON string for the serialization:
| 1 2 3 4 5 6 7 8 9 10 11 12 13 | @State(Scope.Benchmark) public static class SerializationDataProvider { private String jsonString; @Setup(Level.Invocation) public void setup() { this.jsonString = "{\"firstName\": \"Mike\", \"lastName\":\"Duke\", \"hobbies\": [{\"name\": \"Soccer\", " + "\"tags\": [\"Teamsport\", \"Ball\", \"Outdoor\", \"Championship\"]}], \"address\":" + " { \"street\": \"Mainstreet\", \"streetNumber\": \"1A\", \"city\": \"New York\", \"country\":\"USA\", " + "\"postalCode\": 1337}}"; } } |
You can use this state classes to e.g. set up and populate random data for your benchmark methods. These state classes can then be passed to your benchmark as a method argument as you might have already recognized it in the Jackson benchmark.
The whole benchmark code looks like this:
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 | public class SerializationBenchmark { private static final ObjectMapper objectMapper = new ObjectMapper(); private static final Gson gson = new Gson(); @Benchmark @OutputTimeUnit(TimeUnit.MILLISECONDS) @Warmup(iterations = 5) @Measurement(iterations = 10) @BenchmarkMode(Mode.AverageTime) public User benchmarkSerializationWithJackson(SerializationDataProvider serializationDataProvider) throws IOException { return objectMapper.readValue(serializationDataProvider.jsonString, User.class); } @Benchmark @OutputTimeUnit(TimeUnit.MILLISECONDS) @Warmup(iterations = 5) @Measurement(iterations = 10) @BenchmarkMode(Mode.AverageTime) public User benchmarkSerializationWithGSON(SerializationDataProvider serializationDataProvider) { return gson.fromJson(serializationDataProvider.jsonString, User.class); } @State(Scope.Benchmark) public static class SerializationDataProvider { private String jsonString; @Setup(Level.Invocation) public void setup() { this.jsonString = "..."; } } } |
Running the benchmarks
For executing the benchmark, you just have to build the Maven project with:
| 1 | mvn clean package |
And then run:
| 1 | java -jar target/benchmarks.jar |
During the execution, you'll get statistics for both the warmup and the actual benchmark phase:
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 | # Run progress: 40.00% complete, ETA 00:15:04 # Fork: 5 of 5 # Warmup Iteration 1: 0.002 ms/op # Warmup Iteration 2: 0.002 ms/op # Warmup Iteration 3: 0.002 ms/op # Warmup Iteration 4: 0.002 ms/op # Warmup Iteration 5: 0.002 ms/op Iteration 1: 0.002 ms/op Iteration 2: 0.002 ms/op Iteration 3: 0.002 ms/op Iteration 4: 0.002 ms/op Iteration 5: 0.002 ms/op Iteration 6: 0.002 ms/op Iteration 7: 0.002 ms/op Iteration 8: 0.002 ms/op Iteration 9: 0.002 ms/op Iteration 10: 0.002 ms/op Result "de.rieckpil.blog.SerializationBenchmark.benchmarkSerializationWithGSON": 0.002 ±(99.9%) 0.001 ms/op [Average] (min, avg, max) = (0.002, 0.002, 0.003), stdev = 0.001 CI (99.9%): [0.002, 0.002] (assumes normal distribution) |
After the benchmark has completed, you'll see a final output for all of the executed benchmark methods:
| 1 2 3 4 5 6 7 8 9 10 11 | # Run complete. Total time: 00:25:30 REMEMBER: The numbers below are just data. To gain reusable insights, you need to follow up on why the numbers are the way they are. Use profilers (see -prof, -lprof), design factorial experiments, perform baseline and negative tests that provide experimental control, make sure the benchmarking environment is safe on JVM/OS/HW level, ask for reviews from the domain experts. Do not assume the numbers tell you what you want them to tell. Benchmark Mode Cnt Score Error Units SerializationBenchmark.benchmarkSerializationWithGSON avgt 50 0.002 ± 0.001 ms/op SerializationBenchmark.benchmarkSerializationWithJackson avgt 50 0.002 ± 0.001 ms/op |
Final thoughts
With the example above you should be able to write your own benchmarks. In addition, here are my key takeaways after writing some benchmarks:
- try to use a (near) prod identical environment for your benchmarks (e.g. VM options, memory, CPU, OS, other settings …)
- always return data within your benchmark method, so the JVM won't optimize it
- use the
@Setupannotation to prepare data for your benchmark (use the exact sample data if you want to compare two algorithms, methods …) - have a close look at the setup if you want to compare to benchmark results
You can find the sample benchmark on GitHub and more information about JMH here. If you are looking for a simple way to load-test e.g. a REST API, consider using Apache Benchmark.
Have fun benchmarking your methods,
Phil
