If your Spring application uses AWS components like S3, SNS or SQS you might wonder how to write effective integration tests. Should I mock any interaction with AWS? Should I duplicate the AWS infrastructure for testing? Is there maybe a local AWS clone available? With this blog post, you’ll learn how to write integration tests for your Spring application using AWS components with Testcontainers and LocalStack. The demo application uses both SQS and S3 to process order events for which we’ll write an integration test.
Spring Boot application setup
The application is a basic Spring Boot project using Java 11 and Spring Cloud. I’m using the spring-boot-starter-aws-messaging to have almost no setup for connecting to an SQS queue. Furthermore, this messaging starter has a transitive dependency to spring-cloud-starter-aws that provides the core AWS support:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 | <?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <parent> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-parent</artifactId> <version>2.2.6.RELEASE</version> <relativePath/> <!-- lookup parent from repository --> </parent> <groupId>de.rieckpil.blog</groupId> <artifactId>spring-boot-aws-integration-tests</artifactId> <version>0.0.1-SNAPSHOT</version> <name>spring-boot-aws-integration-tests</name> <description>Writing integration tests while using AWS S3</description> <properties> <java.version>11</java.version> <spring-cloud.version>Hoxton.SR3</spring-cloud.version> <testcontainers.version>1.13.0</testcontainers.version> <awaitility.version>4.0.2</awaitility.version> </properties> <dependencies> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-web</artifactId> </dependency> <dependency> <groupId>org.springframework.cloud</groupId> <artifactId>spring-cloud-starter-aws-messaging</artifactId> </dependency> <!-- Test Dependencies --> </dependencies> <dependencyManagement> <dependencies> <dependency> <groupId>org.springframework.cloud</groupId> <artifactId>spring-cloud-dependencies</artifactId> <version>${spring-cloud.version}</version> <type>pom</type> <scope>import</scope> </dependency> </dependencies> </dependencyManagement> <build> <plugins> <plugin> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-maven-plugin</artifactId> </plugin> </plugins> </build> </project> |
For writing integration tests we need the following test dependencies:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 | <dependency> <groupId>org.testcontainers</groupId> <artifactId>localstack</artifactId> <version>${testcontainers.version}</version> <scope>test</scope> </dependency> <dependency> <groupId>org.testcontainers</groupId> <artifactId>junit-jupiter</artifactId> <version>${testcontainers.version}</version> <scope>test</scope> </dependency> <dependency> <groupId>org.awaitility</groupId> <artifactId>awaitility</artifactId> <version>${awaitility.version}</version> <scope>test</scope> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-test</artifactId> <scope>test</scope> <exclusions> <exclusion> <groupId>org.junit.vintage</groupId> <artifactId>junit-vintage-engine</artifactId> </exclusion> </exclusions> </dependency> |
Let’s have a look at why we need these dependencies and what they are doing.
As our demo application makes use of AWS services (S3 and SQS), we have to provide this infrastructure for the integration tests. You can either duplicate the AWS services for your tests or provide a mock infrastructure. Option one is using real AWS services for your tests and hence making sure the application can work with them. It has the downside of additional AWS costs and once two developers execute the integration tests, they might run into issues.
With option two you have the drawback of using a mock infrastructure and a more artificial environment (e.g. almost no latency, not using the real AWS services, …). On the other hand, it’s cheap and can run on multiple machines in parallel. For this tutorial I’m choosing option two and make use of LocalStack:
- Easy-to-use test/mocking framework for developing AWS applications
- It spins up an environment with the same functionality and APIs as AWS
- Available as Docker containers and supports all core AWS services in the community edition
To manage the lifecycle of the LocalStack Docker container, I’m using Testcontainers:
- De-facto standard when it comes to spinning up containers for JUnit tests
- Comes with an own LocalStack module
Next, as we want to verify the main functionality of the application (processing of SQS messages), we have to test asynchronous code. Fortunately, there is a great Java library available for this Awaitility:
- Java Utility library to test asynchronous code
- Express expectations using a DSL
Finally, the Spring Boot Starter Test serves multiple purposes:
- Swiss-army knife for testing Spring applications
- Includes basic testing infrastructure: test engine, assertion libraries, mocking library, etc.
- Exclude the
junit-vintage-engineto only use JUnit 5
Spring application walkthrough
First, let’s have a look at how the connection to AWS is configured. There are multiple ways (e.g. system properties, env variables, etc.) to configure the AWS credentials. As we are using Spring Boot and the Spring Cloud Starter for AWS, I’m using the application.yml to define both keys:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 | cloud: aws: region: static: eu-central-1 stack: auto: false credentials: secretKey: ABC accessKey: XYZ sqs: orderEventQueue: order-event-queue s3: orderEventBucket: order-event-bucket |
Apart from this, I’m also specifying the AWS region and both SQS and S3 resource names. If you are using the Parameter Store of the AWS Systems Manager (SSM), you can also define such configuration values in AWS. As the application is not running inside an AWS stack (e.g. EC2), automatic stack detection is disabled with cloud.aws.stack.auto.
The actual processing logic of this demo application happens inside the SQS listener. As we are using the messaging starter for AWS, we can easily subscribe to an SQS queue using @SqsListener. The logic for each incoming message is dead-simple. We’ll log each incoming OrderEvent and upload it to S3.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 | @Component public class SimpleMessageListener { private final AmazonS3 amazonS3; private final ObjectMapper objectMapper; private final String orderEventBucket; public SimpleMessageListener(@Value("${s3.orderEventBucket}") String orderEventBucket, AmazonS3 amazonS3, ObjectMapper objectMapper) { this.amazonS3 = amazonS3; this.objectMapper = objectMapper; this.orderEventBucket = orderEventBucket; } @SqsListener(value = "${sqs.orderEventQueue}") public void processMessage(@Payload OrderEvent orderEvent) throws JsonProcessingException { System.out.println("Incoming order: " + orderEvent); amazonS3.putObject(orderEventBucket, orderEvent.getId(), objectMapper.writeValueAsString(orderEvent)); System.out.println("Successfully uploaded order to S3"); } } |
The OrderEvent is a POJO containing information about the order:
1 2 3 4 5 6 7 8 9 10 11 12 13 | public class OrderEvent { private String id; private String product; private String message; private boolean expressDelivery; @JsonFormat(shape = Shape.STRING, pattern = "yyyy-MM-dd HH:mm:ss") private LocalDateTime orderedAt; // constructor, getters & setters } |
While the raw SQS payload is a String, the AWS messaging dependency allows you to convert it to a Java object. We are already doing this, as the processMessage method takes OrderEvent as a parameter. Behind the scenes, this conversion is done using the MappingJackson2MessageConverter.
By default, this message converter instantiates its own Jackson ObjectMapper. As the OrderEvent uses a Java 8 LocalDateTime we need the Java Time Module registered inside the ObjectMapper.
To override this default behavior, we can provide our own MappingJackson2MessageConverter and set the ObjectMapper. I’m using the auto-configured ObjectMapper from Spring Boot for this as it contains all required Jackson modules:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 | @Configuration public class MessagingConfig { @Bean public MappingJackson2MessageConverter mappingJackson2MessageConverter(@Autowired ObjectMapper objectMapper) { MappingJackson2MessageConverter jackson2MessageConverter = new MappingJackson2MessageConverter(); jackson2MessageConverter.setObjectMapper(objectMapper); return jackson2MessageConverter; } @Bean public QueueMessagingTemplate queueMessagingTemplate(@Autowired AmazonSQSAsync amazonSQS) { return new QueueMessagingTemplate(amazonSQS); } } |
The second bean inside this configuration is optional, but we’ll use the QueueMessagingTemplate to send a message to SQS during test execution.
Next, let’s see how we can write integration tests for our Spring application using these AWS components with Testcontainers and LocalStack.
Integration test setup with Testcontainers and LocalStack
Now we can focus on testing this application logic with an integration test. During the test execution, we need access to SQS and S3. For this we’ll use the LocalStack module of Testcontainers and use the JUnit Jupiter extension to manage the lifecycle of the LocalStack container:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 | @Testcontainers @SpringBootTest @Import(AwsTestConfig.class) public class SimpleMessageListenerIT { @Container static LocalStackContainer localStack = new LocalStackContainer("0.10.0") .withServices(S3, SQS) .withEnv("DEFAULT_REGION", "eu-central-1"); @BeforeAll static void beforeAll() throws IOException, InterruptedException { localStack.execInContainer("awslocal", "sqs", "create-queue", "--queue-name", QUEUE_NAME); localStack.execInContainer("awslocal", "s3", "mb", "s3://" + BUCKET_NAME); } private static final String QUEUE_NAME = "order-event-test-queue"; private static final String BUCKET_NAME = "order-event-test-bucket"; // ... actual test } |
As our application expects an SQS queue to subscribe to and an S3 bucket to write data to, we need to set them up. I’m doing this set up inside a method annotated with @BeforeAll and hence before the JUnit tests are executed. Testcontainers allows us to execute commands inside the running Docker container and we can use this mechanism to create the infrastructure using awslocal (thin AWS CLI wrapper of LocalStack).
As the queue and bucket names differ from the one we used for the production profile, we can specify the new names inside src/test/resources/application.yml:
1 2 3 4 5 6 7 8 9 10 11 12 | cloud: aws: region: static: eu-central-1 stack: auto: false sqs: orderEventQueue: order-event-test-queue s3: orderEventBucket: order-event-test-bucket |
Verify the AWS SQS message processing of the Spring application
The remaining set up for our test is to configure the AWS clients properly. Without manually specifying the required clients as Spring beans, the auto-configure mechanism would try to connect to AWS during application startup. As we don’t want to connect to the actual AWS services and rather use our mocked environment of LocalStack, we can provide the beans for ourself:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 | @TestConfiguration public class AwsTestConfig { @Bean public AmazonS3 amazonS3() { return AmazonS3ClientBuilder.standard() .withCredentials(localStack.getDefaultCredentialsProvider()) .withEndpointConfiguration(localStack.getEndpointConfiguration(S3)) .build(); } @Bean public AmazonSQSAsync amazonSQS() { return AmazonSQSAsyncClientBuilder.standard() .withCredentials(localStack.getDefaultCredentialsProvider()) .withEndpointConfiguration(localStack.getEndpointConfiguration(SQS)) .build(); } } |
Both clients will now point to the local S3 and SQS instance. We import this @TestConfiguration for our test context using the @Import(AwsTestConfig.class) annotation at the class level of our integration test.
Finally, we can now write the test for our order event processing.
First, we put a message into the local SQS queue using the QueueMessagingTemplate and then expect to find an object in the S3 bucket with the given order id:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 | public class SimpleMessageListenerIT { // ... setup like described @Autowired private AmazonS3 amazonS3; @Autowired private QueueMessagingTemplate queueMessagingTemplate; @Test public void testMessageShouldBeUploadedToS3OnceConsumed() { String orderId = UUID.randomUUID().toString(); OrderEvent orderEvent = new OrderEvent(orderId, "MacBook", "42", LocalDateTime.now(), false); queueMessagingTemplate.convertAndSend(QUEUE_NAME, orderEvent); given() .ignoreException(AmazonS3Exception.class) .await() .atMost(5, SECONDS) .untilAsserted(() -> assertNotNull(amazonS3.getObject(BUCKET_NAME, orderId))); } } |
The given() part comes from Awaitility. As this message processing happens asynchronously, we can’t expect the object to be present in S3 right after we put the message in the queue. Therefore we’ll wait for five seconds to find the object in S3 and otherwise fail the test. The .ignoreException() part is necessary, as the S3 client will throw this exception whenever it can’t find the request object, which might be the case in the first milliseconds of trying to find.
With the setup above you are now able to write integration tests for your Spring application and can include AWS components using LocalStack and Testcontainers. Furthermore, this setup also works for any other AWS component that LocalStack supports e.g. DynamoDB or SNS.
Have fun writing integration tests for your Spring application using AWS components,
Phil