When developing applications that interact with cloud services like Azure Blob Storage, we need to have an efficient testing mechanism in place to ensure reliability and consistency during development and Continuous Integration (CI).
Traditional unit tests are insufficient to validate the integration between our application and the cloud service. On the other hand, running tests against an actual cloud service can be expensive, slow and may lead to unwanted side effects.
This is where a tool like Azurite comes into play. It is an open-source, lightweight emulator provided by Microsoft that simulates Azure Storage API, allowing us to write and test applications without the need for an actual Azure subscription.
In this article, we will explore how we can use Azurite along with Testcontainers to write integration tests for our Spring Boot application that interacts with Azure Blob Storage. The working code referenced in the article can be found in this Github repository.
Sample Spring Boot Application Setup
In order for us to test interactions with Azure Blob Storage, we first need to have an application that interacts with it. Our sample application is going to be a Spring Boot project that exposes a service layer for uploading, retrieving, and deleting files in an Azure Blob Storage container.
Let's start by looking at the Maven project setup and the dependencies required for our application:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 | <properties> <azure.blobstore.version>12.25.3</azure.blobstore.version> </properties> <dependencies> <!-- API dependencies --> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-web</artifactId> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-validation</artifactId> </dependency> <!-- Azure dependencies --> <dependency> <groupId>com.azure</groupId> <artifactId>azure-storage-blob</artifactId> <version>${azure.blobstore.version}</version> </dependency> <!-- Devtools dependencies --> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-configuration-processor</artifactId> <optional>true</optional> </dependency> <dependency> <groupId>org.projectlombok</groupId> <artifactId>lombok</artifactId> <optional>true</optional> </dependency> </dependencies> |
With the above dependencies, we are all set to develop our application that connects and interacts with a provisioned Azure Blob Storage container. We will look at how each of these dependencies is used within our application in the upcoming sections.
The remaining dependencies we need to declare are specific to the testing scope, where we will be running our integration tests:
1 2 3 4 5 6 7 8 9 10 11 | <!-- Test dependencies --> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-test</artifactId> <scope>test</scope> </dependency> <dependency> <groupId>org.testcontainers</groupId> <artifactId>testcontainers</artifactId> <scope>test</scope> </dependency> |
The Spring Boot Starter Test a.k.a. the swiss-army knife for testing gives us the basic testing toolbox as it transitively includes JUnit and other utility libraries, that we will require for writing assertions and running our tests.
And Testcontainers, which allows us to run the Azurite emulator inside a disposable Docker container ensuring an isolated environment for our integration tests.
Azure Spring Boot Configuration
In order to facilitate connection between our application and the Blob Storage container, we need the name of the provisioned containerand its corresponding connection string.
We will store these two properties in our project's application.yml
file and make use of @ConfigurationProperties
to map the defined values to a POJO, which our application will reference to connect to Azure cloud.
1 2 3 4 5 6 7 8 9 10 11 12 13 | @Getter @Setter @Validated @ConfigurationProperties(prefix = "de.rieckpil.azure.blob-storage") public class AzureConfigurationProperties { @NotBlank(message = "Blob Storage container name must be configured") private String containerName; @NotBlank(message = "Blob Storage connection string must be configured") private String connectionString; } |
We have also added the @NotBlank
annotation, to validate the required properties are available when the application starts. The absence of any one of which, would result in the Spring Application Context failing to start up.
Below is a snippet of our application.yaml
file where we will be defining the required properties which will be automatically mapped to the above defined class:
1 2 3 4 5 6 | de: rieckpil: azure: blob-storage: container-name: ${BLOB_STORAGE_CONTAINER_NAME} connection-string: ${BLOB_STORAGE_CONNECTION_STRING} |
The only thing left for us to do is to define a bean of class BlobContainerClient
that is provided in the com.azure:azure-storage-blob
dependency we had declared before. This class provides various methods to communicate with the cloud service and will be autowired into our service layer.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 | @Configuration @RequiredArgsConstructor @EnableConfigurationProperties(AzureConfigurationProperties.class) public class AzureConfiguration { private final AzureConfigurationProperties azureConfigurationProperties; @Bean public BlobContainerClient blobContainerClient() { var connectionString = azureConfigurationProperties.getConnectionString(); var containerName = azureConfigurationProperties.getContainerName(); var blobServiceClient = new BlobServiceClientBuilder().connectionString(connectionString).buildClient(); return blobServiceClient.getBlobContainerClient(containerName); } } |
By defining a bean of BlobContainerClient
in the AzureConfiguration
class, we have now successfully given our application the capability to interact with the provisioned Blob Storage container.
Azure Blob Storage Spring Boot Service Layer
With the necessary configurations in place, we can now create a service class that leverages the above declared BlobContainerClient
bean as a dependency and exposes methods for interacting with the Azure Blob Storage container.
The service we will create provides the following functionalities:
- Uploading objects to the Blob Store
- Retrieving objects from the Blob Store
- Deleting objects from the Blob Store
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 | @Service @RequiredArgsConstructor public class StorageService { private final BlobContainerClient blobContainerClient; public void save(MultipartFile file) { var blobName = file.getOriginalFilename(); var blobClient = blobContainerClient.getBlobClient(blobName); blobClient.upload(file.getInputStream()); } public InputStreamResource retrieve(String blobName) { var blobClient = getBlobClient(blobName); var inputStream = blobClient.downloadContent().toStream(); return new InputStreamResource(inputStream); } public void delete(String blobName) { var blobClient = getBlobClient(blobName); blobClient.delete(); } private BlobClient getBlobClient(String blobName) { var blobClient = blobContainerClient.getBlobClient(blobName); if (Boolean.FALSE.equals(blobClient.exists())) { throw new IllegalArgumentException("No blob exists with given name"); } return blobClient; } } |
With the creation of the above service class, we are done with our sample project setup and can proceed with the main agenda of this article: to test the interactions between our application and Azure Blob Storage.
Azurite Testing Setup with Spring Boot
Prerequisite: Running Docker
The prerequisite for running the Azurite emulator via Testcontainers is, as you've guessed it, an up-and-running Docker instance. We need to ensure this prerequisite is met when running the test suite either locally or when using a CI/CD pipeline.
Running Integration Tests with Maven Failsafe Plugin
We will be using the Maven Failsafe Plugin to run our integration tests during the integration-test phase of the build. The plugin executes test methods in classes whose names end with IT
(a convention for integration test classes).
Let's add the plugin to our project's pom.xml
file:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 | <build> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-failsafe-plugin</artifactId> <executions> <execution> <goals> <goal>integration-test</goal> </goals> </execution> </executions> </plugin> </plugins> </build> |
With the addition of this plugin, we can execute the command mvn integration-test verify
to run the integration tests that we will be writing ahead and verify the reliability of our application. To maintain consistency, I would recommend executing this command in CI/CD pipelines before deploying the application to any environment.
Starting Azurite via Testcontainers
At the time of this writing, the latest version of the Azurite image mcr.microsoft.com/azure-storage/azurite
is version 3.29.0
. We will be using this version in our test class.
Since Testcontainers does not have a dedicated module for Azurite, we will be using the GenericContainer
class, which allows us to start any Docker image.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 | @SpringBootTest class StorageServiceIT { @Autowired private StorageService storageService; @Autowired private BlobContainerClient blobContainerClient; private static final String AZURITE_IMAGE = "mcr.microsoft.com/azure-storage/azurite:3.29.0"; private static final GenericContainer<?> AZURITE_CONTAINER = new GenericContainer<>(AZURITE_IMAGE) .withCommand("azurite-blob", "--blobHost", "0.0.0.0") .withExposedPorts(10000); private static final String DEFAULT_AZURITE_CONNECTION_STRING = "DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://127.0.0.1:%s/devstoreaccount1;"; private static final String CONTAINER_NAME = RandomString.make().toLowerCase(); private static final String CONTAINER_CONNECTION_STRING; static { AZURITE_CONTAINER.start(); var blobPort = AZURITE_CONTAINER.getMappedPort(10000); CONTAINER_CONNECTION_STRING = String.format(DEFAULT_AZURITE_CONNECTION_STRING, blobPort); var blobServiceClient = new BlobServiceClientBuilder().connectionString(CONTAINER_CONNECTION_STRING).buildClient(); blobServiceClient.createBlobContainer(CONTAINER_NAME); } @DynamicPropertySource static void properties(DynamicPropertyRegistry registry) { registry.add("de.rieckpil.azure.blob-storage.container-name", () -> CONTAINER_NAME); registry.add("de.rieckpil.azure.blob-storage.connection-string", () -> CONTAINER_CONNECTION_STRING); } } |
At first glance, the above code in our test class StorageServiceIT
looks a little complex, so let's break down the operations we perform:
- Start a new instance of the Azurite container using the
GenericContainer
class, exposing the default port of10000
for the Blob Service. - Construct the connection string for the Azurite container using the mapped port.
- Create a new Blob Storage container within the Azurite instance using the generated connection string.
- Dynamically configure Azure configuration properties needed by the application to create the required connection bean using
@DynamicPropertySource
.
With this setup, our application will use the started Azurite container for all Blob Storage interactions during the execution of our integration test cases, providing an isolated and ephemeral testing environment.
NOTE: The Azurite container created will be automatically destroyed once the test suite is completed, hence we do not need to worry about manual cleanups.
Test Cases: Azure Blob Storage Service
Now that we have configured the Azurite container successfully, we can test the functionalities exposed by StorageService
class with appropriate test cases:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 | @Test void shouldSaveBlobSuccessfullyToContainer() { // Prepare test file to upload var blobName = RandomString.make() + ".txt"; var blobContent = RandomString.make(50); var blobToUpload = createTextFile(blobName, blobContent); // Invoke method under test storageService.save(blobToUpload); // Verify that the file is saved successfully in Azure Blob Store var blobClient = blobContainerClient.getBlobClient(blobName); assertThat(blobClient.exists()).isTrue(); assertEquals(blobContent, new String(blobClient.downloadContent().toBytes())); } |
In the above test case, we create a random file to upload to the Blob Store and invoke the save()
method exposed by our service class. Then, we use the BlobContainerClient
to verify if a blob with the same name and content has been created successfully.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 | @Test void shouldFetchSavedBlobSuccessfullyFromContainer() { // Prepare test file and upload to azure blob container var blobName = RandomString.make() + ".txt"; var blobContent = RandomString.make(50); var blobToUpload = createTextFile(blobName, blobContent); storageService.save(blobToUpload); // Invoke method under test var retrievedBlob = storageService.retrieve(blobName); // Read the retrieved content and assert integrity assertEquals(blobContent, new String(retrievedBlob.getContentAsByteArray())); } |
Similar to the previous test method, we prepare a random file and upload it to the Blob Store. We then invoke the retrieve()
method exposed by our service class and verify if the content of the retrieved blob is the same as the one we uploaded.
Now let's proceed with validating the delete functionality of our service class:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 | @Test void shouldDeleteBlobFromContainer() { // Prepare test file and upload to azure blob container var blobName = RandomString.make() + ".txt"; var blobContent = RandomString.make(50); var blobToUpload = createTextFile(blobName, blobContent); storageService.save(blobToUpload); // Verify that the blob has been saved in the container var retrievedBlob = storageService.retrieve(blobName); assertThat(retrievedBlob).isNotNull(); assertEquals(blobContent, new String(retrievedBlob.getContentAsByteArray())); // Invoke method under test storageService.delete(blobName); // Verify that the blob does not exist in the container post deletion var exception = assertThrows(IllegalArgumentException.class, () -> storageService.retrieve(blobName)); assertThat(exception.getMessage()).isEqualTo("No blob exists with given name"); } |
In this test case, we repeat the same step of creating and uploading a random file to the Blob Store. Post completion of which, we invoke the delete()
method exposed by our service class with the same name of the saved file. To validate the successful delete operation, we attempt to retrieve the deleted blob with it's corresponding name and assert that appropriate exception is thrown.
By writing these comprehensive test cases, we ensure that our StorageService
class functions correctly and interacts with the provisioned cloud service as expected. We can confidently deploy our application to production environments, knowing that it has been thoroughly tested and validated against a simulated Azure Blob Storage service. And the best part? we didn't even open the Azure portal to achieve this.
Pro-tip: Reduce Boilerplate Testcontainer Code
In every test class that directly or indirectly tests a functionality related with the storage layer, we will have to repeat the long testcontainer setup that we saw above.
This process leads to repetitive boilerplate code and increased test suite execution time since a dedicated Azurite container is started and stopped for each test class.
In this section I'll be sharing a solution for these problem statments, one that makes use of the lifecycle callback extension available in JUnit 5 called BeforeAllCallback
.
Inside our test folder itself, we will be creating a class AzureBlobStorageInitializer
which implements the BeforeAllCallback
interface and fulfills the responsibility of starting and configuring the Azurite container:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 | @Slf4j public class AzureBlobStorageInitializer implements BeforeAllCallback { private static final AtomicBoolean INITIAL_INVOCATION = new AtomicBoolean(Boolean.TRUE); private final String AZURITE_IMAGE = "mcr.microsoft.com/azure-storage/azurite:3.29.0"; private final GenericContainer<?> AZURITE_CONTAINER = new GenericContainer<>(AZURITE_IMAGE) .withCommand("azurite-blob", "--blobHost", "0.0.0.0") .withExposedPorts(10000); private final String DEFAULT_AZURITE_CONNECTION_STRING = "DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://127.0.0.1:%s/devstoreaccount1;"; private final String CONTAINER_NAME = RandomString.make().toLowerCase(); private String CONTAINER_CONNECTION_STRING; @Override public void beforeAll(final ExtensionContext context) { if (INITIAL_INVOCATION.getAndSet(Boolean.FALSE)) { AZURITE_CONTAINER.start(); initializeBlobContainer(); addConfigurationProperties(); } } private void initializeBlobContainer() { var blobPort = AZURITE_CONTAINER.getMappedPort(10000); CONTAINER_CONNECTION_STRING = String.format(DEFAULT_AZURITE_CONNECTION_STRING, blobPort); var blobServiceClient = new BlobServiceClientBuilder().connectionString(CONTAINER_CONNECTION_STRING).buildClient(); blobServiceClient.createBlobContainer(CONTAINER_NAME); } private void addConfigurationProperties() { System.setProperty("de.rieckpil.azure.blob-storage.container-name", CONTAINER_NAME); System.setProperty("de.rieckpil.azure.blob-storage.connection-string", CONTAINER_CONNECTION_STRING); } } |
We use an AtomicBoolean
flag to track the initial invocation to ensure that the Azurite container is started only once in our test suite.
Now we could annotate our test classes with @ExtendWith(AzureBlobStorageInitializer.class)
but I think we can do a better job of improving the readability of this solution and making it “cool”.
We simply create a custom marker annotation @InitializeAzureBlobStorage
and meta-annotate it with @ExtendWith(AzureBlobStorageInitializer.class)
:
1 2 3 4 5 | @Target(ElementType.TYPE) @Retention(RetentionPolicy.RUNTIME) @ExtendWith(AzureBlobStorageInitializer.class) public @interface InitializeAzureBlobStorage { } |
All that is left now, is to remove the boilerplate Azurite setup from our test class and leverage our newly created annotation @InitializeAzureBlobStorage
:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 | @SpringBootTest @InitializeAzureBlobStorage class StorageServiceIT { @Autowired private StorageService storageService; @Test void shouldSaveBlobSuccessfullyToContainer() { // test case } // more test cases } |
And just like that, we have successfully eliminated the need for writing boilerplate setup code in each of our test classes and significantly reduced the execution time of our test suite by using a shared Azurite container that is started only once.
Similar to the previous setup, the shared container will be destroyed post execution of our test suite.
Conclusion
Through this article, we have explored how we can create an isolated testing environment with Azurite and Testcontainers to write integration tests for our application that interacts with Azure Blob Storage.
Using this detailed approach, we gain confidence in deployments, enjoy faster feedback cycles, and benefit from a cost-effective testing solution. I hope you would benefit from this approach. Let me know your thoughts or questions in the comment section.
The source code demonstrated throughout this article is available in this Github repository. I would encourage you to explore the codebase.
Joyful testing,
Hardik Singh Behl