In December 2023, Elastic Mint celebrated five years of partnership with Preqin, a leading financial services company providing data and insights in the global alternatives community. During this period, we were involved in a number of projects related to Preqin Pro, their alternative assets data platform.
Testing APIs Dependent on Elasticsearch
Preqin have built a significant amount of their system around the capabilities of Elasticsearch. One of their key APIs had a large number of tests which verified interactions with Elasticsearch. The only problem was that these tests communicated with a networked deployment of Elasticsearch leading to a number of problems:
- Inability to run tests when not connected to their network
- Very slow test runs
- Test runs triggered by different people interfering with each other
- Builds in their CI platform intermittently failing
Elastic Mint investigated and introduced an open-source library called Elasticsearch-Inside to resolve the problem. This enabled tests to be run against a local copy of Elasticsearch, which was spun up at the beginning of the test run and then torn down afterwards. Our engineers also built a framework to make it easy for each test to initialise the data they needed.
Introducing Docker
A few years later, Docker and Kubernetes were introduced into Preqin’s technology stack, and we thought it was time to have another look at this API.
First, we created a Dockerfile for the project containing all the tests, including the command to run the tests. Then we configured a Docker Compose YAML file with all the API’s dependencies. These included the specific version of Elasticsearch used in production. In the YAML we specified that the test project depended on the other services. This enabled us to run the tests from a command line with all the tests running inside Docker using an instance of Elasticsearch also inside Docker. These tests could also be run easily from within Preqin’s CI platform.
While this isn’t how Docker is intended to be used, this pattern has proved to be invaluable to Preqin’s engineers in testing many other APIs in their platform.
Technologies: ASP .Net Core, Elasticsearch, XUnit, Docker
