AUTO1 Group

Integration Test Speedup with Spring

By Boris Faniuk

Boris is a former member of Engineering of AUTO1 Group.

< Back to list
Coding Mar 11

Problem statement

Over time we realized that our microservices built on top of Spring framework take tons of time on the integration test step. Every single PR runs all tests and this takes 10-20 minutes for some services. To decrease PR build time we decided to research why this happens. One of the problems that this research highlighted is ramping up heavy spring context several times during execution of all module’s tests.

As a proof of concept we have taken 3 modules and cleaned up their tests to achieve better integration test performance.

The results are pretty much self-explanatory:

  1. Module A now takes 3 minutes to test versus 11 before
  2. Module B now takes 3 minutes to test versus 8 before
  3. Module C now takes 4 minutes to test versus 12 before

Below I list all factors that produce many spring contexts and how we managed to overcome them.

Spring features to be avoided

@ActiveProfiles and @ContextConfiguration with different attributes

Our spring-based microservices actively use spring profiles and configuration classes. Many components from common libraries are hidden under profiles and configuration classes to not be active for every single module / environment without need. So, integration test authors define profiles and configuration classes that should be activated for their tests using these annotations:

  1. @ActiveProfiles({"integration-test", "pg-test", "mongo-test-two-nodes"})
  2. @ContextConfiguration(classes = {Application.class, AwsTestConfiguration.class})

Here mongo-test-two-nodes activates special mode for mongo based services to operate with two separate mongo instances. As not all tests need this, the profile is turned off by default and only those tests that need this behaviour would explicitly turn it on. But this, unfortunately, leads to multiple spring contexts being created.

The same happens with AwsTestConfiguration that deploys special test-component to mock AWS calls.

The solution was to make all tests use the same set of active profiles and context configuration classes for every single integration test. The drawback is that some components are deployed, even for the tests that do not need it. But deploying single component takes O(1) time comparing with the whole spring context.

Another point to keep in mind is that order of profiles (configuration classes) matters. @ActiveProfiles({"integration-test", "pg-test"}) @ActiveProfiles({"pg-test", "integration-test"})

Two settings mentioned above result in 2 different spring contexts.

@SpyBean and @MockBean

One interesting feature in spring-boot-test is mocking / spying real spring beans. This was mostly used when we don’t care what some method does in test but we need to check that it was called by parent components. Unfortunately this also makes test to create separate spring context. What we’ve done here is replacing @SpyBean annotation with configuration class that wraps real spring bean into Mockito.spy(..):

@Configuration
public class SomeComponentSpyFactory {
	@Autowired
	private SomeComponent someComponent;

	@Bean
	@Primary
	public SomeComponent someComponentSpy() {
    		return Mockito.spy(someComponent);
	}
}

Another option is to use the combination of @Autowired and @Spy annotations.

	@InjectMocks
	@Autowired
	private ParentComponent parentComponent;

	@Spy
	@Autowired
	private ChildComponent childComponent;

However, the drawback of the last solution is that, parentComponent needs to have a setter for childComponent which is not good in terms of encapsulation. But if you don’t use autowiring on constructor level it is fine. Anyway, one can choose whatever fits the situation better.

@DirtiesContext

One of the services used this annotation to clear the caches between different tests. Clearing a cache is really a problem (see section below), but this solution comes to separate spring context per test class or even test method (depending on annotation parameters). Context per method was actually the case for a few test classes. Instead in this modules we added methods to clear the caches programmatically without deploying the whole new context. One can use java reflection API instead of adding special production-level methods, again whatever fits you better.

Challenge: test execution order and cleaning collections (caches)

After all this “multiple context” factors are gone one may face (and we did) a challenge to fix or re-organize some tests. Before above enhancements were done all tests were running in separate spring contexts, therefore used separate instances of every component and therefore those problems could be hidden.

As an example some tests in mongo-based service were testing getAll()-like operations. Those tests were running functions that fetched all records from some collections. This was not a problem as embedded mongo DB was different instance. After all tests started to exploit the same spring context some of them started to fail intermittently. The failures were intermittent because test execution order is different. One service may break another, but not vice versa. So, to overcome this, we added missing @Before operations that cleaned those collections.

The rule that I would suggest here is that every test should clean up whatever it needs to be clean and not rely on fairness of other tests. That was a problem in our case. Some tests had @After operations that cleaned up, but not everything produced by the test was cleaned up. This caused intermittent failures of other tests.

Tools we created to reduce test execution time

So, now after we know all this, we created two tools (and may create more in future) that help to identify those problems.

Jenkins PR pipeline step

After every PR commit we run one additional step that analyzes test execution log. If more than one spring context is created this leaves some traces and a notification is sent to the author of the commit via slack

As an enhancement, the jenkins jobs could be modified to fail the build if the number of spring contexts is greater than a certain threshold. The threshold could start with a high number and can gradually be reduced as the team cleans up the existing tests by adopting the aforementioned practices/fixes

IntelliJ IDEA plugin

We wrote an IntelliJ plugin to identify and solve the aforementioned problems. There will be a separate blog post about this plugin coming very soon.

Stories you might like:
CodingApr 4
By Chirag Swadia

How we use ES6 generators instead of thunk to simplify our React Redux application code and...

CodingFeb 5
By Mariusz Sondecki

Analysis of Spring 5.X candidate component index applicability to boost our application startup time