(*) If your career as a QA - like the author’s - started without a computer science degree, you will hopefully love to hear and learn about the different theories and notations of runtime analysis. Check wikipedia’s page about O-notation here

Speed… Did you ever think how you can make your tests faster and more reliable? If you're anything like me or my colleagues at AUTO1, you face this challenge all the time. Here at AUTO1 we are constantly looking for new ways to improve our tools and to make testing fast and efficient. In this article, I’m going to take test suite that takes ~70 min and reduce the execution time to 1 min!!! Interesting, right? :)

Intro

Let’s imagine we are testing some Web Shop Scenario. We need to test that users can do some actions with items they purchased (submit feedback, return, submit claim, etc.). And imagine that in this case we have exactly 20 scenarios that go like this:

Steps:
- Create item, put item in certain state
- Create user
- Create user delivery address
- Create user billing address
- Create user payment details
- Sell item to user
- User does something with an item

Back in times…

Let’s look at the timeline of our test

In this case, if you have twenty tests and you run them sequentially, it will take ~73 min.

Time to Improve!

What we can improve here? Of course everybody will say “Pffff, let’s run in parallel!”. This is everyone’s first call, but for parallel execution you must always consider multiple aspects:

• Your test should neither rely nor be affected by tests running in parallel. It should be completely isolated.
• You must consider the application under tests. Is it capable of handling parallel execution? Do you have any external dependencies that can prevent you from this? Will your application behave in the same way if you run 20 tests concurrently? You can run 20, 50 or 1000 parallel tests on your application, nevertheless under some circumstances you might want to reduce number of parallel threads.

Let’s imagine we would like to keep ten threads for start.

We significantly increased the speed. From ~73 to ~8 minutes. This already looks like a game changer! But can we do more? Spoiler: Yes we can!

Removing dependencies

If we check our test scenario again, we can clearly see that the actual validation of our test scenario happens only after the third minute! And everything we have up until then it’s just creating preconditions. What we actually want to test is: “User can do certain action with item purchased on our platform”. We should not care HOW we got this item, we should have another test, testing that user should be able to purchase. This just creates dependency for us that makes our tests:

1. Slower
2. Less stable. It becomes less stable because we have to go through multiple steps (in some cases maybe through multiple applications) with creating item, creating user, adding user info and so on. If something will fail in the middle (which can happen easily) it doesn’t really mean that user couldn't leave a feedback on the item.

P.S. While writing this article I tried to re-create all these conditions and I didn’t have any successful build. And it was always something related to other dependencies.

Solution? There are many possible answers, the one I chose for this example is to move all logic with test data preparation to the API layer. Let’s try this and review our timeline again

Amazing, this is how our test looks now! two minutes just gone.

And this is how our suite looks now comparing to other runs.

Again we manage to halve the test execution time! Considering that it was already pretty fast, it's a great improvement. So at this point you might think we are done here, it’s time to go to bar, grab a beer and celebrate. But then you come back the next day and think "there must be more"...

Having test preparation using API is a great step here, but does it scale from 20 tests to 1000 without having side-effects? At the end, your test data preparation might look like this:

Assuming you have around 40 calls with a reasonable fan-out for every scenario and you usually run tests in multiple threads, you might involuntary start a load test there. Aaaaand it’s time to look for a solution again!

Introducing Q-Service

What if we have our test data waiting for us somewhere? Wouldn’t be amazing?! Our solution for this problem was to create a Web Service that would keep the data in a queue for us. It provides couple of huge benefits for us:

1. Instead of doing all these calls in runtime, we do 1 single call to our API to get test data. And instead of 1 min in test preparation we got <1 sec!
2. We don’t overload the system while executing tests. Which also means for us we can run more tests in parallel!
3. Tests are way more stable. Now we just need to take our test data and check that user can do certain action with that item.

Let’s look at our timeline again. This is how our test scenario changed:

And our test suite looks like this right now:

We barely can see it on our timeline :)

• 1 test now takes ~30s to run (instead of 3m 40s). And it’s “slow” and “unstable”(as it’s usually called) UI end-to-end test.
• Entire suite takes ~1m
• We can easily double the number, if we need, since these tests are producing a very little amount of our resources

How does it work?

Stay tuned! We will open-source it soon!

Summary

In AUTO1 Group, every day we are looking for smarter ways to do software development and testing. And the main point in this article is that you should never stop innovating and looking for new solutions to existing problems. Something that works for us, is not necessarily going to work for you, but maybe you will ask yourself: “Where is our next level? Can we do something better?”