(*) If your career as a QA - like the author’s - started without a computer science degree, you will hopefully love to hear and learn about the different theories and notations of runtime analysis. Check wikipedia’s page about O-notation here
Speed… Did you ever think how you can make your tests faster and more reliable? If you're anything like me or my colleagues at AUTO1, you face this challenge all the time. Here at AUTO1 we are constantly looking for new ways to improve our tools and to make testing fast and efficient. In this article, I’m going to take test suite that takes ~70 min and reduce the execution time to 1 min!!! Interesting, right? :)
Let’s imagine we are testing some Web Shop Scenario. We need to test that users can do some actions with items they purchased (submit feedback, return, submit claim, etc.). And imagine that in this case we have exactly 20 scenarios that go like this:
Steps: - Create item, put item in certain state - Create user - Create user delivery address - Create user billing address - Create user payment details - Sell item to user - User does something with an item
Let’s look at the timeline of our test
In this case, if you have twenty tests and you run them sequentially, it will take ~73 min.
What we can improve here? Of course everybody will say “Pffff, let’s run in parallel!”. This is everyone’s first call, but for parallel execution you must always consider multiple aspects:
Let’s imagine we would like to keep ten threads for start.
We significantly increased the speed. From ~73 to ~8 minutes. This already looks like a game changer! But can we do more? Spoiler: Yes we can!
If we check our test scenario again, we can clearly see that the actual validation of our test scenario happens only after the third minute! And everything we have up until then it’s just creating preconditions. What we actually want to test is: “User can do certain action with item purchased on our platform”. We should not care HOW we got this item, we should have another test, testing that user should be able to purchase. This just creates dependency for us that makes our tests:
P.S. While writing this article I tried to re-create all these conditions and I didn’t have any successful build. And it was always something related to other dependencies.
Solution? There are many possible answers, the one I chose for this example is to move all logic with test data preparation to the API layer. Let’s try this and review our timeline again
Amazing, this is how our test looks now! two minutes just gone.
Again we manage to halve the test execution time! Considering that it was already pretty fast, it's a great improvement. So at this point you might think we are done here, it’s time to go to bar, grab a beer and celebrate. But then you come back the next day and think "there must be more"...
Having test preparation using API is a great step here, but does it scale from 20 tests to 1000 without having side-effects? At the end, your test data preparation might look like this:
Assuming you have around 40 calls with a reasonable fan-out for every scenario and you usually run tests in multiple threads, you might involuntary start a load test there. Aaaaand it’s time to look for a solution again!
What if we have our test data waiting for us somewhere? Wouldn’t be amazing?! Our solution for this problem was to create a Web Service that would keep the data in a queue for us. It provides couple of huge benefits for us:
Let’s look at our timeline again. This is how our test scenario changed:
And our test suite looks like this right now:
We barely can see it on our timeline :)
Stay tuned! We will open-source it soon!
In AUTO1 Group, every day we are looking for smarter ways to do software development and testing. And the main point in this article is that you should never stop innovating and looking for new solutions to existing problems. Something that works for us, is not necessarily going to work for you, but maybe you will ask yourself: “Where is our next level? Can we do something better?”