AUTO1 Group

Java assertion libraries comparison

Eliza Korab

By Eliza Korab

Eliza is a Senior Test Automation Engineer at AUTO1 Group.

< Back to list
QA

In this article I will briefly present a couple of assertion libraries to show the possibilities we have when writing tests with those. The goal here is to compare them and present key features to help you decide on your assertion library to use with automation.

I will choose libraries supporting Java and present all of examples using Java but remember that many of those are supporting also other programming languages.

Currently there are many libraries available which are supporting the creation of assertion in test automation which are alternatives to the “assert” keyword introduced in Java 1.5. Would you consider using those if you would know what those libraries are providing to us and if this is actually matching your needs while writing tests?

While doing manual tests assertions are usually very intuitive. If you think about automated testing... what makes them checking if the actual result matches our expectations? What makes them more effective than manual tests?

Automation tests are fast, repeatable and decrease effort needed for regression. Key value for your tests is a properly done assertion - predicate which is evaluating our logical proposition. Assertions are our must haves, test script without assertion doesn’t add big value to your testing.

Let’s start!

There are multiple libraries available in the Java ecosystem, some of the popular ones are: Hamcrest, AssertJ, TestNG, JUnit, Truth. For Maven projects it's enough to add dependency to your pom to start using those. In AUTO1 we have many test automation repositories, depending on needs - teams are using different libraries for their testing. There are multiple benefits of using assertions libraries such as: custom methods so our assertions may be more sophisticated (especially for collection-related asserts and grouped asserts), messaging, possibility to do hard/soft assertions.

Hard assertions throw an error during execution when the condition is not being met so test cases are marked as failed. Soft assertions do not throw an exception automatically on assertion failure but at the very end of the scenario and then show all failures within the scenario, not just the first one.

Comparison

I took three libraries with their characteristics, reviewing available methods and checks we can use. As an addition I checked if the codebase for those libraries have active communities - means many commit contributions and frequent releases. This information may be worth to consider. In case we have similar features for given libraries which are your “must haves” - it’s always easier to get support when a community is big, active and we frequently receive new versions for the library. More details can be found in a table below.

So we have here:

  1. Hamcrest which is a library for writing matcher objects allowing ‘match’ rules to be defined. There are a number of situations where matchers cannot be applied for UI test, or data filtering. Hamcrest has a wide audience, available since 2007.

  2. AssertJ is a library providing a rich set of assertions, truly helpful error messages and is designed to be easy to use within your favorite IDE. AssertJ’s assertions are super easy to use: just type assertThat followed by the actual value in parentheses and a dot, then any Java IDE will show you all assertions available for the type of the object. (see picture below)

Image title

  1. TestNG is a testing framework inspired by JUnit and NUnit. The design goal of TestNG is to cover a wider range of test categories: unit, functional, end-to-end, integration, etc., with more powerful and easy-to-use functionalities. TestNG features are: annotation support, XML configurations, flexible execution model and simple set of assertions for your testing. As TestNG is a framework and not only an assertion library, in the context of this article TestNG will be considered and compared as well but please remember that in case you do not need other TestNG features then using TestNG only for assertions may be optimal.
  1. JUnit is a unit testing framework for Java and is part of a family of unit testing frameworks xUnit. Similarly as TestNG – in the context of this article will be considered as a tool for assertions only.

  2. Truth is a library to perform assertions. Truth is very similar to AssertJ, since it was inspired by FEST, of which AssertJ is a fork of the 2.0 development line. In comparison to AssertJ - main difference is that Truth has simpler API resulting also in less methods available with comparison to assertJ but have more clear messages.

While comparing you may see that depending on data types there are more or less methods available for, soft assertions are available/not available and we can also in some case write own assertions. Please find table below for more details (click for original size).

For Truth - as it’s very similar to AssertJ I’m not presenting examples separately. AssertJ and Truth differs for specific/advanced features which is not the focus in this article.

Image title

Let’s see a couple of examples for different libraries:

Example A
Let’s assume we have 2 lists, and we want to check if the lists are containing exactly the same elements, regardless of order.

List list1 = Arrays.asList(1, 5, 9, 6);
List list2 = Arrays.asList(1, 6, 5, 9);

assertThat(list1, Matchers.containsInAnyOrder(list2.toArray())); //Hamcrest
assertThat(list1).hasSameElementsAs(list2); //AssertJ
assert.assertEqualsNoOrder(list1, list2); //TestNG
assertLinesMatch(list1, list2) //JUnit

Example B
In our test we would like to check if actual date returned by service is in the past.

assertThat(currentDate, ZonedDateTimeMatchers.before(actualDate)); //Hamcrest
assertThat(currentDate).isBefore(actualDate);//AssertJ
assert.AssertTrue(actualDate – currentDate > 0); //TestNG needs to convert dates and use substract here

Remember to not call methods in assertion conditions and instead assign the result of the method to a local variable and use that variable with assert, as this increases readability.

Example C
In our test we would like to validate file first row of file content which is created by automation test.

Check file content:

assertThat(contentOf(uploaded File)).startsWith("Version:"); //AssertJ

Example presented only with assertJ as it provides the easiest way to do this out of the box.

Example D
In AUTO1 in we use API calls as a setup for our tests, sometimes it happens that we need to do multiple things to get test data in state in which we can do check with UI automation.
Let’s consider the following scenario:

  1. Add certain type of wheels to a purchased car

  2. Sell car to retail merchant

  3. Import car data

  4. Verify that wheels information is available in application

We would like to validate the last step in application over UI but other steps cannot be really completed within this application, so steps 1-3 can be done over API. Out focus in this scenario is to validate information in UI in our app but without preceding steps it cannot be accomplished. So, for that case we can try to control execution of our test with soft/hard assertions and do hard assertions for our “setup” and soft assertions for multiple UI checks we need to do over UI. The test will fail immediately if we, for example, cannot completely import test data. We need to check data for each wheel in the last step so that the usage of soft assertions here could help us. In case we check for only 1 wheel fail we would be able to get more precise information with soft assertion. Find code from this scenario below:

@Test
public void validateWheelsTest() {

Car car = get(TestDataTypeV2.WKDA_PURCHASED);
wheelsService.addWheelsToACar(wheelsDto);
assertThat(carService.getWheels(car)).isNotEmpty; //hard assert over API to validate preceding step

retailService.sellCar(car);
loginPage.openLoginPage();
loginPage.loginAs(userEmail, UserConstants.PASSWORD);
carDetailsPage.open(car);

SoftAssertions softly = new SoftAssertions();
softly.assertThat(carDetailsPage.getCarOrigin()).isNotNull();
softly.assertThat(carDetailsPage.getWheels(0).format()).isEqualTo(wheelsDto.getFrontLeft());
softly.assertThat(carDetailsPage.getWheels(1).format()).isEqualTo(wheelsDto.getFrontRight());
softly.assertThat(carDetailsPage.getWheels(2).format()).isEqualTo(wheelsDto.getBackLeft());
softly.assertThat(carDetailsPage.getWheels(3).format()).isEqualTo(wheelsDto.getBackRight());
softly.assertAll(); //soft assert on car detail checks within the application
}

In some cases it becomes really handy for tests with multiple checks. The test is running until method assertAll() is called to throw assertion errors.

Example E
We want to validate value in invoice, we have there an assumption that value and its precision on invoice is shown with or without decimals (rounded) depending on some business rules. Our assumption is that we'll do numeric assertions with precision:

assertThat(totalBrutto).isEqualTo(100, withPrecision(1d)); //assertJ
final double delta = 0.9;
assertEquals(totalBrutto, 100, delta); //TestNG & JUnit - we need to use additional DELTA variable, syntax is exactly the same in this case :)
assertThat(totalBrutto, closeTo(100, delta)); //Hamcrest

Conclusion

Remember that if assertions are not present then your test is not adding a huge value. The most important thing is that you choose a tool you are comfortable with and which satisfies your needs in tests. Keep in mind that when writing good assertions, analytical skills and properly set goal for the test is as much important as the tools you’re using :) Choose the best way to boost your tests, analyse what kind of checks you will need while writing tests and if you need custom messages, soft assertions, or which library will fulfill those needs in a way you will be satisfied with the tests.

Stories you might like:
By Alexander Gyulai

Are you sure your tests are fast enough?