The OpenAPI journey

The situation

The AUTO1 microservice landscape is growing in size day by day, with new services from a spectrum of languages being erected and usually communicating with each other by using REST contracts.

In a fast paced environment, such as AUTO1, often the service providing the REST API and the service consuming this REST API are developed in parallel, by independent teams. On top of that the business requirements over the lifetime of a project evolve, thus impacting the stability of the API. Having these moving parts in picture can make the development process quite challenging, especially when aiming for high productivity, zero downtime and independent development of the client and server parts. Hence, having a clearly documented and language agnostic API is vital for the success of the whole process.

The light at the end of the tunnel

After extensive research, we found a great candidate seemingly accommodating all the aforementioned requirements - OpenAPI. In short the OpenAPI specification defines a language agnostic standard allowing to describe REST APIs in a human readable form, imposing a unified structure, clearly and comprehensively aligning expectations between the consumer and provider sides.

Below is an example of such an API described using the OpenAPI 3.0 standard:

Fig 1. OpenAPI specification example

Design first, code later

In general, we distinguish two approaches when developing APIs. The code first approach, which is basically about developing the source code fulfilling the business requirements and afterwards, optionally, generating the API specification from it, which for OpenAPI can be done using the swagger annotations. The second approach known as design first, is all about describing the API in a human readable language first, by using for example the OpenAPI specification, and then optionally, based on this artifact generating the source code for server and/or client. Naturally, both approaches come with pros and cons.

As previously mentioned, in AUTO1 it often happens that changes that are being introduced in the API on the server side need to be implemented at more or less the same time on the client side. In such a situation, if the code first approach would be practiced, the consumers would need to wait for the server side implementation to be delivered before starting their part, that would be simply inefficient. Here, for the rescue comes the design first approach, where from the moment when the API specification is agreed upon, both server and consumer side can kick-off their side of the implementation, independently. Additionally, this allows to spot misalignments already at a very early stage, and rectify them before the actual implementation from either of the sides is started, hence less expensive to fix.

The ecosystem

The OpenAPI comes with a rich ecosystem, from UI tools such as Swagger UI allowing to visualize and test ad-hoc the specifications to AWS integration providing a fast lane to expose the specification via AWS API Gateway.

Fig 2. Swagger rendered view of an OpenAPI specification

Another very helpful set of tools are the code generators, that can generate client side libraries and server side stubs from the OpenAPI definitions. These extendible generators support a variety of languages and frameworks, which lets one move faster, especially in a polyglot microservice environment. For our Spring Java applications we have decided to go for a generator that is very well configurable and easily customizable. It can be executed in various ways such as via cli, maven/gradle plugin, etc. Since our Java projects use maven for the build automation process, the maven openapi generator plugin was the natural choice for us.

Fig 3. OpenAPI generator plugin configuration example

The code generators have become pretty fast a very prominent tool among our developers, for various reasons:

• They immensely reduce writing boilerplate code required for setting up the communication channel with the REST APIs, hence letting the developers to focus on the business code rather than the glue code.

• They generate repeatable and consistent code of the API model and related REST client definition.

• Being highly configurable, they can be used out of the box and fine tuned without writing a single line of code in many cases. A good example here could be the Spring generator that allows us to define things such as the time API to use for the generated code, whether to wrap optional parameters in a Java Optional container or whether to use any framework specific features such as Bean Validation 2.0 from JSR 380.

Fig 4. OpenAPI generator plugin configuration example

Retrofitting OpenAPI

Despite the vast configuration possibilities offered by the OpenAPI code generators, in order to integrate OpenAPI into our AUTO1 development world seamlessly and make the generated API and models classes adhere to our coding standards we needed to extend the default Spring generator that we’ve decided to go with in first place, to include some AUTO1 specific code structures.

To be more productive, concise and spare some boilerplate code we use in AUTO1 the Lombok library, a lot, hence we couldn’t afford not having the Lombok annotations included in the generated classes as well. To follow the OpenAPI generator practices, we have added a configuration property to control enabling and disabling the usage of Lombok annotations:

Fig 5. OpenAPI generator plugin configuration example enabling lombok annotations

The above setting would result in generating the following annotations in the output classes:

Fig 6. OpenAPI generated lombok annotations

Sometimes the need arises to define custom annotations but only on some specific classes, for that purpose we have introduced a new attribute to our OpenAPI specifications called x-extra-annotation, that hints the generator to add additional annotations to the generated code. For example:

Fig 7. OpenAPI specification with java annotation hints

The above attribute would result in generating the following annotations in the output classes:

Fig 8. OpenAPI generated extra annotations

Generics is a very powerful feature that is available and widely used in the Java language for almost two decades now. Unfortunately, as of the time of writing of the article the OpenAPI does not support out of the box inheritance together with generics. To make it available to our developers we have introduced yet another attribute in the API specification named x-make-parent-generic specifying the genericType sub-attribute that indicates a generic type of the parent class (in this case PageDTO):

Fig 9. OpenAPI specification with java generic type hints

The above attribute would result in generating the following generic parent in the output classes:

Fig 10. OpenAPI generated extra generic parent inheritance

Yet another limitation of the OpenAPI generator, was that the key of a generated Java API Map needed to be of type String, this was something that was not sitting well with us, hence we’ve extended the generator to support custom key types via the x-map-key attribute like so:

Fig 11. OpenAPI specification with custom map key definition

The above attribute would result in generating a map with a custom key type, as depicted below:

Fig 12. OpenAPI generated map with a custom key type

As mentioned at the beginning of this article, our Java services are using the Spring Cloud stack with declarative Feign REST clients, originally the Spring Cloud Netflix Feign module was used for that purpose, however, with recent upgrades to newer Spring versions it was replaced by the Spring Cloud OpenFeign module that resulted in changes in the imported packages (import org.springframework.cloud.netflix.feign.* was replaced with import org.springframework.cloud.openfeign.*). So we ended up in a situation where services use either the old Feign clients (located in their respective packages) or the new ones, yet share the same OpenAPI specification. To support both cases we made it configurable for consumers to go with either one:

Fig 13. OpenAPI generator configuration with OpenFeign code enabled

Another shortcoming of the default OpenAPI Spring MVC generator was the lack of support for making the generated classes implement specific interfaces (using the “implements” keyword), as of version 4.3.1 only the “extends” keyword was supported. We have overcome this limitation by introducing a custom attribute x-codegen-implements allowing us to specify a number of interfaces the generated class should implement.

Fig 14. OpenAPI generator configuration with custom interfaces

The generated output for the above configuration can be found below:

Fig 15. OpenAPI generated class implementing custom interfaces

Teams often run into the situation where they have to use multiple OpenAPI specifications to generate the necessary contracts to communicate with the upstream services. Unfortunately, the org.openapitools:openapi-generator-maven-plugin requires a separate configuration for each specification, which was a bit too verbose and too cumbersome for us.

Fig 16. OpenAPI configuration with the default openapi maven plugin

In order to circumvent this issue we have created our own maven plugin The main aim of it was to reduce the boilerplate configuration, yet keep the flexibility of the original plugin:

Fig 17. Reduced OpenAPI configuration with the custom openapi maven plugin

We have decided to keep all of the OpenAPI specifications in a single place - a GitHub repository. In order to have proper release management and versioning of the OpenAPI specs we have made it part of our Jenkins pipeline. Upon a merge to the main GitHub branch we pack and upload the changed, and automatically versioned, specification to the artifactory. From there, it can be simply imported as any other maven dependency and used to generate the required API classes.

Fig 18. Example OpenAPI specification import

Once having the OpenAPI spec dependencies defined in the pom file, our custom maven plugin iterates over the all available OpenAPI specs and generates code for them, honouring the defined configuration.

Fig 19. Example OpenAPI maven plugin configuration

When dealing with a large number of microservices a common case is that they share common features, hence common models, within a single bounded context. In a typical Code First approach operating in the Java ecosystem this problem would be solved by sharing a common jar which would be then included by the interested parties. We were interested in achieving similar functionality but from the OpenAPI perspective. We achieved this by simply extracting part of the common functionality into separate specification files and referencing them from the individual OpenAPI specifications that would like to reuse these definitions.

Fig 20.Example specification reusing “common types” stored in separate definition files

The challenge in having OpenAPI specifications referencing other OpenAPI specifications was the dependency management and versioning part. It was especially hard from the consumer perspective, since consumers would need to include both specifications in their respective versions (the referencing one and the referenced one) in order for the code generator to succeed. The solution for this was building API bundling, by adding to the specification a meta attribute called x-bundle-apis we could make all referenced definitions to be included in a single final artifact that was then uploaded in the CI pipeline to artifactory. This meant, from the consumer perspective, that they just needed to include a single dependency now, making their life significantly easier.

By default, when bundling the API we’ve always included the latest referenced specification. Since we have had cases where we wanted to give more control to the API owners to specify which concrete version of a “common API” they would like to reference, we’ve included another attribute x-bundle-imports which could be used as following:

Fig 21 .Example specification using pinned versions of “common types”

Challenges ahead

Despite the vast number of improvements we have already made in order to retrofit OpenAPI and its tooling into our existing AUTO1 ecosystem and practices, there is still room for improvements. Just to name a few. Although OpenAPI specification itself is language agnostic, we have language specific generators which in some cases need to be instructed on how to generate the output code, and unfortunately these generator hints often make sense only in context of a specific language. As an example, instructing the generator to make the generated Java classes implement some specific java interface. Since the OpenAPI specifications are enriched directly with these hints, it results in obscuring and cluttering the contract definition.

Yet another challenge would be to generate more complex classes resembling the structure as seen when written manually in the times from before the OpenAPI approach was adopted.

The verdict

As presented in the article, OpenAPI is a very powerful tool that enabled our AUTO1 organization to move forward faster in a more structured and unified way in terms of defining and evolving the REST APIs. It’s not a silver bullet, as the design first approach and the OpenAPI specification come with a number of challenges by themselves. However, with the talented engineering force we have we were able to resolve most of them in an efficient manner, yet every organization planning to adopt this tool needs to weigh in the benefits and the shortcomings and make the call for themselves.