Back

TechnologyJun 11, 2014

When to Consider an Integration Framework: Real World Example

Jon Pierce

In a previous entry I covered when to use an integration framework. But I recently ignored my own advice and the experience makes a good, real world example for when to consider an integration framework.

Background

Here’s a little history to make the situation clearer. Since December of 2013, I’ve led a small team of developers that updates and maintains one of our core applications. The application was built about five years ago by a completely different team and has been passed around ever since. Even though it’s old by technology standards, it’s using modern libraries like Spring MVC and JPA.

The Opportunity

Recently we had the opportunity to integrate with a third party REST API. Our front end (JSTL + JQuery) would need to make AJAX calls and retrieve data from an external system. There were a few considerations in the implementation that made it anything but trivial:

1.  The external system didn’t really understand REST/JSON. It knew how to build a JSON response, but a GET always returned an HTTP 200 response—even in the case of an error. And in the case of an error, it returned an error message as the payload. Furthermore, some of the REST calls required POST data that the view layer didn’t need to know about.

2.  The external system required authentication that was not related to the user viewing the data. We had to use an API account and that account could not be in any way visible to the user.

3.  The external system performed poorly. While this was the least of our worries, long running AJAX calls are annoying, and multiple long running AJAX calls can cause problems for a back end system and generally make things worse. We needed a way to decouple the outbound side of things from the inbound side, and not interfere with the client.

First Pass

To address the above considerations, we made a really simple architectural decision. We would decouple the service consumer (the AJAX client) from the service provider (the external system) by using a service proxy. A service proxy is an implementation of Service Virtualization, an enterprise integration pattern, in which the service consumer interacts with the service proxy instead of the service provider. We would implement a service proxy with a simple Spring MVC REST/JSON controller.

The proxy allowed us to expose REST endpoints that could be accessed with simple GET requests and call a Spring REST Template client to handle interaction with the external service provider from the server. The client no longer had to know anything about the external server, effectively hiding authentication, client proprietary data, and managing the malformed REST behavior. Lastly, we could easily cache the responses on the server, eliminating the potential to overload the external system with multiple long running requests. Problem solved!

New Requirements, New Consideration

From my perspective, this solution was really elegant—until I demoed it for our customer. The results of the demo were new requirements. It was awesome to see that data in our app, but now we also needed other data from other external systems that also had REST APIs. What did this mean? More proxies, which meant more REST endpoints, more REST clients, and more code to maintain and test. At first glance, the pattern we had established with the initial implementation was a good one, but could we do it without writing all of that code over and over? We could if we used an integration framework.

Refactoring First Pass

While the first pass architectural solution was elegant, the implementation was mostly unnecessary. Spring Integration can handle almost everything above with nothing more than POJOs. In order to replace the initial implementation, we need the flow to look like this:

AJAX HTTP Request -> Inbound http gateway (Service Proxy) -> transformation / enrichment -> outbound gateway -> external service

Inbound Gateway / Service Proxy XML:

What it does:

1.  **Create the inbound gateway with request and reply channels, long timeout (30 seconds), the supported HTTP method GET, and bind it to a path.

2.   **Create the associated channels.

Transformation:

What it does:

1.  **Create a transformer to handle the message transformation for the outbound service, bind it to the bean “externalServiceATransformer” using the method “transform”, set the input to the request channel from the previous step, and set the output to a transformed channel.

2.  **Define the bean mentioned above and bind it to the class “com.credera.transformer.ExternalServiceATransformer”. We would have to create this class and implement the method “transform”. 3.  **Create the associated channel. See more info on transformers.

Outbound Gateway:

What it does:

1.  **Create the outbound gateway with request and reply channels, same timeout as the inbound gateway, and using the HTTP method POST (per the requirements).

2.  **Define the expected response type as a string (it’s returning JSON).

3.  **Define a request factory in order to add the username and password authentication.

4.  **Set up caching with a cache named response, caching the response to methods named “handle*Message”, which will catch the method “handleRequestMessage” invoked by the outbound gateway. Key is used for looking up this message in the cache, and in this instance the key is the payload of the request (i.e., if the same request is sent, the response should be returned from the cache).

5.  **Create the associated channel. See more info related to the method “handleRequestMessage”.

Authentication:

What it does:

1.  **Create a bean using class “org.springframework.ws.transport.http.HttpComponentsMessageSender” and instantiate it with a pre-defined “org.apache.http.auth.UsernamePasswordCredentials”

2.  **Create a bean using class “org.springframework.http.clent.HttpComponentsClientHttpRequestFactory” and instantiate it with an httpClient that uses a reference to the class created in #1’s httpClient. This completes the definition of clientHttpRequestFactory from the previous configuration.

Response Transformation:

What it does:

1.  **Create a transformer to handle the message transformation for the response, using the same class as the request transformation, but the method “transformResponse”. The output is then sent back to the Inbound Gateway’s reply channel.

The new integrations would work exactly the same way. We no longer need to build or maintain REST endpoints or custom REST clients. From the AJAX client’s perspective, the client does not have to know anything about the external services allowing us to standardize the request type and just change specific attributes or endpoints in order to get the desired response. The amount of configuration required is determined by the requirements as some endpoints may require more or less transformation or different types of authentication.

Conclusion

As a result of our hard work, we now have a solution that addresses all of the original requirements, as well as the new requirement requested by our customer. Furthermore, the new solution is easily extended in the case of additional external systems without writing a lot of new code. Less code to write means less code to maintain, which I personally see as a huge win.

References:

Spring.IO – http://spring.io

Enterprise Integration Patterns –http://www.eaipatterns.com/

Artem Bilan – Spring Integration Outbound Gateway Cache – http://stackoverflow.com/a/20746911

Have a Question?

Please complete the Captcha