Consider an Integration project in which the timelines of the different layers make it hard to synchronise the development and test phases across different systems. Or an application integrating with an external system that charges for every call made. Wouldn't it be desirable to limit the paid invocations to such systems during the development phase? How about a multi-environment application which only has one downstream environment available to interact with? These and many such scenarios make it necessary for a reusable, lightweight and intuitive application which can, in essence, be used as a replacement for one or more of the interfacing systems, applications or layers.
Stubs are well-known in the Test-Driven Development world. Essentially, a stub is an implementation which the System Under Test or SUT can call to get predetermined responses/errors. They serve as an alternative to the intended endpoint and can be used for testing the implementation of the SUT. Stubs are different from Mocks in that a mock implements the same interface as the SUT object. There are other entities such as Spies and Dummies used for similar and different purposes. In the context of large-scale integration projects (AKA Digital Transformation), the application of TDD takes a new, multipurpose and multifaceted form. The SUT becomes a whole application or even a complete layer of the Integration Architecture. The stubs have to play the roles of all the other characters from the TDD stable. In this blog, we will visit one of the more intricate and versatile approaches to using a stub.
Presenting the Smart Stubbing Framework: Stubs with an API powered brain
Christened by Ali Nikkhah, Principal at Deloitte Platform Engineering, the Smart Stubbing Framework was implemented at one of the recent major digital transformation initiatives delivered by Deloitte. The framework proved instrumental in the Integration Platform as a Service, or iPaaS layer. There were many purposes served by the framework over the course of the project:
- The downstream systems were undergoing changes in parallel with the MuleSoft applications. As such, during the development phase, developers would need to have a way to hit an application and get the expected results without hitting the downstream applications.
- Paid calls to external systems were redirected to the stubs during the development phase so as to curb avoidable expenses in licensing.
- During triages or while trying to recreate a scenario during tests, when specific data needed to be configured in the downstream systems it took a longer time. Adding dummy data in the stubs was a faster alternative.
- During the test phases, several environments of the iPaaS layer were available for the upstream systems while there was only one end to end environment available. The stubs were the commonly used endpoints among the multiple iPaaS environments.
The Framework
In essence, the stubs are realised as a MuleSoft application. They would use the same contract as the system/layer being stubbed. They would implement all the flows in scope. Each of these flows would have choice routers serving different responses. The configurations are designed to be as close to the endpoints being stubbed. This would ensure that the switch from stubs to the real endpoints is as frictionless as possible.
Design
The design phase of the stubs would involve two major steps:
- Identification of the system(s) being stubbed. This step ensures that only necessary stubs are created. Similar stubs should be consolidated. If we are discussing MuleSoft applications, it would be practical to keep the number of deployable packages low.
- Extensive introspection into the project requirements to come up with the various scenarios for which we expect to be catered. The use cases should be documented appropriately. A sample use case documentation is shown below.
| Integration Interface & URI | Actual API | Request to Response Correlation Logic | Covered Test Cases |
|
Validate Email Addresses URI: /email/validation/v1 |
Validation → External (Experian) REST | Correlation of the incoming email address to specific statically created response payloads. |
|
|
Validate Phone Number URI: /phone/validation/v1 |
Validation → External (MSISDN) REST | Correlation of the incoming phone number to specific statically created response payloads. |
|
Implementation
Implementation of the stubs would be similar to that of a Mule application with the key difference appearing at the business logic implementation stage. The project creation and packaging should follow best-practices. Depending on whether the application contract is a WSDL or a RAML, the flows can be generated. In Anypoint Studio (MuleSoft’s IDE), it is possible to generate the flows automatically. Alternatively, the entry point to the stub can be an HTTP listener which delegates the routing strategy to an APIKIT router (or an APIKIT for SOAP router).
A sample entry flow for the stubbing flow would look something like this:
Subsequently, the router would take the control into the respective flow wherein the conditional responses are returned. This is illustrated in the sample flow below:
The choice routers can be based on business logic which would drive the different response scenarios. The responses can be populated by using Parse Templates. Other alternatives are to use payload transformers in Dataweave, Groovy or Spring beans.
An example of Groovy Engine being used for payload creation:
The response should conform to the contract being used. The error responses and negative test scenario responses were aligned to the interface contracts of the integrated systems.
Implementing the same contracts also means that the same API Policies can be applied to the stubs as the intended system.
Additional Considerations
Over the course of the project, several occasions prompted a thorough analysis of the way we were using the stubs:
- When the stubs were used for performance tests, it made sense to increase the resources allocated to the application (vCores in MuleSoft parlance).
- Attention should be paid to the API Identifiers being used for Policy enforcement since clients are provided access to specific environments.
- Stubs to overtly large datasets should be avoided in order to keep the stubs lightweight.
- Stubs should not be used for performance benchmarking of the overall integration.
It was a learning experience to be a part of the iPaaS team which entailed having full ownership of the Smart Stubbing Framework. The first phase was implemented by Ali himself. Other notable contributors were Becky McCreath, Vittoria Capuano and a very enthusiastic vacationer cohort. I would like to thank the entire iPaaS development and testing team for contributing to the Smart Stubbing Framework in varying capacities.
The most powerful aspect of the framework is its simplicity. Keeping it aligned to the business requirements helps to ensure that they can be maintained and augmented as the requirements evolve. As the entire business logic is driven by conditional programming and pre-determined responses, once the framework is established, it is easy to enhance or modify. It can be adapted to practically any architecture and any technology stack for similar results. Should you wish to know more, we would be glad to share details about the journey. Please get in touch with any queries or comments about the framework.