Introduction

Boomi can mediate interactions with Microsoft Dynamics 365 Custom Engagement by invoking the Dataverse API. In situations where applications need to support real-time synchronous interactions, measures need to be in place to ensure that threshold limits that are imposed by the Dataverse API are not reached. This article presents a solution to address one of these service limits, implemented in Boomi to enable the load balancing of the application users that call the Dataverse API.

Solution Context 

When integrating to Microsoft Dynamics 365 (D365) via the Dataverse API, several service protection limits  have been introduced that can affect customers of the D365 platform, by resulting in 429 HTTP responses when request volume exceeds a certain threshold. In summary, the following API rate limits are imposed by D365 on a per web server basis: 

In Azure API Management (APIM), several rate-limiting policies such as ‘rate-limit-by-key' and/or ‘quota-by-key' are available for use. From the list above, the number of requests can be addressed by utilising the request throttling features of an API gateway. While the execution time limit will not be an issue for synchronous interactions, the limit of 52 concurrent requests per user will affect applications that need to support many real-time synchronous patterns – which will be the main focus of this article.

In the context of a mobile application, a single request can result in an orchestration of multiple backend calls to the Dataverse API. Each of these calls will be using credentials of a single D365 application user for server-to-server authentication, and in such a situation the 52 limit can be easily reached.

This article presents a solution that we have implemented in Boomi to enable the load balancing of the application users that call the Dataverse API. The platform architecture includes Azure APIM and Boomi hosted on Azure VMs, connecting to D365 using Oauth2 client credentials authentication method. In D365, the concept of an Application User allows programmatic access to D365 data and entities, without the need for an additional license. As pre-requisites, the application (in this case Boomi) must be registered in Azure Active Directory and the Application User must be created in D365. 

Solution Approach

An incoming request would be intercepted in Azure APIM, through the use of an APIM Policy to do the following: 

• Use a hashing algorithm to randomly generate an integer between 0 to x-1, where x is the number of client credential sets required to load balance between.
• Pick the corresponding client Id and secret based on the integer. 
• Pass the client Id and secret as custom headers to Boomi backend process, where the Boomi Web Service's listener endpoint will receive these values . 
• Set the Dynamic Process Connection properties to pass the client Id and secret during runtime, prior to calling the Dataverse API from Boomi. 
  

Click to enlarge

Implementation in Boomi

A pre-requisite to the implementation of this functionality is that sufficient standard connections must be available before deployment since one license is required per unique connection per runtime.  

As mentioned earlier, the client Id and secret are passed into Boomi from APIM as custom headers. Since the transport layer between APIM and Boomi is HTTPS, this information is securely transmitted. The headers are retrieved and saved as Dynamic Process Properties in the Boomi process, which allows them to be referenced later on in the process flow.

In Boomi, the connection properties of a Connector shape are set at design time via the canvas, or through extensions during deployment. Boomi enables setting up environment-specific configuration parameters via the concept of extensions. For instance, the D365 connection information can be parameterised against each individual environment via environment extensions.  To address our problem that requires load balancing of the D365 user credentials, we need to allow the connection properties to be dynamically configured at runtime. 

Boomi Dynamic Connection to the Rescue 

In general, Boomi connectors have a limitation whereby only one user credential can be used for authentication purposes with any backend system, including D365. Recently, Boomi has introduced the concept of Dynamic Connections, available for customers as an Add On feature that allows us to dynamically pass credentials per user (not bound to only one). The use of each Dynamic Connector is limited to the Customer's connectivity to a unique application instance, in this case, it is MS Dynamics 365, accessible via the 'Microsoft Dynamics 365 for Customer Engagement' Application Connector. Depending on the purchase agreement, the customer will have access to a certain number of Standard/Test licenses available to be used for this feature.  

Configure the Connection

By enabling the 'Allow dynamic connector properties' option on the Microsoft Dynamics 365 for Customer Engagement Connector, it is possible to override certain connection settings and extension values through the use of the “Set Properties” shape.

This allows us to override client credential details each time a call from Boomi is made to D365. For instance, a pool of 5 client Id/secret pairs can be used to load balance the API calls between five D365 application users, which increases the request volume available to the application.  

Set Dynamic Process Connection Properties

The values for client Id and client secret are set dynamically using a set of previously defined Dynamic Process Properties. A typical Boomi process requiring this would follow this pattern:

The Set Properties shape can be used to configure the client Id and client secret as Dynamic Process Connection Properties as shown below, with the values being set to the corresponding values of the previously configured Dynamic Process Properties:

It is worth noting that while the Azure APIM Policy was used in this instance for load balancing between different sets of client credentials, there is an alternative approach in scenarios where Azure APIM is not present. This is to use a Javascript or Groovy script to perform this logic within Boomi and selecting the client Id/secret pair prior to calling the Dataverse API. The drawback of this approach is that it can affect the performance of the processes, therefore an informed decision should be taken where this operation takes place, based on the platform architecture.

Conclusion

This article presented how to ensure Dataverse API service limits are not reached through a solution implementation that used a combination of Boomi and Azure APIM. Boomi provides the capability to enable special connector features that can help overcome these types of scenarios with minimal effort, and as such is one of the advantages of utilising a low code integration platform.