Sonoma Partners Microsoft CRM and Salesforce Blog

Dynamics 365: Speeding up API Access with the Async Service

Today's blog post was written by Angel Shishkov, Development Principal at Sonoma Partners.

When we need to do a recurring custom calculation or integration in CRM, we usually build a console app, or Azure Job. These solutions run on a schedule and use the CRM SDK along with a service account to connect to CRM and perform the necessary retrieves, creates, and updates. Since these are external to CRM, there is some performance overhead when data is sent over the internet to the CRM server.

It is generally much more efficient to interact with CRM on the server-side; through plugins, workflow activities, custom actions, etc.

I was wondering how much more efficient and how feasible this would be, so read on to find out.

Setting up a test scenario

I specifically wanted to test how the CRM Async Service will perform as a substitute to a recurring calculation job. The theory being that the Async Service physically lives very close to the CRM database (in the same data center, most likely) and will therefore have much lower overhead when transferring data. On top of that, the Async Service does various caching and other optimizations, that allow it to connect to the database without going through the normal authentication steps that a console app would. So, I wanted to test a bulk update scenario using a console app, versus the same scenario using the Async Service, and see how much of a difference in performance there was. If you’re not interested in the technical details, scroll down to the Results section.

The test scenario I chose was a mock case of reading data and creating records. The requirement would be to read some data from CRM, with a FetchXML query, then create a small record with the results of that query. We would then do this several thousand times. This roughly simulates a bulk calculation scenario (like a custom calculated field that is refreshed daily) or an integration scenario (reading and/or writing to CRM).

For the query, I used a sum aggregation of the extendedamount of all the salesorderdetails in the system. Basically – get the total amount of all the Orders in CRM.

For the record creation, I used a custom entity with two fields; an Index field to hold the index number of the record and a Total field to hold the calculated total amount. The Index was just incremental numbers, starting at zero and the Total amount was always the same value repeatedly retrieved by the query above.

Implementing the console app

The console app was a basic application using the CRM SDK to create a connection to the server and perform the queries and creates. It loops 20,000 times and each time it runs the aggregate query, gets the resulting sum, and creates a new_loadtest record with that sum as the total and an incremental number as the Index. I decided to use ExecuteMultipleRequest in batches of 100, since this approach is standard for these kinds of apps and it would be an unfair comparison not to use it. Here is the loop code:

Implementing the async job

The async service implementation was more interesting. The idea here was to leverage the CRM Async Service to do the work, so we would need an async plugin. CRM Online has a 2min timeout on server-side custom code and this includes plugins, workflow activities, etc. On top of that, CRM has a mechanism to detect and prevent infinite loops on the server side where a component could trigger itself indefinitely, like a plugin that fires on the creation of a record, which triggers itself again by creating a new instance of that record.

Based on the above, it becomes difficult to write an async plugin that runs for more than 2min, so the work would need to be broken up into chunks. Since the test case was very simple - run the same fetch query, create a record with an incremental index – I opted for a simple approach as well. I created a new_asynctrigger entity with a new_startindex and new_endindex field. An async plugin would fire when a new_asynctrigger record is created and process the indexes between start and end in one chunk, the same way the console app above did – run the same fetch query, create a record with an incremental index. A separate console app would be needed to create these new_asynctrigger records with the correct start and end index, so that the Async service can be triggered. This is definitely an overhead, compared to the pure console app approach. Furthermore, we need to keep the chunks small enough that each can be processed in under 2min, plus a generous buffer for safety. In this case I used a chunk size of 1000.

Note that the plugin does not use ExecuteMultiple, like the console app does. Since the Async service executes multiple jobs in parallel, we could trigger the CRM ExecuteMultiple throttling (2 concurrent threads max) and cause our Async jobs to error out.

Here is a snippet of how the console app creates new_asynctriggers.

And here is the async plugin code that processes each chunk.

Summary

To summarize, we have a console app going up against the Async Service in the slightly unfair race to query CRM and create records. It’s like one of those timed cooking challenges, except one cook is in the kitchen and the other one starts five blocks away.

Let’s summarize the pros and cons, only as related to speed and efficiency, of the two approaches.

Console App

Pros:

  • Needs to only connect to CRM one time, and can reuse that same connection.
  • Can create all the records in one loop, does not incur overhead to chunk records.
  • Can use ExecuteMultiple.

Cons:

  • Communicates with CRM over the internet, which involves network latency as well as the performance overhead of accessing the external API.
  • Cannot run more than 2 threads in parallel. By default, CRM Online allows 2 concurrent ExecuteMultiple requests. In this test, I did not implement any concurrency.

Async Service

Pros:

  • Has an internal connection to CRM that does not require the usual OAuth handshake.
  • Presumably communicates with CRM over an internal network with much lower latency and higher bandwidth.

Cons:

  • Requires breaking up records into chunks, as it can only process 2min at a time.
  • Requires overhead of creating records in CRM to trigger the async plugins, because of CRM’s infinite loop detection.
  • Does not use ExecuteMultiple, creates records one at a time.

Results

As I stated in the summary, this test strongly favors the Async service. All of the “work” is accessing CRM, not external systems or databases and the requests to CRM are small and numerous, which is bad for the overhead incurred by a console app. Either way, I was a bit surprised by the results.

Console App

On average, 20k iterations completed in 74min.

Async Service

On average, 20k iterations completed in 52sec.

That seems like a very large difference. The two main inequalities I think, are the level of parallelism and the speed of CRM access. The console app runs one thread, while the async approach quickly creates 20 jobs (20k records / 1k chunk size) that are queued to execute together. I can’t say for sure how many of those run in parallel inside the Async service, but likely quite a few, if not all of them. The rest can be attributed to much faster access to CRM from the Async service.

Overall, I wouldn’t recommend using the CRM Async service to process large migration data, as that is not its function. For smaller integrations or recurring calculations, it seems to be clearly the faster choice, but because it is not fully under our control, I would still tend towards the console app/Azure job approach, unless there is a specific performance limitation that needs to be addressed.

Thanks for reading!

Topics: Microsoft Dynamics 365