Sometimes we need to get iteration index in Foreach loop for data processing, one resolution is to set the "Concurrency Control" to 1 and use a variable as index which always be added by 1 for each iteration.
This approach can achieve what we would like to have, but it will impact the performance since the foreach is in sequential mode.
In order to keep the "Foreach" action running in parallel, there's a new approach as following:
Steps
1. Generate a new array which save the all the index of the original array which need to be processed. For example, if we have a 10 items array, then the index array need to be generated as [ 0, 1, 2 ... 9 ] (start from 0, end with [array length - 1] ).
Here I'm using Azure Monitor KQL expression "range" (reference : range operator - Azure Data Explorer | Microsoft Learn) to generate the index array as per the original array length (in screenshot, the variable "TestData" is my original array).
The query result is following:
Meanwhile, if you don't have Log Analytics Workspace, the index array also can be generated by various ways (e.g. Azure Function, Azure Automation, Logic App Foreach action, etc.)
2. In the Foreach action, we need to iterate for the newly generated array
3. Use expression to pickup the original items directly refer to "index".
Since we are directly get items by index which has been generated outside the foreach loop, so it will not be impacted by the parallel iterations.
Performance comparison
In my local environment, for processing 1000 elements array, the new approach can save around 90% time.
For more testing runs, result as following:
|
RunID |
Parallel_Action |
Parallel_Duration |
Sequential_Action |
Sequential_Duration |
|
08585263805686940847269633553CU04 |
Scope_-_Parallel |
26292 |
Scope_-_Sequential |
382392 |
|
08585263805721763244014366564CU05 |
Scope_-_Parallel |
28928 |
Scope_-_Sequential |
386164 |
|
08585263806421048241938496487CU18 |
Scope_-_Parallel |
33225 |
Scope_-_Sequential |
418940 |
|
08585263806457138684407646963CU03 |
Scope_-_Parallel |
29956 |
Scope_-_Sequential |
390212 |
|
08585263806629391869208870544CU12 |
Scope_-_Parallel |
30782 |
Scope_-_Sequential |
403036 |