Logic Apps Data Mapper - Compiling and Testing Your Map

[ParikshitGupta]

The Logic Apps Data Mapper provides manual testing capabilities within its designer interface. This allows you to execute your map against a specific input and observe the resulting output after transformation, serving as a valuable tool during the map development process. However, to ensure the accuracy and reliability of your maps, it’s crucial to conduct thorough testing with a variety of inputs and implement logic to validate the resulting outputs. This is particularly important before deploying your maps to production environments.

Moreover, as your maps undergo enhancements and updates, regression testing becomes an essential part of the CI/CD process to prevent any potential issues from reaching the production system. In this blog post, we will delve into the APIs that power the map designer and explore how these APIs and libraries can be leveraged to create automated tests for your maps. This will ensure that your maps are not only functioning correctly but are also ready for deployment in a production environment.

For the Logic Apps Data Mapper, the Workflow design-time API, which serves the map designer, is initiated in the background. This consists of three main APIs: the schema, the XSLT generator, and the map tester.

1. Schema API: This API compiles, resolves, and parses the schema to provide the necessary schema tree for map design.
2. Generate XSLT API: This API compiles your map code, which is defined in the Logic Apps Mapping Language (LML), into XSLT 3.0. The generated XSLT is self-contained, meaning it includes user-defined functions code and inline XSLT. This XSLT is the only thing required at runtime during workflow execution for transformation.
3. Test Map API: This API executes the map (XSLT) against a given input message and sends the output message in response. We use the transform engine (Saxonica .NET framework engine) to run XSLT 3.0 through the test API to mimic how the XSLT map will be executed by the workflow run, ensuring there are no surprises at runtime.

The design-time API is initiated at the first start of any workflow designer or map designer. Alternatively, you can also start it outside of the VS Code development environment. The prerequisite for this is to have the function core tools installed.

You need to keep two files in the working folder: 'Host.json' and 'LocalSettings.json'. You can start the design-time API using the command 'func host start –port <port number>'.

Here's what the 'Host.json' file looks like:

{
  "version": "2.0",
  "extensionBundle": {
    "id": "Microsoft.Azure.Functions.ExtensionBundle.Workflows",
    "version": "[1.*, 2.0.0)"
  },
  "extensions": {
    "workflow": {
      "settings": {
        "Runtime.WorkflowOperationDiscoveryHostMode": true
      }
    }
  }
}

And here's the 'LocalSettings.json' file:

{
  "IsEncrypted": false,
  "Values": {
    "AzureWebJobsSecretStorageType": "Files",
    "FUNCTIONS_WORKER_RUNTIME": "dotnet-isolated",
    "ProjectDirectoryPath": "REPLACE_WITH_LOGICAPPS_PROJECT_PATH"
  }
}

The 'ProjectDirectoryPath' in 'LocalSettings.json' is required and should point to the folder where the 'Artifacts' folder resides. This folder structure inside the 'Artifacts' folder in the LogicApps standard project is expected and required by the design time and runtime.

Here is an example showing the folder structure with map definitions, user defined functions and inline XSLT contents:

The LogicApps standard project already includes the 'Schemas' and 'Maps' folders. The 'MapDefinitions' folder contains the map code in LogicApps Mapping Language (LML) format. The 'DataMapper\Extensions' folder contains user-defined function definitions and any inline XSLT content required for map definitions. These are custom code extensions provided for mapping, described briefly in the last section of the article.

All of these files - map definitions, custom functions, and inline XSLT - are design-time files only. Once the map is compiled to XSLT, everything required for the map is embedded and inlined in the generated XSLT. The generated XSLT is kept in the 'Maps' folder along with existing Maps for Logic Apps standard, and should be deployed with schemas and maps. The XSLT name is kept the same as the map name for association.

Testing Your Map with the Test Map API

The API can be accessed at 'http://localhost:<port>/runtime/webhooks/workflow/api/management/maps/{MapName}/testMap' using the POST method. The content type should be set to 'application/json'. The body of the request should look like this:

{
    "InputInstanceMessage": {
        "$content-type": "application/xml" or “application/json”,
        "$content": "base64 encoded message"
    }
}

The response will look like this:

{
    "outputInstance": {
        "$content-type": "application/json" or “application/xml”,
        "$content": "base64 encoded message"
    }
}

To use the Test Map API programattically, you'll need the Workflow Extensions NuGet package. You can find sample code in the DataMapper-CodeSamples repository on GitHub, which shows how to invoke the API using an HTTP client in the 'MapperAPIClient' project.

Alternatively, you can achieve the same result by directly invoking library code. However, the input and output will be different. You can find sample code for this in the 'MapTester' project. Both this code and the Test Map API execute the Saxonica XSLT engine for .Net framework.

The 'DataMapperMapTests' project provides some example unit tests for a couple of maps using this code. The tests are organized by input/output type: XML to XML transform, XML to JSON transform, etc. However, you can organize the tests based on your needs, such as one test method for each of your maps.

The test strategy is simple: it executes the map against different inputs and compares the output against the expected output. The comparison in the example is done using XNode’s DeepEquals function for XML output and JToken’s DeepEquals function for JSON output. When the map has dynamic content like some value based on current date/time, or auto-generated number/GUID, this kind of comparison will not work. In such cases, you'll need to employ a different way of validation.

Compiling Your Map with the Generate XSLT API

The LogicApps Mapping Language is designed with several goals in mind, including human readability, intuitiveness, and uniformity across formats such as XML and JSON. For minor adjustments or changes to your map, you should be able to directly edit the map code. However, the new code needs to be recompiled into XSLT so that the Test Map/runtime has a new map to work with. You can use the Generate XSLT API to compile the modified map. The updated map code can also undergo a code review process, as it's easy to identify logical changes in the map.

The map compilation (Generate XSLT) can be integrated into the CI/CD process to generate the new XSLT everytime the map code has changed. We are considering integrating such features into the LogicApps DevOps feature.

The Generate XSLT API can be accessed at 'http://localhost:<port>/runtime/webhooks/workflow/api/management/generateXslt' using the POST method. The content type should be set to 'application/json'. The body of the request should look like this:

{
    "MapContent": "Map content as string"
}

The response will look like this:

{
    "xsltContent": "XSLT content as string"
}

Alternatively, you can achieve the same result by directly invoking the library code to compile the map code into XSLT. You can find sample code for this in the 'GenerateXslt' project.

Custom Code in Mapping

There are three levels of custom code in mapping:

1. Inbuilt XPath Function: This function can execute any given free-form XPath (3.1) expression. The input is a fixed string which is passed as it is to the XSLT in the map compilation process. If there is a need for ad-hoc complex functionality which is used once, the inbuild XPath function can be used.
2. User-Defined Functions: These functions can be written and placed in the 'Functions' folder. They start showing up as functions in the designer and can be used in map code like inbuilt functions. They can take input from source schema nodes and any other functions depending on how the function is defined. If there are reusable repeating mapping patterns to get the node’s value, define them as custom functions. Custom functions can be grouped together in one custom functions file.
3. Inline XSLT Snippets: If you have a need to generate multiple nodes with their schema structure, you can use snippets of custom XSLT and then use them with the inline-xslt function. The inline XSLT snippets should be placed in the 'InlineXslt' folder. Each file contains one specific inline XSLT functionality.