SAP Digital Manufacturing (DM) helps manufacturers to optimize their production processes by connecting the business systems with the shop floor equipment. SAP DM provides global visibility across all plants and orchestrates the execution and monitoring of the production operations. It also offers a powerful concept for extending the core features of the solution with customer-specific functionality. As outlined in Katja Huschle‘s blog post, custom extensions fall into the following categories: Business process extensions, User Interface (UI) extensions, and Machine Learning (ML) extensions.

This blog post series explains how to develop, integrate, and deploy business process extensions for SAP DM on the Microsoft Azure platform. This first part introduces a reference architecture for such extensions, which is applied in part II to a real-world visual inspection scenario to detect defects or anomalies in product pictures taken by an industry camera during the production process. The underlying ML model to perform this task, along with the orchestration and data transformation logic to receive, process, and reply to the visual inspection requests from SAP DM, is deployed as a service on Azure.

I want to especially thank Katja Huschle, Philipp Raub and Bastian Ulke. Without their support, deep technical knowledge and manufacturing industry expertise, this blog post series wouldn’t have been possible.

Reference architecture overview

The proposed reference architecture provides a set of guidelines and recommendations for implementing SAP DM business process extensions on Azure. It addresses common questions around security, interoperability, reliability and performance to improve when designing a business-critical solution that spans across different platforms and technologies. Figure 1 shows the key components in SAP DM, SAP Business Technology Platform (BTP) and the extension on Azure based on a microservice architecture.

Figure 1: Reference architecture

Let’s take a closer look at the reference architecture in the highlighted areas (indicated by the numbers in figure 1):

1. Triggered from a Production Operator Dashboard (POD), the Production Process Runtime runs the Production Process which controls the customized execution of the operations on the shop floor. It calls the extension on Azure registered as an external service in the Service Registry of DM.
2. Integrating the extension requires a Destination created by an administrator in the SAP DM subscription’s BTP subaccount. The Destination takes care for authorizing the request to the extension’s public endpoint on Azure. It uses the OAuth 2.0 Authorization Framework (OAuth) to obtain a security token from Azure Active Directory (Azure AD) before the request is sent with the production order data. This includes context information like the Shop Floor Control (SFC) or plant ID which is used later in the process to correlate the response from the extension.
3. Azure API Management (APIM) service implements the single-entry point for DM to call the extension.  This follows the API Gateway design pattern for microservice-based applications to centralize cross-cutting concerns such as protection from security threats by enforcing authentication and data encryption with standardized protocols (for example OAuth and Transport Layer Security, TLS). The API Management gateway acts as a reverse proxy and can limit the number of requests that clients (such as DM) can make within a certain time period to prevent overloading or abuse of the extension (throttling). All rules enforced by the API gateway to the requests and responses of the extension are configured as XML-based APIM policies.
4. The orchestration and data transformation logic of the extension is implemented as a set of loosely-coupled microservices. This reference architecture proposes a service granularity according to the processing stages of a request (see section Microservice design) and uses the following communication messaging patterns:
   • The external communication between SAP DM and the extension’s public endpoint is based on a variant of the asynchronous request-reply messaging pattern to decouple the POD frontend on SAP DM from the extension’s backend request processing. Due to the command-like nature of the message sent by SAP DM to the extension to perform an operation within the scope of the running production process, the extension acknowledges the reception of the message immediately. This helps the POD to remain responsive and the execution of the production process is not blocked by the extension to complete the processing of the request which may vary from milliseconds to minutes or even hours depending on the business scenario, current load, available capacity etc.
   • Instead of using a status endpoint as defined by the asynchronous request-reply messaging pattern that the client (here SAP DM) polls to check for the result of long running operations, the extension uses a callback endpoint exposed by SAP DM’s public APIs to send the results when processing is complete.
   • The extension’s internal communication between the microservices is based on an event-driven communication pattern. Events in the extension context are different types of message that announce the fact that a request from SAP DM is either ready for processing, or the results after processing are available for sending them back to SAP DM. These messages are sent and received asynchronously using message queues that store the messages until they are retrieved by the receiving microservice. This enables loose-coupling of the microservices that can exchange data without requiring to be active and available at the same time. The message queues also act as a buffer at times with high volumes of messages sent by SAP DM following the queue-based load leveling pattern to avoid a service downtime due to an overload situation.
5. Depending on the business domain, existing skills and preferences, corporate standards and other decision criteria, the development team(s) for the extension can choose the programming language and development framework, and Azure compute service from a broad set of options, for example the portable and platform-independent Distributed Application Runtime (Dapr), managed Kubernetes clusters, container instances and serverless functions, or infrastructure-level services such as virtual machines.
6. Independent of the use case there is a set of Azure platform services that are required to protect the extension against attacks and keep it running in a stable and secure production environment:
   • Azure Event Hubs provides the scalable, reliable, and secure message broker service for transferring the event messages asynchronously between the extension microservices.
   • Azure Key Vault securely stores and controls access to confidential data required by the microservices at runtime. This includes the service key credentials to authorize the response message sent to SAP DM’s API after processing the request.
   • Application Insights is a feature of Azure Monitor to collect and analyse telemetry data in real-time, assess the extension’s health and performance, proactively monitor its availability and responsiveness, and much more.
   • Microsoft Entra ID (formerly known as Microsoft Azure Active Directory) helps to control the access to the extension’s API. It manages the credentials to authorize requests from SAP DM by registering an application secret for it. SAP DM can request an OAuth access token from Entra ID with this credential which is verified by the API Management gateway. Secure access from the microservices runtime environment to other Azure resources including the services listed above (Key Vault etc.) is controlled by managed identities that are bound to the lifecycle of their resource and used to obtain Entra tokens without having the microservices to manage any credentials.

   The extension may integrate with additional services depending on the use case, for example using an ML model in Azure AI Custom Vision.

7. The extension returns the results from the request to the callback endpoint exposed by SAP DM’s public APIs. This requires a service instance and service key for SAP DM in the SAP BTP subaccount. The service key credentials (client ID and secret) are stored in Key Vault and used by the extension to obtain an OAuth access token from SAP BTP’s Authorization and Trust Management Service (XSUAA) to authorize the API call.

Microservice design

Decomposing the extension into autonomous (micro)services follows common design principles. Each microservice should have a clear and well-defined purpose to achieve high cohesion and loose coupling which enables independent development and deployment lifecycles of the services. This reference architecture proposes a decomposition based on the three processing stages of a request: Reception from SAP DM, processing of the production order data sent with the request, and responding the results back to SAP DM. Reaching one processing stage results in an event which triggers the execution of the next step by the corresponding microservice. This leads to the following design that should be applicable to many (if not all) use cases for SAP DM extensions on Azure:

• The RequestReception microservice receives SAP DM’s request from the API management gateway. It may extract the production order data sent with the request and stores it in a database or blob storage before writing an event to the ProcessRequests message queue to trigger the next processing step. The RequestReception service applies the asynchronous request-reply pattern by acknowledging to SAP DM with an HTTP 202 (Accepted) status code that the request has been received and submitted for processing.
• The RequestProcessing microservice gets triggered by new messages from the ProcessRequests queue. By using the queue as a buffer for all requests added by the RequestProcessing microservice, the RequestProcessing microservice can handle the messages at its own pace regardless of the volume of requests sent by SAP DM. If the ProcessRequests queue size growths, additional instances of the RequestProcessing service can be created to meet the demand. The service may also require other Azure services to fulfill its task. Upon successful completion it reports the results by sending an event to the ProcessResults queue.
• The ResultProcessor microservice gets triggered by new messages from the ProcessResults queue. It is the ResultProcessor‘s task to send the response for the initial request from SAP DM by calling SAP DM’s public APIs. This requires the ResultProcessor to request an OAuth access token from SAP BTP’s XSUAA service using the service key credentials stored securely in Key Vault. It must also make sure that the results are converted to the input data schema of the chosen SAP DM API. To correlate the extension’s response with the original request, context information from the request such as the plant ID or SFC ID is copied over to the response. Transient failures that may occur when sending the response, for example temporary loss of network connectivity or unavailability of SAP DM’s API endpoint, should be handled gracefully by the ResultProcessor. Instead of reporting an exception it can continue to send the failing response for a maximum number of retries with an delay factor interval to delay subsequent retries as proposed by the retry pattern.

Network security

The extension is deployed in a virtual network (VNet) with two subnets: The first hosts the APIM instance, the second the application services runtime (Azure Function, Azure Kubernetes service etc.) for running the extension microservices. This enables the use of the private endpoints and a more fine-granular control of the in- and outbound communication using subnet-specific Network Security Groups (NSGs).

A private endpoint is a network interface with a private IP address from the consumer’s VNet that connects it to services supporting Azure Private Link. In other words, enabling a private endpoint for a service brings the service into the consumer’s VNet. This limits the external attack surface of the extension by reducing the exposed services on the Internet.

For example, the APIM instance in figure 1 can only reach the extension’s compute service over the private endpoint deployed by the service in the APIM subnet. This helps to keep the microservices secured and hidden inside their virtual network without making them accessible from public Internet. The same concept is applied to the communication between the microservices and the services they consume, for example Key Vault or Event Hubs. Strictly following this approach for all service communication of the extension limits its external endpoints to the APIM public IP address only.

Reliability

Designing the extension’s architecture to reliably provide consistent access and deliver the expected quality of service to SAP DM is of high importance for business critical processes, especially under (near) real-time conditions in manufacturing. This can be achieved by increasing the infrastructure and application components resilience to failures, for example by making them highly available to avoid downtimes and unplanned outages. However, planning for (high) availability and resilience is always a trade-off and must reflect the surrounding business requirements and consider the cost implications.

By following the application communication and messaging patterns for queue-based load leveling, asynchronous request-reply, retry strategy and request throttling, the extension can already achieve a high level of availability and resilience. In addition, the following measures on infrastructure level can further improve the extension’s reliability:

• Intra-regional resiliency with availability zones: Depending on the Azure region where the extension is deployed to, the supporting platform services of the reference architecture (Azure API Management, Azure Event Hubs, Azure Key Vault, Azure Monitor and Application Insights) can be replicated across availability zones for redundancy. Availability zones are physically separate datacenter(s) within an Azure region that have their own power, cooling, and networking infrastructure. They are connected by a high-performance network with low latency which helps data and services stay synchronized and accessible when an hardware or datacenter failure occurs in a single availability zone.
• Cross-regional replication: Many Azure regions (e.g. Germany West Central) are paired with another, secondary region (e.g. Germany North) based on proximity and other factors for implementing a disaster recovery strategy. For example, in case of a natural disaster affecting a whole primary region, it is recommended to setup automated backup and restore operations for services such as API Management. Other services like Key Vault offer built-in failover support between the primary and secondary region.
• Multi-regional deployment: The extension can be deployed redundantly in a region pair, or even across non-paired regions (e.g. to East US and North Europe) if requests should be load-balanced and routed globally to the closest available endpoint. Azure Front Door provides a global load balancing for HTTP(S) and fast fail-over. It can be used in front of API Management to route requests sent by SAP DM to extension instances in other region(s) if one region becomes unavailable.
• DDoS protection: Distributed Denial-of-Service (DDoS) attacks are among the fastest-growing attack vectors worldwide as documented in this report (p. 38 ff). Enabling a DDoS protection plan for the extension’s network resources (public IP address, VNet) can mitigate common TCP-, UDP- and other network-level attacks to improve the extension’s availability.

Summary and outlook

Part I of this blog series introduced the reference architecture for common extension scenarios of SAP DM with the Microsoft Azure platform. Part II applies this architecture to a real-world use case for visual inspection with SAP DM by setting up a custom image analysis model with Azure AI Custom Vision.