Introduction

Our production landscape, characterized by its complex integrations, requires ongoing maintenance and support. Within this landscape, we utilize multiple Adapter Engines (SAP PO 7.50), with one dedicated to internal communication and another for external communication. To enhance the quality of integration and gain superior visibility across various integration engines, we have opted to integrate a monitoring solution using the ELK stack. This article presents our solution, GoGoXi, which encompasses a specialized export tool and monitoring reports. Additionally, we will share insights and findings gleaned from utilizing GoGoXi.

Integration landscape

This is an overview of integration landscape. Internal systems are both SAP and non-SAP-based. The tool developed does not depend on number or location of SAP PO servers, but this gives the reader an overview of system landscape.

System Landscape

Reasoning

We posed several questions underscoring the necessity for an external monitoring solution:

• What is the volume of data processed by our integration solution? Does this align with our initial sizing and assumptions?
• Which integration scenarios are used infrequently and have low volume? Is there a need to revise the integration design or address underlying issues?
• How reliable is the external system’s availability? Are the maintenance windows of the external system in line with our expectations and their announced schedules? Do these maintenance periods impact our integration processes?
• What is the level of quality of service provided by our integration solution? Are we overlooking crucial information by relying solely on SAP PO’s standard Message Monitor?
• How can we evaluate these questions for any historical period? Is it possible to make comparisons between past and present data?

GoGoXi Architecture

GoGoXi Architecture overview

GoGoXi solution includes the following components:

• GoGoXi exporter tool

Custom tool that exports XI message headers, User-Defined Search attribute values and Auditlog entries in JSON format. Details are available below.

Exporter tool should be started on the daily or weekly basis. Number of times it should be started depends on persistence settings of SAP PO and requirements on data freshness in ELK stack.

• XI messages and Audit log entries

Specialized files exported by GoGoXi exporter tool. File format description is shown below.

• NWA log files

NWA logs as seen downloaded from NWA Log Viewer (List Log Format 2.0). NWA log files should be downloaded manually from NWA Log Viewer. For our purposes we used the following group of log records:

• • SAP logs
  • Developer traces
  • HTTP logs
  • Change logs

Logs should be downloaded periodically so that there will be no gaps in files, meaning before any unprocessed old data is overwritten by newer log data.

• Beats

Elastic component which is highly optimized for parallel incremental load of text files. It monitors folder and sends files it finds line-by-line to Logstash for processing. Documentation is available at official website.

• Logstash

Elastic component which is designed for advanced data processing for events coming from variety of sources and passing to receiver system for storage. Documentation is available at official website.

This component performs the following tasks in relation to GoGoXi:

• • parsing of NWA log files and XI / Auditlog export files
  • auditlog entries augmentation based on known events
  • database segment (index) determination
  • data deduplication based on event hash calculation

• Elasticsearch

Non-relational database management system (DBMS). Performs long-term data storage and data indexing. Documenation is available at official website.

• Kibana

Component is responsible for data visualization and DBMS management. Documenation is available at official website.

GoGoXi exporter tool architecture

The Diagram below describes internal working of the exporter tool and its ability to scale up.

GoGoXi exporter tool inner working

Once tool is started with specific configuration it goes through the list of PO servers as defined in configuration file. Number of threads (by default — 2) is started to extract “Overview” data from SAP PO (contains summary for specific period (DAY, WEEK, MONTH or YEAR) grouped by sender and receiver. This information is passed further down the pipeline.

Number of additional threads (by default — 5) is started to extract XI message headers from SAP PO using “Message Search” call passing sender, receiver and date period from previous step. Threads handle correctly cases when there are more than 10000 messages for a given time period. Xi message headers are passed further down the pipeline.

Number of additional threads (by default — 5) is started to extract User-Defined Search attributes (if there are any) for each XI message header. Extracted UDS attribute values are appended to XI mesage header. Complete XI message header is then saved in Xi Messages file as JSON. Also Xi message header is passed further down the pipeline.

Number of additional threads (by default — 5) is started to extract audit log entries for each XI message header. These entries are saved to Audit log file as JSON.

After all of the above threads finish their work task the tool moves to next SAP PO server. If last SAP PO server is processed, GoGoXi exporter tool terminates.

Export file formats

Tool exports XI Messages and Auditlog in the following format. All files are newline-delimited JSON files. One line represents one XI message or one audit log entry.

An example of XI message is as follows:

{
    "ximessage": {
        "cancelable": false,
        "connectionName": "SOAP_http://sap.com/xi/XI/System",
        "direction": "OUTBOUND",
        "editable": false,
        "endTime": "2023-11-08T11:09:13.377+03:00",
        "endpoint": "<local>",
        "headers": {
            "original": "content-length=5450\nhttp=POST\ncontent-type=multipart/related; boundary=SAP_11271d92-7e0e-11ee-9105-00000c9d89ea_END; type=\"text/xml\"; start=\"<[email protected]>\"\n",
            "contentLength": 5450
        },
        "interface": {
            "name": "PurchaseRequest_Out",
            "namespace": "urn:company:Purchase"
        },
        "isPersistent": true,
        "messageID": "e1f49308-b5d9-1eee-9fc1-c22063df6044",
        "messageKey": "e1f49308-b5d9-1eee-9fc1-c22063df6044\\OUTBOUND\\878415850\\EO\\0\\",
        "messageType": "Send",
        "nodeId": "878415850",
        "persistUntil": "2024-05-06T11:08:57.744+03:00",
        "protocol": "XI",
        "qualityOfService": "EO",
        "receiverName": "EXTERNAL_p",
        "receiverParty": {
            "agency": "http://sap.com/xi/XI",
            "name": "",
            "schema": "XIParty"
        },
        "referenceID": "",
        "restartable": false,
        "retries": 10,
        "retryInterval": 300,
        "scheduleTime": "2023-11-08T11:08:57.748+03:00",
        "senderName": "ERP_S4HANA_P",
        "senderParty": {
            "agency": "http://sap.com/xi/XI",
            "name": "",
            "schema": "XIParty"
        },
        "sequenceNumber": 0,
        "serializationContext": "",
        "serviceDefinition": "",
        "softwareComponent": "",
        "startTime": "2023-11-08T11:08:57.744+03:00",
        "status": "success",
        "timesFailed": 0,
        "transport": "Loopback",
        "version": "0",
        "wasEdited": false,
        "scenarioIdentifier": "dir://ICO/f10d5af248d83dbe9ac62b827e8e223a",
        "parentID": "",
        "duration": 15.633,
        "size": 5450,
        "messagePriority": 1,
        "rootID": "",
        "sequenceID": "",
        "passportTID": "e78c57187e0d11eeb07100000c9d89ea",
        "logLocations": [
            "Receiver JSON Request",
            "AM"
        ],
        "UDS": {
            "RequestNumber": [
                "0048588891"
            ]
        },
        "UDSKeys": [
            "RequestNumber"
        ]
    },
    "gogoxi": {
        "component": "af.pop.sappop",
        "extractedOn": "2023-11-08T12:02:25.7824767+03:00"
    }
}

An example of Audit log is as follows:

{
    "ximessage": {
        "cancelable": false,
        "connectionName": "SOAP_http://sap.com/xi/XI/System",
        "direction": "OUTBOUND",
        "editable": false,
        "endTime": "2023-11-08T11:09:13.377+03:00",
        "endpoint": "<local>",
        "headers": {
            "original": "content-length=5450\nhttp=POST\ncontent-type=multipart/related; boundary=SAP_11271d92-7e0e-11ee-9105-00000c9d89ea_END; type=\"text/xml\"; start=\"<[email protected]>\"\n",
            "contentLength": 5450
        },
        "interface": {
            "name": "PurchaseRequest_Out",
            "namespace": "urn:company:Purchase"
        },
        "isPersistent": true,
        "messageID": "e1f49308-b5d9-1eee-9fc1-c22063df6044",
        "messageKey": "e1f49308-b5d9-1eee-9fc1-c22063df6044\\OUTBOUND\\878415850\\EO\\0\\",
        "messageType": "Send",
        "nodeId": "878415850",
        "persistUntil": "2024-05-06T11:08:57.744+03:00",
        "protocol": "XI",
        "qualityOfService": "EO",
        "receiverName": "EXTERNAL_p",
        "receiverParty": {
            "agency": "http://sap.com/xi/XI",
            "name": "",
            "schema": "XIParty"
        },
        "referenceID": "",
        "restartable": false,
        "retries": 10,
        "retryInterval": 300,
        "scheduleTime": "2023-11-08T11:08:57.748+03:00",
        "senderName": "ERP_S4HANA_P",
        "senderParty": {
            "agency": "http://sap.com/xi/XI",
            "name": "",
            "schema": "XIParty"
        },
        "sequenceNumber": 0,
        "serializationContext": "",
        "serviceDefinition": "",
        "softwareComponent": "",
        "startTime": "2023-11-08T11:08:57.744+03:00",
        "status": "success",
        "timesFailed": 0,
        "transport": "Loopback",
        "version": "0",
        "wasEdited": false,
        "scenarioIdentifier": "dir://ICO/f10d5af248d83dbe9ac62b827e8e223a",
        "parentID": "",
        "duration": 15.633,
        "size": 5450,
        "messagePriority": 1,
        "rootID": "",
        "sequenceID": "",
        "passportTID": "e78c57187e0d11eeb07100000c9d89ea",
        "logLocations": [
            "Receiver JSON Request",
            "AM"
        ],
        "UDS": {
            "RequestNumber": [
                "0048588891"
            ]
        },
        "UDSKeys": [
            "RequestNumber"
        ]
    },
    "auditlog": {
        "messageID": "e1f49308-b5d9-1eee-9fc1-c22063df6044",
        "timestamp": "2023-11-08T11:09:35.869+03:00",
        "status": "SUCCESS",
        "localizedText": "Message status set to DLNG",
        "textKey": "STATUS_SET_SUCCESS",
        "textKeyParams": {
            "0": "DLNG"
        }
    },
    "gogoxi": {
        "component": "af.pop.sappop",
        "extractedOn": "2023-11-08T12:02:25.7824767+03:00"
    }
}

Note that “ximessage” substructure is identical in both structures. This allows you to filter and visualize audit logs based on XI message attributes like source \ destination system or payload size. Some remarks apply to export format, please read details in GitHub repository (link at the end).

Kibana Dashboards

Here are examples of dashboards that we have developed on the SAP PO exported data. These dashboards and their respective panels are used all the time by maintenance team.

XI Quality of Service

Data filter options and quality indicators

Messages grouped by duration time buckets (anything longer than 1 minute is a reason for concern)

Manual message restarts (indicates amount of manual effort required to support integration scenarios)

Amount of storage space required on SAP PO side (accounts for internal message compression) grouped by sender

XI REST Receiver adapter audit log

REST responses grouped by response text (requires traceResponse option in REST adapter)

Number of REST requests grouped by HTTP status code

Discoveries we made using GoGoXi

Here are examples of the discoveries we made while using GoGoXi.

1. Multiple retries before successful delivery

Retry attempts grouped by receiver system

We found one system which requires many restart attempts to deliver messages to it. In the end the issue was attributed to way OAuth tokens were issued by receiver system.

2. Incorrect “Successful” message status

REST adapter responses grouped by response text (text above is etiher JSON-encoded or incorrectly handled as Latin-1)

We quickly found large number of asynchronous messages which were marked as “Processed successfully” but in fact were rejected by the receiver. The issue was attributed to HTTP 200 status code which the receiver system returned even if error occured. We added JSON response check in REST Receiver adapter to mitigate receiver’s behavior.

3. Message cannot be delivered to SAP PO

Incoming HTTP Requests grouped by status codes

HTTP 500 requests — zoomed in

Every five minutes external system tried to push one document to SAP PO with no luck for more than two weeks. Problem was attributed to error in SAP code to parse incoming JSON value. SAP PO was not able to receive JSON since it was considerd as non-valid JSON. Message was finally captured with XPI Inspector for investigation. In the end we added REST adapter option escapeNonXML10Symbols to fix the issue.

4. Incomplete scenario configuration

Auditlog entries marked as ERROR

We are using GetPayloadValueBean and PutPayloadValueBean adapter modules for some of the integration scenarios. Scenarios were working however there are mistakes in its configuration. These mistakes are clearly labeled in audit logs. Configuration was updated, and integration scenario was fixed.

Outro

With the introduction of the GoGoXi tool, we have enhanced our ability to monitor the intricate SAP PO system landscape. This enhancement enables us to identify potential problems in both business and technical integration scenarios before they escalate into significant incidents. Consequently, we can implement corrective and compensatory actions proactively. Ultimately, this approach helps us prevent needless escalations, avoid hastily deploying fixes in the production environment, and fosters an improvement in the overall quality and transparency of integration solution.

Useful links

GoGoXi exporter GitHub repository: https://github.com/platinumICE/GoGoXi

PIMON-2023 GoGoXi webinar (in Russian language with autogenerated subtitles, recorded on September 2023):