This blogpost is to showcase how to implement secure mutual TLS (mTLS) communications with SAP BTP Kyma runtime environment with:

• SAP Kyma mTLS gateways (with managed istio and api-gateway modules) and
• SAP BTP destinations (with SAP IAS generated client certificates) to describe the mTLS targets

Quoting after CloudFare:

Mutual TLS, or mTLS for short, is a method for mutual authentication. mTLS ensures that the parties at each end of a network connection are who they claim to be by verifying that they both have the correct private key.

Thus, assuming, your SAP Kyma workloads are exposed to the internet via a dedicated VirtualService and that you have defined an appropriate AuthorizationPolicy to have the requests authorized against a client’s certificate DN (Distinguished Name).

Let’s see how to simplify the implementation burden of a client certificate authentication (mTLS) by leveraging SAP BTP destinations with SAP IAS generated certificates.

1. Configure SAP BTP destinations for x509 authentication

SAP BTP destinations a are a convenient way of describing target systems.

1.1 Generate a certificate keystore for mTLS authentication

The idea is to use SAP IAS to generate certificates to implement either or both:

• client certificate authentication with kyma workloads
• supported SAP IAS OAuth flows over mTLS (namely, using an x509 certificate instead of a secret to authenticate against the SAP IAS authentication server)

Let’s use the SAP IAS client certificate configuration screen and generate a SAP Public-Key Infrastructure (PKI) Cloud Root CA-signed certificate as follows:

Please make sure you can remember the password used to protect the generated certificate!

1.1.1 Look up the certificate keystore with openssl

Let’s have a closer look at the generated certificate using openssl commands:

$ openssl pkcs12 -info -in poster-quovadis.p12 -nodes
Enter Import Password:
MAC Iteration 10000
MAC verified OK

The generated keystore contains a private key, one public x509 certificate valid for one year and a cacert bundle composed of both intermediate and root CA certificates

PKCS7 Data
Shrouded Keybag: Bag Attributes
    friendlyName: 1
    localKeyID: 54 69 6D 65 20 31 37 30 32 39 33 34 39 35 36 31 37 34 
Key Attributes: <No Attributes>
-----BEGIN PRIVATE KEY-----
MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQCdwV7Hy/ujSGTe
2wO03ox7EuxZ0DHufAYHPfI=
-----END PRIVATE KEY-----

.......... the public x509 certificate valid one year
PKCS7 Encrypted data: Certificate bag
Bag Attributes
    friendlyName: 1
    localKeyID: 54 69 6D 65 20 31 37 30 32 39 33 34 39 35 36 31 37 34 
subject=/C=DE/O=SAP SE/OU=SAP Cloud Platform Clients/OU=Canary/OU=8e1affb2-62a1-43cc-a687-**********/L=********.accounts400.ondemand.com/CN=poster-quovadis (P000000)
issuer=/C=DE/L=EU10-Canary/O=SAP SE/OU=SAP Cloud Platform Clients/CN=SAP Cloud Platform Client CA
-----BEGIN CERTIFICATE-----
MIIF8TCCA9mgAwIBAgIQSmSVJYGM/5ZswXFF/ku+rjANBgkqhkiG9w0BAQsFADCB
tLi9rwrGaLvswUwRd0O8v98b4dyeZ13PNVVCKrxMC433+9jI8A==
-----END CERTIFICATE-----

... both intermediate (client's x509 issuer) and root CA (cacert bundle) ..............

Certificate bag
Bag Attributes
    friendlyName: CN=SAP Cloud Platform Client CA,OU=SAP Cloud Platform Clients,O=SAP SE,L=EU10-Canary,C=DE
subject=/C=DE/L=EU10-Canary/O=SAP SE/OU=SAP Cloud Platform Clients/CN=SAP Cloud Platform Client CA
issuer=/C=DE/L=Walldorf/O=SAP SE/CN=SAP Cloud Root CA
-----BEGIN CERTIFICATE-----
MIIGaDCCBFCgAwIBAgITcAAAAAWaX7qDX+136AAAAAAABTANBgkqhkiG9w0BAQsF
2JX4S0blcGoU+np5
-----END CERTIFICATE-----
Certificate bag
Bag Attributes
    friendlyName: CN=SAP Cloud Root CA,O=SAP SE,L=Walldorf,C=DE
subject=/C=DE/L=Walldorf/O=SAP SE/CN=SAP Cloud Root CA
issuer=/C=DE/L=Walldorf/O=SAP SE/CN=SAP Cloud Root CA
-----BEGIN CERTIFICATE-----
MIIFZjCCA06gAwIBAgIQGHcPvmUGa79M6pM42bGFYjANBgkqhkiG9w0BAQsFADBN
LvHPhNDM3rMsLu06agF4JTbO8ANYtWQTx0PVrZKJu+8fcIaUp7MVBIVZ
-----END CERTIFICATE-----

Good to know:

• There is another way to generate a certificate keystore for the purpose of the mTLS communication, namely using the destination service.
• Otherwise, one can use their own certificate provider

1.2 Upload the generated certificate keystore into the destination service

We can upload the generated client certificate keystore directly into the destination service vault on a BTP sub-account level as depicted below:

1.3 Create a ClientCertificateAuthentication destination definition

Let’s create a ClientCertificateAuthentication destination definition, as follows:

The above destination will yield a client certificate with both private key and x509 certificate in its payload.

"owner": {
    "SubaccountId": "",
    "InstanceId": null
  },
  "destinationConfiguration": {
    "Name": "httpbin-x509",
    "Type": "HTTP",
    "URL": "https://httpbin-x509.mtls.quovadis-******.com",
    "Authentication": "ClientCertificateAuthentication",
    "ProxyType": "Internet",
    "KeyStorePassword": "KeyStorePassword",
    "HTML5.DynamicDestination": "true",
    "KeyStoreLocation": "poster-quovadis.p12"
  },
  "certificates": [
    {
      "Name": "poster-quovadis.p12",
      "Content": "MIIaegIBAzCCGiQGCSqGSIb3DQEHAaCCGhUEghoRMIIaDTCCBaQGCSqGSIb3DQEHAaCCBZUEggWRMIIFjTCCBYkGCyqGSIb3DQEMCgECoIIFQDCCBTwwZuQ18z/yXxxg/LaekF8VFSXHdoKLz1uCvYTn+Ia/DWBBT6D0wdVDSYiqfypLkToCBxoyXYngICJxA=",
      "Type": "CERTIFICATE"
    }
  ]
}

Good to know:

• When using the above destination with either SAP approuter or if using SAP Cloud SDK (direclty or with CAP), the ClientCertificateAuthentication destination payload will be automatically transferred to the backend server for validation against the cacert bundle

2. Expose and secure kyma workloads with client certificates

The following kyma tutorial details a number of pre-requisite steps, namely:

• have a dedicated custom domain for mTLS communication with a mutual TLS gateway
• create a cacert bundle for the server side certificate validation.

Let’s see how this can be done with both a kyma cluster custom domain and your own custom business domains.

2.1 with kyma-system defined mTLS gateways for a kyma cluster domain

As aforementioned, any managed kyma cluster domain is also a custom domain (managed by SAP).

Thus, it is possible to make use of this kyma cluster domain “as is” by adding a dedicated mutual TLS gateway, as described in the following gist:

• Expose and secure kyma workloads with client certificates and kyma-system defined mTLS gateways | gist

2.2. with mTLS gateways for custom business domains

However, the above approach with a kyma cluster domain has two main caveats, namely:

• no wildcard domains
• no business domains

Thus, If you wanted to have proper business domains and/or wildcard custom domains with mTLS gateways, please follow the steps described in this gist:

• mTLS gateways with custom business domains | gist

2.2.1 Example with a pod and a function in the same namespace

This scenario may be of interest as it showcases how to manage mTLS communications with the workloads of different nature, namely deployments and functions, deployed in one single namespace.

• mTLS gateway for a wildcard custom domain

• Virtual Services
  • httpbin
  • telldus

• AuthorizationPolicy with the PolicyTargetReference as a target resource selector

1. Smoke test using curl command

In order to be able to use the curl command we need to retrieve both the private key and the public x509 certificate from the keystore, as follows:

$ openssl pkcs12 -in poster-quovadis.p12 -out poster-quovadis.key -nodes -nocerts
Enter Import Password:
MAC verified OK

$ openssl pkcs12 -in poster-quovadis.p12 -out poster-quovadis.crt -nokeys
Enter Import Password:
MAC verified OK

Eventually we can run the curl command. Please notice the -ik options are not used in the below smoke test

$ curl --key poster-quovadis.key --cert poster-quovadis.crt   https://httpbin-x509.mtls.quovadis-******.com/headers
{
  "headers": {
    "Accept": "*/*", 
    "Host": "httpbin-x509.mtls.quovadis-*****.com", 
    "Test": "true", 
    "User-Agent": "curl/8.4.0", 
    "X-Client-Ssl-Cn": "CN=poster-quovadis (P000000),L=*****.accounts400.ondemand.com,OU=8e1affb2-62a1-43cc-a687-*********,OU=Canary,OU=SAP Cloud Platform Clients,O=SAP SE,C=DE", 
    "X-Client-Ssl-Issuer": "CN=SAP Cloud Platform Client CA,OU=SAP Cloud Platform Clients,O=SAP SE,L=EU10-Canary,C=DE", 
    "X-Envoy-Attempt-Count": "1", 
    "X-Envoy-Expected-Rq-Timeout-Ms": "300000", 
    "X-Envoy-Internal": "true", 
    "X-Forwarded-Client-Cert": "Hash=8c5007a39120e597707e4ed8cd2ee34b294*********************;Cert=\"-----BEGIN%20CERTIFICATE-----%0AMIIF8TCCA9mgAwIBAgIQSmSVJYGM%2F5ZswXFF%2Fku%2BrjANBgkqhkiG9w0BAQsFADCB%0AgDELMAkGA1UEBhMCREUxFPb04Nzc7u%2FW%0AtLi9rwrGaLvswUwRd0O8v98b4dyeZ13PNVVCKrxMC433%2B9jI8A%3D%3D%0A-----END%20CERTIFICATE-----%0A\";Subject=\"CN=poster-quovadis (P000000),L=******.accounts400.ondemand.com,OU=8e1affb2-62a1-43cc-a687-*********,OU=Canary,OU=SAP Cloud Platform Clients,O=SAP SE,C=DE\";URI=,By=spiffe://cluster.local/ns/quovadis/sa/httpbin;Hash=cd11387d4fcfb6d9d3f1b864dc99a7662cce8f8ae94cd73c30e503ae791f998c;Subject=\"\";URI=spiffe://cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account", 
    "X-Forwarded-Host": "httpbin-x509.mtls.quovadis-******.com"
  }
}

2. Smoke test using ClientCertificateAuthentication destination

2.1 with the managed approuter

https://<tenant>.launchpad.cfapps.<region>.hana.ondemand.com/dynamic_dest/httpbin-x509/headers

{
  "headers": {
    "Dnt": "1", 
    "Host": "httpbin-x509.mtls.quovadis-********.com", 
    "Referer": "https://***.authentication.eu12.hana.ondemand.com/", 
    "Tenant-Host-Pattern": "(^(.*)(?=-sapdelim-([^\\.]*)(\\.m)?\\.launchpad\\.cfapps\\.eu12\\.hana\\.ondemand\\.com)|^([^\\.]*)(?=(\\.m)?\\.launchpad\\.cfapps\\.eu12\\.hana\\.ondemand\\.com)|^(.*)(?=-sapdelim-([^\\.]*)(\\.m)?\\.eu12\\.start\\.cloud\\.sap)|^([^\\.]*)(?=(\\.m)?\\.eu12\\.start\\.cloud\\.sap))", 
    "Test": "true", 
    "X-Client-Ssl-Cn": "CN=poster-quovadis (P000000),L=***.accounts400.ondemand.com,OU=8e1affb2-62a1-43cc-a687-***,OU=Canary,OU=SAP Cloud Platform Clients,O=SAP SE,C=DE", 
    "X-Client-Ssl-Issuer": "CN=SAP Cloud Platform Client CA,OU=SAP Cloud Platform Clients,O=SAP SE,L=EU10-Canary,C=DE", 
    "X-Envoy-Attempt-Count": "1", 
    "X-Envoy-External-Address": "10.****.0.***", 
    "X-Forwarded-Client-Cert": "Hash=8c5007a39120e597707e4ed8cd2ee34b2940e0a21d046ec6b0c1687c4d4741ac;Cert=\"-----BEGIN%20CERTIFICATE-----%0AMIIF8TCCA9mgAwIBAgIQSmSVJYGM%2F5ZswXFF%2Fku%2BrjANBgkqhkiG9w0BAQsFADCB%0AgDELMAkGA1UEBhMCREUxFEND%20CERTIFICATE-----%0A\";Subject=\"CN=poster-quovadis (P000000),L=***.accounts400.ondemand.com,OU=8e1affb2-62a1-43cc-a687-****,OU=Canary,OU=SAP Cloud Platform Clients,O=SAP SE,C=DE\";URI=,By=spiffe://cluster.local/ns/quovadis/sa/httpbin;Hash=4720ba0dec622918487d9f09b49c7f62e0d0d476b8170c5eefa1e059dae7c8c9;Subject=\"\";URI=spiffe://cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account", 
    "X-Forwarded-Host": "httpbin-x509.mtls.quovadis-********.com", 
    "X-Forwarded-Path": "/dynamic_dest/httpbin-x509/headers", 
  }
}

2.2 with the multi-tenant approuter deployed to a kyma cluster

Just include the following route in the approuter’s xs-app.json, as depicted below:

{
    "source": "^/dynamic_dest/([^/]+)/(.*)$",
    "target": "$2",
    "authenticationType": "xsuaa",
    "preferLocal": true,
    "destination": "$1"
}

and then you can invoke your destination similar to the managed approuter’s dynamic_dest construct, namely:

<your multi-tennt approuter api rule>/dynamic_dest/<destination name>/<path>

for instance:
<your multi-tennt approuter api rule>/dynamic_dest/httpbin-x509/headers

Good to know:

• Given the approuter multi-tenancy, the destination definition could be at a provider or at any subscriber as either subaccount or instance level destination definition

3 with SAPI API Management API provider and proxy

This involves first uploading the keystore to your APIM tenant and creating an mTLS API Provider.

Next step is to create an API proxy and protect it with OAuth policy as well.

Additional resources

• S/4HANA Cloud APIs with ClientCertificateAuthentication. | SAP Blogs
• Export Certificates and Private Key from a PKCS#12 File with OpenSSL
• What is mutual TLS (mTLS)?