“Elevate Your SAP Experience: A Guide to Maintenance Without Interruption for SAP Applications”

In various instances, the necessity arises to conduct meticulous maintenance on the SAP HANA database, whether it be for operating system enhancements or database updates, among other considerations. It is imperative that such maintenance activities align seamlessly with your SAP systems, especially when implemented in conjunction with a High Availability/Disaster Recovery (HA/DR) solution.

Let’s contemplate a scenario wherein your operational framework aligns with a three-tier landscape configuration, delineated as follows –

HANA DB Architecture

Site1 – Primary (Running on Azure Zone 1)

Site2 – Secondary (Running on Azure Zone 1)

Site3 – DR(Disaster Recovery running on Azure Zone 2)

HANA DB – SAP HANA SPS06 DB including XSA (RHEL 8x)

Below 2 parameters are active in DB –

[inifile_checker]/enable = true

[inifile_checker]/replicate = true

XSA Configuration –

Default Domain – Hostname<Site1>

API_URL – Hostname<Site1>:port

Replication Scenario –

Site1 >>> SYNC Replication >>> Site2

Site2 >>> ASYNC Replication >>> Site3

You have completed HSR setup as per below URL’s and it’s successfully tested as well –

https://learn.microsoft.com/en-us/azure/sap/workloads/sap-hana-high-availability-rhel

https://learn.microsoft.com/en-us/azure/sap/workloads/disaster-recovery-sap-hana

During approved maintenance, you need to handover the VM’s to OS team for patching and reboot.

Assumptions and Prerequisites for SAP HANA Database Maintenance –

In order to mitigate the risk of a Single Point of Failure, it is assumed that at all times, two database (DB) servers should maintain an active replication status. This strategic redundancy ensures a robust and fault-tolerant system architecture.

Before embarking on the maintenance journey, certain prerequisites must be diligently addressed:

1. Replication Vigilance:
– It is essential to consistently validate the replication status within HANA Studio and at the operating system level. This verification process ensures that the replication mechanism is not only active but also performing optimally.

2. WMM Configuration in Solution Manager:
– To preemptively counter potential alerts, it is imperative to configure Work Mode Management (WMM) in the Solution Manager. This proactive measure aids in maintaining a harmonious operational environment and minimizes disruptions during the maintenance phase.

3. Mitigating External Monitoring Alerts:
– To streamline the maintenance process further, consider suppressing alerts from third-party monitoring tools. This precautionary step prevents unnecessary interruptions, allowing for a focused and unencumbered maintenance operation.

Execution Steps Sequence :-

HANA DB DR(Site3) Server –

• Shutdown DR system via HDB stop command
• Handover VM to OS team for patching and rebooting
• Post reboot, change the replication status from Site2 >> Site3 to Site1 >> Site3(ASYNC)

hdbnsutil -sr_register --name=Site3 --remoteHost=<Site1 host> --remoteInstance=<Site1 instance nu> --replicationMode=async --operationMode=logreplay

• Navigate to custom config directory (/usr/sap/<SID>/SYS/global/hdb/custom/config) and edit global.ini file to remove the hooks before starting the HANA DB
• Mark # for hooks in global.ini file

#[ha_dr_provider_SAPHanaSR]
#path = /hana/shared/myHooks
#execution_order = 1
#provider = SAPHanaSR

• HDB Start and validate process and replication status in HANA Studio/OS level

sapcontrol -nr <instance num> -function GetProcessList

• DR should be connected with Site1(Primary Instance)

New Replication Scenario –

Site1 >>> SYNC Replication >>> Site2

Site1 >>> ASYNC Replication >>> Site3

HANA Secondary (Site2) Server –

• Set the cluster maintenance mode – TRUE to avoid cluster fail issue

          pcs property set maintenance–mode=true

• Shutdown Secondary(Site2) system via HDB stop command
• Handover VM to OS team for patching and rebooting
• HDB Start and validate process and replication status in HANA Studio/OS level

sapcontrol -nr <instance num> -function GetProcessList

• Secondary(Site2) should be connected with Site1(Primary Instance)
• Remove maintenance mode, cluster should be OK, is there any error clear the resource

          pcs property set maintenance–mode=false

No Change in Replication Scenario –

Site1 >>> SYNC Replication >>> Site2

Site1 >>> ASYNC Replication >>> Site3

Now we are moving to Primary DB restart hence it’s requires to change the replication setup of DR system from Site1 to Site2 to keep 2 systems replication active.

This step is optional, as per requirement – 

• Shutdown DR system via HDB stop command
• Change the replication status from Site1 >> Site3 to Site2 >> Site3(ASYNC)

hdbnsutil -sr_register --name=Site3 --remoteHost=<Site2 host> --remoteInstance=<Site2 instance nu> --replicationMode=async --operationMode=logreplay

• Navigate to custom config directory (/usr/sap/<SID>/SYS/global/hdb/custom/config) and edit global.ini file to remove the hooks before starting the HANA DB
• Mark # for hooks in global.ini file

#[ha_dr_provider_SAPHanaSR]
#path = /hana/shared/myHooks
#execution_order = 1
#provider = SAPHanaSR

• HDB Start and validate process and replication status in HANA Studio/OS level

sapcontrol -nr <instance num> -function GetProcessList

• DR should be connected with Site2(Secondary Instance)

New Replication Scenario –

Site1 >>> SYNC Replication >>> Site2

Site2 >>> ASYNC Replication >>> Site3

HANA Primary (Site1) Server –

• Move cluster resource from Primary to Secondary (Site2 should be Primary Now)

          pcs resource move <resource_name> <node_name>

• Validate SAP system should be available and all user sessions are established
• Validate replication status in HANA Studio/OS level
• Validate the pacemaker cluster status

pcs cluster status (if secondary resource is in stopped status, keep in stopped status)

• Set the cluster maintenance mode – TRUE and remove location constraint from cluster config file (location constraint created due to manual resource move hence remove manually or else resource will not be moved automatically during actual switchover/failover situation)

          pcs property set maintenance–mode=true

• Shutdown Secondary(Site1) system via HDB stop command
• Handover VM to OS team for patching and rebooting
• Post reboot (Site1), re register HSR to Site1 >> Site2(SYNC)

hdbnsutil -sr_register --name=Site1 --remoteHost=<Site2 host> --remoteInstance=<Site2 instance nu> --replicationMode=sync --operationMode=logreplay

• HDB Start and validate process and replication status in HANA Studio/OS level

sapcontrol -nr <instance num> -function GetProcessList

• Secondary(Site1) should be connected with Site2(Primary Instance)
• Remove maintenance mode, cluster should be OK, is there any error clear the resource

          pcs property set maintenance–mode=false

          New Replication Scenario –

Primary Site2 >>> SYNC Replication >>> Site1 Secondary

Primary Site2 >>> ASYNC Replication >>> Site3 DR

You can keep the Primary Site2 and change the DR replication from Site2 to Site1 as per above given DR replication steps or else changing the Primary Site from Site2 to Site1 will follow all the switchover steps.

Troubleshooting –

1. XS controller service is not starting up post primary switchover due to old host default domain and api_url (as per configuration)

To fix this situation, follow the below steps – (this will not stop the INDEXSERVER service on primary system)

• Stop the XSA service via <sid>adm user in primary system

XSA disable

• Set the cluster maintenance mode TRUE
• Stop the HANA DB on secondary and DR system
• Unregister the secondary and DR system

hdbnsutil –sr_unregister

• Disable system replication on the primary system

hdbnsutil –sr_disable

• Navigate to location on primary system/usr/sap/<SID>/SYS/global/hdb/custom/config and edit the xscontroller.ini

Change the Default Domain and API_URL to local hostname

• Start the XSA service via <sid>adm user in primary system and validate the XSA service to get start successfully

XSA enable

• Post start of XSA, enable the replication in primary system

hdbnsutil -sr_enable

• Reconnect the secondary and DR system as per replication scenario
• Validate replication status in HANA Studio/OS level
• Set the cluster maintenance mode FALSE

2. Error while registering DR from Standby/Primary Server

Error –

INFO: no copy check for systemPKI SSFS files as primary does not support this (older version than secondary) or primary has deactivated it explicitly
got request for system replication site registration, but system replication is not enabled.

To fix this situation, follow the below steps –

• Enable the replication on Site1/Site2 from DR server need to connect and re register DR system

hdbnsutil -sr_enable –name=Site2 or Site1

References –

2964676 – How to disable XSA (SAP HANA XS Advanced)

https://help.sap.com/docs/SAP_HANA_PLATFORM/2c1988d620e04368aa4103bf26f17727/73596bf87326455e8f2c10b83580d91b.html

https://blogs.sap.com/2020/05/27/how-to-change-sap-hana-cockpit-url-from-physical-hostname-to-a-dns-alias-name/