Configure Disaster Recovery for Omnissa Access

Omnissa Access supports disaster recovery using a backup-and-restore model with a shared NFS mount. In this model, a secondary cluster has access to backup data produced by the primary cluster. Upon a declared DR event, the secondary cluster is manually promoted to primary and handles read/write traffic. Failback to the original site is a planned manual operation; it does not happen automatically. When the original primary becomes available, it remains in the secondary role until planned failback is executed.

This topic describes how to configure NFS backup storage, manage backup schedules, back up required bootstrap node data, and restore Omnissa Access from backup during a failover.

---

Prerequisites

Before configuring disaster recovery, ensure that the following conditions are met:

• A replicated NFS share is configured and network-accessible to both the primary cluster nodes and the secondary cluster nodes.
• The secondary cluster will be deployed using the same asset bundle version as the source cluster.
• You have administrative access to the bootstrap node of the primary cluster to back up secrets and configuration files.

---

Configure NFS Exports on the NFS Server

Because backup services run as root, the NFS export must include the no_root_squash option. On the NFS server, verify the export configuration in /etc/exports:

cat /etc/exports

The output should include an entry in the following format:

<NFS-MOUNT-POINT> <NFS-CLIENT-IP>(rw,sync,no_subtree_check,no_root_squash)

<NFS-CLIENT-IP> is the IP range or set of IPs that covers all cluster nodes permitted to access the mount. For example, if the source cluster nodes use addresses in the range 10.48.34.100–10.48.34.105, set <NFS-CLIENT-IP> to 10.48.34.0/24 to cover the full subnet. When the secondary cluster is promoted to primary during a failover, its node addresses must also fall within the permitted IP range.

---

Configure profile.yml for NFS Backup

The source cluster must be configured to write backups to a shared NFS mount before services are deployed. These backups are used to restore the secondary cluster during a failover.

1. On the bootstrap node of the source cluster, open /opt/wss/<cluster_name>/profile.yml.

2. Locate the following lines, populate them with your NFS server details, and uncomment them:

   nfs_host: <NFS-HOST-IP>
   nfs_path: <NFS-MOUNT-POINT>
   nfs_version: 4
   

3. Save the file, then proceed with deployment — either using the guided Phase 3, Option 1 - Deploy Omnissa Access Using the Access Wizard (which also asks an NFS shared storage question in Section 6, Optional Services) or manually per Phase 3, Option 2 - Deploy Omnissa Access Manually.

   Note: If NFS details were not included in profile.yml before the initial service deployment, add them afterward and redeploy the backup-dependent services:

     > ```bash
     > wso services deploy -s postgres
     > wso services deploy -s control-plane-backup
     > wso services deploy -s opensearch
     > ```
   

4. To confirm that the NFS mount is accessible from another machine on secondary cluster, run the following commands from a separate virtual machine:

   # Mount the NFS share manually to test connectivity
   mount -t nfs <NFS-IP>:<NFS-MOUNT-POINT> /mnt
   
   # List the NFS exports visible to the client
   showmount -e <NFS-IP>
   

   The output of showmount -e should include an entry for the configured mount point:

   <NFS-MOUNT-POINT> <NFS-IP>
   

---

Back Up Bootstrap Node Data

The following items from the primary bootstrap node must be available on the secondary bootstrap node before a restore can proceed. Back up these items to a secure location before a DR event occurs.

Omnissa recommends storing this data on the same NFS server used for service backups, but on a separate NFS mount, to avoid any interference with automated backup operations.

---

Configure Backup Frequency and Retention

You can control how frequently backups are taken and how long they are retained by setting environment variables in the additional_env_vars.env file on the bootstrap node. These settings determine your recovery point objective (RPO) — the maximum amount of data that could be lost in the event of a failure.

---

Database Backup Types

Omnissa Access uses two complementary backup types for the PostgreSQL database.

Full Backup

A full backup captures a complete snapshot of the entire PostgreSQL data directory at a specific point in time. Full backups are self-contained and can be restored without any additional backup files. By default, a full database backup runs once daily at 5:00 AM UTC and serves as the baseline for delta backups.

Delta Backup

A delta backup captures only the data pages that changed since the most recent full backup. Delta backups are smaller and faster to complete than full backups, but they are not self-contained — restoring from a delta backup requires both the most recent full backup and the delta backup. By default, a delta backup runs once daily at 5:00 PM UTC.

Together, the full and delta backup jobs provide complete daily coverage. In a failure that occurs between scheduled backups, restoration uses the most recent full backup with the latest delta applied on top, limiting potential data loss to within approximately 12 hours.

Delta Backup Chain Depth

The maximum number of delta backup layers that can be chained on top of a single full backup is fixed at 2. This limit is enforced internally by WALG_DELTA_MAX_STEPS and is not configurable.

Full Backup (base)
└── Delta 1 (changes since Full Backup)
    └── Delta 2 (changes since Delta 1)  ← maximum chain depth

When the chain reaches the configured maximum, WAL-G automatically promotes the next scheduled delta backup to a new full backup.

You can adjust the schedule for full and delta backups, but you must ensure that no more than two delta backups run between successive full backups. Exceeding this limit is not supported.

---

FQDN and Networking Requirements

To support a failover without requiring changes to end-user or connector configuration, the secondary cluster must meet the following requirements before a DR event occurs:

• TLS certificates on secondary cluster endpoints must be valid for the same fully qualified domain name (FQDN) as the primary cluster. Certificate rotation processes must cover both the primary and secondary sites consistently.
• Firewall rules must permit outbound traffic to both primary and secondary cluster endpoints. Published IP address ranges must be actively maintained for both sites.
• Connector session state must be fully rebuildable on the secondary gateway, with no hard dependency on session storage that exists only on the primary cluster.

---

Restore Omnissa Access from Backup

This procedure describes how to activate the secondary cluster as the new primary during a failover. When the restore is complete, the secondary cluster handles all read and write traffic. The original primary cluster, once it is available again, is treated as the new secondary until a planned failback is performed.

Important: Before beginning this procedure, ensure that the original primary cluster is not reachable from the secondary cluster. If the original cluster is still reachable during restoration, OpenSearch and PostgreSQL might attempt to add nodes from the original cluster to the restored cluster.

Note: Omnissa recommends performing timed restore drills in a non-production environment before deploying disaster recovery in production. This establishes realistic recovery time objective (RTO) estimates based on your actual data volumes.

Prerequisites

Before beginning the restore procedure, confirm:

• The secondary cluster will be deployed using the same asset bundle version as the source cluster, to ensure that all Omnissa Access service images match.
• The bootstrap node data listed in Back Up Bootstrap Node Data is accessible on the secondary bootstrap node.

Procedure

Step 1: Deploy the Secondary Cluster

Provision new virtual machines and deploy the control plane asset bundle on them, using the same asset bundle version as the source cluster. The secondary cluster must have the same configuration as the source cluster.

Important: In this step, deploy only the CP platform services — Vault, Consul, and Nomad. Do not deploy Omnissa Access services at this stage. Access services are restored later in the procedure.

Before proceeding, back up the directory /opt/wss/<cluster_name>/cp-cluster/secrets/cp-cluster on the secondary bootstrap node.

Estimated time: 30 minutes

---

Step 2: Snapshot the Secondary Cluster Nodes

After the control plane is successfully deployed, take virtual machine snapshots of all secondary cluster nodes. These snapshots provide a known-good restore point if the disaster recovery procedure needs to be restarted.

Note: If you revert to these snapshots at any point during the restore procedure, confirm that Vault is unsealed on the secondary cluster before continuing. Log in to the Vault UI using the appropriate credentials to confirm that the Vault status is unsealed.

---

Step 3: Copy Secrets onto the Secondary Cluster

Copy the tls and vault folders from your backup location to the secondary cluster's backup_secrets directory:

• Copy backed-up tls folder → /opt/wss/<cluster_name>/cp-cluster/backup_secrets/tls
• Copy backed-up vault folder → /opt/wss/<cluster_name>/cp-cluster/backup_secrets/vault

After copying, verify that the backup_secrets directory on the secondary bootstrap node contains both folders:

ls -lrt backup_secrets/
total 0
drwxr-x---. 2 root root 64 May  7 07:46 tls
drwxr-x---. 4 root root 30 May  7 07:46 vault

---

Step 4: Plan the Backup Restore

From the secondary cluster directory, run the following command to review the backup and confirm the restore point:

wso cp plan-restore-backup

To restore from a specific point in time, use the -a flag with a time duration. For example:

This flag sets the RPO for the restore operation.

---

Step 5: Restore Control Plane Components

This step restores Consul key-value data and Vault secrets onto the secondary cluster.

1. Ensure that Vault is unsealed on the secondary cluster.

2. Before running the restore command, confirm that the tls and vault folders are present in backup_secrets on the secondary bootstrap node:

   ls -lrt /opt/wss/<cluster_name>/cp-cluster/backup_secrets/
   

   The output should show both the tls and vault directories:

   drwxr-x---. 2 root root 64 <date> tls
   drwxr-x---. 4 root root 30 <date> vault
   

3. From the cluster directory, run:

   wso cp restore-backup
   

   The command restores Consul key-value data and Vault secrets onto the secondary cluster. Expected output:

   Imports KV in case of disaster recovery
   Backing up control plane credentials
   Restoring vault
   Restoring control plane credentials
   Resetting vault integration tokens
   Resetting vault integration token in nomad
   Resetting vault PKI integration token
   Updated vault with secrets after restore during Disaster recovery
   Adding PKI root and intermediate CA to truststore
   Deploying vault
   Restarting vault servers
   Unsealing Vault
   Enabling consul and nomad integration with vault
   Deploying consul
   Configuring consul connect CA provider
   Restarting the consul leader
   Restarting nomad scheduler agent
   Cleaning up
   Generating environment file with cluster details and secret tokens
   

   Note: If an error occurs during this step, revert all secondary cluster virtual machines to the snapshots taken in Step 2, confirm that Vault is unsealed on the secondary cluster, and restart the procedure from Step 3.

   Estimated time: 15 minutes

---

Step 6: Restore Services

This step restores the database and all services. Backups are not taken during this step.

1. Copy the environment variables file from the backup location to the secondary bootstrap node:

   /opt/wss/<cluster_name>/additional_env_vars.env
   

2. Copy the Access service profile to the secondary bootstrap node:

   /opt/wss/<cluster_name>/access/access-profile.yml
   

3. In the access-profile.yml file, perform the following:

   • Update the ip_ignore_list_for_xff_header and fqdn.ip fields to reflect the IP addresses and configuration of the secondary cluster.
   • Add the Nomad subnet to the ip_ignore_list_for_xff_header list.
     • Note: The default value is 172.26.64.0/20. If you chose to deploy with a custom subnet, update the subnet value in the ip_ignore_list_for_xff_headerfield accordingly.

4. Update the DNS server to direct the FQDN to the secondary cluster.

5. Confirm that the original primary cluster is offline or not reachable.

6. Run the following command to restore all services:

   wso services restore --cps_image <cps-image>
   

   Note: If any job fails during this step, purge it and reissue the same command.

   To restore a specific service only, include the -s flag:

   wso services restore -s <service-name> --cps_image <cps-image>
   

   Where <cps-image> is the control plane services image tag that matches the source cluster deployment.

   Note: If an error occurs during this step, revert all secondary cluster virtual machines to the snapshots taken in Step 2, confirm that Vault is unsealed on the secondary cluster, and restart the procedure from Step 3.

   Estimated time: Minimum 75 minutes. Actual time scales linearly with the volume of data in PostgreSQL and OpenSearch.

---

Step 7: Complete the Restore and Resume Backup Operations

Run the following command to verify the restored data and enable backup operations on the newly promoted primary cluster:

wso services restore-complete --cps_image <cps-image>

Estimated time: 10 minutes

---

Recovery Time Objective

The total recovery time objective (RTO) depends on the volume of data in PostgreSQL and OpenSearch. The minimum expected RTO is approximately two hours, with actual time increasing proportionally with data volume. Omnissa recommends running timed restore drills to establish RTO figures specific to your deployment before relying on this procedure in a production failover.