Cluster Reboot or Planned Maintenance of Omnissa Access Cluster

After any node reboot, the node does not automatically rejoin the Omnissa Access cluster. You must run wso cp unseal from the bootstrap node to restore cluster membership.

Note: This behavior applies to the current interim build only.

Use the appropriate procedure option below based on your maintenance scenario.

Option 1 - Planned Maintenance: Reboot One Node at a Time

Reboot one node at a time to maintain cluster availability. The cluster remains functional at degraded capacity during the reboot.

Omnissa Access Nodes

1. Bring down the node to be maintained.

2. Monitor cluster health and service status.

   The cluster should remain healthy with one instance of each service running at reduced capacity.

3. Complete maintenance and bring the node back online.

4. From the bootstrap node, run:

   wso cp unseal
   

5. From the bootstrap node, run the following command to verify cluster health.

   Allow time for all services to come back online:

   wso healthcheck
   

6. Verify that two instances of each service are running.

7. Run the following command to confirm all services are ready to serve traffic:

   wso access check-service-readiness
   

If Both Access Nodes Go Down (Unplanned):

1. Once both nodes are back online, run the following command from the bootstrap node:

   wso cp unseal
   

2. Run wso healthcheck from the bootstrap node and allow time for all services to come back online.
3. Verify that two instances of each service are running.
4. Run wso access check-service-readiness to confirm services are ready to serve traffic.

Infrastructure/Platform Nodes

Reboot one node at a time. Infrastructure services (PostgreSQL, Kafka, Redis, and OpenSearch) remain functional with two instances during the reboot.

1. Bring down the node to be maintained.

2. Monitor cluster health.

   The cluster should remain functional with two instances of each infrastructure service.

3. Complete maintenance and bring the node back online.

4. From the bootstrap node, run:

   wso cp unseal
   

5. From the bootstrap node, run wso healthcheck to verify cluster health.

   Allow time for all services to come back online.

6. Verify that three instances of each infrastructure service are running.

If all Infrastructure/Platform Nodes Go Down (Unplanned):

1. Once all nodes are back online, run the following command from the bootstrap node:

   wso cp unseal
   

2. Run wso healthcheck from the bootstrap node and allow time for services to come back online.
3. Verify that three instances of each infrastructure service are running.

Bootstrap Node Reboot

Rebooting the bootstrap node does not affect cluster services.

Option 2 - Planned Maintenance: Without Degraded Performance (Buffer Node)

To perform maintenance without service degradation, add a temporary buffer node before beginning. This allows services to migrate off the node being maintained.

1. Deploy a new node with the same configuration as the existing nodes and add it to the cluster.

2. From the bootstrap node, update cp-cluster.ini with the new node IP and run:

   wso cp deploy -t <new-node-ip>
   

3. Drain the node to be maintained from the Nomad UI (Clients > select node > Drain).

   This action migrates all services to other nodes, including the buffer node, with no performance impact.

4. Perform maintenance on the existing nodes one at a time.

5. After maintenance is complete, optionally remove the buffer node by deleting its entry from cp-cluster.ini and shutting it down.

   You can also retain it as a standby node for future maintenance.

6. From the bootstrap node, run:

   wso cp unseal
   

7. Run wso healthcheck to verify cluster health.

8. Verify that two instances of each Access service and three instances of each Infrastructure/Platform service are running.

9. Run wso access check-service-readiness to confirm all services are ready to serve traffic.

   Note: After a node reboot and wso cp unseal, allow time for all services to return online and form a healthy cluster. Monitor cluster health during this period. If a service fails to recover, check its logs in the Nomad UI, purge the service, and redeploy it:

   wso services deploy -s <service-name>