Cluster Reboot or Planned Maintenance of Omnissa Access Cluster

After any node reboot (except for the bootstrap node), if the node does not automatically rejoin the Omnissa Access cluster. You must run wso cp unseal from the bootstrap node to restore cluster membership.

Use the appropriate procedure option below based on your maintenance scenario.

Option 1 - Planned Maintenance: Reboot One Node at a Time

Reboot one node at a time to maintain cluster availability. The cluster remains functional at degraded capacity during the reboot.

Omnissa Access Nodes

Bring down the node to be maintained.
Monitor cluster health and service status.

The cluster should remain healthy with one instance of each service running at reduced capacity.
Complete maintenance and bring the node back online.
From the bootstrap node, run the following command to verify cluster health.
```
wso healthcheck
```
Note: Allow time for all services to come back online.
Verify that two instances of each service are running.
Run the following command to confirm all services are ready to serve traffic:
```
wso access check-service-readiness
```

If Both Access Nodes Go Down (Unplanned)

If both access nodes are down, then do not proceed to the next step. Ensure both access nodes are up and running. Proceed to the next step, when all services are running and healthy.
From the bootstrap node, run the following command to verify cluster health.
```
wso healthcheck
```
Note: Allow time for all services to come back online.
Verify that two instances of each service are running.
Run wso access check-service-readiness to confirm services are ready to serve traffic.

Infrastructure/Platform Nodes

Reboot one node at a time. Infrastructure services (PostgreSQL, Kafka, Redis, and OpenSearch) remain functional with two instances during the reboot.

Bring down the node to be maintained.
Monitor cluster health.

The cluster should remain functional with two instances of each infrastructure service.
Complete maintenance and bring the node back online.
From the bootstrap node, run:
```
wso cp unseal
```
From the bootstrap node, run the following command to verify cluster health.
```
wso healthcheck
```
Note: Allow time for all services to come back online.
Verify that three instances of each infrastructure service are running.

If One Infrastructure Node Goes Down (Unplanned)

In this scenario, the cluster should remain operational. However, it will likely run in a degraded state. To restore cluster operation, refer to the following steps:

Run the following command from the bootstrap node:
```
wso cp unseal
```
From the bootstrap node, run the following command to verify cluster health.
```
wso healthcheck
```
Note: Allow time for all services to come back online.

If all Infrastructure/Platform Nodes Go Down (Unplanned)

Power-on the nodes.
Once all nodes are back online, run the following command from the bootstrap node:
```
wso cp unseal
```
From the bootstrap node, run the following command to verify cluster health.
```
wso healthcheck
```
Note: Allow time for all services to come back online.
Verify that three instances of each infrastructure service are running.

If Recovered Infrastructure Node is Back Online, but Service Interruptions Continue

In this scenario, an infrastructure node was brought back online, but there are still service interruptions. Services appear to be running, however the following symptoms still persist:

Directory user logins consistently fail.
Local admin logins work intermittently.
Configured policy rules show as undefined in the UI.
Access functionality remains partially broken, even though platform/service health appears normal.

Refer to the following steps to restore the cluster to full operation.

From the bootstrap node, restart the following services:

wso cp unseal
wso services deploy -s authcontrol
wso services deploy -s federation
wso services deploy -s saas

From the bootstrap node, run the following command to verify cluster health.
```
wso healthcheck
```
Note: Allow time for all services to come back online.

Bootstrap Node Reboot

Rebooting the bootstrap node does not affect cluster services.

Option 2 - Planned Maintenance: Without Degraded Performance (Buffer Node)

To perform maintenance without service degradation, add a temporary buffer node before beginning. This allows services to migrate off the node being maintained.

Deploy a new node with the same configuration as the existing nodes and add it to the cluster.

From the bootstrap node, update cp-cluster.ini with the new node IP.

When adding a new infra node, update the following sections:

consul_server_linux
vault_server_linux
nomad_server_linux
kafka_controller_linux
kafka_server_linux
postgres_linux
opensearch_leader_linux
opensearch_data_linux
general_compute_linux

When adding a new access node, update the following sections:
```
general_compute_access_linux
general_compute_nginx_http
```

From the bootstrap node, run wso cp deploy -t <new-node-ip>.
```
wso cp deploy -t <new-node-ip>
```
Drain the node to be maintained from the Nomad UI (Clients > select node > Drain).

This action migrates all services to other nodes, including the buffer node, with no performance impact.
Perform maintenance on the existing nodes one at a time.
After maintenance is complete, optionally remove the buffer node by deleting its entry from cp-cluster.ini and shutting it down.

You can also retain it as a standby node for future maintenance.
From the bootstrap node, run:
```
wso cp unseal
```
From the bootstrap node, run the following command to verify cluster health.
```
wso healthcheck
```
Note: Allow time for all services to come back online.
Verify that two instances of each Access service and three instances of each Infrastructure/Platform service are running.
Run wso access check-service-readiness to confirm all services are ready to serve traffic.

Note: After a node reboot and wso cp unseal, allow time for all services to return online and form a healthy cluster. Monitor cluster health during this period. If a service fails to recover, check its logs in the Nomad UI, purge the service, and redeploy it:
```
wso services deploy -s <service-name>
```

War diese Seite hilfreich?