EnglishFrançaisDeutschEspañolItaliano日本語한국어NederlandsPortuguês (Brasil)中文 (简体)中文 (繁體, 台灣)
Skip to main content

11 de noviembre de 2026

Managing Omnissa Access Control Plane Cluster Nodes

This topic describes how to replace an existing Control Plane deployment by adding or removing Omnissa Infrastructure/Platform nodes and Access nodes, and explains what happens when a node becomes unhealthy and is removed from the cluster.

Prerequisites

  1. A working directory/cluster directory exists containing all cluster configuration files from bootstrap machine.

  2. New node IPs are added to under the appropriate host group(s), prior to running the command in the cluster inventory file.

  3. The new nodes are reachable via SSH (Same ssh password as existing nodes) from the bootstrap/bastion machine.

  4. The nodes to be replaced, have to be powered off, prior to cp deploy.

  5. The inventory file does not need to be updated, if you are replacing node with same IP for same category (access/infra).

  6. Things to consider, while replacing infra nodes:

    • If the cluster environment is pointing to a node that you want to replace is powered-off, then it will have to be pointed to another active node serving consul, nomad, and vault.
    vi /root/<cluster_name>/<cluster_name>/<sluster_name>.env
# cluster
CLUSTER_NAME=cp-cluster
DATACENTER_NAME=cp-cluster-dc

NOMAD_ADDRESS=https://<active_infra_ip>:4646
CONSUL_ADDRESS=https://<active_infra_ip>:8501
VAULT_ADDRESS=https://<active_infra_ip>:8202
CONSUL_HTTP_PORT=8500

NOMAD_TOKEN=<existing_token>
CONSUL_TOKEN=<existing_token>
VAULT_TOKEN=<existing_token>
    • If the infra IP to replace is the first listed in the consul_server_linux_group of the inventory file, then replace the IP with any other active infra server.
    # Provide IPs to management server nodes
[consul_server_linux]
X.X.X.X   <<- first IP should not be the one you want to replace 
<active_infra_ip>
<active_infra_ip>


# Change the ordering from replaced being the first 
[consul_server_linux]
<active_infra_ip>
X.X.X.X   
<active_infra_ip>

Important:

  • Do not add the bootstrap IP to the inventory. The operation will fail with a validation error if the bootstrap IP is detected.
  • For replacing infra IPs, power-off and replace with new node one at a time. This is to ensure the cluster raft isn't broken (mandatory for 3 infra nodes). Always there should be <number_of_infra_cluster>/2+1 infra cluster, to maintain quorum.

Add New Access/Infra Nodes (with New IPs)

  1. Update the inventory file

    Open the /opt/wss/<cluster_name>/cp-cluster/cp-cluster.ini file and add the new node IP address under the appropriate host group.

    • Access Node: If you are adding a new Omnissa Access node, add it under these sections:
    [general_compute_access_linux]
[general_compute_nginx_http]
    • Infrastructure/Platform Node: If you are adding a new Infra node, add it under these sections:
    [consul_server_linux]
[vault_server_linux]
[nomad_server_linux]
[kafka_controller_linux]
[kafka_server_linux]
[postgres_linux]
[opensearch_leader_linux]
[opensearch_data_linux]
[general_compute_linux]

  2. Deploy the control plane

    wso cp deploy

  3. If access node, update NGINX_HTTP_NODE_COUNT and redeploy nginx services

    vi /root/<cluster_name>/additional_env_vars.env
wso services stop  -s nginx-http 
wso services stop  -s nginx-stream
wso services deploy  -s nginx-http 
wso services deploy  -s nginx-stream

  4. Check health

    wso healthcheck
wso access check-service-readiness

  5. Move workloads (services) to the new node

    By default, no workloads will be running on the newly added node. You should log in to the Nomad UI using the Nomad Token and perform the below steps

    1. Log in to the Nomad UI at https://10.0.0.x:4646/ui/jobs
    2. Click Clients.
    3. Click the Node ID that you want to move the workloads from.
    4. Click Drain to move the workloads to new machine

    Note: Upon completion of the drain operation, all the workloads will be moved to new machine. You cannot move selected services to the new node.

Replace Access Nodes (with different IPs)

  1. Update the /opt/wss/<cluster_name>/cp-cluster/cp-cluster.ini file, replace the IP with the new one in both the general_compute_access_linux and general_compute_nginx_http

  2. Power-off the node you want to replace

  3. Run wso cp deploy -v command

    wso cp deploy -v

    In case of a failure, run with full option.

    wso cp deploy -f -v

  4. Remove the old node from nomad membership

    After powering-off the old machine, nomad marks this node as down. The allocations should be deployed on the new node. Mark the node as ineligible and purge it.

    source /root/cp-cluster/cp-cluster/cp-cluster.env
curl -sk -H "X-Nomad-Token: $NOMAD_TOKEN" \
  "$NOMAD_ADDRESS/v1/nodes" \
  | jq -r '.[] | select(.Status=="down") | "\(.ID) \(.Name) \(.Address)"'

NODE_ID=<full-node-UUID>

# Make the old client ineligible, purge it, and run garbage collection:
curl -sk -X POST -H "X-Nomad-Token: $NOMAD_TOKEN" \
  -d '{"Eligibility":"ineligible"}' \
  "$NOMAD_ADDRESS/v1/node/$NODE_ID/eligibility"

curl -sk -X POST -H "X-Nomad-Token: $NOMAD_TOKEN" \
  "$NOMAD_ADDRESS/v1/node/$NODE_ID/purge"

curl -sk -X PUT -H "X-Nomad-Token: $NOMAD_TOKEN" \
  "$NOMAD_ADDRESS/v1/system/gc"

  5. Remove the old node from consul membership

    OLD_IP="<old_ip>"

NODE_NAME=$(curl -sk -H "X-Consul-Token: $CONSUL_TOKEN" \
  "$CONSUL_ADDRESS/v1/agent/members" \
  | jq -r --arg ip "$OLD_IP" \
    '.[] | select(.Addr == $ip) | .Name' \
  | head -n1)

curl -sk -X PUT -H "X-Consul-Token: $CONSUL_TOKEN" \
  "$CONSUL_ADDRESS/v1/agent/force-leave/${NODE_NAME}?prune=true"

  6. Run health checks

    wso healthcheck
wso access check-service-readiness

Replace Infra Nodes (with different IPs)

  1. Replace the unreachable infra-node IP with the new machine IP in the /opt/wss/<cluster_name>/cp-cluster/cp-cluster.ini file

  2. If the replaced node hosts asset-server

    On the asset-server node, stop its Telegraf service prior to deployment.

    wso services stop -s telegraf-asset-server

  3. Deploy the control plane

    wso cp deploy

  4. Purge the old nomad client

    source /root/cp-cluster/cp-cluster/cp-cluster.env

# Fetch powered off node id
curl -sk -H "X-Nomad-Token: $NOMAD_TOKEN" \
  "$NOMAD_ADDRESS/v1/nodes" \
  | jq -r '.[] | select(.Status=="down") | "\(.ID) \(.Name) \(.Address)"'

NODE_ID=<full-node-UUID>

# Make the old client ineligible, purge it, and run garbage collection:
curl -sk -X POST -H "X-Nomad-Token: $NOMAD_TOKEN" \
  -d '{"Eligibility":"ineligible"}' \
  "$NOMAD_ADDRESS/v1/node/$NODE_ID/eligibility"
curl -sk -X POST -H "X-Nomad-Token: $NOMAD_TOKEN" \
  "$NOMAD_ADDRESS/v1/node/$NODE_ID/purge"

# Trigger gc to clear unused instances  
curl -sk -X PUT -H "X-Nomad-Token: $NOMAD_TOKEN" \
  "$NOMAD_ADDRESS/v1/system/gc"

  5. Remove the failed Nomad server member

    Applicable when the replaced node was also a Nomad server:

    FAILED_SERVER=$(curl -sk \
  -H "X-Nomad-Token: $NOMAD_TOKEN" \
  "$NOMAD_ADDRESS/v1/agent/members" \
  | jq -r '.Members[] | select(.Tags.role=="nomad" and .Status=="failed") | .Name')
echo "$FAILED_SERVER"

# Confirm this prints the old server name, then remove it:
curl -sk -X PUT -H "X-Nomad-Token: $NOMAD_TOKEN" \
  "$NOMAD_ADDRESS/v1/agent/force-leave?node=$FAILED_SERVER&prune=true"

  6. Remove the old Vault peer

    # List the Vault Raft members:
curl -sk -H "X-Vault-Token: $VAULT_TOKEN" \
  "$VAULT_ADDRESS/v1/sys/storage/raft/configuration" \
  | jq -r '.data.config.servers[] | "\(.node_id) \(.address)"'

# Remove the old peer using its node_id:
curl -sk \
  -H "X-Vault-Token: $VAULT_TOKEN" \
  -H "Content-Type: application/json" \
  -X POST \
  -d '{"server_id":"<old-node-id>"}' \
  "$VAULT_ADDRESS/v1/sys/storage/raft/remove-peer"

  7. Verify the cluster

    wso healthcheck

Replace Access/Infra Nodes (with same IPs)

  1. Run wso cp deploy -f command

    wso cp deploy -f

  2. Purge the old nomad client

    source /root/cp-cluster/cp-cluster/cp-cluster.env

# Fetch powered off node id
curl -sk -H "X-Nomad-Token: $NOMAD_TOKEN" \
  "$NOMAD_ADDRESS/v1/nodes" \
  | jq -r '.[] | select(.Status=="down") | "\(.ID) \(.Name) \(.Address)"'

NODE_ID=<full-node-UUID>

# Make the old client ineligible, purge it, and run garbage collection:
curl -sk -X POST -H "X-Nomad-Token: $NOMAD_TOKEN" \
  -d '{"Eligibility":"ineligible"}' \
  "$NOMAD_ADDRESS/v1/node/$NODE_ID/eligibility"
curl -sk -X POST -H "X-Nomad-Token: $NOMAD_TOKEN" \
  "$NOMAD_ADDRESS/v1/node/$NODE_ID/purge"

# Trigger gc to clear unused instances  
curl -sk -X PUT -H "X-Nomad-Token: $NOMAD_TOKEN" \
  "$NOMAD_ADDRESS/v1/system/gc"

  3. Remove the failed Nomad server member

    Applicable when the replaced node was also a Nomad server:

    FAILED_SERVER=$(curl -sk \
  -H "X-Nomad-Token: $NOMAD_TOKEN" \
  "$NOMAD_ADDRESS/v1/agent/members" \
  | jq -r '.Members[] | select(.Tags.role=="nomad" and .Status=="failed") | .Name')
echo "$FAILED_SERVER"

# Confirm this prints the old server name, then remove it:
curl -sk -X PUT -H "X-Nomad-Token: $NOMAD_TOKEN" \
  "$NOMAD_ADDRESS/v1/agent/force-leave?node=$FAILED_SERVER&prune=true"

  4. Check cluster and service status

    Run the following commands to check if the cluster is healthy:

    wso healthcheck
wso access check-service-readiness

Replace Bootstrap Node

Overview

Bootstrap node is not part of the CP cluster. It is not possible to use wso commands to replace the bootstrap node. However the bootstrap node holds the important cluster config files. The config files are needed to replace bootstrap node with the new node.

Prerequisites:

  • Deploy new bootstrap machine, with same size as old bootstrap machine.
  • All the cluster config files should be backed up from old bootstrap.

  1. From existing bootstrap machine create tarball (preserve permissions)

    cd /root/cp-cluster
tar czpvf /root/cp-cluster-migration-$(date +%Y%m%d).tar.gz \
  profile.yml \
  logging \
  telegraf_plugin \
  ansible_extra_vars.yml \
  access \
  cp-cluster \
  additional_env_vars.env \
  deploy_services.log \
  deployment-config.yml

  2. Copy to new bootstrap machine

    scp /root/cp-cluster-migration-*.tar.gz configuser@<new-bootstrap-machine>:/home/configuser/

  3. Extract on new machine

    mkdir -p /root/cp-cluster
cd /root/cp-cluster
tar xzpvf /home/configuser/cp-cluster-migration-*.tar.gz

  4. Only one of the two options:

    • Option 1: Existing asset bundle

      Asset bundle download and configure

      mkdir -p assets
cd assets
curl --output asset-bundle-2605.zip \
  https://artifactory.air-watch.com/artifactory/control-plane-local/access-asset-bundles/26.05.0/offline-assets-bundle-ws1access-26.05.0-2026.06.04-395-29e945d.zip
unzip asset-bundle-2605.zip
cp cli-distribution/linux/wso /usr/bin/
wso eula
wso configure /root/cp-cluster/assets/

    • Option 2: New asset bundle

      Ensure bootstrap machine OS is up-to-date

      dnf update -y
reboot

      After machines completes reboot, validate health

      wso version
wso healthcheck
wso access check-service-readiness

Remove an Omnissa Access Node from the Cluster

There is no dedicated CLI command to remove a node from the Control Plane cluster. When a node becomes unavailable or unhealthy, the cluster handles removal automatically:

  • The node stops receiving new workload allocations after approximately two missed heartbeats.
  • Existing workloads on the node are rescheduled to healthy nodes.
  • No manual removal command is required.
  1. Remove the old node IP from the cp-cluster.ini file from all relevant sections.
  2. Power off the removed node.

Note: Cluster membership cleanup happens on a different timeline than failure detection:

  • Consul automatically purges inactive nodes from its member list from anywhere between 24 to 72 hours.
  • Nomad removes stale nodes during its next garbage-collection cycle.
  • Because of this, a removed node might remain visible in the Nomad or Consul UI for some time after it stops receiving traffic, even though it is no longer used for scheduling.

Password Management for Control Plane Cluster Nodes

The default password expiry for the configuser and root accounts is 60 days. You must change the password on every Control Plane cluster node before it expires.

The root password and the configuser password do not need to match each other, but each must be identical across every node in the cluster. For example, the root password must be the same on every node, and the configuser password must be the same on every node.

Reset the configuser or root password

Single node, logged in as root or via sudo:

passwd configuser
passwd root

Across all nodes at once, from the Bootstrap node:

export TARGETS='general_compute_linux:general_compute_access_linux'
wso control-plane ansible -- -b -m shell -a "echo 'configuser:NEW_PASSWORD' | chpasswd" "${TARGETS}"
wso control-plane ansible -- -b -m shell -a "echo 'root:NEW_PASSWORD' | chpasswd" "${TARGETS}"

Replace NEW_PASSWORD with the actual password before running, and ${TARGETS} with the Ansible inventory group or host pattern for the nodes to update — don't leave the literal placeholder in a saved script or your shell history.

Important: After changing the configuser password, update the same password in cp-cluster.ini on the Bootstrap node, or wso CLI operations against the cluster will start failing authentication. Active configuser sessions do not need to be invalidated after a reset.

Change the password expiration period

The 60-day expiration is controlled by password-aging fields on each account.

Note: If running as configuser, use sudoin front of the commands.

Check the current setting with:

chage -l configuser
chage -l root

Change it per account (single node):

chage -M <days> configuser
chage -M <days> root

-M sets the maximum number of days a password stays valid; -1 disables expiration (not generally recommended for root or service accounts).

Recover a completely forgotten root password

This requires console (hypervisor) access to the affected machine — it cannot be done over SSH, and only recovers one node at a time.

  1. Reboot the machine and interrupt at the GRUB menu.

  2. Highlight the boot entry and press e to edit it.

  3. Find the line starting linux (or linux16), go to the end of it, and append:

    rd.break enforcing=0

  4. Press Ctrl+X (or F10) to boot with the modified parameters — this drops you into the Dracut emergency shell.

  5. Remount the root filesystem read-write and chroot into it:

    mount -o remount,rw /sysroot
chroot /sysroot

  6. Set the new password:

    passwd root

  7. Flag the filesystem for an SELinux relabel on next boot, since SELinux was in permissive mode for this session:

    touch /.autorelabel

  8. Exit the chroot and reboot:

    exit
reboot

    The first boot after this takes longer than usual because of the SELinux relabel.

SSH Authentication for Replacement Nodes

When adding a replacement node, configure SSH authentication according to the authentication method used by the existing cluster.

Password-based authentication

If using password for authentication, when adding a new machine (with same or different IP), it must be configured with same ssh username and password.

SSH key-based authentication

If using ssh-key for authentication, copy the same ssh public key from the bootstrap node to the designated machine.

su configuser
ssh-copy-id configuser@<new-vm-ip>

Replacing the Bootstrap Node

If the bootstrap node itself is being replaced, run the cluster configuration update command using the appropriate INI file:

update-cluster-ini INI_FILE=<file-path>

Troubleshooting

IssueCauseResolution
Bootstrap IP in inventoryThe bootstrap node IP was added to cp-cluster.ini.Remove the bootstrap IP from the inventory file.
Deployment skippedThe node was already onboarded (hash match).Use -f (--force) to re-run.
SSH connection refusedThe node is unreachable, or SSH is not configured.Verify connectivity and SSH keys.
Failure at the Consul or Vault stepThe Consul or Vault cluster is unhealthy.Verify cluster health before adding a node.

¿Le resultó útil esta página?

Enviar comentarios sobre este tema

¿Le resultó útil este tema?

No incluya información personal ni confidencial.

Generando el enlace…