Omnissa Access Cluster Upgrade Hangs at the RPMDB Rebuild Stage

Resolve a stalled cluster upgrade caused by a secondary node that stops responding during the RPMDB rebuild stage. The procedure involves scaling the cluster down to a single upgraded node and cloning it to restore the cluster.

Problem

During an upgrade of an Omnissa Access cluster, a secondary node stops progressing and remains unresponsive at the successfully rebuilt RPMDB stage. New SSH connections to the affected node cannot be established. Other nodes in the cluster may have completed the upgrade successfully.

Cause

A node-specific environment issue during the RPM database rebuild process prevents the upgrade from completing on the affected node.

Solution

1. Take a snapshot of each cluster node before making changes.

   Note: If your deployment uses an external database, back up the database before proceeding.

2. Power off the unresponsive node or nodes.

3. On the working node, verify that the upgrade completed successfully, then take a snapshot of the upgraded node.

4. In vCenter, clone the upgraded node once for each node that you powered off.

5. For each cloned node, open the virtual machine settings in vCenter and update the network adapter MAC address to match the MAC address of the original node it is replacing.

6. Power on the cloned node and log in as root.

7. Update the /etc/hosts file with the correct IP address and hostname for this node.

8. Run sudo ln -sf /run/systemd/resolve/resolv.conf /etc/resolv.conf to restore the DNS resolver configuration.

9. Edit the /etc/systemd/network/10-eth0.network file and set the IP address to the correct address for this node.

10. Edit the /etc/hostname file and replace the existing entry with the correct hostname for this node.

11. Reboot the node.

12. Verify that the reconfigured node joined the cluster successfully and that all Omnissa Access services are running.

    Note: Repeat steps 5 through 12 for each additional cloned node.