Logging and Troubleshooting Information for Omnissa Access

After you install Omnissa Access, you can use different approaches when you encounter issues. For example, you can troubleshoot issues by accessing the logs, you can restart a service, and you can implement the workarounds for known issues.

Omnissa Access Service Logs

This procedure provides you access to the service logs.

Procedure:

1. Open the Nomad interface
2. Go to Jobs
3. Select service
4. Find Allocation ID
5. Log in to the node
6. Navigate to: /opt/nomad/alloc/<allocation-id>/alloc/data

Restart a Service

This procedure restarts the service of your choice.

Procedure:

1. Stop and purge job in Nomad UI
2. Redeploy service from the Bootstrap node:

   wso services deploy -s <service-name>
   

A Service is in a Not Ready State

A service can enter a NOT_READY state at any time, not only during initial deployment. This can also happen after a completed deployment or during an upgrade.

If you deployed manually, see the wso access check-service-readiness step in Deploying the Omnissa Access and Infrastructure Services.

The troubleshooting steps below apply regardless of which path you used.

Problem and Cause

The specific problem and cause varies according to the service. See the solution provided below for an example of how to solve this type of issue.

Solution

The following example demonstrates how you can use Nomad to troubleshoot the greenbox service when it is listed in a NOT_READY state. Other services may store their logs under a different task name or directory, so use this as a model for the general approach rather than an exact match for what you will see in your environment.

1. Run from the bootstrap node the following command: sudo cat /opt/wss/<cluster_name>/cp-cluster/cp-cluster.env to reveal the Nomad URL and token, then open the Nomad UI in a browser and click Jobs

2. Click greenbox-job

3. Under Recent Allocations, right-click each allocation ID that appears in the ID column and select Open Link in New Tab

4. For each Allocation tab opened, click the Files tab and navigate to alloc > data and open the appropriate log file, most likely the largest sized file and the most recently modified, such as greenbox_web.log

5. Again, for each Allocation tab opened, click Tail, then click the play icon to stream the log live, and review the output to identify the cause of the failure

   

NTP and Clock Synchronization

Omnissa Access depends on time clock sychronization for healthy operation. This section helps troubleshoot NTP-related issues.

Problem and Cause

A common symptom of time clock sychronization issues is a certificate error. This will often appear as an expired certificate failure. As an example, wso cp deploy will fail, if there is an NTP issue:

wso cp deploy

Here is sample output of the error:

2026-08-04 14:48:31,740 p=4684 u=root n=ansible INFO| fatal: [127.0.0.1]: 
FAILED! => {"changed": true, "cmd": "vault operator init -format json", 
"delta": "0:00:00.087964", "end": "2026-08-04 14:46:09.578602", 
"msg": "The command exited with a non-zero return code.", "rc": 2, 
"start": "2026-08-04 14:46:09.490638", 
"stderr": "Get \"https://127.0.0.1:8202/v1/sys/seal-status\": 
tls: failed to verify certificate: x509: certificate has expired or is not yet valid: 
current time 2026-08-04T14:46:09Z is before 2026-08-04T14:47:41Z", 
"stderr_lines": ["Get \"https://127.0.0.1:8202/v1/sys/seal-status\": 
tls: failed to verify certificate: x509: certificate has expired or is not yet valid: 
current time 2026-08-04T14:46:09Z is before 2026-08-04T14:47:41Z"], "stdout": "", 
"stdout_lines": []}

Solution

Running wso cp sync-time will bring the cluster back into time clock sychronization. This command references the NTP server settings in the profile.yml file:

wso cp sync-time

Commands wso control-plane ansible and wso cp failing, when using ssh-key to login

Omnissa Access supports multiple methods of ssh authentication for logging in to cluster nodes: password and ssh-key. The ssh-key method is recommended, for the following reasons:

• More secure
• Never expire (no rotation required)

Problem and Cause

Release 26.07 introduced an issue where commands wso control-plane ansible and wso cp fail to run properly, when using ssh-key to log into to any node in the cluster. This is caused by both commands attempting to mount differing host directories to the same point: /workdir.

Solution

This issue was resolved with release 26.10. However, clusters starting-out with a 26.07 fresh deployment may exhibit this problem. To resolve, run the following commands:

cd <workdir>/cp-cluster/
mkdir -p cp-cluster
ln -sfn ../private-key cp-cluster/private-key

Port Collision

Omnissa Access services bind listeners to serveral ports. This section helps troubleshoot scenarios where there is a port conflict.

Problem and Cause

A port collision can occur during a fresh deployment or upgrade. This issue most commonly occurs while running the wso services deploy command. This error can be viewed from the Nomad UI:

Solution

Port collision errors can be resolved by performing the following steps from the bootstrap node:

1. export TARGETS="general_compute_linux" (core) or general_compute_access_linux (access)

2. export PORT=<colliding_port>

3. source <cluster.env>

4. Find the JobID holding the port:

   wso control-plane ansible -- -b -m shell -a \
   "for n in \$(NOMAD_ADDR='$NOMAD_ADDRESS' NOMAD_TOKEN='$NOMAD_TOKEN' NOMAD_SKIP_VERIFY=true /usr/local/bin/nomad node status -json | jq -r '.[].ID'); do for a in \$(curl -sk -H \"X-Nomad-Token: $NOMAD_TOKEN\" \"$NOMAD_ADDRESS/v1/node/\$n/allocations\" | jq -r '.[] | select(.ClientStatus==\"running\" or .ClientStatus==\"pending\") | .ID'); do curl -sk -H \"X-Nomad-Token: $NOMAD_TOKEN\" \"$NOMAD_ADDRESS/v1/allocation/\$a\" | jq -r \"select([(.AllocatedResources.Shared.Ports // [])[].Value] | index($PORT)) | .JobID\"; done; done | sort -u" \
   "$TARGETS"
   

5. wso services stop -s <service>

6. wso services deploy -s <service>

7. Continue with wso services deploy -t full

PostgreSQL Point-in-Time "Last Known Good" Rollback

Omnissa Access utilizes a PostgreSQL for cluster configuration and logging. This section helps revert potentially-unwanted changes in the database.

Problem and Cause

Nodes in an Omnissa Access cluster support highly-available deployments. This fault-tolerant architecture helps protect against potential infrastructure failures, such as an outage. However, this does not help prevent a bad data change. Unwanted changes can still find their way into a node and propagate through-out the cluster, via a typical replication or synchronization.

Restoring from a point in time is the best recourse for such a data corruption. These steps walkthough the proper procedure for rolling-back the database to a last known good state.

Scope

This design does not cover data corruption in Vault, Consul, or Nomad clusters. This does not cover point-in-time-recovery for Nomad, Consul, Vault, and Opensearch.

Prerequisites

These roll-back procedures are dependent on the presence of NFS.

Solution

At a high level, the recovery executive flow will consist of:

• Expose a recovery target time

  Add --recovery-target-time <timestamp> (optionally: a relative age such as 2d or 6h) to wso services restore, threaded through to the existing image logic. To ensure the change is backward-compatible and opted-in, default setting: empty = LATEST.

• Clear the data directory as an explicit, operator-run pre-step (not automated in CPS)

  The destructive PGDATA delete is intentionally kept out of the automated restore. It is run deliberately via wso control-plane ansible, with confirmation, before wso services restore. The active data directory is discovered at runtime (setting the nomad variable is not required).

Procedure:

1. The ordered flow, recovery_mode=true

   Phase A - Recovery preparation. Run manually (via wso control-plane ansible):
     1. Discover active PGDATA host path (no hardcoding), record on each node
     2. Stop the Postgres Nomad job (prevents reschedule) — run once
     3. Move live PGDATA and split-WAL aside to *_bkp (do NOT delete yet)
   Phase B - Services restore.
     4. Only now does recovery_target_time do anything (restore from NFS via WAL-G) 
     5. After restore succeeds and Postgres is healthy, free disk — drop *_bkp
   Phase C - Restore complete.
     6. Reconstructed dir is new live PGDATA; replicas re-clone from the promoted leader
   

2. Recovery preparation. Run manually, via wso control-plane ansible

   The destructive delete is deliberately not automated in pre-deploy.sh or CPS. A new POSTGRESQL_HOST_DATA_DIR nomad variable is not needed. The active directory is automatically discovered at runtime. from the running container. Script for command line (no Nomad UI):

   export TARGETS="general_compute_linux"
   export JOB="postgres"
   
   # 1. Discover active PGDATA host path (no hardcoding), record on each node
   wso control-plane ansible -- -b -m shell -a \
     'cid=$(docker ps -q -f status=running -f name="^'"${JOB}"'-task-"); \
      dd=$(docker exec "$cid" printenv POSTGRESQL_DATA_DIR); \
      wd=$(docker exec "$cid" printenv POSTGRESQL_WAL_DIR 2>/dev/null); \
      echo "/opt/control-plane/data/postgres/${dd##*/}" > /var/tmp/pg_data_dir; \
      if [ -n "$wd" ]; then echo "/opt/control-plane/data/postgres-wal/${wd##*/}" > /var/tmp/pg_wal_dir; else : > /var/tmp/pg_wal_dir; fi; \
      cat /var/tmp/pg_data_dir; cat /var/tmp/pg_wal_dir' "${TARGETS}"
   
   # 2. Stop the Postgres Nomad job (prevents reschedule) — run once
   source /opt/wss/access/cp-cluster/cp-cluster.env
   wso services stop -s postgres
   
   # 3. Move live PGDATA and split-WAL aside to *_bkp (do NOT delete yet).
   #    Recreate empty dirs so the next restore/deploy can remount them.
   #    Never touch the NFS backup. wal-g-restore.sh only *warns* about stale
   #    WAL left in the split dir — leaving the live WAL in place defeats PITR,
   #    so it must be moved aside with PGDATA.
   wso control-plane ansible -- -b -m shell -a \
     'd=$(cat /var/tmp/pg_data_dir); \
      rm -rf "${d}_bkp"; mv "$d" "${d}_bkp"; \
      if [ -s /var/tmp/pg_wal_dir ]; then \
        w=$(cat /var/tmp/pg_wal_dir); \
        rm -rf "${w}_bkp"; mv "$w" "${w}_bkp"; \
      fi' \
     "$TARGETS"
   
   # 4. Only now does recovery_target_time do anything (restore from NFS via WAL-G)
   wso services restore -s postgres --recovery_target_time "<age>"
   
   # 5. After restore succeeds and Postgres is healthy, free disk — drop *_bkp.
   #    Do NOT run this if restore failed; keep *_bkp for investigation / rollback.
   wso control-plane ansible -- -b -m shell -a \
     'rm -rf "$(cat /var/tmp/pg_data_dir)_bkp"; \
      [ -s /var/tmp/pg_wal_dir ] && rm -rf "$(cat /var/tmp/pg_wal_dir)_bkp"' \
     "$TARGETS"
   

   Notes:

   • Step 1 reads POSTGRESQL_DATA_DIR from the running container and maps it to the host path. This step is version-agnostic. It is decoupled from the upgrade data_dir_old and data_dir_old_new promoter variables.

   • Step 2 stops the job (not a single alloc) so Nomad does not reschedule it. The wiped directories stay empty, until Phase B.

   • Step 3 is destructive. The step is wrapped with an explicit warning and requires a typed confirmation, which prints the resolved path before deleting.

3. Resulting operator flow

   The process is organizated into three phases:

   # Phase A — recovery-prep (operator-run, with confirmation): 3 wso control-plane ansible calls
     discover active PGDATA → nomad job stop postgres → rm -rf PGDATA
   # Phase B — restore
     wso services restore -s postgres --recovery_target_time "2026-07-26 03:00:00 UTC"
   # Phase C — validate, then
     wso services restore-complete -s postgres        # recovery_mode=false → normal operation
   

   Note: It is recommended to take a fresh wal-g backup-push of current state first, so the roll-back itself is reversible. Confirm a base backup older than the target exists and the target is within WALG_RETENTION_DAYS.