Gracefully powering off a cluster

Last Updated : Sep 25, 2023 |

About this task

Use this procedure to gracefully shut down your solution cluster.

Warning:

  • If you shut down your cluster using a different method than what is described in this procedure, file corruption can occur.

Before you begin

  • You must plan a maintenance window to perform this task.

  • Stop call events across the solution.

    • To stop all Avaya Oceana® traffic for an Avaya Analytics™ cluster, run the kubectl scale deployment orca-ref-input-adaptor --replicas=0 command using an account with root privileges.

  • Back up Common Services using the ccm backup command. If you do not perform a backup, you risk a full reinstallation of the solution.

  • Back up product application data as described in your solution documentation.

Procedure

  1. Log into Cluster Control Manager.
  2. Check the health of the pods:
    1. Enter k get pods -A
    2. If any pod is listed as other than Running or Completed, do not proceed any further. Contact Avaya Support.
    3. Enter ccm status --health
    4. Check that all products are listed as Healthy.
    5. Check that historical reporting is working.
    6. Check that the supervisor desktop shows data.
  3. Create a metadata backup:
    1. Enter the following commands: 
      ccm release orca analytics
    Historical Reporting 
        Backup Metadata 
            Backup Metadata
            Export Metadata
    2. Wait for the command to complete before continuing.
    3. Note the name and location of the backup.
    4. Use winscp or a similar tool to copy that backup off ccm to a safe location.
  4. Create a database backup:
    1. Enter the following commands: 
      ccm release orca analytics
    Database 
        Database Backup 
            Remote Backups
    2. Wait for the command to complete before continuing.
    3. Verify that the new backup file exists on the external backup server.
  5. Identify which node runs the registry-pod:
    1. Enter kubectl get pods -n image-registry -o wide
    2. Note which master node has the registry-pod.
  6. Stop the data flow to the cluster:
    1. Enter kubectlscale --replicas=0 deployment orca-ref-input-adaptor
    2. Verify that the input ref-adaptor pod has stopped by entering kubectl get pods | grep orca-ref-input-adaptor until no orca-ref-input-adaptor pods are shown.
  7. Create a CCM backup:
    1. Enter ccm backup
    2. Wait for the command to complete before continuing.
    3. Verify that the new backup file exists on the external backup server.
  8. Stop the authorization database:
    1. Gracefully shutdown the Common Services database by entering pre-infra-upgrade
    2. Wait for the command to complete before continuing.
  9. Determine the node roles:
    1. Enter ccm version -k
    2. Determine role of each node (worker or controller-worker).
    3. Make a note of the node roles as you need them during later procedures.
  10. Determine which nodes contain a second disk and which are diskless.
    1. Run the checkInfra -Sd | grep LVM_THIN command. 
      [cust@flex190-129 ~]$ checkInfra -Sd | grep LVM_THIN
| pool_sds             | flex190-132.dr.example.com | LVM_THIN | vg_sds/sds_thinpool |   341.00 GiB |    464.76 GiB | True         | Ok    |            |
| pool_sds             | flex190-133.dr.example.com | LVM_THIN | vg_sds/sds_thinpool |   341.00 GiB |    464.76 GiB | True         | Ok    |            |
[cust@flex190-129 ~]$
    2. Note the disk status of each node. Determine which node is not listed. That node is the diskless node.
  11. As root user, enter kubectl get pods -n image-registry -o wide
  12. Make a note of the cluster node hosting the image registry.
  13. Log in to vCenter as an administrator or with the account used to deploy the cluster.
  14. Click on the VMs and Templates tab.
  15. Power off the worker nodes:
    1. Right-click on the first worker node and click Power > Shut Down Guest OS.
    2. Check the VM status in vCenter. Wait until the node has powered down before continuing to the next node.
    3. Repeat these steps for the next worker node until all worker nodes are powered off.
  16. Power off the controller-worker node without a second disk:
    1. Right-click on the controller-worker node without a second disk and click Power > Shut Down Guest OS.
    2. Check the VM status in vCenter. Wait until the node has powered down.
  17. Power off the controller-worker nodes with second disks:
    1. Right-click on each controller-worker node containing a second disk and click Power > Shut Down Guest OS. You can power these off together, no wait time is needed between nodes.
  18. Power off Cluster Control Manager:
    1. Click on the VMs and Templates tab.
    2. Locate and click on Cluster Control Manager in the folder you designated for the cluster.
    3. Right-click Cluster Control Manager and then click Power > Shut Down Guest OS.