Removing a non-initial node from the cluster if the node is unavailable

Last Updated : Jan 10, 2025 |

About this task

Use this procedure to remove an unavailable non-initial cluster node, which cannot be removed using the standard node uninstallation procedure. For example, use this procedure if you cannot log in to the node using an SSH connection or if the Cassandra service is not running on the node.

Do not use this procedure to remove the initial cluster node.

Important:

After performing this procedure, do not start the removed node. If you start the node, it will re-appear in the list of cluster nodes.

Before you begin

Enable EASG functionality. EASG is required to obtain the sroot privileges, which you must use to perform commands in this procedure.

Procedure

  1. If the virtual machine with the node that you want to remove is still running, stop the virtual machine.
  2. Log in to the initial node as an administrator using an SSH connection.
  3. Run the following command to obtain the sroot privileges:
    su - sroot
  4. Run the following commands:
    cdto cassandra
    cd bin
  5. Run the following command to obtain the UUID of the unavailable node in the Cassandra cluster:
    ./nodetool status

    The unavailable node has the DN status. The node UUID is displayed in the Host ID column. The following is an example of the command output:

    [root@aawg bin]$ ./nodetool status
    Datacenter: datacenter1
    =======================
    Status=Up/Down
    |/ State=Normal/Leaving/Joining/Moving
    --  Address      Load       Tokens       Owns (effective)  Host ID                               Rack
    UN  233.252.0.2  5.07 MiB   256          100.0%            d1e8b3dc-f116-4bbc-851d-122c9e400bc3  rack1
    DN  233.252.0.3  5.07 MiB   256          100.0%            e2e779ae-d34d-2a10-23de-5ac14c4af783  rack1
    UN  233.252.0.4  5.07 MiB   256          100.0%            12cc1379-5164-8e74-81c0-bc8da8d09341  rack1

    In this example, the unavailable node has UUID e2e779ae-d34d-2a10-23de-5ac14c4af783.

  6. Run the following command:
    ./nodetool removenode <UUID>

    In this command, UUID is the UUID of the unavailable Cassandra database node. For example:

    ./nodetool removenode e2e779ae-d34d-2a10-23de-5ac14c4af783
  7. Run the ./nodetool status command again and ensure that the Cassandra cluster does not contain the unavailable node.
  8. Run the following command:
    cdto misc
  9. Run the following command to obtain the UUID of the unavailable Avaya Aura® Web Gateway node:
    app listnodes

    The unavailable node has the down status. The node UUID is displayed in the Server identifier column. The following is an example of the command output:

    [root@aawg misc]$ app listnodes
    Node IP address   Status  Server identifier                     Singleton  Version
    233.252.0.2       live    2523a75a-11a2-45ff-ba39-e23af17c316d  master     3.11.0.0
    233.252.0.3       down    3edfc879-b374-1c12-a045-a1456cfc277a             3.11.0.0
    233.252.0.4       live    6e430dff-7411-f2c4-9a5a-beec23916707             3.11.0.0
    

    In this example, the unavailable node has UUID 3edfc879-b374-1c12-a045-a1456cfc277a.

  10. Run the following command:
    ./clitool-csa.sh removeClusterNode <UUID>

    In this command, UUID is the UUID of the unavailable Avaya Aura® Web Gateway node. For example:

    ./clitool-csa.sh removeClusterNode 3edfc879-b374-1c12-a045-a1456cfc277a
  11. Run the app listnodes command again and ensure that the Avaya Aura® Web Gateway cluster does not contain the node.