Restarting Kafka pod after multiple K8s node restarts

Last Updated : Dec 27, 2022 |
Prolog information
Metadata
Keywords:
Use this procedure to restart Kafka pod after multiple K8s Node Restarts.
  1. To restart the pod, delete the following command: kubectl delete pod -n avaya-kafka eventing-kafka-cp-kafka-0(/1/2)
  2. Wait for a while and if the pod still not coming to ready state do the following:
    1. Scale down Kafka service to 0
      kubectl scale sts -n avaya-kafka eventing-kafka-cp-kafka --replicas=0
    2. From the pod describe output (done in the issue identification step) get the PV name.
    3. k exec --namespace=piraeus deployment/piraeus-op-piraeus-operator-cs-controller -- linstor r l -r pvc-d2a17871-27b1-46e7-b64b-196547ff51ca
      • Replace the PV Name in the above command with the pod describe output
      • From the output find the Node on which the PVC is "inuse"
      +--------------------------------------------------------------------------------------------------------------------------------+

      | ResourceName                             | Node                       | Port | Usage  | Conns |    State | CreatedOn           |

      |================================================================================================================================|

      | pvc-d2a17871-27b1-46e7-b64b-196547ff51ca | node5336.puneccq.avaya.com | 7024 | Unused | Ok    | UpToDate | 2022-07-16 05:53:09 |

      | pvc-d2a17871-27b1-46e7-b64b-196547ff51ca | node5337.puneccq.avaya.com | 7024 | Unused | Ok    | UpToDate | 2022-07-16 05:52:59 |

      | pvc-d2a17871-27b1-46e7-b64b-196547ff51ca | node5338.puneccq.avaya.com | 7024 | InUse  | Ok    | Diskless | 2022-07-16 05:53:03 |
  4. Get the name of the satellite pod running on the node identified in previous step
    ADDITIONAL INFORMATION: k get pods -n piraeus -o wide | grep ns

    piraeus-op-piraeus-operator-ns-node-chdh9                     2/2     Running   4 (2d13h ago)   6d8h    10.133.53.37    node5337.puneccq.avaya.com   <none>           <none>

    piraeus-op-piraeus-operator-ns-node-sk5dh                     2/2     Running   2 (3d5h ago)    6d8h    10.133.53.38    node5338.puneccq.avaya.com   <none>           <none>

    piraeus-op-piraeus-operator-ns-node-vh59k                     2/2     Running   4 (2d12h ago)   6d8h    10.133.53.36    node5336.puneccq.avaya.com   <none>           <none>
  5. Disconnect the InUse resource using the drdbadmin cli
    ADDITIONAL INFORMATION:
    kubectl exec -n piraeus <satellite pod name> -c linstor-satellite -- drbdadm disconnect <pv-name> –force
  6. Check the status of the resource again and it should be in Unused state (excuse the command in step 2.c. and the usage should be "Unused" for all 3).
  7. Connect the resource back
    ADDITIONAL INFORMATION:
    kubectl exec -n piraeus <satellite pod name> -c linstor-satellite -- drbdadm connect <pv-name>
  8. Scale up Kafka
    ADDITIONAL INFORMATION:
    kubectl scale sts -n avaya-kafka eventing-kafka-cp-kafka --replicas=3
  9. Wait for few minutes and check whether all the Kafka pod are in ready state
    ADDITIONAL INFORMATION:
    kubcetl get pods -n avaya-kafka -o wide