Error message seen in backup logs for Incremental backups

Last Updated : Jul 11, 2022 |
Prolog information
Metadata
Keywords:

Condition

Below error message seen in backup logs for Incremental backups:
time="2022-07-07T15:24:41Z" level=info msg="stderr=[ERROR: [064]: unable to write to remote-0 process on 'analyticsdb-node-0-8sl2-0.analyticsdb-pods.default.svc.cluster.local.' write: [32] Broken pipe\nWARN: unable to write to remote-0 process on 'analyticsdb-node-0-8sl2-0.analyticsdb-pods.default.svc.cluster.local.' write: [32] Broken pipe\n]" time="2022-07-07T15:24:41Z" level=fatal msg="command terminated with exit code 64"

Cause

One of the database pods is not in a Running state. For Incremental Backups to complete successfully, all database pods needs to be in Running state with no errors.

Solution

  1. pgBackRest re-runs an Incremental Backup, if it fails at the first time.
  2. If a database pod is restarting when the backup is taken, you can monitor if there are new backup jobs created. Run the following command as a root user:
    ADDITIONAL INFORMATION: kubectl get po | grep analytics
    analyticsdb-backup-4m92--1-7qn2b 1/1 Running 0 3m31s
    analyticsdb-backup-4m92--1-b2fxm 0/1 Error 0 3m41s
    analyticsdb-backup-4m92--1-lswnz 0/1 Error 0 4m20s
  3. You can see 2 failed backup jobs with the latest job running. To view its log, you can run the following command as a root user:
    ADDITIONAL INFORMATION: kubectl get logs analyticsdb-backup-4m92--1-7qn2b
  4. To confirm the incremental backup is successful, do the following:
    1. Run the ccm release orca analytics script.
    2. Select List Incremental Backups option.