Use this procedure to clear a Data replication pg_xlog usage more than 1 GB alarm.
Caution:
This procedure is service interrupting and will cause down time on the deployment. Try to do this during low or no traffic periods.
Condition
Alarm message text
Data replication pg_xlog usage more than 1 GB
Alarm level
Critical
Solution
Procedure
Scenario 1 — Check the replication status between the SBC systems to see if the Data Replication is broken alarm is active.
Do one of the following depending on the type of system you are troubleshooting:
On an SBC system, run the following commands:
psql -U postgres -d sbcedb
sbcedb=# select * from alarms;
On an EMS system, run the following commands:
psql -U postgres -d commondb
commondb=# select * from alarms;
If the alarm is seen, then follow the steps in Clearing a “Data Replication is broken” alarm to clear the alarm. This should also fix the Data replication pg_xlog usage more than 1 GB issue.
Scenario 2 — The number of pg_xlogs will grow if the peer SBC or EMS system is down for a long time. To prevent the disk space from getting full, you must act immediately when this alarm is seen.
Check the database alarms table. If the table has the alarm Peer SBC is unreachable, confirm that the peer SBC or EMS system is powered up and in operation. It must be in operation so that database replication can occur.
On an SBC system, run the following commands:
psql -U postgres -d sbcedb
sbcedb=# select * from alarms;
On an EMS system, run the following commands:
psql -U postgres -d commondb
commondb=# select * from alarms;
If the peer SBC or EMS has a true hardware failure, break the replication on the SBC or EMS system that is in operation so that pg_xlogs will no longer grow on that system. To break the replication, downtime is required.
On the operational SBC or EMS system, do the following:
Scenario 3 — Disk space of /var becomes 100% full and the SBC or EMS system goes down. This happens when the alarms Data Replication is broken, Data replication pg_xlog usage more than 1 GB, and Peer SBC is unreachable have occurred and were not attended to.
Run the following command to check the /var partition files to see which files are causing the disk space issue:
cd /var
du –sh
This command lists the file size of each file in the / root directory. Confirm whether any pg_xlog files are more than 1 GB verify that the cause of the issue is really due to log size or replication being down.
Do a backup of the entire database. In a HA system or active-active EMS system, do a backup of the database on both systems, if possible. In the SBC or EMS that is operational, run the following command:
Confirm that the operational SBC or EMS system comes back online and starts all processes.
On the SBC or EMS system that is down or has database issues, confirm that both application processes and the database are down. Run the following commands to delete the database directory, recreate the database directory, recreate the database, break replication, and reboot the server:
