Centralized log locations for troubleshooting

Last Updated : Apr 25, 2023 |
This section provides a centralized overview of key log categories, their locations, and common indicators to look for during diagnostic analysis.

OS and Kernel logs

These logs are fundamental for diagnosing hardware, driver, and core operating system issues. The following are the primary locations:
  • Standard Linux Logs: /var/log/ (includes messages, syslog, kern.log, and auth.log)
  • SBC-routed OS/Kernel Logs: /archive/syslog/ipcs/ (includes kern.log, daemon.debug.log, and cm.log)
You can check for the following issues or failures:
  • Kernel/Hardware Issues: Search for lines containing kernel: BUG:, Oops:, Call Trace:, I/O errors, or NIC/driver resets (such as link is down, resetting).
  • System Resource Issues: Search for messages like invoked oom-killer, Out of memory:, hung task, or soft lockup.
  • Service Failures: Check for repeated systemd[1]: for failing units messages, indicating a service is failing to start.

SBC core application logs (SIP, TURN, Media)

These logs are critical for debugging call flows, signaling issues, and core SBC processes. The following are the primary locations:
  • The following are the main application logs (elog):
    • SBC Processes: /archive/log/ipcs/ss/logfiles/elog/
    • EMS Processes: /archive/log/ipcs/sems/logfiles/elog/
  • The following are the SIP and PPM Traces:
    • SIP: /archive/log/tracesbc/tracesbc_sip/
    • PPM: /archive/log/tracesbc/tracesbc_ppm/
  • The following are the specialized logs:
    • TURN Controller: /archive/log/turnserver/
    • ICE/Media (RvToolkit): /archive/log/rvtoolkit/RvIce.log
You can check for the following errors or issues:
  • General Errors: Search for ERROR, CRIT, FATAL, ASSERT, or Segmentation fault.
  • Call/Signaling Issues: In tracesbc_sip or signaling_log, look for failed SIP transactions (4xx, 5xx, 6xx), repeated 408 Request Timeouts, or transport errors like no route or ICMP unreachable.
  • System Monitoring: In sysmon.log, check for CPU, memory, or disk threshold alerts and process restart loops.

ICU, upgrade, and hardware verification logs

These logs are essential for troubleshooting installation, upgrade, rollback, and initial configuration problems.
The primary location is /archive/log/icu/.
The following are the key files:
  • Generic installation and utility logs: icu.log, init.log.
  • Main upgrade and rollback logs: upgrade.log, ursbce.log.
  • Peon service and API layer logs: peon.log.
  • Hardware detection and validation logs: hw_verify.log, hw_verify_error.log.
You can check for the following issues:
  • Upgrade Failures: Search for ERROR, FAILED, DISK_FAILURE, INVALID_SIGNATURE, or TARBALL_EXTRACTION_FAILED.
  • Hardware Issues: In hw_verify_error.log, look for REQ_FAIL (hard requirement not met) or RCM_FAIL (recommended specification level not met).

Database logs (savondb)

Database logs track schema changes, data import/export, and replication status.
The following are the primary locations:
  • /archive/log/db/
  • /var/log/sbcedb.log
The following are the key files:
  • Database orchestration log:manager.log.
  • DB schema upgrade logs: upgrade.log.
  • Data import logs: import.log, importALL.log.
You can check for the following issues:
  • Schema/Upgrade Failures: Search for ERROR:, FATAL:, duplicate key, or relation does not exist.
  • Import/Backup Failures: Review error_log_importALL*.dat for detailed errors; check for pg_dump or pg_restore command failures.

System crash and dump files

When a process crashes unexpectedly, the system generates dump files that are critical for root cause analysis by Avaya Support.
The core dumps path is /archive/crash.
The process memory dumps (smdumps) path is /usr/local/ipcs/smdump/.