Providing Great Debug
- Kelli Higdon
- Lorne Gaetz (Unlicensed)
Many times a support technician or a developer will ask you to provide logs or "debug". You may think to yourself, "I have no idea what logs they want?" or "I've barely used this PBX because everything has been working fine up until this point so I have no idea how to help!"
That's the main reason this post was created.
There are a couple different types of debug:
FreePBX CLI Debug
If you are running FreePBX 13 or higher and are executing a command through fwconsole you can use the --verbose option to output a stack trace that is especially helpful for developers to be able to fix problems
FreePBX Debug
If you think the error you are facing is a FreePBX GUI issue (white screen or blank page or something not working right in the GUI) then you will want to provide developers the debug log of the gui.
To do this simply run (from the SSH CLI)
fwconsole dbug |
This will generate output similar to what is shown below:
[root@dahdidev1 ~]# fwconsole dbug
Fetching FreePBX settings with gen_amp_conf.php..
trying to run as user asterisk:
[FPBXDBUGDISABLE] already set to [0]
==> /var/log/asterisk/freepbx_dbug <==
==> /var/log/httpd/error_log <==
mv: cannot stat `/var/lib/asterisk/agi-bin/license-*.zl': No such file or directory
[Wed Dec 03 00:05:09 2014] [error] [client 125.64.35.67] File does not exist: /var/www/html/zc
[Wed Dec 03 01:14:26 2014] [error] [client 125.64.35.67] File does not exist: /var/www/html/ucp/zc
[Wed Dec 03 01:14:26 2014] [error] [client 125.64.35.67] File does not exist: /var/www/html/restapi
[Wed Dec 03 01:14:26 2014] [error] [client 125.64.35.67] File does not exist: /tftpboot/zc
[Wed Dec 03 01:14:26 2014] [error] [client 125.64.35.67] File does not exist: /var/www/html/aastra
[Wed Dec 03 01:14:26 2014] [error] [client 125.64.35.67] File does not exist: /var/www/html/restapps
[Wed Dec 03 01:59:48 2014] [error] [client 94.102.49.82] script '/var/www/html/recordings/index.php' not found or unable to stat
[Wed Dec 03 01:59:50 2014] [error] [client 94.102.49.82] File does not exist: /var/www/html/admin/modules/admindashboard
[Wed Dec 03 01:59:51 2014] [error] [client 94.102.49.82] client denied by server configuration: /var/www/html/admin/page.framework.php
==> /var/log/asterisk/freepbx.log <==
[2014-Dec-03 09:44:02] [PHP-WARNING] (/var/www/html/admin/modules/endpoint/functions.inc/functions_epm_ucp.php:0) - No license for this product (PBXact) - make sure zend_loader.license_path is properly configured in your ini file!
[2014-Dec-03 09:44:02] [PHP-WARNING] (/var/www/html/admin/modules/endpoint/functions.inc/functions_epm_ucp.php:0) - License check failed!
[2014-Dec-03 09:57:01] [WARNING] (libraries/modulefunctions.legacy.php:7) - Depreciated Function _module_distro_id detected in /var/www/html/admin/modules/sysadmin/functions.inc/general.php on line 107
[2014-Dec-03 09:57:01] [PHP-NOTICE] (/var/www/html/admin/modules/sysadmin/functions.inc/license.php:53) - Undefined index: register
[2014-Dec-03 09:57:02] [WARNING] (libraries/modulefunctions.legacy.php:7) - Depreciated Function _module_distro_id detected in /var/www/html/admin/modules/sysadmin/functions.inc/general.php on line 107
[2014-Dec-03 09:57:02] [PHP-NOTICE] (/var/www/html/admin/modules/sysadmin/functions.inc/license.php:53) - Undefined index: register
[2014-Dec-03 09:58:20] [INFO] (bin/freepbx_setting:39) - trying to run as user asterisk:
[2014-Dec-03 09:58:20] [INFO] (bin/freepbx_setting:40) -
[2014-Dec-03 09:58:21] [INFO] (bin/freepbx_setting:61) - [FPBXDBUGDISABLE] already set to [0]
[2014-Dec-03 09:58:21] [INFO] (bin/freepbx_setting:62) - |
Now the the initial interface has started up it's time to go to whatever GUI issue you are having and replicate it. Once you have done that copy and past what is shown to you in the output of this command and send it to a developer or support technician.
Browser Debug (Console Log)
CHROME:
Press either CTRL + SHIFT + J to open the “console” tab of the Developer Tools.
Alternative method:
Press either CTRL + SHIFT + I or F12 to open the Developer Tools.
Press ESC (or click on “Show console” in the bottom right corner) to slide the console up.
Note: In Chrome’s dev tools, there is a “console” tab. However, a smaller “slide-up” console can be opened while any of the other tabs is active.
SAFARI:
Press CTRL + ALT + I to open the Web Inspector.
See Chrome’s step 2. (Chrome and Safari have pretty much identical dev tools.)
Note: Step 1 only works if the “Show Develop menu in menu bar” check box in the Advanced tab of the Preferences menu is checked!
IE9:
Press F12 to open the developer tools.
Click the “console” tab.
FIREFOX:
Press CTRL + SHIFT + K to open the Web console (COMMAND + SHIFT + K on Macs).
or, if Firebug is installed:
Press F12 to open Firebug.
Click on the “console” tab.
OPERA:
Press CTRL + SHIFT + I to open Dragonfly.
Click on the “console” tab.
Asterisk Logs
If you think this is an Asterisk bug or a FreePBX dial plan generation bug then you need to get the Asterisk Logs
To get these logs you will need to run (from SSH CLI):
tail -f /var/log/asterisk/full |
This will then give you output like:
Now make a phone call or cause the disruption to happen and copy and paste that output to a developer or support technician.
Asterisk Logs - Part II
Suppose you want a call trace from a specific call that has already happened, so it's too late to see it in the console live. First locate the call in the CDR, and get the uniquieid from the system column for the call in question:
Then grep the Asterisk full logs for that value:
This will return a few lines, which will include the Asterisk CALL-ID (not to be confused with CallerID), the second number in the square brackets. It will also return a file name, as full logs are rotated daily and purged weekly. Perform a second grep on the CALL-ID and filename like:
This will return the full call trace from beginning to end of a specific call. You can then search through the trace and send to pastebin for a shareable URL:
If you wish to remove sensitive information before sending to pastebin you can use sed. Suppose you had a caller ID (or DID) of 3145551212 and you want to replace it with NPANXXXXXX, you can do so using this command:
Using grep to get a call trace in this way, requires that every line in the trace contain the string that is being grepped. If you have SIP debug enabled, and you also need SIP trace information for debug purposes, it will be filtered out by the above grep commands.
Backtraces (Segfaults/Core Dumps/Asterisk Crashing)
You must run this against the version of Asterisk that is crashing. EG if you had Asterisk 13.17 and it core dumped and then you rolled back to 13.16 you can't run GDB against this as the debug tools will be on 13.16 and won't be able to correctly parse the core dump from 13.17
If you've gotten this far, you hit a rare occurrence with Asterisk in that it's randomly dying on you. The first thing to do is to determine if you are able to obtain a backtrace. To do this look for a file named core.* in your tmp directory. Please note the date and time of the core file you use as to not use an old core dump.
If you're running FreePBX Distro 7, you can simply install the 'sangoma-devel' package and then use the new 'debuginfo-install' command to download all the required debug packages. Note that you should replace the asterisk version in the command with the actual asterisk version that you are running ('asterisk11' or 'asterisk14', for example)
If you're running an older version of FreePBX Distro, you will need to install these packages manually, again replacing the asterisk version in this example with the actual version of asterisk you are running.
If the asterisk version is 13.14.0 and 14.3.0 above then use the ast_coredumper tool to obtain the backtrace.
If you want a backtrace from a running instance of Asterisk, possibly it's hung but has not crashed and not generated a core dump, you can do so using:
Otherwise you can then run the command below to obtain a backtrace.txt file which can be investigated and uploaded to the ticket
If you are unable to locate the core file, we recommend you check out the Asterisk Wiki on Obtaining Backtraces.
Uploading Logs
You can then upload your logs to pastebin.freepbx.org or by piping to the pastebin command in the distro: