Troubleshooting Techniques

Work with whatever tools you are comfortable with

You will notice that there are a mix of command-line linux commands and applications specifically referenced in this page for troubleshooting NCA issues. This is because these are the specific tools that we have become comfortable using to troubleshoot various NCA issues. The worst mistake anyone can make is assuming that the exact steps outlined here are the only way to analyze NCA logs. When verbose logging is enabled, NCA produces an incredible amount of logging information - so much that it is very easy to get overwhelmed and not know how to start to find the root cause of an issue. The important thing to take away from these steps is the overall process to take to isolate different types of issues.

Linux and "grep"

grep -rH "mystring" $(pwd)

    *Note: replace "mystring"

This useful command will recursively search all files from the current directory for a specific string.  This is a very useful tool for when you are experiencing some errors and you are looking to see how many different calls experience the same issue. 

When NCA call-logging is enabled, it will create a different file for every call. 

By using the "-rH" flags the grep command will show the filename alongside every instance of the string, and you can very quickly take a glance at the output to see if this is a major issue effecting most of your calls, or if it is isolated to just a small portion of your total call volume.

When using this command, it is most efficient to run the command from the /logs/call-logs directory, however there are times when you want to run the command on a smaller portion of your total calls.  This can easily be accomplished by changing directories into the /logs/call-logs/<year>/<month>/<day> or even the /logs/call-logs/<year>/<month>/<day>/<hour> folder so that the command only picks up on the logs for the day or hour.

Wireshark

Wireshark cannot actually interpret any of the NCA logs, however by taking a .pcap trace of the ethernet traffic on the machine running NCA, it is possible to use Wiresharks powerful filtering tools to quickly hone in on an issue with either the SIP signalling, or identify an issue with the RTP.  There is no information present in a .pcap trace that cannot be found in the verbose NCA logs, but it can be a useful tool if you are comfortable and familiar with it.

In all newer versions of wireshark, if you go to Telephony > Voip Calls, you will get a popup window with every SIP call that was initiated after the capture had started.  By selecting both the inbound dialog between the Dialer and NCA, and the outbound dialog between NCA and the intended destination and clicking on "follow", Wireshark will create a call-flow-diagram that shows in chronological order all SIP messages for a single outbound call attempt from the Dialer.  Additionally, if you click on "player" you can listen to the RTP that is hitting the system for analysis, and you can check to see if there are any irregularities such as excessive jitter or packet loss that might explain an issue with NCA's accuracy.

Note:  There is specific information in the NCA logs that is not found in a pcap trace, so you cannot expect to fully debug an NCA issue with only a pcap trace.

Call-Recordings and Audacity

There is a lot information to be gained by enabling call recording and opening up the files in a program like Audacity that allows you to look at the spectrum of an audio file.

Most importantly, the audio recorded in these files are the exact audio packets as NCA receives them.  This means that when listening to these files you are essentially analyzing the exact information that NCA used to come up with it's disposition.  By isolating potential issues both by visually inspecting the spectrum and by listening o the playback of the audio, it is possible to come up with logical hypotheses concerning why NCA ended up with any given disposition.

By opening up many audio files at the same time as separate "tracks" in the application, you can quickly inspect the spectrum analysis and overall length of the file for any similarities.  This is specifically useful for confirming the issue when a single 'Answering Machine' message is being hit many times, and is causing skewed results.

diff in "vi" or "notepad++"

If you have the logs of a "failed" call as well as well as logs from a "good" call, by doing a diff between the two files and looking for large chunks of the file that are present in one that are not present in the other, you can quickly isolate probable locations in the file where you will find information about why the bad call failed.  This happens because when NCA operates as expected almost every log file will follow the same format, being almost the exact same length and having the SIP Messages and internal logs coming in the same order.

Know your RFC's

SIP is an extensive protocol that is comprised of many different RFC's, none of which have been universally standardized.

NCA only conforms to RFC 3261, which is just a small subsection of the commonly implemented RFC's of todays new SIP devices.  It is possible to have a fully functioning SIP network that uses signalling that NCA does not support.  In this case, in order to fully integrate NCA into the solution steps will have to be made to rework the network to allow for NCA to be inserted into the signalling path and ensure that it receives only supported methods.

Pen and Paper

There is no substitute to drawing out a full SIP call-flow-diagram similar to those found on the NCA and NAT page, the NCA user guide, and on most documentation explaining SIP messages (including the RFC documention).  There is a good reason why this format has become standardized for representing SIP call flows, as it provides a huge amount of information in a human readable format.  If you are unable to find an issue by going through the NCA call-logs in their text readable format, spending the time to draw out a rough call-flow-diagram makes the information much easier to parse and to gain with useful insights about the operation of NCA from that information.