Audio Issues

The following information will help you troubleshoot general audio issues experienced during inbound and outbound call scenarios.

For the purpose of this guide, we will be using the following Network diagram. Your network setup may be different but the troubleshooting steps still apply:

Open

Basic Steps

Whenever you experience an audio issue on calls, the first step is to verify the bad audio is not being generated by the device you are calling or listening from.

The best way to begin troubleshooting is to reduce call load down to only your test call. This way its easier to identify and diagnose the the problem.  If you have many calls flowing through your system while trying to troubleshoot it will be very difficult.

Also, when reproducing the issue, always use the same device(s) and the same call path.  This means always call the same phone number from the same device and to the same device. Again this will help us target the issue more quickly.

Verify with another handset/device

Verify speakerphone is NOT being used, which can cause echo and distortion bouncing of environmental items.

Wireshark trace

The first step to diagnosing your audio issue is to take a wireshark trace on the sip side of your network. Since this is a quickest and easiest steps do it first to attempt to isolate the issue.

The wireshark trace on the sip side will identify if the bad audio is coming from any of the sip devices on your network, or if the bad audio is being generated somewhere on the SS7 signaling side. The below diagram shows where the wireshark trace should be taken from:

Open

There are many areas on the sip side where the wireshark can be taken from.  The best place to take it from is on the NSG SS7 Gateway since it has the built-in wireshark trace utility.

This is not to say that the issue is due to something on NSG Gateway, but that it provides troubleshooting capabilities to diagnose where ever the issue is.

• Log into your NSG Gateway webgui by typing the IP address of NSG Gateway in a web browser.

• Select Reports > Network Capture

• uncheck 'All Network Interfaces' and select the ethernet interface that has a cable connected to it going to your sip side device
  
  
  

• Click the Capture button at the bottom to begin tracing on the sip side
  
  

• Reproduce your issue

• Stop the wireshark trace by click on the Stop Capture
  
  

• Download the wireshark trace to your local machine by clicking on the Download button  
  
  

  • The downloaded file will be called prot-capture.tgz

• Open the archived file which will show you the wireshark trace you captured.

  • it will be called 'prot-capture-eth0.pcap'

• To open your wireshark trace, make sure you have wireshark on your local system then double click on the file which will open up in wireshark

  • If you don't have wireshark, you can download it from here: https://www.wireshark.org/download.html

Now its time to listen to the audio from within the wireshark trace.

• Within wireshark click on Telephony > VoIP Calls

• Highlight each row that shows up (each row is one leg of the call) by holding the the CTRL key on your keyboard as you click on them:
  
  

• Click on Play Streams located at the bottom of the window
  
  

You will now be presented with the audio streams of your call.  

You will be able to play each stream separately. Each stream is the audio from one side of the call. For example, one stream will be the audio from your SIP endpoints towards the NSG Gateway and the other stream will be the audio from the SS7 side of NSG towards your SIP endpoints.  Listen to both of them

If the audio stream from the SIP end points towards NSG sounds bad, the issue is being caused by any of your sip end points devices. If  this is your case, you have fully diagnosed your audio issue and do NOT continue with the information below.

If the audio from the NSG towards your sip end points sounds bad, then we need to continue troubleshooting. If this is your case, continue with the next section.

Ftdm Trace

NSG Gateway has a built-in feature to capture an audio trace from within the software.  This audio trace is called Ftdm trace.  Taking an Ftdm trace is useful to verify if there is bad audio coming from the SS7 telco lines or anywhere between the cables connected to NSG up to the NSG Software.

To take an ftdm trace, log into the SSH console of your NSG Gateway:

• Use any terminal application of your choice. For example, we use putty (which can be easily downloaded)

• Type the IP address of your NSG Gateway in the host section
  
  

   

•  Type in the command to enable the ftdm trace, taking care to fill in the correct parameters based on which channel and span the test call will be on:
  
  ftdm trace <path> <span_id> <chan_id> 
  
  NOTE: The <path> parameter includes both the path for output recordings and the prefix for naming the files. Eg. /root/test will output files named test-in-s1c1 and test-out-s1c1 in /root for example.
  
  Example: if this is the only test call on the NSG Gateway the parameter will be 1 1 (for span 1 chan 1)
  -> ftdm trace /root/test 1 1 
   

• After running the above command the trace will begin

• Make your test call and reproduce the issue

• once completed, type the following to disable the ftdm trace 
  -> ftdm notrace <span_id> <chan_id> 

If you are unable to identify which channel the call is being made on, you can block all the channels accept the one in question. To do this follow the procedural commands found here: Channel Based Operation#Blocking/UnblockingVoiceCICs

Analysing Recordings using Audacity

Once you have the recordings from the NSG Gateway you can open them in a sound editor such as Audacity.  The files output are raw audio files. They don't have a container which specifies the codecs and formats, so you'll have to tell Audacity how to play them. Open up Audacity and select a stream to import by selecting File->Import->Raw Data... Browse to the recording files you took from the NSG Gateway. You'll have to import the Tx and Rx streams one at a time. When you select the file, Audacity will ask you for some information about the audio stream. Use the following settings:

Here is a sample of what you will see when you open your streams in audacity:

If the RX stream has bad audio this verifies that the bad audio is coming from the SS7 telco or the connection of the SS7 lines to the NSG Gateway (meaning cable related issue).
If the TX stream is  bad, this means the audio from the SIP side towards NSG Gateway is bad.

Taking an SS7 line trace 

Taking an SS7 line trace is great to capture the audio that comes directly from the telco ss7 lines plugged into NSG

• Log into the command line of the NSG Gateway

• Type the following command to begin a trace
  
  -> wan_ec_client wanpipe1 monitor <channel number>
       <channel number> is a channel number obtained by viewing the TDM Status: Operating the Gateway#TDMStatus

• wanpipe1 is port 1. If the call is on port 2, replace with wanpipe2 (and so on..).

• the utility wan_ec_client will save an audio record inside the local directory (where you ran the trace) 

   If your audio issues requires a longer trace to be taken, use the 2 minute trace, seen below:
  To record for 2 minutes run:
        ->  wanpipe1 monitor120 <channel number>

The details of the above syntax is the same as for the 15 second trace

Verifying NSG Gateway System Status 

If the audio issues are being generated by the NSG Gateway, the below information should help diagnose

Verify cable strength

If the cables are not connected properly to NSG, even a slight drop in cable strength can cause audio issues to and from NSG Gateway.

• To verify cable strength navigate to Overview > TDM Status
  
  

   

• Hover the mouse over the port you wish to check. For example, hover the mouse over port 1 and a small window will pop up to display line strength status
  
  

   

• The line stength is indicated at the bottom where it says Rx Level :> -2.5db.  If the line strength says anything different than 2.5 db, this can be the cause of your audio issues. To fix this, try to re-make the cable by crimping an new rj45 connector to it and re-test

Verifying Clocking

This trouble shooting page assumes that "wanrouter status" command displays that port is in Connected state.

Once the wanpipe port is connected, the voice/b-channels start transmitting and receiving data.  At the interrupt level, the card is fully operational and voice DMA's are passing data to and from the memory. In order to confirm that wanpipe port interrupts are performing up to spec without irq slips, we must view ifconfig statistics.

Note that in voice communications the interrupts occur at 10ms intervals and are handed by a single core at a time. This means that an 8 core system would not help in processing interrupt requests from a single card.  As a single card is attached to a single PCI interrupt.

Ifconfig 

The ifconfig command will display all network interfaces.
Sangoma network interfaces start with "w".  
w1g1 - indicates span 1
w2g1 - indicates span 2

Reset ifconfig statistics: execute stats command once
-> /etc/wanpipe/scripts/stats.sh clear

Execute "ifconfig" command repeatedly every 1 sec in Shell Command:
-> ifconfig 

Alternatively

Execute above stats command
-> /etc/wanpipe/scripts/stats.sh

Look for "overrun" statistics in w* interfaces.
Confirm that overrun statistics are NOT INCREMENTING.
Its ok if few overruns are present as overruns are normal until all ports become connected.
However after "connected" condition there should be no more overruns increasing.

What we are looking for here, is that overruns are not incrementing every few seconds.

In case the overrun counters are incrementing:

1. System cannot handle the voice 10ms interrupt load

2. System is misconfigured.  Kernel does not support the chipset.

3. Run:
   -> cat /proc/interrupts
   -> Check that all interrupts are running as IO-APIC.  
       Make sure there are no X-PIC interrupts.

4. Overrun Troubleshooting