PRI-Overruns & Performance Issues

Test for Overruns

Type the following command in your Linux command line to check for the status of the  'overruns' for the Wanpipe ports (i.e. w1g1, w2g1..etc):
-> ifconfig

If the numbers beside the word 'overruns' keeps increasing (over time), this means that either there is a clock synchronization issue or your system is under performing.
     Note: It is only important if the numbers increase over time.  

To check if overruns increase over time, simply type "ifconfig" multiple times and check if there is a difference.  Or type the following watch command to view live:

Below is an output of "ifconfig" where overruns are incrementing (over time): 

If you see the same scenario happening to you, skip directly to the next section "Is this a Timing related issue or system overloaded"
If you do not see any overruns incrementing over time, continue with the following steps.

     Note: under normal operation overruns will incrementing for the first 30 second after the Sangoma Wanpipe driver is started.  Disregard any incrementing overruns during this period. 

Is my Server optimized for Sangoma cards

Your server's hardware/software configuration must be ideal when using real time TDM devices, otherwise data will be lost causing severe issues starting with noise.

Below are steps to help identify and resolve issues due to your server.  Steps 1-3 are quick system checks:

1. Run the following command in your Linux CLI to verify your system DMA engine is running efficiently. 
   DMA is used to transfer data from the Sangoma card to the kernel (and vice versa).

   Type: 
   

   hdparm -t /dev/sda

   Your output must display rates above 45mb/sec.  Anything lower means your system DMA engine is not efficient enough and your solution is to update your kernel chipset drivers

   Below is sample output of ideal conditions:

   

   Note: If you are using IDE harddrive type: hdparm -t /dev/hda, instead.
             Also ensure that your IDE hard drive is using DMA, otherwise this means your IDE hard drive is monopolizing CPU causing your issue
             To ensure DMA on IDE hard drive, type: hdparm /dev/hda
   

2. Verify that you do not have PCI power savings enabled through your server BIOS settings, which can prevent performance issues of data transfer To/From Sangoma card.
   If you do not know how to check/change this, add acpi=off in your system bootloader configuration file (as seen below)

   Also verify your system is using the proper interrupt handler (IO-APIC) for the Wanpipe ports, for the same reason mentioned above. 
   To check if you are using IO-APIC, type cat /proc/interrupts and check:

   

    

   If you are not using IO-APIC you can make the appropriate change by adding apic=on in your bootload configuration file (as seen below)
   To add acpi=off, or apic=on in your bootloader configuration file:
   -> open /etc/grub.conf
   -> add the parameters at the end of your active kernel line:

   

   bootloader with BOTH adjustments added

   Note: Server must be rebooted if to apply any of the above changes
    

3. Verify your server is balancing load across all CPU cores. 
   This can be easily checked by typing "cat /proc interrupts" in your Linux command line and verifying you see numbers across your CPU cores (which means your Input/output devices on your server are utilizing your CPU efficiently)

   cat /proc/interrupts
    

   

   Figure 1: Balanced system load

   

   Figure 2: unbalanced system load

   If your system does not have balanced interrupt load  (as referenced from the above two figures), you must correct this immediately.

   To solve this issue, install and run IRQ Balance.  This is a utility that is typically automatically installed with most RedHAT Linux distributions (i.e. CentOS).

   To install IRQ Balance, type the following in your Linux command line to auto-download from your package manager (you must have internet access for this step):
   -> For RedHAT (i.e. CentOS): yum install irqbalance
   -> For Debian/Ubuntu: apt-get install irqbalance 

   When completed, start IRQ Balance by typing: "irqbalance start"
   Check that your system load is now balanced by typing cat /proc/interrupts again.  You may require a system reboot.

   If you still have an audio issue skip to "Taking an audio trace!" 

Is this a timing related issue or system overloaded

This section will help diagnose audio issues if you see 'overruns' incrementing for the Wanpipe ports when using the "ifconfig" command.

For example:

At this point, either you have multiple clock situation, or your system is overloaded.
To diagnose, we will need to run the Wanpipe load test (below) 

Before running the Wanpipe load test, verify your clock setting configuration.

• If all your ports on your Sangoma card are connected to the telco, your Sangoma card should be set to receive the clock from the remote side (the telco)

  • In all your Wanpipe configuration files, located in /etc/wanpipe(i.e. Wanpipe1.conf, Wanpipe2.conf, Wanpipe3.conf...etc), you must have the following set:

    • TE_CLOCK= NORMAL

    • TE_REF_CLOCK= 0

      

    • This is automatically selected when configuring your Sangoma card with wancfg_dahdi 

  If all your ports on your Sangoma card are connected to another PBX (or channel bank), your Sangoma card should be set to drive the clock for the remote side

  • In all your Wanpipe configuration files, located in /etc/wanpipe (i.e. Wanpipe1.conf, Wanpipe2.conf, Wanpipe3.conf...etc), you must have the following set:

    • TE_CLOCK= MASTER

    • TE_REF_CLOCK= 0

      

• If some of your ports on your Sangoma card are connected to the telco and some to another PBX or channel bank:

  • For ports connected directly to the telco:

    • TE_CLOCK= NORMAL

    • TE_REF_CLOCK= 0

  •  For ports connected to PBX or channel bank

    • TE_CLOCK= MASTER

      • TE_REF_CLOCK= 1

  • As seen, all the ports on the Sangoma card that drive a clock for the remote side use port 1 for clock reference. Since port 1 is connected to telco, this will sync the card using 1 clock. If TE_REF_CLOCK=0 for the MASTER ports, the card would use its internal oscillator for clock source (which of course will have its own unque clock), and cause multiple clock scenario.

If your clock settings are configured properly, we need to isolate the system from the connected lines in order to identify what the issue is (overloaded system or clock issue on one of the connected ports)
We need to run the Wanpipe load test.

Wanpipe load Test
 

The following steps will isolate your server from the connected lines, and run a load test to determine if the reasons for your issues are due to your server hardware and/or configuration.

Proceed to the Wanpipe Load Test page in order to being 

Important details to understand from performing the Wanpipe load test:

• your current configuration will be overwrite, so it is very important not to forget to back up your /etc/wanpipe directory (as detailed)

• If you experience 'overruns', this means that the symptoms you are experiencing are due hardware or software configuration on your server. 

  • If your Sangoma Card has a hardware echo canceller, try changing the DAHDI chunk size, to reduce the load on your system.  This may completely resolve your issue

    • NOTE: you may also change the DAHDI chunk size if your card does not have a hardware EC, if you do not need EC in your environment (hardware or software)

  • if you have re-verified the steps at the beginning of this page, you may need to try new hardware. 

  • contact sangoma support at support.sangoma.com and create a ticket.

• If everything was successful with no errors, then most likely the reason for your symptoms is due to at clock issue on at least one of the lines connected to the Sangoma card

  • proceed to the next section if this is your scenario.

Timing related issue due to clock synchronization on the lines 

 If you have reached this point in the guide, the reason for your symptoms is due to a deviation of the clock on at least one of the lines connected to the Sangoma card.

The steps below will allow you to monitor for 'overruns' counter while starting each port on the card, one-by-one.  

There will be one port that will be started that will cause the 'overruns' counter to increment in this, and all other ports.  It is the line plugged into this port that is the culprit of your entire issue. 

1.  Stop the Wanpipe driver

   1. Type: wanrouter stop

2. Plug in all your lines into the ports on the card

3. Open a separate Window (or terminal) and type the following command in order to watch the 'overruns' counter for the Wanpipe ports, live

4. Start only the first Wanpipe port, and load into dahdi + asterisk

   1. Type: wanrouter start wanpipe1

   2. Type: dahdi_cfg 
      *ignore any error messages here.  DAHDI is simply complaining that it configured without the rest of the Wanpipe ports

   3. Type: asterisk

   4. monitor the 'overruns' for wanpipe 1 for a period of 4 minutes.

      1. Verify the overruns do not increment over the 4 minutes

         Below is an example from a test system

         

         

         NOTE: If is normal to see the overruns increment for the first 30 seconds, until the clock synchronizes 

   5. If no overruns increment within the 4 minutes, continue with the next port, Wanpipe 2

      1. type: wanrouter start wanpipe2

      2. type: dahdi_cfg

      3. monitor overruns for all started ports (wanpipe 1 and wanpipe2)

         1. Note: you may noticed a few overruns increment for these ports for the first 30 seconds, but must stop incrementing afterwards

   6. Continue the same process (as in 'e') for the rest of the ports until you see overruns sporadically increment across the ports

      1. note down the last port started 

      2. This port has caused a multiple clock scenario on the card and must be removed. If you have another Sangoma card, place this line on there to resolve the issue

      3. if you are not convinced that this line is causing the issue, move this line to another port and vise versa and see if the issue now occurs when starting that port

         1. the ports that you switch around should have same physical layer configuration to make this process easy

For more information please refer to Overruns.