Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

where 'X' stands for the wanpipe number that is showing 'disconnected'.

Please see E1-T1 Alarms for a sample output of the above command, that describes the meaning of each alarm.

...

Code Block
...
Jun 14 18:12:21 kernel: [20491.813839] wanpipe1: Warning: Excessive Fifo Errors:Resync (rx=128/tx=128)
Jun 14 18:12:21 kernel: [20491.814841] wanpipe2: Warning: Excessive Fifo Errors:Resync (rx=127/tx=127)
Jun 14 18:12:21 kernel: [20491.816119] wanpipe3: Warning: Excessive Fifo Errors:Resync
...

Overruns are caused by:

  1. Clocking: The timing of the information being received by the Sangoma Card (specific to digital cards: T1/E1, BRI)

  2. Processing interrupts: The ability of the computer system to be able to process the hardware interrupts generated by card in a synchronous fashion

Troubleshooting Interrupts

  1. Since the hardware interrupts generated by the Sangoma card are handled by the system's DMA engine, its best to run a dma engine test on the system. 

    If you have a serial-ata (SATA) hard drive or type the following in your linux cli:

    hdparm -t /dev/sda

    Code Block
    /dev/sda:
     Timing buffered disk reads:  354 MB in  3.01 seconds = 117.69 MB/sec

    If you have a parallel-ata (IDE) hard drive use

    hdparm -t /dev/hda

    Code Block
    /dev/hda:
     Timing buffered disk reads:  240 MB in  3.01 seconds =  79.73 MB/sec#hdparm  /dev/hda
    /dev/hda: multcount    = 16 (on)
     IO_support   =  0 (default 16-bit)
     unmaskirq    =  0 (off)
     using_dma    =  1 (on)   Insure this is set to "1 (on)"
     keepsettings =  0 (off)
     readonly     =  0 (off)
     readahead    = 256 (on)
     geometry     = 65535/16/63, sectors = 80026361856, start = 0

    The output above must be over 45 mb/sec

    If 'using_dma' /=1, then your hard drive is monopolizing your CPU time.

    The above test uses your hard drive to test the efficiency of the dma engine on the system.

  2. If your output is less than the required rate then you must resolve this issue before you may use Sangoma cards.

    *Note: Most likely, you might require to update the kernel chipset drivers on your system 
    ---------

  3. if the above test passes, then make certain that you do not have any PCI power save modes enabled that would prevent full capacity PCI functionality / Disable ACPI.  You also want to use/enable the APIC interrupt handler (if you run "cat /proc/interrupts" and notice anything other than IO-APIC for your wanpipe interrupts, follow this step).

    Go to your bootloader configuration file and at the end of the kernel command line and add "apic=on acpi=off".  The bootloader config file will be system dependent.  An example of what this change should look like is:

    vi /etc/grub.conf

    Code Block
    ...
    kernel /vmlinuz-2.6.18-194.el5PAE ro root=/dev/VolGroup00/LogVol00 rhgb apic=on acpi=off
    ....

    *Note: you must reboot your system in order for the above changes to be applied

    ---------

  4. If overruns persist, then there may be some other processes sharing the same IRQ as the wanpipe interfaces, hindering wanpipe performance. View the current running IRQ settings:

    cat /proc/interrupts

    Code Block
            CPU0      CPU1      CPU2      CPU3
    74:     189        0          0     67018402      PCI-MSI eth0
    90:     987        0          0        0          PCI-MSI HDA Intel
    98:     25         0       252880      0          PCI-MSI eth1
    169:    0          0          0        0          IO-APIC-level uhci_hcd:usb3
    177:    84     77660975    1187926   6290         IO-APIC-level wanpipe1, wanpipe2, wanpipe3, wanpipe4
    185:    0          0          0        0          IO-APIC-level ehci_hcd:usb1, uhci_hcd:usb5, uhci_hcd:usb8

    *NOTE: the above is an ideal view of /proc/interrupts

    If you notice any other device being shared on the same IRQ as your wanpipe interface(s), then try isolating your wanpipe(s) to their own IRQ by one of the two methods:
    -> There is potentially an option in your system BIOS to accomplish this.
    -> placing the Sangoma card in another slot, then run "cat /proc/interrupts" to verify

    If you notice that not all processors/cores are being used to handle interrupts concurrently you can adjust smp_affinity settings for your specific IRQ :
    -> Guide on how to do this here: http://www.cs.uwaterloo.ca/~brecht/servers/apic/SMP-affinity.txt------
    ---------

  5. If overruns continue, and your Sangoma Card has a Hardware Echo cancellor, and you are using Asterisk with Dahdi, try changing the Dahdi Chunk Size:
    (If your Sangoma card does not have a hardware Echo Cancellor, skip to step 5)

    To check if your card has a hardware echo cancellor run, run "wanrouter hwprobe" and check for HWEC=X

    wanrouter hwprobe

    Code Block
    -------------------------------
    | Wanpipe Hardware Probe Info |
    -------------------------------
    1 . AFT-A108-SH : SLOT=1 : BUS=5 : IRQ=177 : CPU=A : PORT=1 : HWEC=256 : V=31

    If HWEC= <anything but 0>, then you have a Hardware Echo Cancellor

    Changing the Dahdi Chunk Size reduces the amount of hardware interrupts caused by the card.  This allows your system more time to process interrupts before the next one is called.  Please see Reduce see Reduce hardware interupts & context switching by 70% to 70% to adjust your dahdi chunk size.

    *Note: This step involves re-installing the wanpipe driver.
    ------------

  6. If overruns continue, then investigate the CPU usage on an interrupt level using the "top" application
    (NOTE: only one CPU can be used per interrupt) 

    top

    Code Block
    top - 22:10:59 up 16 days,  8:03,  2 users,  load average: 0.00, 0.02, 0.00
    Tasks: 130 total,   1 running, 129 sleeping,   0 stopped,   0 zombie
    Cpu0  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
    Cpu1  :  0.0%us,  0.7%sy,  0.0%ni, 99.3%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
    Cpu2  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
    Cpu3  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st

    -> press the '1' key once in the top screen output and a few extra lines should populate near the top of the screen with information based on each CPU on the system:

    Watch the value beside "hi" for each CPU. This indicates the CPU usage for Interrupts.
    If this number is very high, then this may be the evidence that your system is not able to process interrupts in time.
    -----------
    -

  7. If overruns continue to increment we need to isolate the Sangoma cards from the TDM lines to see where the issue is being caused.

    1. Put the all the ports in loop back by create loop back plugs for all configured ports ( see HERE for loop back pin outs for your card)

    2. In all wanpipe configuration files (/etc/wanpipe/wanpipeX.conf) change the clock source from NORMAL to MASTER (TE_REF_CLOCK= MASTER)

    3. Restart wanrouter (wanrouter restart)

    The above scenario demonstrates how to allow the card to create its own clock and communicate to itself (as if there were lines plugged into the card)
    If the above scenario stops the overruns from incrementing, then the issue is due to multiple clock signals from the connected E1/T1 lines: proceed to Troubleshooting Due to Clocking from the PSTN lines.

...

  1. Watch the live output of "ifconfig" in a separate window

    watch -n 1 "ifconfig"

  2. Stop all wanpipe interface(s)

    wanrouter stop

  3. Start each wanpipe interface in ascending order, with 30 second intervals between each, and only proceed when you see "overruns" from ifconfig not-incrementing 

    wanrouter start wanpipe1
    wanrouter start wanpipe2
    .......

  4. If the next interface you start causing "overruns" to increment on that wanpipe interface only, or on all wanpipe interface, the line connected to that wanpipe interface has a different clock than the rest of the lines connected. Verify that the line is in good health by using the wanpipemon utility and the wanpipemon utility and contact your telco if there is an issue with the clock.  Otherwise, you will need to connect this line to a separate Sangoma card to meet the "1 clock per card" requirement

...

-> Missing Wanpipe requirements
    Please click on the following links to verify all requirements met:
         -> Wanpipe Requirement

 If you have verified the requirements the reason would be indicated inside the setup_drv_compile.log located in the source directory of the wanpipe driver location:
     i.e: /usr/src/wanpipe-3.X.X/setup_drv_compile.log

...