High Availablity Failover

Feature Overview 

The High Availability component provides the ability to join two SBC Servers in an Active/Passive cluster with automatic migration of service in case of a hardware or network failure.  Also, allowing the system administrator to configure both servers as the primary one and copy the configuration to the secondary server with a single command.  During the migration process, established calls drop and no new calls are processed.  The migration period is very quick, typically lasting only 10-15 seconds.

High-Availability Sangoma SBCs is a group of 2 SBC Servers that support the SBC application that can be reliably utilized with a minimum of down-time. The SBC operates by using specially designed high availability software to harness the redundant SBC Server to provide continued service when system components or network connectivity fail. Without clustering, if a SBC Server crashes, the SIP Trunking, Remote Phone or other Call Flow applications will be unavailable until the crashed SBC Server is fixed.  HA resolves this situation by detecting hardware, software, and network faults, and immediately migrates the SBC application on another SBC Server without requiring administrative intervention, a process known as Failover. As part of this process, the SBC application shares the same configuration and uses common "Floating" IP Addresses to create a singular SBC application.  The Sangoma SBC makes use of Floating IPs, these are created for each of the network segments (WAN, LAN, DMZ) connected to the SBCs.  These Floating IPs are common/shared on both Primary and Secondary SBCs as the common point of entry and exit for SIP Call Flow.  These Floating IPs allow for the Failover to occur, when the Master Node fails, the Salve Node becomes the Master Node and take Active control of the Floating IPs.

SBC HA clusters are often used for critical Trunking and Remote Phone applications on a Carrier network, Enterprise Network, and various VoIP business applications.  SBC HA cluster implementations add redundancy to eliminate single points of failure, including multiple network connections and Call Flow applications.  There is a Dedicated Network for the purpose of a "heartbeat" which is used to monitor the health and status of the other SBC Server in the cluster, as well as copy the configuration to the Secondary server within the SBC application.

Requirements

These are the minimum requirements for proper operation of High Availability on the Sangoma SBC,

1. Minimum Software release 2.3

2. The cluster requires one Dedicated Ethernet Network Link between both SBC Appliances and one Ethernet Network Link as a Backup Link.  The Backup Link can be either a Second Dedicated Link or one Ethernet Interface connected to an internal network subnet, where both servers can reach each other on the same Subnet via a Multicast link

3. The High Availability feature is only supported on Sangoma SBC Appliances and SBC VM Software with 3 Ethernet Network Links or more.

4. When using SBC VMs, creating the VMs both VMs have to have the same number of interfaces since first boot, same exact number Ethernet physical interfaces, number assignment (Eth0, Eth1...) and default gateway configuration.

5. Multi Service Business Gateway (MSBG, SBC with Telecom Cards) is not Supported.

Site Planning

Pick your Network Topology

There are a large number of Network Topologies that the Sangoma SBC supports.  Some of the most common are:

1. WAN - LAN : One Ethernet Interface on the Public WAN - One Ethernet Interface on the Private LAN

2. DMZ - LAN : One Ethernet Interface on the Public/Private DMZ - One Ethernet Interface on the Private LAN

3. DMZ : One Ethernet Interface on the Public/Private DMZ

4. LAN : One Ethernet Interface on the Private LAN

IP Address Assignment

For the specific application of HA Failover, there is an increase of IP Address usage. 

1. Each of the Primary and Secondary SBC Servers have their own unique IP Address for direct access to the specific SBC server.

2. When LAN and WAN interfaces are needed, each of the Primary and Secondary SBC Servers have their own unique WAN IP Address for Default Gateway assignment

3. Floating IP Addresses are created for each SIP Signalling interface (SIP application bound to a shared IP address).  And these Floating IPs are created for each of the network segments (WAN, LAN, DMZ) connected to the SBCs.  These Floating IPs are common/shared on both Primary and Secondary SBCs as the common point of entry and exit for SIP Call Flow.  These Floating IPs allow for the Failover to occur, when the Master Node fails, the Salve Node becomes the Master Node and take Active control of the Floating IPs.

4. You may need more than one Floating IP for different applications.  SIP Trunking may us one Floating IP, and Remote Phones may use another Floating IP.  On each LAN and WAN side. (or at least different SIP Port usage)

5. A common Dedicated Network Interface is created on each of the Primary and Secondary Server, each with it's own IP Address within a unique network segment.

Open

Configuration

Within these instructions, SBC Servers which are being configured will be referenced as Primary - the SBC Server where all the common configuration is performed - and Secondary - the one receiving the common configuration via copying the configuration from the Primary server.  The SBC Server which are part of a configured and running cluster will be referenced as Nodes - either Master Node - for the SBC Server where resources are actively running, or Slave Node - for the SBC Server acting as a Failover Server. This is the same nomenclature used by the user interface and at the system logs within the SBC configuration.

The following steps assume the SBC is "out-of-the-box", without any previous configuration or operation.

Configuring the Primary SBC Server

First Time LogIn

By default, Sangoma's SBC Appliances have the following Default IP Settings.

IP: 192.168.168.2
Netmask: 255.255.255.0
Gateway: N/A

You can login to the web interface by going to http://192.168.168.2/ and login using the following default credentials:

Username: root
Password: sangoma

License

It is important to note there is no special licensing for HA Failover on the Sangoma SBC products.

Sangoma SBC Appliances should come already pre-installed with Max-Calls licenses.  The SBC VM Software will require a License Key and process to bind the key to a MAC Address.

Configure Network Settings 

Configuration  | IP Setting | Network

The Interface tab is where all of the network interfaces on the SBC.  Local Host (lo), DSPs (sngdsp), and physical Ethernet Interfaces (ethX). You see the MAC address and Network Statuses.  You may "Edit" any of the physical Ethernet Interfaces to change Ethernet Speed and Duplex.

The IP tab is where IP Addresses, Subnet Mask, and IP versions are assigned to physical Ethernet Interfaces.

The Network box is where the Default Gateway and DNS Servers are assigned.

Open

Primary Server interface is the Ethernet Interface and IP Address of the Primary Server.  Not used for SIP Call Processing, simply to be used as direct administration to the Primary Server.

Dedicated Network Interface is the Ethernet Interface and IP Address of the Primary Server to be used as the link between both SBC Appliances for the purpose of a "heartbeat" which is used to monitor the health and status of the other SBC Server in the cluster, as well as copying the configuration within the SBC application.  This Ethernet Interface and IP Address MUST use a completely separate Subnet that does not conflict with standard network routing of the system.  For example, does not conflict with WAN, LAN and DMZ network segment scopes.  For a more specific example, if you are using 192.168.0.0/24 on the LAN, use 10.0.0.0/30 on the Dedicated Network Interface.  /30 Mask works well as only two IP Addresses are needed, one for each SBC Server.

Media interfaces are special DSP's (Digital Signal Processor) which are accessible through any Ethernet network any of the SBC interfaces are attached to. These media interfaces are often embedded within a Sangoma PCI(e) card (ie D500, D100 devices) and in a SBC VM Software Hybrid, the DSPs are completely stand-alone processors that are just connected to the same network (D150).

Configure "Primary Server Interface"

Configuration  | IP Setting | Network

In the IP tab, click on "ADD" to add the IP Address of the Primary Servers Interfaces for administration. You can click "Edit" for each network interface you want to reconfigure.

Configure the IP Address

1. Select the Physical Ethernet Interface,

   1. LAN side Ethernet interface

   2. Optional - depending on SBC setup in the network topology.  When completing adding the LAN side, Also add an additional WAN side Ethernet Interface if needed.

2. IPv4 or IPv6 type

3. Enter the IP Address and Mask of the Primary Server

Example of LAN Side

Example of WAN Side

Configure "Primary Dedicated Network Interface"

Again, this is the unique network between the Primary Server and the Secondary Server.  An Ethernet Cable is connected between these two Ethernet Interfaces.

See picture above for the different network connections.  Plan the IP Addresses requirements.

Configuration  | IP Setting | Network

In the IP tab, click on "ADD" to add the IP Address of the Primary Dedicated Network Interfaces for "Heartbeat", copying configuration to the Peer node and Status.

Configure the IP Address

1. Select the Physical Ethernet Interface, any Interface that is assigned to be the "heartbeat" interface.

2. IPv4 or IPv6 type

3. Enter the Non-conflicting IP Address and Mask of the Primary Dedicated Network Interface - it is helpful to keep the network scope small since there are only two IPs in use.

Configure Default Gateway and DNS Servers

From the Network box, you can also set the hostname, default gateway and the DNS servers. If you use DHCP for any of the interfaces you won't be able to specify a default gateway or DNS servers.

The Default Gateway is always the way to the Internet.

1. Hostname: Assign a unique Host Name for the Primary SBC.   This name is used to identify the Node in the HA Cluster please refer to Hostname Change section.

2. Default Gateway Interface: Select the Ethernet Interface connected to the WAN or Internet

3. Default Gateway IPV4 (or IPV6) Gateway: Assign the default Gateway on the Interface selected above

4. Static DNS: Assign DNS Servers

5. Press Save

 Hostname Change

Steps to follow while modifying hostname:

1. Goto Configuration>IP Settings>Network and click on Edit under Network section.

2. Update the host name and save the changes.

3. Now click on Apply Network and after applying, click on Restart Network.

4. Once the Network is restarted, we need to reboot the SBC in order to change the hostname.

Configure Media Interfaces

Configuration | IP Settings | Media Interfaces

Media interfaces are the DSPs that perform RTP streaming, transcoding, and SRTP encryption. These media interfaces are also network devices and therefore require IP configuration (IP Address, Netmask, Gateway etc).  For the Sangoma Appliances using a D100 (media interface without an external ethernet port) the IP address assigned can be any IP because the interface will remain "Hardware Hidden" within the appliance and the RTP packets end up using the IP of the signaling network interfaces.

The first step to configure media interfaces is select the media mode in which NSC will operate. There is 3 media interface IP modes: "Hardware Hidden", "Hardware Exposed" and "Software".  When using Sangoma SBC Appliances the recommended setting is "Hardware Hidden".

The "Hidden" mode is simpler to operate. In this mode all the media interfaces are hidden within the system and all the IP traffic generated by the media interfaces is routed/forwarded through the SBC host operating system and NATed. This mode is simpler because you don't have to worry about multiple IP addresses for your media interfaces. The media interfaces will still need an IP, the SBC automatically assigns Private no routable IPs.

The "Exposed" mode is typically used in SBC VM Hybrid Software, when the D150 is external to the VM Server.  This requires more careful configuration as the media interfaces will be exposed to your network.

The "Software" mode is used in the SMB SBC Appliance, and the SBC VM Software.  When there is no DSP card present.

Detect Modules

The first time you modify the media interfaces configuration you must go through a discovery procedure to find all media interfaces.

Press "Detect Modules", when discovered the DSP cards are assigned and provisioned with IP Addresses.

If you select the "Exposed" IP mode, the web ui will allow you to configure the IP settings for the media interfaces it finds.  In "Hidden" mode you are only asked to provide a starting UDP port range for the RTP streams. You can leave the default if you don't require a particular port range.

Once you click "Save", the web ui will perform the device discovery procedure which will take a few seconds. The discovery procedure will send ethernet broadcast messages to auto-discover Sangoma media interfaces attached to the same network(s) of the selected ethernet interfaces. Once done, you will receive a report of the hardware found.

Configure Placeholder SIP Profile

In order to successfully Apply a SBC configuration there needs to a SIP Profile.  In this step, we are configuring a place-holder SIP Profile to be deleted later.  Typically there is a Default Profile that is pre-configured, this is fine to continue to use.  Or if the IP Address or Ethernet port has changed, please modify the SIP Profile to the new Interface and IP Address.

Configuration | Signaling | SIP Profiles

Ensure there is a Name for the Profile and the new Ethernet Interface and IP Address is selected.

Apply the Configuration 

Configuration | Management | Apply

The SBC Configuration is not nearly complete at this stage, Simply, there is enough configuration to APPLY the settings to the SBC. This will update any IP Addresses and DSP configurations.

Configuring the Secondary SBC Server

This process initially starts as commissioning an independent SBC Server that is going to be identified as the Secondary Server.  The initial goal is to complete the commissioning steps to result in having a completely new and unique SBC Server that uses a different Secondary Server Interface, a corresponding peer for the Dedicated Network Interface, Media Interface and placeholder SIP Profile.

Follow the same steps as above to commission the Secondary SBC Server.

First Time LogIn

License

Configure Network Settings 

Configure "Secondary Server Interface"

Using a different IP Address than the Primary Server.  The Secondary Server will most likely be located on the same Subnet as the primary, but needs a different IP Address.

Configuration  | IP Setting | Network

In the IP tab, click on "ADD" to add the IP Address of the Secondary Servers Interfaces for administration. You can click "Edit" for each network interface you want to reconfigure.

Configure the IP Address

1. Select the Physical Ethernet Interface,

   1. LAN side Ethernet interface

   2. Optional - depending on SBC setup in the network topology.  When completing adding the LAN side, Also add an additional WAN side Ethernet Interface if needed.

2. IPv4 or IPv6 type

3. Enter the IP Address and Mask of the Secondary Server

Secondary LAN Example

Secondary WAN Example

Configure "Secondary Dedicated Network Interface"

Again, this is the unique network between the Primary Server and the Secondary Server.  A Ethernet Cable is connected between these two Ethernet Interfaces.  Here we are configuring the Secondary Server.

Configuration  | IP Setting | Network

In the IP tab, click on "ADD" to add the IP Address of the Secondary Dedicated Network Interfaces for "Heartbeat", copying of configuration to the Secondary server and Status. You can click "Edit" for each network interface you want to reconfigure.

Configure the IP Address

1. Select the Physical Ethernet Interface, any Interface that is assigned to be the "heartbeat" interface.

2. IPv4 or IPv6 type

Enter the Non-conflicting IP Address and Mask of the Secondary Dedicated Network Interface - it is helpful to keep the network scope small since there are only two IPs in use.

 Hostname Change

Steps to follow while modifying hostname:

1. Goto Configuration>IP Settings>Network and click on Edit under Network section.

2. Update the host name and save the changes.

3. Now click on Apply Network and after applying, click on Restart Network.

4. Once Network is restarted, we need to reboot the SBC to change the hostname.

Configure Default Gateway and DNS Servers

Configure Media Interfaces

Detect Modules

Configure Placeholder SIP Profile

Apply the Configuration

Registering the Secondary SBC Server

After initial configuration on both the Primary Server and Secondary Server and applying the Network aspect of the configuration, the next step is registering the Secondary Server on the Primary Server.

The process of Registering the Secondary Server to the Primary Server provides the Primary Server the IP Addresses where the Secondary can be reached and authorization keys for the enabling of access from the Primary.

Generating an Access Key

On the the Secondary Server, the Access Key is generated with Configuration | High Availability | Settings

Configuration | High Availability | Settings

Click on Generate Access Key

A prompt asking to create a configuration backup will appear. Click Ok to create the backup.

Once the backup is successful, another prompt will appear to generate the access key. Click Ok to generate the access key.

• Copy the encoded Access Key text from the text box into Clipboard or Notepad or TextEditor

Adding a Peer on the Primary Server

On the Primary Server, go to the Configuration | High Availability | Settings and add the Secondary Server as a Peer Node with the Access Key.

Configuration | High Availability | Settings

Under Peer Nodes, click on Add.

A prompt asking to create a configuration backup will appear. Click Create to create the backup.

Once the backup is successful, another prompt will appear to copy the access key generated by the Secondary Server. Write a meaningful name
that best describes the Secondary Server as the other node and then paste the encoded Access Key text obtained from the Secondary Server.

The system will be registered locally and show on the Peer nodes list - its configuration can be viewed or modified by clicking Modify on the right side of the row.

Configuring HA on the Primary SBC Server

With the Secondary Server registered,

Configure Floating IPs

A Floating IP is an IP Address used as a commonly shared IP Address between the Primary and Secondary Servers.  When the Master Node fails - for whatever reason, the Slave Node takes over as the Master Node using the assigned Floating IPs.  These Floating IPs are defined in the HA Settings configuration and not in the Network Interfaces.  Then these Floating IPs are assigned in the various SIP Profiles used for applications such as SIP Trunking and Remote Phones. 

Configuration | High Availability | Settings

Clicking on Add button at the "Floating IPs" box;

Choosing a meaningful name for the address (ex: External_Trunking_IP)

and then configuring the address.

Repeat these steps for all IP Addressed and Interfaces where the SBC and SIP Profile will be reachable via SIP protocol.  Common applications would be to have One External WAN Trunking IP Address and another Internal LAN Trunking IP Address.

Enable High Availability

In this section, we select the Dedicated Network Interface and the Backup Network Interface and turn on Active/Passive HA.

Configuration | High Availability | Settings

The High Availability feature on the Primary can then be enabled by:

On the High Availability box, click Edit button.

• Dedicated Network / Interface: Select the Ethernet Interface configured earlier for the Dedicated Network - this is the unique network used for "Heartbeat"

• Backup Network / Interface: Select the Ethernet Interface that will be used for the Backup "Heartbeat", typically this is the LAN interface.

• Operational Mode: Select Active/Passive

• Name for current node on peers:  Optionally, selecting a meaningful name for the other Node

• Click on Save button. (all other defaults are ok to leave default)

For instance,  if eth4 is used as a Dedicated Network / Interface and is also a direct cable connection between the Primary and Secondary servers then it must be removed from the Failover triggers table. If the Failover triggers  is not removed, in the instance when the Slave Node goes down due to a reboot for instance, it will cause the Master node to temporarily lost NSC service until the Slave node's Cluster service is started.

Floating IP Assignment in SIP Profiles on Primary SBC Server

Now to continue the configuration of the SBC.  We last left the Primary Server with a "place-holder" SIP Profile. Delete it and make the SIP Profiles we are going to use.

Configuration | Signaling | SIP Profiles

Create SIP Profile

Configuration | Signaling | SIP Profiles

Click Add

• Display Name: Define a name that best represents the purpose of the SIP Profile

• SIP IP Address: Select the Floating IP that will be used for this Profile.

• RTP IP Address: (OPTIONAL) Select (SIP Profile) to use the SIP IP Address, or select a different Floating IP that will be used the RTP IP Address.

There may be many SIP Profiles with each there own unique Floating IP Address.

You may Apply the Configuration as this point, knowing that there is plenty of configuration remaining to correct operation.

Complete the Configuration

Network Configuration, Media Interfaces, and SIP Profile configuration are not enough to complete the configuration of a SBC.  The following sections of configuration are required at a minimum to complete the configuration of the SBC.

Configuration | Signaling | Domains

Configuration | Signaling | SIP Trunks

Configuration | Media | Media Profiles

Configuration | Routing | Call Routing

Once completed the configuration of the SBC,  Apply the Configuration.

Apply the Configuration

Copy configuration from the Primary to the Secondary Server

Once you have successfully applied the configuration, you will be redirected to the Control Panel, you will see a warning message "Configuration has not been copied to Peer".

This means that the configuration changes on the Primary server has not yet been copied over to the Secondary server.

Overview | Dashboard | Control Panel

Clicking on Copy configuration to Peer button will start copying the configuration from the Primary to the Secondary server, where all configuration will be copied over, including all the user interface settings and service startup information.  This process may take a few minutes, please be patient.

After both servers are configured and the configuration has been copied over, the cluster is now fully configured and ready to start operation.

Starting the HA Cluster

Overview | Dashboard | Control Panel

Clicking on Start All Nodes button will start the process joining the HA.  When complete, there will be a Master Node and a Slave Node.  The Master Node is the SBC unit that is actively processing calls, whereas the Slave Node is monitoring the Master Node.

After pressing Start All Nodes, the required Application Services are started.  Services managed directly by the cluster are called Cluster Resources, and will be shown on the Control Panel as soon as the cluster services are initialized.

Application Services will show Running on the Master Node, identified in the appropriate column.

If not, please refer to the Troubleshooting section.

If the system was already configured before, the SIP profiles bound to external IP addresses need to be re-configured to use the newly created floating IPs, and any IP address hardcoded in any part of the system (including the dialplan) will have to be changed to use the floating IP address.

SBC VM Ethernet Manual Setup: dnsmasq is **required** for HA. The dnsmasq config file can be modified if needed, provided it doesn't interfere with any other configuration that is already in place. **If it is ever disabled, things will stop working**.  it's OK to add entries to dnsmasq, but it is NOT OK to bypass it.

Management

Cluster management is performed at at the Control Panel page: when the High Availability module is enabled, this page is augmented with information from all nodes in the cluster and from the common cluster state, the later available when the Cluster Management service is active.

The first table - Cluster Nodes - shows the current configured nodes in the cluster and their state, which are described below:

Normal operational states:

• STOPPED: Cluster Management (clustermon) service is not running on this node;

• STARTING: clustermon service is starting, cluster services not running yet;

• INITIALIZING: clustermon service was started and cluster is being initialized;

• JOINING: clustermon is running and node is attempting to join the cluster;

• ONLINE: node is part of the cluster and its fully operational;

Failure and non-operational states:

• DISABLED: node is disabled mode, either from manual intervention or from a failover trigger;

• OFFLINE or NOT JOINED: clustermon is running but node was unable to join the cluster;

• UNREACHABLE: node is not part of the cluster and could not be reached;

• FAILING: node is part of the cluster but could not be reached - this usually indicates a node that is about to fail;

• UNMONITORED: node is part of the cluster but clustermon is not running - which mean this node would not be able to failover.

Additional information can also be displayed after the state name, as a single letter between parenthesis, which will have the following meaning:

• (C): cluster is being reconfigured;

• (T): Cluster Management service (clustermon) is being terminated.

The second table - Cluster Resources - shows the resources/services which are directly managed by the cluster, which includes the NetBorder Session Controller service and the floating IPs, and which can be in the following states:

• STOPPED: service was not started and is not currently running;

• STARTING: service was requested to start and cluster is proceeding to start it on first available node;

• STARTED: service was successfully started and is currently running;

• STOPPING: service was requested to stop and cluster is proceeding to stop it on the node where it currently runs;

• NOT RUNNABLE: service was requested to start but there are no nodes available for running it;

• NOT RUNNING: an error happened and the service was not started, the cluster will attempt to start the service again within the specified failure timeouts.

For starting the cluster services, the recommended way is to start all nodes from the Control Panel, clicking at the Start all nodes button.

This will start the management service on all nodes and initialize the cluster, which will elect a node for assigning the floating IPs and start the NetBorder Session Controller service. The node where these services run will be the master node, and the other will be the slave node.

Alternatively, the cluster services can be started individually on each node on the Cluster Nodes box, by selecting Start cluster service action on right side of the node's row and clicking on Execute.

Services managed directly by the cluster are called Cluster Resources, and will be shown on the Control Panel as soon as the cluster services are initialized.

NOTE: Services SIP Security Monitor, Media Firewall and RTCP Monitor are automatically started by NetBorder Session Controller on the master node and should not be started on all nodes at the same time.

HA SSH Firewall Automatic Whitelist

During the setup of an HA environment, on the Cluster Nodes when there is an existing IP address in the SSH Firewall Rule Whitelist the IP addresses of its Peer node will be added to the list.

In this example, the IP address 192.168.77.24/32 is added to the SSH Firewall Rule Whitelist on the HA Master node. After this IP address is added, the other IP addresses of the Peer node is automatically added.

When there is any changes to the Cluster Nodes' IP addresses and the system under goes a Copy configuration to Peer node operation, the SSH Firewall Rule Whitelist will add the changed IP address of it's Peer node on top of the previous IP address. This old IP address after the change is still persistent and should be deleted it since it is not being used by the system.

In this example, on the HA Master node the Network IP address 10.253.253.254/32 was the new change and was added to the SSH Firewall Rule Whitelist after Copying configuration to Slave. Since 10.254.254.254/32 is not used anymore it should be deleted on the whitelist.

HA Upgrade Guide

Please refer to the HA Upgrade Guide, to upgrade a pair of SBC servers in High Availability setup