This information on this page applies only to PBX versions 15.x and above. For version 14.x or lower please see: Warm Spare Setup
Introduction
This guide will walk you through the process of setting up a warm spare server.
APPLICATION NOTE
You will need 2 PBX servers of the same model with identical hardware including analog and digital cards.
This article assumes the following:
You have an existing PBX system that will be your primary server.
You have an identical PBX system that will be your secondary server.
The two servers can communicate on an IP level.
The Warm Spare setup on PBX 15+ has changed from the way it used to be in older versions.
We are now using the Filestore , API and Backup & Restore modules for the Warm spare Backup configuration, so we need to ensure that all 3 mentioned modules are present in both Primary and Secondary server.
We have to follow below mentioned steps in order to setup PBX 15+ Warmspare setup.
NOTE - one primary difference in this latest iteration of warm spare functionality is that the majority of the configuration is done on the PRIMARY system and not the Warmspare!
How to add the Warm Spare Server on the Primary Server
the process needs to send data from the primary to the warmspare. We can either use FTP or SSH protocol to perform this task using Filestore module (SSH is the recommended path)
SSH Configuration
In this section, we will describe how to configure "SSH" to use for Warm Spare setup.
First, we will set up shared keys between the two servers so they can communicate across SSH on port 22. PBX 15+ onward there is no need of generating SSH Keys for the Primary server. Backup & Restore module will take care of generating SSH Keys by itself. But we have to copy the Primary Server SSH Keys to Secondary server to ensure that Primary can communicate with Secondary server easily.
We can use any one of the following 2 methods to copy the SSH keys to Secondary (Warm spare) server.
Manually copy the SSH Keys to Secondary Server
"FreePBX GUI - > Admin → Backup & Restore → Global Settings" has server SSH keys which we can copy to Secondary server manually. (note some browsers may not let you copy this data)
You may copy this key to spare/Secondary server manually to /root/.ssh/authorized_keys file. Create a file called "authorized_keys" (if not present) and add your Public Key in that file. If that file already exists just add your Key to the end of the file (make sure each key is separated by a new line!)
Using SSH CLI command to copy Primary SSH keys to Secondary server (preferred method)
Please note if you are doing this from a fresh install and have never visited the backup and restore module these steps will FAIL.
Before proceeding go to the backup and restore module in the GUI , this triggers creation of keys referenced below.
Once in the module click on the global settings tab and confirm the key is present as shown above
Login to your Primary server with an SSH client such as PuTTy, SecureCRT, or other SSH client.
We will copy the key to the Secondary server with the help of the following command so that the primary server can SSH to the secondary server without needing a password.
At the primary server Linux CLI prompt type: sudo -u asterisk ssh-copy-id -i /home/asterisk/.ssh/id_rsa.pub root@SecondaryServerIP and enter the password when prompted.
APPLICATION NOTES
Make sure you replace the SecondaryServerIP with the IP Address of your Secondary PBX. (use IP and not a hostname that may be common to both primary and warmspare; if fqdns are desired create 3 records the common name , specific name for primary and specific name for warm spare - ie mypbx.company.com , mypbx1.company.com , mypbx2.company.com)
If the Firewall is configured, pay attention to creating the right rule allowing the two servers to talk to each other.
If this command completes without error, you are ready to test:
At the prompt type: ssh -i /home/asterisk/.ssh/id_rsa root@SecondaryServerIP
If all went well, you should now be logged in to the Secondary server.
Filestore Module Setup
Once SSH Setup done, we have to configure the Filestore module for the SSH.
Please follow the guide Filestore Module#SSH to perform the SSH configuration on the primary server.
FTP Configuration
We have to enable FTP protocol on the Warm Spare (Secondary) server by following System Admin - Provisioning Protocols#ProvisioningProtocols-Enable/DisableFTP
filestore FTP
If you have selected the FTP filestore, make sure that your FTP username and password are not removed from the Warm Spare Server
during the Restore process(System Admin→Provisioning Protocols) to overcome this situation, you can use the same FTP username and password
on the Warm Spare Server as the Primary Server. Or, you can create a separate FTP user for the Filestore module to access the server.
Filestore Module Setup
Once FTP Setup as mentioned as above done then we have to configure "Filestore" module for the FTP.
Please follow the guide Filestore Module#FTP to perform the FTP configuration i.e. Add Warm Spare (Secondary) server.
Generating API Credentials on the Warm Spare Server
We need to generate API credentials on Warm Spare (Secondary) server which will be used by Primary Server to trigger the "restore" process to Warm Spare server.
SSH or FTP method will be used to copy the data but API will be used to initiate the "restore" process.
Please follow the steps below to generate the Warm spare API credentials which will be used later in within the backup job definition on the primary.
FreePBX GUI → Connectivity → API
Create the Application Machine to Machine App
Allow the scope(gql:backup) to access the backup module and then "Add Application".
Copy the Access details to notepad or text editor, you will need this info when we setup the replication job on the Primary server.
SSL Certificates
If you are using self Singed Certificates, DO NOT use HTTPS in the below URLs on primary server Which can result this error (cURL error 60: Issuer certificate is invalid. )
1. Token URL
2. Authorization URL
3. GraphQL URL
4. Rest URL
HTTP / HTTPS ports
When the PBX WebUI is configured to use non-standard ports for HTTP (80) / HTTPS (443), then you need to include the port number in the URLs.
For example when port 2001 is configured for HTTP the Token URL will look like this:
http://192.168.3.249:2001/admin/api/api/token
How to setup a Warm Spare Backup job
Build your backup job as you normally would however make sure you are adding your Warm Spare server as one of the backup Storage Locations (note you can select multiple Storage Locations here)
Enable the "Warm Spare" option.
Once we enable the "Warm spare" option. Now you can see the Warmspare configuration options as shown below.
Warm Spare Server Input fields
 | The Warm Spare settings will show up once "Enable" is set to "Yes" |
 | |
 | Should NAT settings from primary system be restored to the Warm Spare system? |
 | Should Bind Address settings on the primary system be restored to the Warm Spare Server ? |
 | Should DNS settings on the primary system be restored to the Warm Spare Server ? |
 | Should we run "Apply Configs" on the Warm Spare Server after a restore is completed? |
 | The Warm Spare Server which is added in the Filestore module (FTP/SSH) |
 | Warm Spare server Access Token URL |
 | Client ID generated on the Warm Spare Server API |
 | Client Secret generated on the Warm Spare Server API |
 | GraphQl URL Generated on the Warm Spare Server API |
 | This is the Access Token which will be generated on the Warm Spare Server |
 | By clicking on this button, you can TEST the connectivity between the servers |
 | Exculde Certman while restoring the backup If this option is set to yes then certificate will not be restored which require HTTPS config to be rebuild manually with spare server certificate. |
|---|---|
 | Exclude Trunks While restring the backup exculde the trunks from primary server |
Execute Warm spare job
Once the warm spare backup job has been created, we can see the backup job in Backup & Restore grid.
We can either choose to run Warm Spare via "Scheduling option" while creating or editing backup job or we can run directly from the grid.
Warm spare job output will be something like below and at the end "Restore Status" will display the response from the API (which triggered the restore process in warm spare server).
IMPORTANT Operational notes:
be certain to omit the backup module from the warm spare job's list of modules to backup and restore; otherwise the job will be replicated to the warm spare in an active state and will attempt to run causing general chaos and mayhem.
if you see warm spare job failures related to a truncated key or connectivity issues insure that the IP of both the primary itself and the warm spare are defined in firewall as trusted ON THE PRIMARY... if not during the restore those items configured in firewall are on the warm spare are overwritten by that which is restored from the primary and if the primary's IP is not defined as trusted on itself , when those rules are replicated to the warm spare access from the primary to the warm spare can be impacted causing the job to fail
Sipstation users need to have a primary and a warm spare trunk group; trunks are to be individually configured on the respective deployment with the primary trunk group set up to failover to the warm spare trunk group. Additionally within the warm spare options, "Exclude Trunks" must be flagged as YES and the sipstation module must be omitted from the warm spare job.