FreePBX HA-Increasing Volume Size

Warning

Be extremely careful when increasing volume sizes. It's possible that you may fatally corrupt your filesystem if you're not extremely cautious.

ALWAYS PERFORM A BACKUP BEFORE ATTEMPTING TO INCREASE VOLUME SIZES

YOU MUST HAVE BOTH NODES ONLINE, AND ALL DISKS IN 'UPTODATE' STATE BEFORE CONTINUING

With that said, if you're careful and patient you can do an on-line hot upgrade of your filesystems without an outage.

1 Stage 1: Increase block device
2 Stage 2: Tell DRBD to update its internal counters and resync.
3 Stage 3: Wait
4 Stage 4: Increase filesystem size
5 Stage 5: Wait, again
6 Stage 6: You're almost done!

Stage 1: Increase block device

First, verify that you have free space in your LVM ON BOTH MACHINES

[root@freepbx-a ~]# vgdisplay
  --- Volume group ---
  VG Name               vg_ha
  System ID
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  7
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                6
  Open LV               6
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               465.14 GiB
  PE Size               4.00 MiB
  Total PE              119077
  Alloc PE / Size       36576 / 142.88 GiB
  Free  PE / Size       82501 / 322.27 GiB
  VG UUID               WemjWb-sskA-31xv-H5bu-CKj0-UktP-1T97WZ

The second line from the bottom - 'Free PE' - says that this volume group has 322G of unused space that we can allocate to any volume. The other machine says a similar amount, so I'm certain that I can add space on both machines.

Run the exact same command on both machines. In this case, we're extending the 'asterisk' filesystem (or, to start with, the bottom level block device).

lvextend -L250G /dev/vg_ha/drbd_asterisk

This sets the volume size, rather than abstractly increasing it. As the volume sizes may have been created slightly differently on install (depending on disk sizes), you must explicitly set a size.

After running this command, verify on both nodes that both volumes are now exactly the same size.

[root@freepbx-b ~]# lvdisplay /dev/vg_ha/drbd_asterisk
  --- Logical volume ---
  LV Path                /dev/vg_ha/drbd_asterisk
  LV Name                drbd_asterisk
  VG Name                vg_ha
  LV UUID                sBnghY-YHLh-03De-jard-CVgy-fNYV-TRt23Y
  LV Write Access        read/write
  LV Creation host, time freepbx-a, 2014-01-09 23:02:57 -0600
  LV Status              available
  # open                 2
  LV Size                250.00 GiB
  Current LE             64000
  Segments               2
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:2

The 'Current LE' (Current Logical Extents) is the number that should match. If they don't, you must resolve this before continuing. Attempting to continue if they don't match is guaranteed to corrupt your filesystem, and will also probably crash the machine. In our case, they matched (and they will, if you don't try to use the 'add' command to lvextend, and use the 'set' option as above).

Both nodes have 64000 Logical Extends, so we can proceed.

Stage 2: Tell DRBD to update its internal counters and resync.

You only need to perform this command on one node.

Replace 'asterisk' with whatever volume you're increasing the size of, obviously. This will update DRBD on both nodes, you do not need to do anything on the other one. Checking via the Web UI will show that the volume is now resyncing. This will perform a complete resync, and attempt to read and write to every block of available space to ensure that the previous stages have been completed. There is no way to abort this, nor do you want to.

Stage 3: Wait

No. Really. Wait for the resync to complete. Whilst you can, if you REALLY MUST, perform the resize now, don't. The server is doing a complete resync now to verify that the next stage isn't going to have problems. Let it finish.

Stage 4: Increase filesystem size

This is the final and - hopefully - anti-climatic part. After all the previous replcation has completed, resize the filesystem on the active node. You need the block device - in our case, it's /dev/drbd1.

You can see that /dev/drbd1 is mounted on /drbd/asterisk, so that is the volume we're going to resize.

THIS TAKES TIME! DO NOT ABORT!

Stage 5: Wait, again

While the filesystem is growing, the machine is fully accessible, and everything will be fine. You can watch the filesystem grow in a different window, by running the 'df' or 'df -h' command.

Note that the '1K-blocks' has increased, which has lead to an increase in 'Available' and the respective shrinking of 'Use %'.

You can also run 'df -h' which will give you a human readable output:

Stage 6: You're almost done!

When the resize has finished, it should exit without errors.

Your final thing to do is to update the database with the size of the volume so that when you replace a node, the new volume is created at the correct size.

This will show you the OLD size.

You now update it with the new size. USE THE SIZE YOU USED ON THE LVEXTEND COMMAND, NOT THE SIZE REPORTED BY 'df.'

PBX GUI