WarningBe extremely careful when increasing volume sizes. It's possible that you may fatally corrupt your filesystem if you're not extremely cautious. ALWAYS PERFORM A BACKUP BEFORE ATTEMPTING TO INCREASE VOLUME SIZES YOU MUST HAVE BOTH NODES ONLINE, AND ALL DISKS IN 'UPTODATE' STATE BEFORE CONTINUING |
With that said, if you're careful and patient you can do an on-line hot upgrade of your filesystems without an outage.
Stage 1: Increase block device
First, verify that you have free space in your LVM ON BOTH MACHINES
[root@freepbx-a ~]# vgdisplay --- Volume group --- VG Name vg_ha System ID Format lvm2 Metadata Areas 1 Metadata Sequence No 7 VG Access read/write VG Status resizable MAX LV 0 Cur LV 6 Open LV 6 Max PV 0 Cur PV 1 Act PV 1 VG Size 465.14 GiB PE Size 4.00 MiB Total PE 119077 Alloc PE / Size 36576 / 142.88 GiB Free PE / Size 82501 / 322.27 GiB VG UUID WemjWb-sskA-31xv-H5bu-CKj0-UktP-1T97WZ |
The second line from the bottom - 'Free PE' - says that this volume group has 322G of unused space that we can allocate to any volume. The other machine says a similar amount, so I'm certain that I can add space on both machines.
Run the exact same command on both machines. In this case, we're extending the 'asterisk' filesystem (or, to start with, the bottom level block device).
lvextend -L250G /dev/vg_ha/drbd_asterisk |
This sets the volume size, rather than abstractly increasing it. As the volume sizes may have been created slightly differently on install (depending on disk sizes), you must explicitly set a size.
After running this command, verify on both nodes that both volumes are now exactly the same size.
[root@freepbx-b ~]# lvdisplay /dev/vg_ha/drbd_asterisk --- Logical volume --- LV Path /dev/vg_ha/drbd_asterisk LV Name drbd_asterisk VG Name vg_ha LV UUID sBnghY-YHLh-03De-jard-CVgy-fNYV-TRt23Y LV Write Access read/write LV Creation host, time freepbx-a, 2014-01-09 23:02:57 -0600 LV Status available # open 2 LV Size 250.00 GiB Current LE 64000 Segments 2 Allocation inherit Read ahead sectors auto - currently set to 256 Block device 253:2 |
The 'Current LE' (Current Logical Extents) is the number that should match. If they don't, you must resolve this before continuing. Attempting to continue if they don't match is guaranteed to corrupt your filesystem, and will also probably crash the machine. In our case, they matched (and they will, if you don't try to use the 'add' command to lvextend, and use the 'set' option as above).
[root@freepbx-a ~]# lvdisplay /dev/vg_ha/drbd_asterisk --- Logical volume --- LV Path /dev/vg_ha/drbd_asterisk LV Name drbd_asterisk VG Name vg_ha LV UUID 4CbvFI-pit7-WTMi-deUI-Xhvh-eXg5-PhqBZb LV Write Access read/write LV Creation host, time freepbx-a, 2014-01-02 18:08:16 -0500 LV Status available # open 2 LV Size 250.00 GiB Current LE 64000 Segments 2 Allocation inherit Read ahead sectors auto - currently set to 256 Block device 253:2 |
Both nodes have 64000 Logical Extends, so we can proceed.
Stage 2: Tell DRBD to update its internal counters and resync.
You only need to perform this command on one node.
drbdadm resize asterisk |
Replace 'asterisk' with whatever volume you're increasing the size of, obviously. This will update DRBD on both nodes, you do not need to do anything on the other one. Checking via the Web UI will show that the volume is now resyncing. This will perform a complete resync, and attempt to read and write to every block of available space to ensure that the previous stages have been completed. There is no way to abort this, nor do you want to.
Stage 3: Wait
No. Really. Wait for the resync to complete. Whilst you can, if you REALLY MUST, perform the resize now, don't. The server is doing a complete resync now to verify that the next stage isn't going to have problems. Let it finish.
Stage 4: Increase filesystem size
This is the final and - hopefully - anti-climatic part. After all the previous replcation has completed, resize the filesystem on the active node. You need the block device - in our case, it's /dev/drbd1.
[root@freepbx-a ~]# mount /dev/mapper/vg_ha-slash on / type ext4 (rw) proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) devpts on /dev/pts type devpts (rw,gid=5,mode=620) tmpfs on /dev/shm type tmpfs (rw) /dev/md0 on /boot type ext4 (rw) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) none on /sys/kernel/config type configfs (rw) /dev/drbd3 on /drbd/httpd type ext4 (rw) /dev/drbd1 on /drbd/asterisk type ext4 (rw) /dev/drbd4 on /drbd/spare type ext4 (rw) /dev/drbd2 on /drbd/mysql type ext4 (rw) [root@freepbx-a ~]# |
You can see that /dev/drbd1 is mounted on /drbd/asterisk, so that is the volume we're going to resize.
[root@freepbx-a ~]# resize2fs /dev/drbd1 resize2fs 1.41.12 (17-May-2010) Filesystem at /dev/drbd1 is mounted on /drbd/asterisk; on-line resizing required old desc_blocks = 3, new_desc_blocks = 16 Performing an on-line resize of /dev/drbd1 to 65533991 (4k) blocks. |
THIS TAKES TIME! DO NOT ABORT! |
Stage 5: Wait, again
While the filesystem is growing, the machine is fully accessible, and everything will be fine. You can watch the filesystem grow in a different window, by running the 'df' or 'df -h' command.
[root@freepbx-a ~]# df Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/vg_ha-slash 48380792 2177960 43745232 5% / tmpfs 1993480 58712 1934768 3% /dev/shm /dev/md0 495780 32390 437794 7% /boot /dev/drbd3 20641752 237416 19355796 2% /drbd/httpd /dev/drbd1 116758940 29531216 81297720 27% /drbd/asterisk /dev/drbd4 15481324 169456 14525464 2% /drbd/spare /dev/drbd2 20641752 869796 18723416 5% /drbd/mysql [root@freepbx-a ~]# ... I paused here for a minute ... [root@freepbx-a ~]# df Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/vg_ha-slash 48380792 2178012 43745180 5% / tmpfs 1993480 58712 1934768 3% /dev/shm /dev/md0 495780 32390 437794 7% /boot /dev/drbd3 20641752 237464 19355748 2% /drbd/httpd /dev/drbd1 151722156 29557460 114459100 21% /drbd/asterisk /dev/drbd4 15481324 169456 14525464 2% /drbd/spare /dev/drbd2 20641752 869972 18723240 5% /drbd/mysql [root@freepbx-a ~]# |
Note that the '1K-blocks' has increased, which has lead to an increase in 'Available' and the respective shrinking of 'Use %'.
You can also run 'df -h' which will give you a human readable output:
[root@freepbx-a ~]# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg_ha-slash 47G 2.1G 42G 5% / tmpfs 2.0G 58M 1.9G 3% /dev/shm /dev/md0 485M 32M 428M 7% /boot /dev/drbd3 20G 232M 19G 2% /drbd/httpd /dev/drbd1 165G 29G 128G 19% /drbd/asterisk /dev/drbd4 15G 166M 14G 2% /drbd/spare /dev/drbd2 20G 850M 18G 5% /drbd/mysql [root@freepbx-a ~]# |
Stage 6: You're almost done!
When the resize has finished, it should exit without errors.
[root@freepbx-a ~]# resize2fs /dev/drbd1 resize2fs 1.41.12 (17-May-2010) Filesystem at /dev/drbd1 is mounted on /drbd/asterisk; on-line resizing required old desc_blocks = 3, new_desc_blocks = 16 Performing an on-line resize of /dev/drbd1 to 65533991 (4k) blocks. The filesystem on /dev/drbd1 is now 65533991 blocks long. [root@freepbx-a ~]# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg_ha-slash 47G 2.1G 42G 5% / tmpfs 2.0G 58M 1.9G 3% /dev/shm /dev/md0 485M 32M 428M 7% /boot /dev/drbd3 20G 232M 19G 2% /drbd/httpd /dev/drbd1 247G 29G 206G 13% /drbd/asterisk /dev/drbd4 15G 166M 14G 2% /drbd/spare /dev/drbd2 20G 851M 18G 5% /drbd/mysql [root@freepbx-a ~]# |
Your final thing to do is to update the database with the size of the volume so that when you replace a node, the new volume is created at the correct size.
amportal a m select * from freepbxha where `key`='SIZE_asterisk'; |
This will show you the OLD size.
+---------------+-------+ | key | value | +---------------+-------+ | SIZE_asterisk | 40 | +---------------+-------+ 1 row in set (0.00 sec) |
You now update it with the new size. USE THE SIZE YOU USED ON THE LVEXTEND COMMAND, NOT THE SIZE REPORTED BY 'df.'
mysql> update freepbxha set `value`='250' where `key`='SIZE_asterisk'; Query OK, 1 row affected (0.41 sec) Rows matched: 1 Changed: 1 Warnings: 0 mysql> |