Stalled Upgrade
- Lorne Gaetz (Unlicensed)
Owned by Lorne Gaetz (Unlicensed)
Nov 30, 2023
5 min read
Loading data...
It is possible that your upgrade may hang or around this point:
Cleanup : libtonezone-devel-2.10.0.1-1.shmz65.1.17.x86_64 262/478
Cleanup : bfa-firmware-3.2.21.1-2.el6.noarch 263/478
Cleanup : asterisk-version-switch-2.0.0.0-10.shmz65.1.17.noarch 264/478
Cleanup : dracut-kernel-004-336.el6_5.2.noarch 265/478
Cleanup : dracut-004-336.el6_5.2.noarch 266/478
Cleanup : iSymphonyServerV3-fpbx-3.2.1.11-1.noarch 267/478 |
This is due to a bug that is fixed in the latest version of Pacemaker. If you open another window to the node that has hung, run the command as below:
[root@freepbx-b ~]# tail /var/log/cluster/corosync.log
Dec 09 15:54:12 [1877] freepbx-b crmd: info: crm_xml_cleanup: Cleaning up memory from libxml2
Dec 09 15:54:12 [1866] freepbx-b pacemakerd: error: pcmk_child_exit: Child process crmd (1877) exited: Generic Pacemaker error (201)
Dec 09 15:54:12 [1866] freepbx-b pacemakerd: notice: pcmk_process_exit: Respawning failed child process: crmd
Dec 09 15:54:12 [1866] freepbx-b pacemakerd: info: start_child: Forked child 15957 for process crmd
Dec 09 15:54:13 [1874] freepbx-b lrmd: info: crm_client_new: Connecting 0xdb4a20 for uid=0 gid=0 pid=15957 id=2d58981e-51a2-40d7-a0bb-d4d82352f359
Dec 09 15:54:13 [1874] freepbx-b lrmd: error: crm_abort: lrmd_ipc_dispatch: Triggered assert at main.c:116 : flags & crm_ipc_client_response
Dec 09 15:54:13 [1874] freepbx-b lrmd: error: lrmd_ipc_dispatch: Invalid client request: 0xd9a780
Dec 09 15:54:13 [1866] freepbx-b pacemakerd: info: crm_signal_dispatch: Invoking handler for signal 15: Terminated
Dec 09 15:54:13 [1866] freepbx-b pacemakerd: notice: pcmk_shutdown_worker: Shuting down Pacemaker
Dec 09 15:54:13 [1866] freepbx-b pacemakerd: notice: stop_child: Stopping crmd: Sent -15 to process 15957 |
The bug (that was fixed as part of this upgrade) is that the 'crmd' process that has been asked to exit is not exiting. You need to manually kill that process. In the example above, the process ID that needs to be killed is the last number on the last line - 15957.
[root@freepbx-b cluster]# kill -9 15957
[root@freepbx-b cluster]# |
After running that command, the upgrade will immediately continue on, and process as normal.