Re: [libvirt-users] VM migration upon shutdown in centos 7
[Please keed the list CCed] On 01/10/2018 04:43 PM, Raman Gupta wrote:
Does this command alone succeed? Yes. I have used this command to migrate VMs successfully, without even knowing that spelling has changed.
I don't know enough about systemd but maybe it's not waiting for virsh to finish? Yes I also think virsh or libvirtd or related service does not wait before /root/vm_migrate.sh is called and hence live migration fails. If I replace Live Migration with a simple ping to peer node in the vPreShutdownHook.service, then ping goes thru successfully thus indicating network service was UP at that time.
Can you try to get any logs to see what is going on actually? Jan 8 15:30:58 desktop4 systemd: Stopping Session c1 of user gdm. Jan 8 15:30:58 desktop4 gdm: Freeing conversation 'gdm-launch-environment' with active job Jan 8 15:30:58 desktop4 systemd: Stopped target Sound Card. Jan 8 15:30:58 desktop4 systemd: Stopping Sound Card. Jan 8 15:30:58 desktop4 systemd: Stopping LVM2 PV scan on device 8:3... Jan 8 15:30:58 desktop4 systemd: Removed slice system-getty.slice. Jan 8 15:30:58 desktop4 systemd: Stopping system-getty.slice. Jan 8 15:30:58 desktop4 systemd: Stopping Authorization Manager... Jan 8 15:30:58 desktop4 systemd: Stopping Virtual Machine and Container Registration Service... Jan 8 15:30:58 desktop4 systemd: Stopping GlusterFS brick processes (stopping only)... Jan 8 15:30:58 desktop4 systemd: Removed slice system-selinux\x2dpolicy\x2dmigrate\x2dlocal\x2dchanges.slice. Jan 8 15:30:58 desktop4 systemd: Stopping system-selinux\x2dpolicy\x2dmigrate\x2dlocal\x2dchanges.slice. Jan 8 15:30:58 desktop4 systemd: Stopping Availability of block devices... Jan 8 15:30:58 desktop4 systemd: Stopping Virtual Machine qemu-1-GsmController. Jan 8 15:30:58 desktop4 alsactl[837]: alsactl daemon stopped Jan 8 15:30:58 desktop4 systemd: Stopping Session 1 of user root. Jan 8 15:30:58 desktop4 systemd: Stopping Manage Sound Card State (restore and store)...
I don't see your service being called. Anyway, look at libvirt-guests.service file. Looks like the following lines cause systemd to wait for a command to finish: [Service] EnvironmentFile=-/etc/sysconfig/libvirt-guests # Hack just call traditional service until we factor # out the code ExecStart=@libexecdir@/libvirt-guests.sh start ExecStop=@libexecdir@/libvirt-guests.sh stop Type=oneshot RemainAfterExit=yes StandardOutput=journal+console TimeoutStopSec=0
On Wed, Jan 10, 2018 at 8:46 PM, Michal Privoznik <mprivozn@redhat.com> wrote:
On 01/05/2018 12:00 PM, Raman Gupta wrote:
Hi,
I have CentOS 7, two node system which allows live VM migration between them. Live migration triggered from virsh is happily happening. I am using GlusterFS for replicating VM disk files.
Now I want to automatically do the live migration at the time of reboot/shutdown/halt of the host node and for this I have written a systemd service unit [vPreShutdownHook.service] and placed the live migration command in a migrate script which is invoked from this service unit. The migrate script is invoked but migration does not happen. If someone has some idea please help me to migrate VM upon shutdown.
######## vPreShutdownHook.service ###############
[Unit] Description=vPreShutdownHook Requires=network.target Requires=libvirtd.service Requires=dbus.service Requires=glusterd.service Requires=glusterfsd.service DefaultDependencies=no Before=shutdown.target reboot.target
[Service] Type=oneshot RemainAfterExit=true ExecStart=/bin/true ExecStop=/root/vm_migrate.sh
[Install] WantedBy=multi-user.target
########## Command to migrate ############ /usr/bin/virsh migrate --verbose --p2p --tunneled --live --compressed --comp-methods "mt" --comp-mt-level 5 --comp-mt-threads 5 --comp-mt-dthreads 5 MY_VM qemu+ssh://root@$node2/system
Does this command alone succeed? BTW: unless really needed --live will only make the migration take longer.
I don't know enough about systemd but maybe it's not waiting for virsh to finish? Can you try to get any logs to see what is going on actually?
Michal
Michal
Hi,
I don't see your service being called. It started when system started, see logs: Jan 12 14:59:42 desktop4 systemd: systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN) Jan 12 14:59:42 desktop4 systemd: Detected architecture x86-64. Jan 12 14:59:42 desktop4 systemd: Set hostname to <desktop4>. Jan 12 14:59:42 desktop4 systemd: Started Create list of required static device nodes for the current kernel. Jan 12 14:59:42 desktop4 systemd: Started *vPreShutdownHook*. Jan 12 14:59:42 desktop4 systemd: Starting Create Static Device Nodes in /dev...
Anyway, look at libvirt-guests.service file. Looks like the following lines cause systemd to wait for a command to finish: The libvirt-guests.service is NOT running in either of my nodes, even though VM is working fine.
I tried starting libvirt-guests.service and added it as Requires target in vPreShutdownHook.service, but it did not help. On reboot the VM did not migrated. Thanks On Thu, Jan 11, 2018 at 12:56 PM, Michal Privoznik <mprivozn@redhat.com> wrote:
[Please keed the list CCed]
On 01/10/2018 04:43 PM, Raman Gupta wrote:
Does this command alone succeed? Yes. I have used this command to migrate VMs successfully, without even knowing that spelling has changed.
I don't know enough about systemd but maybe it's not waiting for virsh to finish? Yes I also think virsh or libvirtd or related service does not wait before /root/vm_migrate.sh is called and hence live migration fails. If I replace Live Migration with a simple ping to peer node in the vPreShutdownHook.service, then ping goes thru successfully thus indicating network service was UP at that time.
Can you try to get any logs to see what is going on actually? Jan 8 15:30:58 desktop4 systemd: Stopping Session c1 of user gdm. Jan 8 15:30:58 desktop4 gdm: Freeing conversation 'gdm-launch-environment' with active job Jan 8 15:30:58 desktop4 systemd: Stopped target Sound Card. Jan 8 15:30:58 desktop4 systemd: Stopping Sound Card. Jan 8 15:30:58 desktop4 systemd: Stopping LVM2 PV scan on device 8:3... Jan 8 15:30:58 desktop4 systemd: Removed slice system-getty.slice. Jan 8 15:30:58 desktop4 systemd: Stopping system-getty.slice. Jan 8 15:30:58 desktop4 systemd: Stopping Authorization Manager... Jan 8 15:30:58 desktop4 systemd: Stopping Virtual Machine and Container Registration Service... Jan 8 15:30:58 desktop4 systemd: Stopping GlusterFS brick processes (stopping only)... Jan 8 15:30:58 desktop4 systemd: Removed slice system-selinux\x2dpolicy\x2dmigrate\x2dlocal\x2dchanges.slice. Jan 8 15:30:58 desktop4 systemd: Stopping system-selinux\x2dpolicy\x2dmigrate\x2dlocal\x2dchanges.slice. Jan 8 15:30:58 desktop4 systemd: Stopping Availability of block devices... Jan 8 15:30:58 desktop4 systemd: Stopping Virtual Machine qemu-1-GsmController. Jan 8 15:30:58 desktop4 alsactl[837]: alsactl daemon stopped Jan 8 15:30:58 desktop4 systemd: Stopping Session 1 of user root. Jan 8 15:30:58 desktop4 systemd: Stopping Manage Sound Card State (restore and store)...
I don't see your service being called. Anyway, look at libvirt-guests.service file. Looks like the following lines cause systemd to wait for a command to finish:
[Service] EnvironmentFile=-/etc/sysconfig/libvirt-guests # Hack just call traditional service until we factor # out the code ExecStart=@libexecdir@/libvirt-guests.sh start ExecStop=@libexecdir@/libvirt-guests.sh stop Type=oneshot RemainAfterExit=yes StandardOutput=journal+console TimeoutStopSec=0
On Wed, Jan 10, 2018 at 8:46 PM, Michal Privoznik <mprivozn@redhat.com> wrote:
On 01/05/2018 12:00 PM, Raman Gupta wrote:
Hi,
I have CentOS 7, two node system which allows live VM migration
between
them. Live migration triggered from virsh is happily happening. I am using GlusterFS for replicating VM disk files.
Now I want to automatically do the live migration at the time of reboot/shutdown/halt of the host node and for this I have written a systemd service unit [vPreShutdownHook.service] and placed the live migration command in a migrate script which is invoked from this service unit. The migrate script is invoked but migration does not happen. If someone has some idea please help me to migrate VM upon shutdown.
######## vPreShutdownHook.service ###############
[Unit] Description=vPreShutdownHook Requires=network.target Requires=libvirtd.service Requires=dbus.service Requires=glusterd.service Requires=glusterfsd.service DefaultDependencies=no Before=shutdown.target reboot.target
[Service] Type=oneshot RemainAfterExit=true ExecStart=/bin/true ExecStop=/root/vm_migrate.sh
[Install] WantedBy=multi-user.target
########## Command to migrate ############ /usr/bin/virsh migrate --verbose --p2p --tunneled --live --compressed --comp-methods "mt" --comp-mt-level 5 --comp-mt-threads 5 --comp-mt-dthreads 5 MY_VM qemu+ssh://root@$node2/system
Does this command alone succeed? BTW: unless really needed --live will only make the migration take longer.
I don't know enough about systemd but maybe it's not waiting for virsh to finish? Can you try to get any logs to see what is going on actually?
Michal
Michal
On 01/12/2018 11:08 AM, Raman Gupta wrote:
Hi,
I don't see your service being called.
It started when system started, see logs: Jan 12 14:59:42 desktop4 systemd: systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN) Jan 12 14:59:42 desktop4 systemd: Detected architecture x86-64. Jan 12 14:59:42 desktop4 systemd: Set hostname to <desktop4>. Jan 12 14:59:42 desktop4 systemd: Started Create list of required static device nodes for the current kernel. Jan 12 14:59:42 desktop4 systemd: Started *vPreShutdownHook*. Jan 12 14:59:42 desktop4 systemd: Starting Create Static Device Nodes in /dev...
Oh, these are new.
Anyway, look at libvirt-guests.service file. Looks like the following lines cause systemd to wait for a command to finish: The libvirt-guests.service is NOT running in either of my nodes, even though VM is working fine.
I did not say to run the service. I said to take a look at the service file because it implements what you want - waiting for a command to finish.
I tried starting libvirt-guests.service and added it as Requires target in vPreShutdownHook.service, but it did not help. On reboot the VM did not migrated.
It makes no sense to have your service requiring libvrt-guests. Either libvirt-guests will save all your domains and thus your script has nothing to migrate, or it will do nothing and then it's just NO-OP. Michal
participants (2)
-
Michal Privoznik -
Raman Gupta