[libvirt-users] live migration and config
Hello! After some mistakes yesterday we ( me and my colleague ) think that it will be wise for libvirt to check config file existence on remote side and through error if not, before migrating, otherwise migration will fail and VM fs can be damaged, because it is sort of remove of power plug... We missed twice yesterday :-( Could you tell me is there already such option or any plans to implement this? Thank you!
On Thu, Sep 13, 2018 at 10:35:09 +0400, Dmitry Melekhov wrote:
After some mistakes yesterday we ( me and my colleague ) think that it will be wise for libvirt to check config file existence on remote side
Which config file?
and through error if not,
before migrating, otherwise migration will fail and VM fs can be damaged, because it is sort of remove of power plug...
We missed twice yesterday :-(
Could you tell me is there already such option or any plans to implement this?
You didn't really say anything about what you did and what happened. Thus it's impossible to help you. Jirka
13.09.2018 17:47, Jiri Denemark пишет:
On Thu, Sep 13, 2018 at 10:35:09 +0400, Dmitry Melekhov wrote:
After some mistakes yesterday we ( me and my colleague ) think that it will be wise for libvirt to check config file existence on remote side Which config file?
VM config file, namely qemu. We forgot to mount shared storage (namely gluster volume), on which we have config files, twice... And tried to live migrate VM to another host, where there is no config file, so VM can't be started, so it died. Twice :-(
and through error if not,
before migrating, otherwise migration will fail and VM fs can be damaged, because it is sort of remove of power plug...
We missed twice yesterday :-(
Could you tell me is there already such option or any plans to implement this? You didn't really say anything about what you did and what happened. Thus it's impossible to help you. I hope I explained.
Thank you!
Jirka
On Thu, Sep 13, 2018 at 18:38:57 +0400, Dmitry Melekhov wrote:
13.09.2018 17:47, Jiri Denemark пишет:
On Thu, Sep 13, 2018 at 10:35:09 +0400, Dmitry Melekhov wrote:
After some mistakes yesterday we ( me and my colleague ) think that it will be wise for libvirt to check config file existence on remote side Which config file?
VM config file, namely qemu.
Do you mean the domain XML definition found in /etc/libvirt/qemu/*.xml ?
We forgot to mount shared storage (namely gluster volume), on which we have config files, twice...
If you're talking about the domain XML file, then you should never store them on a shared storage. All XML files in /etc/libvirt are supposed to be local to each host. Sharing them may cause changes done on one host will not be seen by libvirtd on the other host and they can even get lost when libvirtd on the other host decides to write its version of the domain definition to the file. And migration is designed so that the domain definition does not need to be present on the destination host. In fact, even if it's present, libvirtd will ignore it until you stop the migrated domain and start it again.
And tried to live migrate VM to another host, where there is no config file, so VM can't be started, so it died. Twice :-(
Error messages or logs would help. Otherwise we have no clue what failed and why. Jirka
13.09.2018 18:57, Jiri Denemark пишет:
On Thu, Sep 13, 2018 at 18:38:57 +0400, Dmitry Melekhov wrote:
13.09.2018 17:47, Jiri Denemark пишет:
On Thu, Sep 13, 2018 at 10:35:09 +0400, Dmitry Melekhov wrote:
After some mistakes yesterday we ( me and my colleague ) think that it will be wise for libvirt to check config file existence on remote side Which config file?
VM config file, namely qemu.
Do you mean the domain XML definition found in /etc/libvirt/qemu/*.xml ? Yes and no, xml, but at another location.
We forgot to mount shared storage (namely gluster volume), on which we have config files, twice... If you're talking about the domain XML file, then you should never store them on a shared storage. All XML files in /etc/libvirt are supposed to be local to each host.
They are not stored in /etc/libvirt , but in other location, which is suitable for sharing.
Sharing them may cause changes done on one host will not be seen by libvirtd on the other host and they can even get lost when libvirtd on the other host decides to write its version of the domain definition to the file.
And migration is designed so that the domain definition does not need to be present on the destination host. In fact, even if it's present, libvirtd will ignore it until you stop the migrated domain and start it again.
Sounds strange, because if there is no config file on target host migration fails, always.
And tried to live migrate VM to another host, where there is no config file, so VM can't be started, so it died. Twice :-( Error messages or logs would help. Otherwise we have no clue what failed and why.
Unfortunately, my colleague left today for 2 weeks vacations and I have no access to these hosts, so I will be able to look in logs or do some tests only in October. So, I'll write there again after 2 weeks or so. Thank you!
On Thu, Sep 13, 2018 at 19:37:00 +0400, Dmitry Melekhov wrote:
13.09.2018 18:57, Jiri Denemark пишет:
On Thu, Sep 13, 2018 at 18:38:57 +0400, Dmitry Melekhov wrote:
13.09.2018 17:47, Jiri Denemark пишет:
On Thu, Sep 13, 2018 at 10:35:09 +0400, Dmitry Melekhov wrote:
After some mistakes yesterday we ( me and my colleague ) think that it will be wise for libvirt to check config file existence on remote side Which config file?
VM config file, namely qemu.
Do you mean the domain XML definition found in /etc/libvirt/qemu/*.xml ? Yes and no, xml, but at another location.
We forgot to mount shared storage (namely gluster volume), on which we have config files, twice... If you're talking about the domain XML file, then you should never store them on a shared storage. All XML files in /etc/libvirt are supposed to be local to each host.
They are not stored in /etc/libvirt , but in other location, which is suitable for sharing.
I'm even more confused now. How could libvirt even know about such files? Unless you use transient domains, i.e., point libvirt to the right XML when starting the domain.
Unfortunately, my colleague left today for 2 weeks vacations and I have no access to these hosts, so I will be able to look in logs or do some tests only in October. So, I'll write there again after 2 weeks or so.
Yeah, it would be nice if you could share the actual domain XML and the command you used for migrating it to the other host, too. Jirka
14.09.2018 15:43, Jiri Denemark пишет:
On Thu, Sep 13, 2018 at 19:37:00 +0400, Dmitry Melekhov wrote:
13.09.2018 18:57, Jiri Denemark пишет:
On Thu, Sep 13, 2018 at 18:38:57 +0400, Dmitry Melekhov wrote:
13.09.2018 17:47, Jiri Denemark пишет:
On Thu, Sep 13, 2018 at 10:35:09 +0400, Dmitry Melekhov wrote:
After some mistakes yesterday we ( me and my colleague ) think that it will be wise for libvirt to check config file existence on remote side Which config file?
VM config file, namely qemu. Do you mean the domain XML definition found in /etc/libvirt/qemu/*.xml ? Yes and no, xml, but at another location.
We forgot to mount shared storage (namely gluster volume), on which we have config files, twice... If you're talking about the domain XML file, then you should never store them on a shared storage. All XML files in /etc/libvirt are supposed to be local to each host. They are not stored in /etc/libvirt , but in other location, which is suitable for sharing.
I'm even more confused now. How could libvirt even know about such files? Unless you use transient domains, i.e., point libvirt to the right XML when starting the domain.
Yes, it transient, really, we use pacemaker's VirtualDomain http://www.linux-ha.org/wiki/VirtualDomain_(resource_agent) pacemaker creates domain and destroys it. As I can see from sources migration looks like: virsh ${VIRSH_OPTIONS} migrate --live $migrate_opts $DOMAIN_NAME $remoteuri $migrateuri & Don't know which exact options it uses though. Do we need xml on another host if domain is transient for live migration?
Unfortunately, my colleague left today for 2 weeks vacations and I have no access to these hosts, so I will be able to look in logs or do some tests only in October. So, I'll write there again after 2 weeks or so. Yeah, it would be nice if you could share the actual domain XML and the command you used for migrating it to the other host, too.
Don't sure I can do this, I know English a little bit ( my colleague can't write ;-) ), but this is his cluster, not my, so I prefer to wait 2 weeks, not try to request access.... Thank you!
On Fri, Sep 14, 2018 at 16:00:43 +0400, Dmitry Melekhov wrote:
14.09.2018 15:43, Jiri Denemark пишет:
On Thu, Sep 13, 2018 at 19:37:00 +0400, Dmitry Melekhov wrote:
13.09.2018 18:57, Jiri Denemark пишет:
On Thu, Sep 13, 2018 at 18:38:57 +0400, Dmitry Melekhov wrote:
13.09.2018 17:47, Jiri Denemark пишет:
On Thu, Sep 13, 2018 at 10:35:09 +0400, Dmitry Melekhov wrote: > After some mistakes yesterday we ( me and my colleague ) think that it > will be wise for libvirt to check config file existence on remote side Which config file?
VM config file, namely qemu. Do you mean the domain XML definition found in /etc/libvirt/qemu/*.xml ? Yes and no, xml, but at another location.
We forgot to mount shared storage (namely gluster volume), on which we have config files, twice... If you're talking about the domain XML file, then you should never store them on a shared storage. All XML files in /etc/libvirt are supposed to be local to each host. They are not stored in /etc/libvirt , but in other location, which is suitable for sharing.
I'm even more confused now. How could libvirt even know about such files? Unless you use transient domains, i.e., point libvirt to the right XML when starting the domain.
Yes, it transient, really, we use pacemaker's VirtualDomain http://www.linux-ha.org/wiki/VirtualDomain_(resource_agent)
pacemaker creates domain and destroys it.
As I can see from sources migration looks like: virsh ${VIRSH_OPTIONS} migrate --live $migrate_opts $DOMAIN_NAME $remoteuri $migrateuri &
Don't know which exact options it uses though.
Do we need xml on another host if domain is transient for live migration?
No, the XML is only needed when starting the domain. At the beginning of migration libvirt transfers the XML to libvirtd on the other host so that QEMU with the right parameters can be started there. Just to make sure, the disk images used by the domain are stored on a shared storage accessible from both hosts, right? Jirka
14.09.2018 16:15, Jiri Denemark пишет:
14.09.2018 15:43, Jiri Denemark пишет:
On Thu, Sep 13, 2018 at 19:37:00 +0400, Dmitry Melekhov wrote:
13.09.2018 18:57, Jiri Denemark пишет:
On Thu, Sep 13, 2018 at 18:38:57 +0400, Dmitry Melekhov wrote:
13.09.2018 17:47, Jiri Denemark пишет: > On Thu, Sep 13, 2018 at 10:35:09 +0400, Dmitry Melekhov wrote: >> After some mistakes yesterday we ( me and my colleague ) think that it >> will be wise for libvirt to check config file existence on remote side > Which config file? > VM config file, namely qemu. Do you mean the domain XML definition found in /etc/libvirt/qemu/*.xml ? Yes and no, xml, but at another location.
We forgot to mount shared storage (namely gluster volume), on which we have config files, twice... If you're talking about the domain XML file, then you should never store them on a shared storage. All XML files in /etc/libvirt are supposed to be local to each host. They are not stored in /etc/libvirt , but in other location, which is suitable for sharing. I'm even more confused now. How could libvirt even know about such files? Unless you use transient domains, i.e., point libvirt to the right XML when starting the domain. Yes, it transient, really, we use pacemaker's VirtualDomain http://www.linux-ha.org/wiki/VirtualDomain_(resource_agent)
pacemaker creates domain and destroys it.
As I can see from sources migration looks like: virsh ${VIRSH_OPTIONS} migrate --live $migrate_opts $DOMAIN_NAME $remoteuri $migrateuri &
Don't know which exact options it uses though.
Do we need xml on another host if domain is transient for live migration? No, the XML is only needed when starting the domain. At the beginning of migration libvirt transfers the XML to libvirtd on the other host so
On Fri, Sep 14, 2018 at 16:00:43 +0400, Dmitry Melekhov wrote: that QEMU with the right parameters can be started there. Well, then it is really strange... Just to make sure, the disk images used by the domain are stored on a shared storage accessible from both hosts, right?
Yes, we use gluster and connection over gfapi, so VM storage is accessible from all hosts. Usually, when xml are available, migration works without problems. Just thought - it is possible that problem is in pacemaker's VirtualDomain and it needs xml on all hosts, there is such comment : # During a probe, it is permissible for the config file to not be # readable (it might be on shared storage not available during the # probe). In that case, we're # unable to get the domain name. Thus, we also can't check whether the # domain is running. The only thing we can do here is to assume that # it is not running. We have to check, but looks like we have to ask in pacemaker mail list, it has to check if xml is available. Thank you!
Hello! We tested this and problem is not in libvirt, but in pacemaker's virtualdomain. Thank you! 13.09.2018 10:35, Dmitry Melekhov пишет:
Hello!
After some mistakes yesterday we ( me and my colleague ) think that it will be wise for libvirt to check config file existence on remote side and through error if not,
before migrating, otherwise migration will fail and VM fs can be damaged, because it is sort of remove of power plug...
We missed twice yesterday :-(
Could you tell me is there already such option or any plans to implement this?
Thank you!
participants (2)
-
Dmitry Melekhov -
Jiri Denemark