6:46 a.m.
https://bugzilla.redhat.com/show_bug.cgi?id=822052
When doing a live migration, if the destination fails for any
reason after the point in which files should be labeled, then
the cleanup of the destination would restore the labels to their
defaults, even though the source is still trying to continue
running with the image open. Bug 822052 mentioned one source
of live migration failure - a mismatch in SELinux virt_use_nfs
settings (on for source, off for destination); but I found other
situations that would also trigger it (for example, having a
graphics device tied to port 5999 on the source, and a different
domain on the destination already using that port, so that the
destination cannot reuse the port).
In short, just as cleanup of the source on a successful migration
must not relabel files (because the destination would be crippled
by the relabel), cleanup of the destination on a failed migraion
must not relabel files (because the source would be crippled).
* src/qemu/qemu_process.c (qemuProcessStart): Set flag to avoid
label restoration when cleaning up on failed migration.
Signed-off-by: Eric Blake <eblake(a)redhat.com>
---
src/qemu/qemu_process.c | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/src/qemu/qemu_process.c b/src/qemu/qemu_process.c
index 31de759..d727fc4 100644
--- a/src/qemu/qemu_process.c
+++ b/src/qemu/qemu_process.c
@@ -3466,6 +3466,10 @@ int qemuProcessStart(virConnectPtr conn,
* restore any security label as we would overwrite labels
* we did not set. */
stop_flags = VIR_QEMU_PROCESS_STOP_NO_RELABEL;
+ /* If we fail while doing incoming migration, then we must not
+ * relabel, as the source is still using the files. */
+ if (migrateFrom)
+ stop_flags |= VIR_QEMU_PROCESS_STOP_MIGRATED;
hookData.conn = conn;
hookData.vm = vm;
--
1.8.3.1
7:10 a.m.
On 08/20/2013 04:46 PM, Eric Blake wrote:
https://bugzilla.redhat.com/show_bug.cgi?id=822052
When doing a live migration, if the destination fails for any
reason after the point in which files should be labeled, then
the cleanup of the destination would restore the labels to their
defaults, even though the source is still trying to continue
running with the image open. Bug 822052 mentioned one source
of live migration failure - a mismatch in SELinux virt_use_nfs
settings (on for source, off for destination); but I found other
situations that would also trigger it (for example, having a
graphics device tied to port 5999 on the source, and a different
domain on the destination already using that port, so that the
destination cannot reuse the port).
In short, just as cleanup of the source on a successful migration
must not relabel files (because the destination would be crippled
by the relabel), cleanup of the destination on a failed migraion
must not relabel files (because the source would be crippled).
Note that while debugging this, I ran across a bigger design problem.
This patch works as long as source and destination are configured
equally, or where the difference (such as having selinux virt_use_nfs
settings distinct) can be detected immediately upon the attempt to start
qemu on the destination. A much more insidious case occurs if you have
a shared disk image on NFS, but where you have a configuration mismatch
between source and destination. In my case, the mismatch was caused by
running Fedora's pre-packaged libvirtd (which is built using qemu:qemu
as the default user/group for qemu processes) on the source, but a
self-built libvirtd (built with ./autogen.sh --system) on the
destination, and with no edits to /etc/libvirt/qemu.conf on either side.
Note that the default user is a configure option, and that if
/etc/libvirt/qemu.conf does NOT call out a user/group setting, then
libvirtd uses the configure defaults; and since './autogen.sh --system'
currently does not change the configure defaults, it ends up using
root:root to run qemu guests. This is probably a shortcoming of
'./autogen.sh --system' for not matching what the rpm does, but fixing
that is a separate patch.
The problem at hand is that when migrating between different
configurations, the destination first attempts to label files before
even starting qemu - and does a chown() to root:root. The failure case
described above for matched configurations occurred when the destination
did a chown() to root:root as part of the cleanup after failure to
migrate, but with my mismatched configuration, the chown() happened much
earlier. One of the (numerous) annoyances of NFS is that a chown() can
kill access of existing open fds (on a POSIX file system, once you have
an fd via open() or other means, you cannot lose rights to the inode
that the fd is visiting, no matter whether chown(), unlink(), or any
other number of things occur on the file that you originally opened in
the meantime; not so for NFS). But since the forced loss of access
isn't instantaneous, you now have a race - if live migration is fast
enough, the source can still get the guest migrated to the destination;
but if there is lots of I/O and memory churn in the guest, so that the
live migration takes long enough for the forced loss of access to kick
in, then the source will start seeing I/O errors, possibly causing
severe corruption to the guest.
We should really consider whether we need more framework for diagnosing
incompatible configurations between source and destination, and outright
rejecting migration where two machines do not share the same user/group
setup.
--
Eric Blake eblake redhat com +1-919-301-3266
Libvirt virtualization library http://libvirt.org
5:29 p.m.
On Tue, Aug 20, 2013 at 05:10:38PM -0600, Eric Blake wrote:
The problem at hand is that when migrating between different
configurations, the destination first attempts to label files before
even starting qemu - and does a chown() to root:root.
IMHO making libvirt work in the face of such admin mis-configurations
is out of scope.
We should really consider whether we need more framework for
diagnosing
incompatible configurations between source and destination, and outright
rejecting migration where two machines do not share the same user/group
setup.
I tend to view this as a general OS management problem. Tools like
puppet/chef/etc should be used to manage your virt hosts to avoid
inconsistent configs between hosts.
Daniel
--
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org -o- http://virt-manager.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
8:54 p.m.
On Tue, Aug 20, 2013 at 04:46:47PM -0600, Eric Blake wrote:
https://bugzilla.redhat.com/show_bug.cgi?id=822052
When doing a live migration, if the destination fails for any
reason after the point in which files should be labeled, then
the cleanup of the destination would restore the labels to their
defaults, even though the source is still trying to continue
running with the image open. Bug 822052 mentioned one source
of live migration failure - a mismatch in SELinux virt_use_nfs
settings (on for source, off for destination); but I found other
situations that would also trigger it (for example, having a
graphics device tied to port 5999 on the source, and a different
domain on the destination already using that port, so that the
destination cannot reuse the port).
In short, just as cleanup of the source on a successful migration
must not relabel files (because the destination would be crippled
by the relabel), cleanup of the destination on a failed migraion
must not relabel files (because the source would be crippled).
* src/qemu/qemu_process.c (qemuProcessStart): Set flag to avoid
label restoration when cleaning up on failed migration.
Signed-off-by: Eric Blake <eblake(a)redhat.com>
---
src/qemu/qemu_process.c | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/src/qemu/qemu_process.c b/src/qemu/qemu_process.c
index 31de759..d727fc4 100644
--- a/src/qemu/qemu_process.c
+++ b/src/qemu/qemu_process.c
@@ -3466,6 +3466,10 @@ int qemuProcessStart(virConnectPtr conn,
* restore any security label as we would overwrite labels
* we did not set. */
stop_flags = VIR_QEMU_PROCESS_STOP_NO_RELABEL;
+ /* If we fail while doing incoming migration, then we must not
+ * relabel, as the source is still using the files. */
+ if (migrateFrom)
+ stop_flags |= VIR_QEMU_PROCESS_STOP_MIGRATED;
hookData.conn = conn;
hookData.vm = vm;
ACK
Daniel
--
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org -o- http://virt-manager.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
10:12 p.m.
On 08/21/2013 06:54 AM, Daniel P. Berrange wrote:
On Tue, Aug 20, 2013 at 04:46:47PM -0600, Eric Blake wrote:
> https://bugzilla.redhat.com/show_bug.cgi?id=822052
>
> When doing a live migration, if the destination fails for any
> reason after the point in which files should be labeled, then
> the cleanup of the destination would restore the labels to their
> defaults, even though the source is still trying to continue
> running with the image open. Bug 822052 mentioned one source
> of live migration failure - a mismatch in SELinux virt_use_nfs
> settings (on for source, off for destination); but I found other
> situations that would also trigger it (for example, having a
> graphics device tied to port 5999 on the source, and a different
> domain on the destination already using that port, so that the
> destination cannot reuse the port).
>
> In short, just as cleanup of the source on a successful migration
> must not relabel files (because the destination would be crippled
> by the relabel), cleanup of the destination on a failed migraion
> must not relabel files (because the source would be crippled).
>
> * src/qemu/qemu_process.c (qemuProcessStart): Set flag to avoid
> label restoration when cleaning up on failed migration.
>
> Signed-off-by: Eric Blake <eblake(a)redhat.com>
> ---
> src/qemu/qemu_process.c | 4 ++++
> 1 file changed, 4 insertions(+)
>
> diff --git a/src/qemu/qemu_process.c b/src/qemu/qemu_process.c
> index 31de759..d727fc4 100644
> --- a/src/qemu/qemu_process.c
> +++ b/src/qemu/qemu_process.c
> @@ -3466,6 +3466,10 @@ int qemuProcessStart(virConnectPtr conn,
> * restore any security label as we would overwrite labels
> * we did not set. */
> stop_flags = VIR_QEMU_PROCESS_STOP_NO_RELABEL;
> + /* If we fail while doing incoming migration, then we must not
> + * relabel, as the source is still using the files. */
> + if (migrateFrom)
> + stop_flags |= VIR_QEMU_PROCESS_STOP_MIGRATED;
>
> hookData.conn = conn;
> hookData.vm = vm;
ACK
Thanks; pushed.
--
Eric Blake eblake redhat com +1-919-301-3266
Libvirt virtualization library http://libvirt.org
4089
days inactive
4090
days old
4 comments
2 participants
participants (2)
-
Daniel P. Berrange -
Eric Blake