[libvirt] Checking the expected behavior on dynamic image file ownership when live migrating
Hi, today I was debugging an issue that I found with qemu 4.0 and ended up puzzled about file ownership. I'm almost EOD now, but wanted to reach out here for a sanity check before I debug further. The case sumamrized is this: 1. start guest 1.1 image files are changed to libvirt-qemu:kvm (which matches Ubuntus user/group config) 2. live migrate the guest to a different node 2.1 image files go back to root:root which they initially had (ok) 3. migrate guest back to the original node x. image files stay root:root and are not changed back to libvirt-qemu:kvm That is odd/unexpected, but it seems the same applies to older versions and there it was never a problem so far. In my case I stumbled over it because the newer qemu --copy-storage-* options now want to re-open the file at some point and that fails with: error: internal error: unable to execute QEMU command 'drive-mirror': Could not reopen file: Permission denied If I e.g. shut down and start the guest on the source node (which fixes up the ownership at runtime) then migration with copy-storage works fine again. I'm mostly looking for hints if this is a known issue or being worked on to not waste time tomorrow. So hints are welcome. -- Christian Ehrhardt Software Engineer, Ubuntu Server Canonical Ltd
On Mon, Jul 01, 2019 at 05:26:45PM +0200, Christian Ehrhardt wrote:
Hi, today I was debugging an issue that I found with qemu 4.0 and ended up puzzled about file ownership. I'm almost EOD now, but wanted to reach out here for a sanity check before I debug further. The case sumamrized is this:
1. start guest 1.1 image files are changed to libvirt-qemu:kvm (which matches Ubuntus user/group config) 2. live migrate the guest to a different node 2.1 image files go back to root:root which they initially had (ok) 3. migrate guest back to the original node x. image files stay root:root and are not changed back to libvirt-qemu:kvm
That is odd/unexpected, but it seems the same applies to older versions and there it was never a problem so far.
This sounds odd - I see no reason why the first migration should behave differently from the second migration, as libvirt makes no distinction & has no knowledge of previous migrations. Could the two hosts be configured differently in some way. For example is the disk image on ext4 on one host, but nfs on the other host ? With shared filesystems, we'd generally expect the disk to be exposed on a shared filesystem on both hosts. We do take special action if we see an NFS filesystem, as we cannot reset file ownership on the migration source, as that would impact the migration dest too.
In my case I stumbled over it because the newer qemu --copy-storage-* options now want to re-open the file at some point and that fails with: error: internal error: unable to execute QEMU command 'drive-mirror': Could not reopen file: Permission denied
If I e.g. shut down and start the guest on the source node (which fixes up the ownership at runtime) then migration with copy-storage works fine again.
I'm mostly looking for hints if this is a known issue or being worked on to not waste time tomorrow. So hints are welcome.
-- Christian Ehrhardt Software Engineer, Ubuntu Server Canonical Ltd
-- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Mon, Jul 1, 2019 at 7:56 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
On Mon, Jul 01, 2019 at 05:26:45PM +0200, Christian Ehrhardt wrote:
Hi, today I was debugging an issue that I found with qemu 4.0 and ended up puzzled about file ownership. I'm almost EOD now, but wanted to reach out here for a sanity check before I debug further. The case sumamrized is this:
1. start guest 1.1 image files are changed to libvirt-qemu:kvm (which matches Ubuntus user/group config) 2. live migrate the guest to a different node 2.1 image files go back to root:root which they initially had (ok) 3. migrate guest back to the original node x. image files stay root:root and are not changed back to libvirt-qemu:kvm
That is odd/unexpected, but it seems the same applies to older versions and there it was never a problem so far.
This sounds odd - I see no reason why the first migration should behave differently from the second migration, as libvirt makes no distinction & has no knowledge of previous migrations.
I might have been unclear, let me clarify: 1. migration without copy (changes the ownership) lxc exec testkvm-eoan-from -- virsh migrate --live --unsafe kvmguest-eoan-normal qemu+ssh://10.222.144.19/system (and the same type backwards) 2. migration away from here with --copy-storage* fails due to file ownership
Could the two hosts be configured differently in some way. For example is the disk image on ext4 on one host, but nfs on the other host ? With shared filesystems, we'd generally expect the disk to be exposed on a shared filesystem on both hosts.
The two hosts are in fact two LXD containers sharing the same Filesystem (for the first migration) And still two containers, but without shared FS for the second copy-storage-* migration that then eventually fails due to the bad file ownership. But since they in fact use shared storage that might still be the reason - maybe the ordering is important here. Since in the setup for migration #1 they really use shared storage it might be that we have something like 1. source: migrates off guest 2. target: receives guest 3. target: guest is complete and changes file ownership to libvirt-qemu:kvm (correct) 4. source: shuts down the stub that is left after migration and changes file ownership to root:root Due to really using the same storage (not like the same FCP or scsi device but the same FS) that could be the reason the ownership after a migration cycle is reset. Thanks for the hint, worth a check with some slightly altered setups ... ... and confirmed - if I use copied images, but not on a shared FS the ownership is handled correctly. So my way of sharing the FS might be odd. And libvirt does not detect it as such, and due to that above ordering triggers the issue or the ownership being set to root:root by the migration source after the migration is complete. Quoting the virsh man page my setup is similar to "disk images are stored on coherent clustered filesystem, such as GFS2 or GPFS" but my libvirt doesn't know that and therefore changes the ownership. Maybe it would have the same bug on GFS2/GPFS? I don't have such a setup at hand, how is the issue avoided there? Is there a way to make libvirt realize that it really is on "the same" FS to avoid these operations?
We do take special action if we see an NFS filesystem, as we cannot reset file ownership on the migration source, as that would impact the migration dest too.
In my case I stumbled over it because the newer qemu --copy-storage-* options now want to re-open the file at some point and that fails with: error: internal error: unable to execute QEMU command 'drive-mirror': Could not reopen file: Permission denied
If I e.g. shut down and start the guest on the source node (which fixes up the ownership at runtime) then migration with copy-storage works fine again.
I'm mostly looking for hints if this is a known issue or being worked on to not waste time tomorrow. So hints are welcome.
-- Christian Ehrhardt Software Engineer, Ubuntu Server Canonical Ltd
-- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
-- Christian Ehrhardt Software Engineer, Ubuntu Server Canonical Ltd
On Mon, Jul 01, 2019 at 08:20:44PM +0200, Christian Ehrhardt wrote:
On Mon, Jul 1, 2019 at 7:56 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
On Mon, Jul 01, 2019 at 05:26:45PM +0200, Christian Ehrhardt wrote:
Hi, today I was debugging an issue that I found with qemu 4.0 and ended up puzzled about file ownership. I'm almost EOD now, but wanted to reach out here for a sanity check before I debug further. The case sumamrized is this:
1. start guest 1.1 image files are changed to libvirt-qemu:kvm (which matches Ubuntus user/group config) 2. live migrate the guest to a different node 2.1 image files go back to root:root which they initially had (ok) 3. migrate guest back to the original node x. image files stay root:root and are not changed back to libvirt-qemu:kvm
That is odd/unexpected, but it seems the same applies to older versions and there it was never a problem so far.
This sounds odd - I see no reason why the first migration should behave differently from the second migration, as libvirt makes no distinction & has no knowledge of previous migrations.
I might have been unclear, let me clarify: 1. migration without copy (changes the ownership) lxc exec testkvm-eoan-from -- virsh migrate --live --unsafe kvmguest-eoan-normal qemu+ssh://10.222.144.19/system (and the same type backwards) 2. migration away from here with --copy-storage* fails due to file ownership
Could the two hosts be configured differently in some way. For example is the disk image on ext4 on one host, but nfs on the other host ? With shared filesystems, we'd generally expect the disk to be exposed on a shared filesystem on both hosts.
The two hosts are in fact two LXD containers sharing the same Filesystem (for the first migration) And still two containers, but without shared FS for the second copy-storage-* migration that then eventually fails due to the bad file ownership.
But since they in fact use shared storage that might still be the reason - maybe the ordering is important here. Since in the setup for migration #1 they really use shared storage it might be that we have something like
1. source: migrates off guest 2. target: receives guest 3. target: guest is complete and changes file ownership to libvirt-qemu:kvm (correct) 4. source: shuts down the stub that is left after migration and changes file ownership to root:root
Due to really using the same storage (not like the same FCP or scsi device but the same FS) that could be the reason the ownership after a migration cycle is reset. Thanks for the hint, worth a check with some slightly altered setups ... ... and confirmed - if I use copied images, but not on a shared FS the ownership is handled correctly.
So my way of sharing the FS might be odd. And libvirt does not detect it as such, and due to that above ordering triggers the issue or the ownership being set to root:root by the migration source after the migration is complete.
Yes, this is going to be a problem. Libvirt will look at the filesystem and see that it is a local-only filesystem, and so assume it can safely reset ownership when source VM shuts down. It cannot tell that this is going to negatively impact the target VM using the same filesystem. This is an example of one of the problems that makes us say that localhost migration is not supported. If you're going to use containers, you need to make sure that each container either has a separate filesystem mount, or that the container sees a filesystem like NFS so it knows it is shared.
Quoting the virsh man page my setup is similar to "disk images are stored on coherent clustered filesystem, such as GFS2 or GPFS" but my libvirt doesn't know that and therefore changes the ownership. Maybe it would have the same bug on GFS2/GPFS? I don't have such a setup at hand, how is the issue avoided there? Is there a way to make libvirt realize that it really is on "the same" FS to avoid these operations?
Libvirt uses statfs to find the filesystem magic virFileIsSharedFSType checks NFS, GFS2, OCFS, AFS, SMB, CIFS, CEPH & GPFS Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Tue, Jul 2, 2019 at 10:29 AM Daniel P. Berrangé <berrange@redhat.com> wrote:
On Mon, Jul 01, 2019 at 08:20:44PM +0200, Christian Ehrhardt wrote:
On Mon, Jul 1, 2019 at 7:56 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
On Mon, Jul 01, 2019 at 05:26:45PM +0200, Christian Ehrhardt wrote:
Hi, today I was debugging an issue that I found with qemu 4.0 and ended up puzzled about file ownership. I'm almost EOD now, but wanted to reach out here for a sanity check before I debug further. The case sumamrized is this:
1. start guest 1.1 image files are changed to libvirt-qemu:kvm (which matches Ubuntus user/group config) 2. live migrate the guest to a different node 2.1 image files go back to root:root which they initially had (ok) 3. migrate guest back to the original node x. image files stay root:root and are not changed back to libvirt-qemu:kvm
That is odd/unexpected, but it seems the same applies to older versions and there it was never a problem so far.
This sounds odd - I see no reason why the first migration should behave differently from the second migration, as libvirt makes no distinction & has no knowledge of previous migrations.
I might have been unclear, let me clarify: 1. migration without copy (changes the ownership) lxc exec testkvm-eoan-from -- virsh migrate --live --unsafe kvmguest-eoan-normal qemu+ssh://10.222.144.19/system (and the same type backwards) 2. migration away from here with --copy-storage* fails due to file ownership
Could the two hosts be configured differently in some way. For example is the disk image on ext4 on one host, but nfs on the other host ? With shared filesystems, we'd generally expect the disk to be exposed on a shared filesystem on both hosts.
The two hosts are in fact two LXD containers sharing the same Filesystem (for the first migration) And still two containers, but without shared FS for the second copy-storage-* migration that then eventually fails due to the bad file ownership.
But since they in fact use shared storage that might still be the reason - maybe the ordering is important here. Since in the setup for migration #1 they really use shared storage it might be that we have something like
1. source: migrates off guest 2. target: receives guest 3. target: guest is complete and changes file ownership to libvirt-qemu:kvm (correct) 4. source: shuts down the stub that is left after migration and changes file ownership to root:root
Due to really using the same storage (not like the same FCP or scsi device but the same FS) that could be the reason the ownership after a migration cycle is reset. Thanks for the hint, worth a check with some slightly altered setups ... ... and confirmed - if I use copied images, but not on a shared FS the ownership is handled correctly.
So my way of sharing the FS might be odd. And libvirt does not detect it as such, and due to that above ordering triggers the issue or the ownership being set to root:root by the migration source after the migration is complete.
Yes, this is going to be a problem. Libvirt will look at the filesystem and see that it is a local-only filesystem, and so assume it can safely reset ownership when source VM shuts down. It cannot tell that this is going to negatively impact the target VM using the same filesystem.
This is an example of one of the problems that makes us say that localhost migration is not supported.
If you're going to use containers, you need to make sure that each container either has a separate filesystem mount, or that the container sees a filesystem like NFS so it knows it is shared.
Quoting the virsh man page my setup is similar to "disk images are stored on coherent clustered filesystem, such as GFS2 or GPFS" but my libvirt doesn't know that and therefore changes the ownership. Maybe it would have the same bug on GFS2/GPFS? I don't have such a setup at hand, how is the issue avoided there? Is there a way to make libvirt realize that it really is on "the same" FS to avoid these operations?
Libvirt uses statfs to find the filesystem magic virFileIsSharedFSType checks NFS, GFS2, OCFS, AFS, SMB, CIFS, CEPH & GPFS
Thanks for the pointer Daniel! To stat -f [1] it appears to be the same (ext2/ext3) as in the host. I started a discussion with our container people if I could detect/differentiate that somehow. Until then, now that I'm aware I can just let automation clean up ownership after those actions. [1]: http://paste.ubuntu.com/p/jffSrtKg7t/
Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
-- Christian Ehrhardt Software Engineer, Ubuntu Server Canonical Ltd
participants (2)
-
Christian Ehrhardt -
Daniel P. Berrangé