On Mon, Mar 23, 2020 at 10:03:13AM -0500, Joe Muro wrote:

Hi Martin, thanks for the explanation. Now I understand why libvirt doesn't revert the file permissions back to the original. I am running these VMs on an isolated test machine, so I'll disable dynamic file ownership and make sure libvirt has access to image files.

Or access the disks only using libvirt and friends ;) Also, is there no upgrade for the libvirt package? Maybe I don't exactly remember if anything else is needed. Adding Michal to Cc since he'll know more.

Sorry about the message formatting. I modified settings on my client, hopefully it sends plaintext now. (I'll switch to personal email going forward, as the choice of email clients at work is limited.)

No problem, you couldn't know unless you specifically looked for it, don't worry about it ;)

-Joe

From: Martin Kletzander <mkletzan@redhat.com> To: Joe Muro <joemuro@us.ibm.com> Cc: libvirt-users@redhat.com Date: 03/20/2020 04:01 PM Subject: [EXTERNAL] Re: libvirt dynamic file ownership

On Fri, Mar 20, 2020 at 03:38:36PM +0000, Joe Muro wrote:

...

Hi,

Hi, could you please configure your client to send plaintext version as well? We mainly prefer plaintext on this list ;-)

...

I am trying to understand libvirt dynamic ownership behavior. I have a VM that uses a qcow2 image with the following permissions:

$ ll t257kvxg-10-20-101-40.qcow2 -rw-r--r-- 1 jmuro libvirt 2279079936 Mar 20 11:10 t257kvxg-10-20-101-40.qcow2

When I start the domain the permissions are changed:

$ virsh start t257kvxg-10-20-101-40 Domain t257kvxg-10-20-101-40 started $ ll t257kvxg-10-20-101-40.qcow2 -rw-r--r-- 1 libvirt-qemu libvirt 2279079936 Mar 20 11:18 t257kvxg-10-20-101-40.qcow2

This is expected behavior based on the settings in /etc/libvirt/qemu.conf:

user = "libvirt-qemu" group = "libvirt" # Whether libvirt should dynamically change file ownership # to match the configured user/group above. Defaults to 1. # Set to 0 to disable file ownership changes. #dynamic_ownership = 1

However, when I shutdown the domain, the file permissions revert to root.

$ ll t257kvxg-10-20-101-40.qcow2 -rw-r--r-- 1 root root 2282749952 Mar 20 11:20 t257kvxg-10-20-101-40.qcow2

I expect libvirt to revert the file permissions back to the original. Otherwise, a regular user would lose ownership of the image file. FWIW: I am starting the domain as a non-root user under qemu:///system

This has always been the case because the original information is lost (which is actually not that easy to store properly, race-free, etc.) and the safest way to make sure nobody accesses the disks (e.g. another domain running under libvirt-qemu:libvirt, that would get exploited) is to just change it to root:root. Michal finally managed to make this work, in limited cases, but I think it landed in 6.1.0, I'm not sure.

Anyway, there are some workarounds you can do:

a) set relabel=no for the disk in the XML (and make sure the VM will be able to access it),

b) set relabel=no for the whole domain (and make sure the VM will be able to access everything), or

c) if worse comes to worse, just disable the whole dynamic ownership and handle it yourself

If possible, try upgrading libvirt and checking if that helps.

...

I am running on Ubuntu 20.04 LTS (Focal Fossa) with the following libvirt level:

libvirt-clients/focal,now 6.0.0-0ubuntu5 s390x [installed] libvirt-daemon-driver-qemu/focal,now 6.0.0-0ubuntu5 s390x [installed,automatic] libvirt-daemon-driver-storage-rbd/focal,now 6.0.0-0ubuntu5 s390x [installed,automatic] libvirt-daemon-system-systemd/focal,now 6.0.0-0ubuntu5 s390x [installed,automatic] libvirt-daemon-system/focal,now 6.0.0-0ubuntu5 s390x [installed] libvirt-daemon/focal,now 6.0.0-0ubuntu5 s390x [installed] libvirt-glib-1.0-0/focal,now 2.0.0-2 s390x [installed,automatic] libvirt0/focal,now 6.0.0-0ubuntu5 s390x [installed,automatic] python3-libvirt/focal,now 6.0.0-0ubuntu3 s390x [installed]

Thanks

-Joe

[attachment "signature.asc" deleted by Joe Muro/Poughkeepsie/IBM]

"Michal Prívozník" <mprivozn@redhat.com> wrote on 03/23/2020 12:26:14 PM:

From: "Michal Prívozník" <mprivozn@redhat.com> To: Martin Kletzander <mkletzan@redhat.com>, Joe Muro <joemuro@us.ibm.com> Cc: libvirt-users@redhat.com Date: 03/23/2020 12:26 PM Subject: [EXTERNAL] Re: libvirt dynamic file ownership

On 20. 3. 2020 20:57, Martin Kletzander wrote:

...

On Fri, Mar 20, 2020 at 03:38:36PM +0000, Joe Muro wrote:

...

Hi,

Hi, could you please configure your client to send plaintext version as well? We mainly prefer plaintext on this list ;-)

...

I am trying to understand libvirt dynamic ownership behavior. I have a VM that uses a qcow2 image with the following permissions:

$ ll t257kvxg-10-20-101-40.qcow2 -rw-r--r-- 1 jmuro libvirt 2279079936 Mar 20 11:10 t257kvxg-10-20-101-40.qcow2

When I start the domain the permissions are changed:

$ virsh start t257kvxg-10-20-101-40 Domain t257kvxg-10-20-101-40 started $ ll t257kvxg-10-20-101-40.qcow2 -rw-r--r-- 1 libvirt-qemu libvirt 2279079936 Mar 20 11:18 t257kvxg-10-20-101-40.qcow2

This is expected behavior based on the settings in /etc/libvirt/qemu.conf:

user = "libvirt-qemu" group = "libvirt" # Whether libvirt should dynamically change file ownership # to match the configured user/group above. Defaults to 1. # Set to 0 to disable file ownership changes. #dynamic_ownership = 1

However, when I shutdown the domain, the file permissions revert to root.

$ ll t257kvxg-10-20-101-40.qcow2 -rw-r--r-- 1 root root 2282749952 Mar 20 11:20 t257kvxg-10-20-101-40.qcow2

I expect libvirt to revert the file permissions back to the original. Otherwise, a regular user would lose ownership of the image file. FWIW: I am starting the domain as a non-root user under qemu:///system

This has always been the case because the original information is lost (which is actually not that easy to store properly, race-free, etc.) and the safest way to make sure nobody accesses the disks (e.g. another domain running under libvirt-qemu:libvirt, that would get exploited) is to just change it to root:root.  Michal finally managed to make this work, in limited cases, but I think it landed in 6.1.0, I'm not sure.

Anyway, there are some workarounds you can do:

 a) set relabel=no for the disk in the XML (and make sure the VM will be able to     access it),

 b) set relabel=no for the whole domain (and make sure the VM will be able to     access everything), or

 c) if worse comes to worse, just disable the whole dynamic ownership and handle     it yourself

If possible, try upgrading libvirt and checking if that helps.

Remembering of the original owner was enabled even in 6.0.0, but there are some prerequisites:

1) the FS that hosts the image must be capable of XATTRs. Note the NFS still isn't.

The qcow is hosted on an ext3 filesystem. $ findmnt /guestimages TARGET SOURCE FSTYPE OPTIONS /guestimages /dev/mapper/mpatha-part1 ext3 rw,relatime Not sure why we are using ext3. Regardless, it seems user_xattr option is not enabled. I imagine this must be enabled.

2) the disk XML. Can you please share the <disk/> snippet for this disk? It needs to be the top most layer of backing chain (if you have some snapshots over it).

<disk type='volume' device='disk'> <driver name='qemu' type='qcow2'/> <source pool='guestimages' volume='t257kvxg-10-20-101-40.qcow2'/> <target dev='vda' bus='virtio'/> <address type='ccw' cssid='0xfe' ssid='0x0' devno='0x0000'/> </disk> I am using a backing volume, do you need that info as well?

Upgrading to 6.1.0 would help though, because I'm constantly fixing some bugs in that area as I go along.

BTW: you can check if the original owner remembering is enabled for your domain if you look whether the domain status XML has rememberOwner set:

grep rememberOwner /var/run/libvirt/qemu/$domain.xml

It looks like it is enabled. $ sudo grep rememberOwner /var/run/libvirt/qemu/t257kvxg-10-20-101-40.xml <rememberOwner/>

Michal