On 11/23/21 17:25, Elias Mobery wrote: > > Hi everyone! > > I've built a Debian Live ISO with packages qemu and libvirt to run a VM > in the live environment. > > The guest images are placed in /var/lib/libvirt/images and 2GB each. > > Everything works great, except for one issue. > > When starting a VM, libvirt automatically issues a chown command to the > images, changing ownership. > > This results in a copy of the images being created in > /run/live/overlay/rw/var/lib/libvirt/images > > I don't want these copies to be made but can't stop it. > > I've tried editing qemu.conf user/group, dynamic ownership etc. without > any luck. > > Is there a way to STOP libvirt from changing the ownership of these images? > > Setting dynamic_ownership=0 in qemu.conf is the way to go (don't forget to restart the daemon after you made the change). Alternatively, you can set <seclabel/> either for whole domain or individual disks, e.g. like this: <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/var/lib/libvirt/images/vm1.qcow2'> <seclabel relabel='no'/> </source> </disk> or for whole domain: ... </devices> <seclabel type='none'/> </domain> I'm not sure what your setup is, but if chown() is a problem then what if guest tries to write onto its disk? Wouldn't that create a copy in overlay? Michal

Hello Michal, thank you for the reply! I've carefully tested everything you suggested, thanks. I set dynamic_ownership=0 and use these hooks during the live build for permissions. (I googled a lot, and apparently libvirt needs the images to be executable too) chown -R libvirt-qemu:kvm /var/lib/libvirt/images chmod -R g+rwx /var/lib/libvirt/images

Booting the live debian iso everything works in virt-manager, but again, after clicking "run", a copy of the vm image is created in /run/live/overlay/rw/var/lib/libvirt/images and only then does the VM start.

Either it's still being chowned or chmodded somehow, or it's something else, but I can't stop this copy being made. Interestingly, when I boot the Live debian iso and then copy the images into /var/lib/libvirt/images from my USB stick, the VM starts immediately without creating any copies in the /run/live.... directory. So my guess is that maybe the squashfs could be the issue?

Editing the XML <source file='/var/lib/libvirt/images/vm1.qcow2'> <seclabel relabel='no'/> </source> This results in an error: Unsupported Configuration: Security driver model 'null' not available

Here I tried setting security_driver=none in qemu.conf but same error after. </devices> <seclabel type='none'/> </domain>

This also returns an Error but I'm still googling to understand it properly. XML document failed to validate against schema Unable to validate doc against /usr/share/libvirt/schemas/domain.rng Invalid element relabel for element seclabel Extra element seclabel in interleave Element domain failed to validate content Thanks again so much for your helo, I've been messing with this for weeks now and it's killing me. On Tue, Nov 23, 2021, 9:43 PM Michal Prívozník <mprivozn(a)redhat.com&gt; wrote: > On 11/23/21 17:25, Elias Mobery wrote: > > > > Hi everyone! > > > > I've built a Debian Live ISO with packages qemu and libvirt to run a VM > > in the live environment. > > > > The guest images are placed in /var/lib/libvirt/images and 2GB each. > > > > Everything works great, except for one issue. > > > > When starting a VM, libvirt automatically issues a chown command to the > > images, changing ownership. > > > > This results in a copy of the images being created in > > /run/live/overlay/rw/var/lib/libvirt/images > > > > I don't want these copies to be made but can't stop it. > > > > I've tried editing qemu.conf user/group, dynamic ownership etc. without > > any luck. > > > > Is there a way to STOP libvirt from changing the ownership of these > images? > > > > > > Setting dynamic_ownership=0 in qemu.conf is the way to go (don't forget > to restart the daemon after you made the change). > > Alternatively, you can set <seclabel/> either for whole domain or > individual disks, e.g. like this: > > <disk type='file' device='disk'> > <driver name='qemu' type='qcow2'/> > <source file='/var/lib/libvirt/images/vm1.qcow2'> > <seclabel relabel='no'/> > </source> > </disk> > > or for whole domain: > > ... > </devices> > <seclabel type='none'/> > </domain> > > I'm not sure what your setup is, but if chown() is a problem then what > if guest tries to write onto its disk? Wouldn't that create a copy in > overlay? > > Michal > >

Hey Martin & Michal, thank you both for the replies! Yes, sorry I messed up that seclabel entry, thanks. I fixed it and there is no more error, but the images are still copied unfortunately. @Michal yes, I have been reading the Debian Live Manual like a contract, after messing with qemu.conf and permissions for days, I think its something in the live build and not libvirt. @Martin yeah both virsh start and virt-manager cause the images to be copied to /run..... first. Yes, when I boot the Debian Live ISO, plug in the USB (mounted rw), copy the images to the live system from the USB and click run in virt-manager, the VM starts instantly and no copies are made in /run.... I use ISO loopback and toram to load the system directly into ram via tmpfs mount. That's my grub.cfg menuentry "debian-live.iso" { set iso="/ISO/debian-live.iso" set root=(hd0,5) loopback loop $iso linux (loop)/live/vmlinuz-4.19.0-6-amd64 boot=live components toram findiso=$iso initrd (loop)/live/initrd.img-4.19.0-6-amd64 } So, the squashfs is mounted read-only in /run/live/rootfs/filesystem.squashfs and its /dev/loop1 /proc/mounts says: tmpfs /run/live/overlay rw overlay rw lowerdir=/run/live/rootfs/filesystem.squashfs upperdir=/run/live/overlay/rw (same place where the images are copied)

So you think that because the images are part of the squashfs which is read-only, they are first copied to /run/live/overlay/rw... to be written to?

I always thought that tmpfs is mounted on top of squash, so it's as if the squashfs were writable, without having to literally copy between the two filesystems.

Also I made the same build with Virtualbox and there is no copying of the images to /run/live/overlay. I will do some serious research into squashfs and tmpfs rn.

On Wed, Nov 24, 2021, 5:10 PM Martin Kletzander <mkletzan(a)redhat.com&gt; wrote: > On Wed, Nov 24, 2021 at 04:01:34PM +0100, Elias Mobery wrote: > >Hello Michal, thank you for the reply! > > > >I've carefully tested everything you suggested, thanks. > > > >I set dynamic_ownership=0 and use these hooks during the live build for > >permissions. (I googled a lot, and apparently libvirt needs the images to > >be executable too) > > > >chown -R libvirt-qemu:kvm /var/lib/libvirt/images > >chmod -R g+rwx /var/lib/libvirt/images > > > > I do not see why would the images need to be executable, you might need > an executable bit set on the directory so that your and qemu user can > browse it. > > >Booting the live debian iso everything works in virt-manager, but again, > >after clicking "run", a copy of the vm image is created in > >/run/live/overlay/rw/var/lib/libvirt/images and only then does the VM > start. > > > > Does that happen when you try to run it with virsh instead of > virt-manager? Are you connected to the system daemon? > > >Either it's still being chowned or chmodded somehow, or it's something > >else, but I can't stop this copy being made. > > > >Interestingly, when I boot the Live debian iso and then copy the images > >into /var/lib/libvirt/images from my USB stick, the VM starts immediately > >without creating any copies in the /run/live.... directory. So my guess is > >that maybe the squashfs could be the issue? > > > > Oh, interesting, is the USB stick filesystem mounted R/W and the live > ISO filesystem mounted read-only? How is the overlay mounted? > > >Editing the XML > > > ><source file='/var/lib/libvirt/images/vm1.qcow2'> > > <seclabel relabel='no'/> > > </source> > > > >This results in an error: > >Unsupported Configuration: > >Security driver model 'null' not available > > > > For issues like this it is most helpful to check the documentation: > > https://libvirt.org/formatdomain.html > > where you can see it is just a missing attribute `model`, in this case > model="dac". > > >Here I tried setting security_driver=none in qemu.conf but same error > after. > > > ></devices> > > <seclabel type='none'/> > > </domain> > > > > Same here. > > >This also returns an Error but I'm still googling to understand it > properly. > > > >XML document failed to validate against schema > >Unable to validate doc against /usr/share/libvirt/schemas/domain.rng > >Invalid element relabel for element seclabel > >Extra element seclabel in interleave > >Element domain failed to validate content > > > >Thanks again so much for your helo, I've been messing with this for weeks > >now and it's killing me. > > > >On Tue, Nov 23, 2021, 9:43 PM Michal Prívozník <mprivozn(a)redhat.com&gt; > wrote: > > > >> On 11/23/21 17:25, Elias Mobery wrote: > >> > > >> > Hi everyone! > >> > > >> > I've built a Debian Live ISO with packages qemu and libvirt to run a > VM > >> > in the live environment. > >> > > >> > The guest images are placed in /var/lib/libvirt/images and 2GB each. > >> > > >> > Everything works great, except for one issue. > >> > > >> > When starting a VM, libvirt automatically issues a chown command to > the > >> > images, changing ownership. > >> > > >> > This results in a copy of the images being created in > >> > /run/live/overlay/rw/var/lib/libvirt/images > >> > > >> > I don't want these copies to be made but can't stop it. > >> > > >> > I've tried editing qemu.conf user/group, dynamic ownership etc. > without > >> > any luck. > >> > > >> > Is there a way to STOP libvirt from changing the ownership of these > >> images? > >> > > >> > > >> > >> Setting dynamic_ownership=0 in qemu.conf is the way to go (don't forget > >> to restart the daemon after you made the change). > >> > >> Alternatively, you can set <seclabel/> either for whole domain or > >> individual disks, e.g. like this: > >> > >> <disk type='file' device='disk'> > >> <driver name='qemu' type='qcow2'/> > >> <source file='/var/lib/libvirt/images/vm1.qcow2'> > >> <seclabel relabel='no'/> > >> </source> > >> </disk> > >> > >> or for whole domain: > >> > >> ... > >> </devices> > >> <seclabel type='none'/> > >> </domain> > >> > >> I'm not sure what your setup is, but if chown() is a problem then what > >> if guest tries to write onto its disk? Wouldn't that create a copy in > >> overlay? > >> > >> Michal > >> > >> >

Hey Martin, wow thanks for the detailed info! I'll definitely look into this, but I kind of figured it out, I believe. So, I keep the images in the read-only squashfs, boot the live system and use snapshot writeback: virsh snapshot-create-as with options --disk-only --diskspec

This pretty much fixes my problem, and only disk writes occur in tmpfs, instead of the entire image being copied. BUT, now I have to reduce the disk space within the VM to almost 0 to save space.

I'm reading up on qemu-img convert and virt-sparsify to accomplish this, but I will also try your method now, it seems more professional. Thank you for your amazing input and patience, I love this community. Sharing this information is the literal definition of freedom IMO, thank you all!

On Thu, Nov 25, 2021, 11:40 AM Martin Kletzander <mkletzan(a)redhat.com&gt; wrote: > On Wed, Nov 24, 2021 at 06:21:22PM +0100, Elias Mobery wrote: > >Hey Martin & Michal, thank you both for the replies! > > > >Yes, sorry I messed up that seclabel entry, thanks. I fixed it and there > is > >no more error, but the images are still copied unfortunately. > > > >@Michal yes, I have been reading the Debian Live Manual like a contract, > >after messing with qemu.conf and permissions for days, I think its > >something in the live build and not libvirt. > > > >@Martin yeah both virsh start and virt-manager cause the images to be > >copied to /run..... first. > > > >Yes, when I boot the Debian Live ISO, plug in the USB (mounted rw), copy > >the images to the live system from the USB and click run in virt-manager, > >the VM starts instantly and no copies are made in /run.... > > > >I use ISO loopback and toram to load the system directly into ram via > tmpfs > >mount. That's my grub.cfg > > > >menuentry "debian-live.iso" { set iso="/ISO/debian-live.iso" set > >root=(hd0,5) loopback loop $iso linux (loop)/live/vmlinuz-4.19.0-6-amd64 > >boot=live components toram findiso=$iso initrd > >(loop)/live/initrd.img-4.19.0-6-amd64 } > > > >So, the squashfs is mounted read-only in > >/run/live/rootfs/filesystem.squashfs and its /dev/loop1 > > > >/proc/mounts says: > > > >tmpfs /run/live/overlay rw > >overlay rw lowerdir=/run/live/rootfs/filesystem.squashfs > >upperdir=/run/live/overlay/rw (same place where the images are copied) > > > > I do not think this is relevant, but sometimes overlayfs wants to have a > workdir as well. > > >So you think that because the images are part of the squashfs which is > >read-only, they are first copied to /run/live/overlay/rw... to be written > >to? > > > > Are they really copied? I can't think of many more things why this > would happen. Maybe some locking code does that in the backend (both > libvirt and qemu lock parts of the disks). > > >I always thought that tmpfs is mounted on top of squash, so it's as if the > >squashfs were writable, without having to literally copy between the two > >filesystems. > > > > It is not mounted over, if you mount over something you lose the > lowerdir, that's why it needs to be done with overlayfs which does the > copy-on-write. > > That leads me to another idea. Since you want copy-on-write for the > images, I presume, the image will get copied anyway once the guest > writes even a single byte to the image. Unfortunately overlayfs is not > the best way for this scenario due to the granularity it offers > (per-file, not per-block or anything smaller). What I think you > actually want in this case is to have a qcow image with the original > image used as backing. You can create those with libvirt or even > without it, just using qemu-img, for example like this: > > qemu-img create -b /path/to/non-modified.image -f qcow2 -F > format_of_backing_file /path/to/new-image.qcow2 > > For more information please read the qemu-img manual page to make sure > you use it properly, I just wrote the above from the top of my head, not > tested it. The backing file path can also be relative. > > If you specify the new image for the machines then the granularity > changes to way smaller blocks than the whole file making it occupy > less space in case of changes. > > >Also I made the same build with Virtualbox and there is no copying of the > >images to /run/live/overlay. > > > >I will do some serious research into squashfs and tmpfs rn. > > > > If you want to figure out when it happens I would suggest enabling debug > logs for libvirt, starting the qemu VM with --paused, but maybe it will > need some more debugging on lower layers. Anyway I really believe the > approach suggested above is what will lead to better results for you. > > Have a nice day > Martin > > > > > > > > >On Wed, Nov 24, 2021, 5:10 PM Martin Kletzander <mkletzan(a)redhat.com&gt; > wrote: > > > >> On Wed, Nov 24, 2021 at 04:01:34PM +0100, Elias Mobery wrote: > >> >Hello Michal, thank you for the reply! > >> > > >> >I've carefully tested everything you suggested, thanks. > >> > > >> >I set dynamic_ownership=0 and use these hooks during the live build for > >> >permissions. (I googled a lot, and apparently libvirt needs the images > to > >> >be executable too) > >> > > >> >chown -R libvirt-qemu:kvm /var/lib/libvirt/images > >> >chmod -R g+rwx /var/lib/libvirt/images > >> > > >> > >> I do not see why would the images need to be executable, you might need > >> an executable bit set on the directory so that your and qemu user can > >> browse it. > >> > >> >Booting the live debian iso everything works in virt-manager, but > again, > >> >after clicking "run", a copy of the vm image is created in > >> >/run/live/overlay/rw/var/lib/libvirt/images and only then does the VM > >> start. > >> > > >> > >> Does that happen when you try to run it with virsh instead of > >> virt-manager? Are you connected to the system daemon? > >> > >> >Either it's still being chowned or chmodded somehow, or it's something > >> >else, but I can't stop this copy being made. > >> > > >> >Interestingly, when I boot the Live debian iso and then copy the images > >> >into /var/lib/libvirt/images from my USB stick, the VM starts > immediately > >> >without creating any copies in the /run/live.... directory. So my > guess is > >> >that maybe the squashfs could be the issue? > >> > > >> > >> Oh, interesting, is the USB stick filesystem mounted R/W and the live > >> ISO filesystem mounted read-only? How is the overlay mounted? > >> > >> >Editing the XML > >> > > >> ><source file='/var/lib/libvirt/images/vm1.qcow2'> > >> > <seclabel relabel='no'/> > >> > </source> > >> > > >> >This results in an error: > >> >Unsupported Configuration: > >> >Security driver model 'null' not available > >> > > >> > >> For issues like this it is most helpful to check the documentation: > >> > >> https://libvirt.org/formatdomain.html > >> > >> where you can see it is just a missing attribute `model`, in this case > >> model="dac". > >> > >> >Here I tried setting security_driver=none in qemu.conf but same error > >> after. > >> > > >> ></devices> > >> > <seclabel type='none'/> > >> > </domain> > >> > > >> > >> Same here. > >> > >> >This also returns an Error but I'm still googling to understand it > >> properly. > >> > > >> >XML document failed to validate against schema > >> >Unable to validate doc against /usr/share/libvirt/schemas/domain.rng > >> >Invalid element relabel for element seclabel > >> >Extra element seclabel in interleave > >> >Element domain failed to validate content > >> > > >> >Thanks again so much for your helo, I've been messing with this for > weeks > >> >now and it's killing me. > >> > > >> >On Tue, Nov 23, 2021, 9:43 PM Michal Prívozník <mprivozn(a)redhat.com&gt; > >> wrote: > >> > > >> >> On 11/23/21 17:25, Elias Mobery wrote: > >> >> > > >> >> > Hi everyone! > >> >> > > >> >> > I've built a Debian Live ISO with packages qemu and libvirt to run > a > >> VM > >> >> > in the live environment. > >> >> > > >> >> > The guest images are placed in /var/lib/libvirt/images and 2GB > each. > >> >> > > >> >> > Everything works great, except for one issue. > >> >> > > >> >> > When starting a VM, libvirt automatically issues a chown command to > >> the > >> >> > images, changing ownership. > >> >> > > >> >> > This results in a copy of the images being created in > >> >> > /run/live/overlay/rw/var/lib/libvirt/images > >> >> > > >> >> > I don't want these copies to be made but can't stop it. > >> >> > > >> >> > I've tried editing qemu.conf user/group, dynamic ownership etc. > >> without > >> >> > any luck. > >> >> > > >> >> > Is there a way to STOP libvirt from changing the ownership of these > >> >> images? > >> >> > > >> >> > > >> >> > >> >> Setting dynamic_ownership=0 in qemu.conf is the way to go (don't > forget > >> >> to restart the daemon after you made the change). > >> >> > >> >> Alternatively, you can set <seclabel/> either for whole domain or > >> >> individual disks, e.g. like this: > >> >> > >> >> <disk type='file' device='disk'> > >> >> <driver name='qemu' type='qcow2'/> > >> >> <source file='/var/lib/libvirt/images/vm1.qcow2'> > >> >> <seclabel relabel='no'/> > >> >> </source> > >> >> </disk> > >> >> > >> >> or for whole domain: > >> >> > >> >> ... > >> >> </devices> > >> >> <seclabel type='none'/> > >> >> </domain> > >> >> > >> >> I'm not sure what your setup is, but if chown() is a problem then > what > >> >> if guest tries to write onto its disk? Wouldn't that create a copy in > >> >> overlay? > >> >> > >> >> Michal > >> >> > >> >> > >> >