Re: Libvirt + Debian Live = Heart Attack

Hey Martin, wow thanks for the detailed info! I'll definitely look into this, but I kind of figured it out, I believe. So, I keep the images in the read-only squashfs, boot the live system and use snapshot writeback: virsh snapshot-create-as with options --disk-only --diskspec

This pretty much fixes my problem, and only disk writes occur in tmpfs, instead of the entire image being copied. BUT, now I have to reduce the disk space within the VM to almost 0 to save space.

I'm reading up on qemu-img convert and virt-sparsify to accomplish this, but I will also try your method now, it seems more professional. Thank you for your amazing input and patience, I love this community. Sharing this information is the literal definition of freedom IMO, thank you all!

On Thu, Nov 25, 2021, 11:40 AM Martin Kletzander <mkletzan(a)redhat.com&gt; wrote: > On Wed, Nov 24, 2021 at 06:21:22PM +0100, Elias Mobery wrote: > >Hey Martin & Michal, thank you both for the replies! > > > >Yes, sorry I messed up that seclabel entry, thanks. I fixed it and there > is > >no more error, but the images are still copied unfortunately. > > > >@Michal yes, I have been reading the Debian Live Manual like a contract, > >after messing with qemu.conf and permissions for days, I think its > >something in the live build and not libvirt. > > > >@Martin yeah both virsh start and virt-manager cause the images to be > >copied to /run..... first. > > > >Yes, when I boot the Debian Live ISO, plug in the USB (mounted rw), copy > >the images to the live system from the USB and click run in virt-manager, > >the VM starts instantly and no copies are made in /run.... > > > >I use ISO loopback and toram to load the system directly into ram via > tmpfs > >mount. That's my grub.cfg > > > >menuentry "debian-live.iso" { set iso="/ISO/debian-live.iso" set > >root=(hd0,5) loopback loop $iso linux (loop)/live/vmlinuz-4.19.0-6-amd64 > >boot=live components toram findiso=$iso initrd > >(loop)/live/initrd.img-4.19.0-6-amd64 } > > > >So, the squashfs is mounted read-only in > >/run/live/rootfs/filesystem.squashfs and its /dev/loop1 > > > >/proc/mounts says: > > > >tmpfs /run/live/overlay rw > >overlay rw lowerdir=/run/live/rootfs/filesystem.squashfs > >upperdir=/run/live/overlay/rw (same place where the images are copied) > > > > I do not think this is relevant, but sometimes overlayfs wants to have a > workdir as well. > > >So you think that because the images are part of the squashfs which is > >read-only, they are first copied to /run/live/overlay/rw... to be written > >to? > > > > Are they really copied? I can't think of many more things why this > would happen. Maybe some locking code does that in the backend (both > libvirt and qemu lock parts of the disks). > > >I always thought that tmpfs is mounted on top of squash, so it's as if the > >squashfs were writable, without having to literally copy between the two > >filesystems. > > > > It is not mounted over, if you mount over something you lose the > lowerdir, that's why it needs to be done with overlayfs which does the > copy-on-write. > > That leads me to another idea. Since you want copy-on-write for the > images, I presume, the image will get copied anyway once the guest > writes even a single byte to the image. Unfortunately overlayfs is not > the best way for this scenario due to the granularity it offers > (per-file, not per-block or anything smaller). What I think you > actually want in this case is to have a qcow image with the original > image used as backing. You can create those with libvirt or even > without it, just using qemu-img, for example like this: > > qemu-img create -b /path/to/non-modified.image -f qcow2 -F > format_of_backing_file /path/to/new-image.qcow2 > > For more information please read the qemu-img manual page to make sure > you use it properly, I just wrote the above from the top of my head, not > tested it. The backing file path can also be relative. > > If you specify the new image for the machines then the granularity > changes to way smaller blocks than the whole file making it occupy > less space in case of changes. > > >Also I made the same build with Virtualbox and there is no copying of the > >images to /run/live/overlay. > > > >I will do some serious research into squashfs and tmpfs rn. > > > > If you want to figure out when it happens I would suggest enabling debug > logs for libvirt, starting the qemu VM with --paused, but maybe it will > need some more debugging on lower layers. Anyway I really believe the > approach suggested above is what will lead to better results for you. > > Have a nice day > Martin > > > > > > > > >On Wed, Nov 24, 2021, 5:10 PM Martin Kletzander <mkletzan(a)redhat.com&gt; > wrote: > > > >> On Wed, Nov 24, 2021 at 04:01:34PM +0100, Elias Mobery wrote: > >> >Hello Michal, thank you for the reply! > >> > > >> >I've carefully tested everything you suggested, thanks. > >> > > >> >I set dynamic_ownership=0 and use these hooks during the live build for > >> >permissions. (I googled a lot, and apparently libvirt needs the images > to > >> >be executable too) > >> > > >> >chown -R libvirt-qemu:kvm /var/lib/libvirt/images > >> >chmod -R g+rwx /var/lib/libvirt/images > >> > > >> > >> I do not see why would the images need to be executable, you might need > >> an executable bit set on the directory so that your and qemu user can > >> browse it. > >> > >> >Booting the live debian iso everything works in virt-manager, but > again, > >> >after clicking "run", a copy of the vm image is created in > >> >/run/live/overlay/rw/var/lib/libvirt/images and only then does the VM > >> start. > >> > > >> > >> Does that happen when you try to run it with virsh instead of > >> virt-manager? Are you connected to the system daemon? > >> > >> >Either it's still being chowned or chmodded somehow, or it's something > >> >else, but I can't stop this copy being made. > >> > > >> >Interestingly, when I boot the Live debian iso and then copy the images > >> >into /var/lib/libvirt/images from my USB stick, the VM starts > immediately > >> >without creating any copies in the /run/live.... directory. So my > guess is > >> >that maybe the squashfs could be the issue? > >> > > >> > >> Oh, interesting, is the USB stick filesystem mounted R/W and the live > >> ISO filesystem mounted read-only? How is the overlay mounted? > >> > >> >Editing the XML > >> > > >> ><source file='/var/lib/libvirt/images/vm1.qcow2'> > >> > <seclabel relabel='no'/> > >> > </source> > >> > > >> >This results in an error: > >> >Unsupported Configuration: > >> >Security driver model 'null' not available > >> > > >> > >> For issues like this it is most helpful to check the documentation: > >> > >> https://libvirt.org/formatdomain.html > >> > >> where you can see it is just a missing attribute `model`, in this case > >> model="dac". > >> > >> >Here I tried setting security_driver=none in qemu.conf but same error > >> after. > >> > > >> ></devices> > >> > <seclabel type='none'/> > >> > </domain> > >> > > >> > >> Same here. > >> > >> >This also returns an Error but I'm still googling to understand it > >> properly. > >> > > >> >XML document failed to validate against schema > >> >Unable to validate doc against /usr/share/libvirt/schemas/domain.rng > >> >Invalid element relabel for element seclabel > >> >Extra element seclabel in interleave > >> >Element domain failed to validate content > >> > > >> >Thanks again so much for your helo, I've been messing with this for > weeks > >> >now and it's killing me. > >> > > >> >On Tue, Nov 23, 2021, 9:43 PM Michal Prívozník <mprivozn(a)redhat.com&gt; > >> wrote: > >> > > >> >> On 11/23/21 17:25, Elias Mobery wrote: > >> >> > > >> >> > Hi everyone! > >> >> > > >> >> > I've built a Debian Live ISO with packages qemu and libvirt to run > a > >> VM > >> >> > in the live environment. > >> >> > > >> >> > The guest images are placed in /var/lib/libvirt/images and 2GB > each. > >> >> > > >> >> > Everything works great, except for one issue. > >> >> > > >> >> > When starting a VM, libvirt automatically issues a chown command to > >> the > >> >> > images, changing ownership. > >> >> > > >> >> > This results in a copy of the images being created in > >> >> > /run/live/overlay/rw/var/lib/libvirt/images > >> >> > > >> >> > I don't want these copies to be made but can't stop it. > >> >> > > >> >> > I've tried editing qemu.conf user/group, dynamic ownership etc. > >> without > >> >> > any luck. > >> >> > > >> >> > Is there a way to STOP libvirt from changing the ownership of these > >> >> images? > >> >> > > >> >> > > >> >> > >> >> Setting dynamic_ownership=0 in qemu.conf is the way to go (don't > forget > >> >> to restart the daemon after you made the change). > >> >> > >> >> Alternatively, you can set <seclabel/> either for whole domain or > >> >> individual disks, e.g. like this: > >> >> > >> >> <disk type='file' device='disk'> > >> >> <driver name='qemu' type='qcow2'/> > >> >> <source file='/var/lib/libvirt/images/vm1.qcow2'> > >> >> <seclabel relabel='no'/> > >> >> </source> > >> >> </disk> > >> >> > >> >> or for whole domain: > >> >> > >> >> ... > >> >> </devices> > >> >> <seclabel type='none'/> > >> >> </domain> > >> >> > >> >> I'm not sure what your setup is, but if chown() is a problem then > what > >> >> if guest tries to write onto its disk? Wouldn't that create a copy in > >> >> overlay? > >> >> > >> >> Michal > >> >> > >> >> > >> >