Hello Michal, thank you for the reply! I've carefully tested everything you suggested, thanks. I set dynamic_ownership=0 and use these hooks during the live build for permissions. (I googled a lot, and apparently libvirt needs the images to be executable too) chown -R libvirt-qemu:kvm /var/lib/libvirt/images chmod -R g+rwx /var/lib/libvirt/images Booting the live debian iso everything works in virt-manager, but again, after clicking "run", a copy of the vm image is created in /run/live/overlay/rw/var/lib/libvirt/images and only then does the VM start. Either it's still being chowned or chmodded somehow, or it's something else, but I can't stop this copy being made. Interestingly, when I boot the Live debian iso and then copy the images into /var/lib/libvirt/images from my USB stick, the VM starts immediately without creating any copies in the /run/live.... directory. So my guess is that maybe the squashfs could be the issue? Editing the XML <source file='/var/lib/libvirt/images/vm1.qcow2'> <seclabel relabel='no'/> </source> This results in an error: Unsupported Configuration: Security driver model 'null' not available Here I tried setting security_driver=none in qemu.conf but same error after. </devices> <seclabel type='none'/> </domain> This also returns an Error but I'm still googling to understand it properly. XML document failed to validate against schema Unable to validate doc against /usr/share/libvirt/schemas/domain.rng Invalid element relabel for element seclabel Extra element seclabel in interleave Element domain failed to validate content Thanks again so much for your helo, I've been messing with this for weeks now and it's killing me. On Tue, Nov 23, 2021, 9:43 PM Michal Prívozník <mprivozn@redhat.com> wrote:

On 11/23/21 17:25, Elias Mobery wrote:

...

Hi everyone!

I've built a Debian Live ISO with packages qemu and libvirt to run a VM in the live environment.

The guest images are placed in /var/lib/libvirt/images and 2GB each.

Everything works great, except for one issue.

When starting a VM, libvirt automatically issues a chown command to the images, changing ownership.

This results in a copy of the images being created in /run/live/overlay/rw/var/lib/libvirt/images

I don't want these copies to be made but can't stop it.

I've tried editing qemu.conf user/group, dynamic ownership etc. without any luck.

Is there a way to STOP libvirt from changing the ownership of these

images?

...

Setting dynamic_ownership=0 in qemu.conf is the way to go (don't forget to restart the daemon after you made the change).

Alternatively, you can set <seclabel/> either for whole domain or individual disks, e.g. like this:

<disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/var/lib/libvirt/images/vm1.qcow2'> <seclabel relabel='no'/> </source> </disk>

or for whole domain:

... </devices> <seclabel type='none'/> </domain>

I'm not sure what your setup is, but if chown() is a problem then what if guest tries to write onto its disk? Wouldn't that create a copy in overlay?

Michal

On Wed, Nov 24, 2021 at 04:01:34PM +0100, Elias Mobery wrote:

Hello Michal, thank you for the reply!

I've carefully tested everything you suggested, thanks.

I set dynamic_ownership=0 and use these hooks during the live build for permissions. (I googled a lot, and apparently libvirt needs the images to be executable too)

chown -R libvirt-qemu:kvm /var/lib/libvirt/images chmod -R g+rwx /var/lib/libvirt/images

I do not see why would the images need to be executable, you might need an executable bit set on the directory so that your and qemu user can browse it.

Booting the live debian iso everything works in virt-manager, but again, after clicking "run", a copy of the vm image is created in /run/live/overlay/rw/var/lib/libvirt/images and only then does the VM start.

Does that happen when you try to run it with virsh instead of virt-manager? Are you connected to the system daemon?

Either it's still being chowned or chmodded somehow, or it's something else, but I can't stop this copy being made.

Interestingly, when I boot the Live debian iso and then copy the images into /var/lib/libvirt/images from my USB stick, the VM starts immediately without creating any copies in the /run/live.... directory. So my guess is that maybe the squashfs could be the issue?

Oh, interesting, is the USB stick filesystem mounted R/W and the live ISO filesystem mounted read-only? How is the overlay mounted?

Editing the XML

<source file='/var/lib/libvirt/images/vm1.qcow2'> <seclabel relabel='no'/> </source>

This results in an error: Unsupported Configuration: Security driver model 'null' not available

For issues like this it is most helpful to check the documentation: https://libvirt.org/formatdomain.html where you can see it is just a missing attribute `model`, in this case model="dac".

Here I tried setting security_driver=none in qemu.conf but same error after.

</devices> <seclabel type='none'/> </domain>

Same here.

This also returns an Error but I'm still googling to understand it properly.

XML document failed to validate against schema Unable to validate doc against /usr/share/libvirt/schemas/domain.rng Invalid element relabel for element seclabel Extra element seclabel in interleave Element domain failed to validate content

Thanks again so much for your helo, I've been messing with this for weeks now and it's killing me.

On Tue, Nov 23, 2021, 9:43 PM Michal Prívozník <mprivozn@redhat.com> wrote:

...

On 11/23/21 17:25, Elias Mobery wrote:

...

Hi everyone!

I've built a Debian Live ISO with packages qemu and libvirt to run a VM in the live environment.

The guest images are placed in /var/lib/libvirt/images and 2GB each.

Everything works great, except for one issue.

When starting a VM, libvirt automatically issues a chown command to the images, changing ownership.

This results in a copy of the images being created in /run/live/overlay/rw/var/lib/libvirt/images

I don't want these copies to be made but can't stop it.

I've tried editing qemu.conf user/group, dynamic ownership etc. without any luck.

Is there a way to STOP libvirt from changing the ownership of these

images?

...

Setting dynamic_ownership=0 in qemu.conf is the way to go (don't forget to restart the daemon after you made the change).

Alternatively, you can set <seclabel/> either for whole domain or individual disks, e.g. like this:

<disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/var/lib/libvirt/images/vm1.qcow2'> <seclabel relabel='no'/> </source> </disk>

or for whole domain:

... </devices> <seclabel type='none'/> </domain>

I'm not sure what your setup is, but if chown() is a problem then what if guest tries to write onto its disk? Wouldn't that create a copy in overlay?

Michal

On Wed, Nov 24, 2021 at 06:21:22PM +0100, Elias Mobery wrote:

Hey Martin & Michal, thank you both for the replies!

Yes, sorry I messed up that seclabel entry, thanks. I fixed it and there is no more error, but the images are still copied unfortunately.

@Michal yes, I have been reading the Debian Live Manual like a contract, after messing with qemu.conf and permissions for days, I think its something in the live build and not libvirt.

@Martin yeah both virsh start and virt-manager cause the images to be copied to /run..... first.

Yes, when I boot the Debian Live ISO, plug in the USB (mounted rw), copy the images to the live system from the USB and click run in virt-manager, the VM starts instantly and no copies are made in /run....

I use ISO loopback and toram to load the system directly into ram via tmpfs mount. That's my grub.cfg

menuentry "debian-live.iso" { set iso="/ISO/debian-live.iso" set root=(hd0,5) loopback loop $iso linux (loop)/live/vmlinuz-4.19.0-6-amd64 boot=live components toram findiso=$iso initrd (loop)/live/initrd.img-4.19.0-6-amd64 }

So, the squashfs is mounted read-only in /run/live/rootfs/filesystem.squashfs and its /dev/loop1

/proc/mounts says:

tmpfs /run/live/overlay rw overlay rw lowerdir=/run/live/rootfs/filesystem.squashfs upperdir=/run/live/overlay/rw (same place where the images are copied)

I do not think this is relevant, but sometimes overlayfs wants to have a workdir as well.

So you think that because the images are part of the squashfs which is read-only, they are first copied to /run/live/overlay/rw... to be written to?

Are they really copied? I can't think of many more things why this would happen. Maybe some locking code does that in the backend (both libvirt and qemu lock parts of the disks).

I always thought that tmpfs is mounted on top of squash, so it's as if the squashfs were writable, without having to literally copy between the two filesystems.

It is not mounted over, if you mount over something you lose the lowerdir, that's why it needs to be done with overlayfs which does the copy-on-write. That leads me to another idea. Since you want copy-on-write for the images, I presume, the image will get copied anyway once the guest writes even a single byte to the image. Unfortunately overlayfs is not the best way for this scenario due to the granularity it offers (per-file, not per-block or anything smaller). What I think you actually want in this case is to have a qcow image with the original image used as backing. You can create those with libvirt or even without it, just using qemu-img, for example like this: qemu-img create -b /path/to/non-modified.image -f qcow2 -F format_of_backing_file /path/to/new-image.qcow2 For more information please read the qemu-img manual page to make sure you use it properly, I just wrote the above from the top of my head, not tested it. The backing file path can also be relative. If you specify the new image for the machines then the granularity changes to way smaller blocks than the whole file making it occupy less space in case of changes.

Also I made the same build with Virtualbox and there is no copying of the images to /run/live/overlay.

I will do some serious research into squashfs and tmpfs rn.

If you want to figure out when it happens I would suggest enabling debug logs for libvirt, starting the qemu VM with --paused, but maybe it will need some more debugging on lower layers. Anyway I really believe the approach suggested above is what will lead to better results for you. Have a nice day Martin

On Wed, Nov 24, 2021, 5:10 PM Martin Kletzander <mkletzan@redhat.com> wrote:

...

On Wed, Nov 24, 2021 at 04:01:34PM +0100, Elias Mobery wrote:

...

Hello Michal, thank you for the reply!

I've carefully tested everything you suggested, thanks.

I set dynamic_ownership=0 and use these hooks during the live build for permissions. (I googled a lot, and apparently libvirt needs the images to be executable too)

chown -R libvirt-qemu:kvm /var/lib/libvirt/images chmod -R g+rwx /var/lib/libvirt/images

I do not see why would the images need to be executable, you might need an executable bit set on the directory so that your and qemu user can browse it.

...

Booting the live debian iso everything works in virt-manager, but again, after clicking "run", a copy of the vm image is created in /run/live/overlay/rw/var/lib/libvirt/images and only then does the VM start.

Does that happen when you try to run it with virsh instead of virt-manager? Are you connected to the system daemon?

...

Either it's still being chowned or chmodded somehow, or it's something else, but I can't stop this copy being made.

Interestingly, when I boot the Live debian iso and then copy the images into /var/lib/libvirt/images from my USB stick, the VM starts immediately without creating any copies in the /run/live.... directory. So my guess is that maybe the squashfs could be the issue?

Oh, interesting, is the USB stick filesystem mounted R/W and the live ISO filesystem mounted read-only? How is the overlay mounted?

...

Editing the XML

<source file='/var/lib/libvirt/images/vm1.qcow2'> <seclabel relabel='no'/> </source>

This results in an error: Unsupported Configuration: Security driver model 'null' not available

For issues like this it is most helpful to check the documentation:

https://libvirt.org/formatdomain.html

where you can see it is just a missing attribute `model`, in this case model="dac".

...

Here I tried setting security_driver=none in qemu.conf but same error after.

</devices> <seclabel type='none'/> </domain>

Same here.

...

This also returns an Error but I'm still googling to understand it properly.

XML document failed to validate against schema Unable to validate doc against /usr/share/libvirt/schemas/domain.rng Invalid element relabel for element seclabel Extra element seclabel in interleave Element domain failed to validate content

Thanks again so much for your helo, I've been messing with this for weeks now and it's killing me.

On Tue, Nov 23, 2021, 9:43 PM Michal Prívozník <mprivozn@redhat.com> wrote:

...

On 11/23/21 17:25, Elias Mobery wrote:

...

Hi everyone!

I've built a Debian Live ISO with packages qemu and libvirt to run a

...

in the live environment.

The guest images are placed in /var/lib/libvirt/images and 2GB each.

Everything works great, except for one issue.

When starting a VM, libvirt automatically issues a chown command to

VM the

...

images, changing ownership.

This results in a copy of the images being created in /run/live/overlay/rw/var/lib/libvirt/images

I don't want these copies to be made but can't stop it.

I've tried editing qemu.conf user/group, dynamic ownership etc. without any luck.

Is there a way to STOP libvirt from changing the ownership of these images?

Setting dynamic_ownership=0 in qemu.conf is the way to go (don't forget to restart the daemon after you made the change).

Alternatively, you can set <seclabel/> either for whole domain or individual disks, e.g. like this:

<disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/var/lib/libvirt/images/vm1.qcow2'> <seclabel relabel='no'/> </source> </disk>

or for whole domain:

... </devices> <seclabel type='none'/> </domain>

I'm not sure what your setup is, but if chown() is a problem then what if guest tries to write onto its disk? Wouldn't that create a copy in overlay?

Michal

On Thu, Nov 25, 2021 at 01:21:23PM +0100, Elias Mobery wrote:

Hey Martin, wow thanks for the detailed info!

I'll definitely look into this, but I kind of figured it out, I believe.

So, I keep the images in the read-only squashfs, boot the live system and use snapshot writeback: virsh snapshot-create-as with options --disk-only --diskspec

Well, this basically does a very similar thing, only in a way that libvirt keeps more metadata about the fact.

This pretty much fixes my problem, and only disk writes occur in tmpfs, instead of the entire image being copied.

BUT, now I have to reduce the disk space within the VM to almost 0 to save space.

Not sure which space you mean. If that is inside the VM, then uninstalling unneeded stuff is the way to go, if it is the image file in the ISO, then virt-sparsify before creating the ISO will punch holes where there is no usable data. If you mean in the overlay/snapshot, then it only records the changes.

I'm reading up on qemu-img convert and virt-sparsify to accomplish this, but I will also try your method now, it seems more professional.

Thank you for your amazing input and patience, I love this community. Sharing this information is the literal definition of freedom IMO, thank you all!

Thanks ;)

On Thu, Nov 25, 2021, 11:40 AM Martin Kletzander <mkletzan@redhat.com> wrote:

...

On Wed, Nov 24, 2021 at 06:21:22PM +0100, Elias Mobery wrote:

...

Hey Martin & Michal, thank you both for the replies!

Yes, sorry I messed up that seclabel entry, thanks. I fixed it and there is no more error, but the images are still copied unfortunately.

@Michal yes, I have been reading the Debian Live Manual like a contract, after messing with qemu.conf and permissions for days, I think its something in the live build and not libvirt.

@Martin yeah both virsh start and virt-manager cause the images to be copied to /run..... first.

Yes, when I boot the Debian Live ISO, plug in the USB (mounted rw), copy the images to the live system from the USB and click run in virt-manager, the VM starts instantly and no copies are made in /run....

I use ISO loopback and toram to load the system directly into ram via tmpfs mount. That's my grub.cfg

menuentry "debian-live.iso" { set iso="/ISO/debian-live.iso" set root=(hd0,5) loopback loop $iso linux (loop)/live/vmlinuz-4.19.0-6-amd64 boot=live components toram findiso=$iso initrd (loop)/live/initrd.img-4.19.0-6-amd64 }

So, the squashfs is mounted read-only in /run/live/rootfs/filesystem.squashfs and its /dev/loop1

/proc/mounts says:

tmpfs /run/live/overlay rw overlay rw lowerdir=/run/live/rootfs/filesystem.squashfs upperdir=/run/live/overlay/rw (same place where the images are copied)

I do not think this is relevant, but sometimes overlayfs wants to have a workdir as well.

...

So you think that because the images are part of the squashfs which is read-only, they are first copied to /run/live/overlay/rw... to be written to?

Are they really copied? I can't think of many more things why this would happen. Maybe some locking code does that in the backend (both libvirt and qemu lock parts of the disks).

...

I always thought that tmpfs is mounted on top of squash, so it's as if the squashfs were writable, without having to literally copy between the two filesystems.

It is not mounted over, if you mount over something you lose the lowerdir, that's why it needs to be done with overlayfs which does the copy-on-write.

That leads me to another idea. Since you want copy-on-write for the images, I presume, the image will get copied anyway once the guest writes even a single byte to the image. Unfortunately overlayfs is not the best way for this scenario due to the granularity it offers (per-file, not per-block or anything smaller). What I think you actually want in this case is to have a qcow image with the original image used as backing. You can create those with libvirt or even without it, just using qemu-img, for example like this:

qemu-img create -b /path/to/non-modified.image -f qcow2 -F format_of_backing_file /path/to/new-image.qcow2

For more information please read the qemu-img manual page to make sure you use it properly, I just wrote the above from the top of my head, not tested it. The backing file path can also be relative.

If you specify the new image for the machines then the granularity changes to way smaller blocks than the whole file making it occupy less space in case of changes.

...

Also I made the same build with Virtualbox and there is no copying of the images to /run/live/overlay.

I will do some serious research into squashfs and tmpfs rn.

If you want to figure out when it happens I would suggest enabling debug logs for libvirt, starting the qemu VM with --paused, but maybe it will need some more debugging on lower layers. Anyway I really believe the approach suggested above is what will lead to better results for you.

Have a nice day Martin

...

On Wed, Nov 24, 2021, 5:10 PM Martin Kletzander <mkletzan@redhat.com>

wrote:

...

...

On Wed, Nov 24, 2021 at 04:01:34PM +0100, Elias Mobery wrote:

...

Hello Michal, thank you for the reply!

I've carefully tested everything you suggested, thanks.

I set dynamic_ownership=0 and use these hooks during the live build for permissions. (I googled a lot, and apparently libvirt needs the images

to

...

...

...

be executable too)

chown -R libvirt-qemu:kvm /var/lib/libvirt/images chmod -R g+rwx /var/lib/libvirt/images

I do not see why would the images need to be executable, you might need an executable bit set on the directory so that your and qemu user can browse it.

...

Booting the live debian iso everything works in virt-manager, but again, after clicking "run", a copy of the vm image is created in /run/live/overlay/rw/var/lib/libvirt/images and only then does the VM start.

Does that happen when you try to run it with virsh instead of virt-manager? Are you connected to the system daemon?

...

Either it's still being chowned or chmodded somehow, or it's something else, but I can't stop this copy being made.

Interestingly, when I boot the Live debian iso and then copy the images into /var/lib/libvirt/images from my USB stick, the VM starts immediately without creating any copies in the /run/live.... directory. So my guess is that maybe the squashfs could be the issue?

Oh, interesting, is the USB stick filesystem mounted R/W and the live ISO filesystem mounted read-only? How is the overlay mounted?

...

Editing the XML

<source file='/var/lib/libvirt/images/vm1.qcow2'> <seclabel relabel='no'/> </source>

This results in an error: Unsupported Configuration: Security driver model 'null' not available

For issues like this it is most helpful to check the documentation:

https://libvirt.org/formatdomain.html

where you can see it is just a missing attribute `model`, in this case model="dac".

...

Here I tried setting security_driver=none in qemu.conf but same error after.

</devices> <seclabel type='none'/> </domain>

Same here.

...

This also returns an Error but I'm still googling to understand it properly.

XML document failed to validate against schema Unable to validate doc against /usr/share/libvirt/schemas/domain.rng Invalid element relabel for element seclabel Extra element seclabel in interleave Element domain failed to validate content

Thanks again so much for your helo, I've been messing with this for weeks now and it's killing me.

On Tue, Nov 23, 2021, 9:43 PM Michal Prívozník <mprivozn@redhat.com> wrote:

...

On 11/23/21 17:25, Elias Mobery wrote: > > Hi everyone! > > I've built a Debian Live ISO with packages qemu and libvirt to run a VM > in the live environment. > > The guest images are placed in /var/lib/libvirt/images and 2GB each. > > Everything works great, except for one issue. > > When starting a VM, libvirt automatically issues a chown command to the > images, changing ownership. > > This results in a copy of the images being created in > /run/live/overlay/rw/var/lib/libvirt/images > > I don't want these copies to be made but can't stop it. > > I've tried editing qemu.conf user/group, dynamic ownership etc. without > any luck. > > Is there a way to STOP libvirt from changing the ownership of these images? > >

Setting dynamic_ownership=0 in qemu.conf is the way to go (don't forget to restart the daemon after you made the change).

Alternatively, you can set <seclabel/> either for whole domain or individual disks, e.g. like this:

<disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/var/lib/libvirt/images/vm1.qcow2'> <seclabel relabel='no'/> </source> </disk>

or for whole domain:

... </devices> <seclabel type='none'/> </domain>

I'm not sure what your setup is, but if chown() is a problem then what if guest tries to write onto its disk? Wouldn't that create a copy in overlay?

Michal