[libvirt-users] Change in treatment of qcow2 with chained backing files in v5.10.0?
Hello list, upon upgrading from libvirt v5.9.0 -> v5.10.0, libvirt started calling qemu differently causing the VM operating system to not be found. I'm wondering whether I'm chasing a bug, or have to otherwise fix my domain config (created by virt-manager). To start with, I have a Windows 7 domain based on the following images: windows7-base.qcow2: QEMU QCOW2 Image (v3), 53687091200 bytes windows7-Office2013.qcow2: QEMU QCOW2 Image (v3), has backing file (path windows7-taintable.qcow2), 80530636800 bytes windows7-taintable.qcow2: QEMU QCOW2 Image (v3), has backing file (path windows7-base.qcow2), 53687091200 bytes The VM must be launched with image windows7-Office2013.qcow2. Using the domain config [1] with a <disk> section <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/home/joj/lib/vm/windows/windows7-Office2013.qcow2'/> <target dev='vda' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </disk> libvirt 5.9.0 is calling qemu 4.2.0 like [2], and libvirt 5.10.0 like [3], the differences being (5.9.0 1st, 5.10.0 2nd lines) [--drive file=/home/joj/lib/vm/windows/windows7-Office2013.qcow2,format=qcow2,if=none,id=drive-virtio-disk0-] {+-blockdev {"driver":"file","filename":"/home/joj/lib/vm/windows/windows7-taintable.qcow2","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap"} -blockdev {"node-name":"libvirt-2-format","read-only":true,"driver":"raw","file":"libvirt-2-storage"} -blockdev {"driver":"file","filename":"/home/joj/lib/vm/windows/windows7-Office2013.qcow2","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"} -blockdev {"node-name":"libvirt-1-format","read-only":false,"driver":"qcow2","file":"libvirt-1-storage","backing":"libvirt-2-format"}+} -device [-virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1-] {+virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=libvirt-1-format,id=virtio-disk0,bootindex=1+} As you can see, 5.10.0 is expanding the backing file chain, not leaving that to qemu. Under 5.10.0, I get an Operating System not Found error, under 5.9.0, with the shorter -drive argument, everything works as desired. Can you perhaps advise me how I either fix the domain spec [1] to get libvirt to use a true equivalent to the old qemu invocation as I don't think this is a qemu problem, or how I can turn the new-style JSON parametrization off? Other suggestions how to tackle this would also be much appreciated. --- [1] https://0x0.st/z0KM.xml [2] https://0x0.st/z0K1.txt [3] https://0x0.st/z0Kj.txt
On Mon, Dec 16, 2019 at 12:49:40 +0100, Jens John wrote:
Hello list,
Hi Jens,
upon upgrading from libvirt v5.9.0 -> v5.10.0, libvirt started calling qemu differently causing the VM operating system to not be found. I'm wondering whether I'm chasing a bug, or have to otherwise fix my domain config (created by virt-manager).
Yes, this change was done by me when enabling the new way to configure block devices in qemu. I think I know what's going on:
To start with, I have a Windows 7 domain based on the following images:
windows7-base.qcow2: QEMU QCOW2 Image (v3), 53687091200 bytes windows7-Office2013.qcow2: QEMU QCOW2 Image (v3), has backing file (path windows7-taintable.qcow2), 80530636800 bytes windows7-taintable.qcow2: QEMU QCOW2 Image (v3), has backing file (path windows7-base.qcow2), 53687091200 bytes
So this output was obtained by 'file'. That unfortunately doesn't give enough information. Please run 'qemu-img info' on those files and look for the 'backing file format:'field. That field is important, because otherwise we'd fall back to image format detection which was deemed unsafe and libvirt no longer allows it. $ qemu-img info pull4.qcow2 image: pull4.qcow2 file format: qcow2 virtual size: 10 MiB (10485760 bytes) disk size: 196 KiB cluster_size: 65536 backing file: pull3.qcow2 backing file format: qcow2 ^^^^^^^^^^^^^^^^^^^^^^^^^^ Format specific information: compat: 1.1 lazy refcounts: false refcount bits: 16 corrupt: false The root of the problem is that the images mentioned above will not have the format section because -F argument for qemu-img was not used when creating them. This would force libvirt to use format detection which is insecure and thus libvirt does not do that and assumes the image is raw. This in normal circumstances causes problems right away, because when running any more advanced security model than unix permissions (selinux apparmor) libvirt would not allow qemu to access the backing file. You don't seem to be running it so it was accidentally working for you until now.
The VM must be launched with image windows7-Office2013.qcow2.
Using the domain config [1] with a <disk> section
<disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/home/joj/lib/vm/windows/windows7-Office2013.qcow2'/> <target dev='vda' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </disk>
libvirt 5.9.0 is calling qemu 4.2.0 like [2], and libvirt 5.10.0 like [3], the differences being (5.9.0 1st, 5.10.0 2nd lines)
[--drive file=/home/joj/lib/vm/windows/windows7-Office2013.qcow2,format=qcow2,if=none,id=drive-virtio-disk0-] {+-blockdev {"driver":"file","filename":"/home/joj/lib/vm/windows/windows7-taintable.qcow2","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap"} -blockdev {"node-name":"libvirt-2-format","read-only":true,"driver":"raw","file":"libvirt-2-storage"} -blockdev {"driver":"file","filename":"/home/joj/lib/vm/windows/windows7-Office2013.qcow2","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"} -blockdev {"node-name":"libvirt-1-format","read-only":false,"driver":"qcow2","file":"libvirt-1-storage","backing":"libvirt-2-format"}+}
-blockdev {"driver":"file","filename":"/home/joj/lib/vm/windows/windows7-taintable.qcow2","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap"} -blockdev {"node-name":"libvirt-2-format","read-only":true,"driver":"raw","file":"libvirt-2-storage"} Here you can see that the /home/joj/lib/vm/windows/windows7-taintable.qcow2 image was considered being 'raw' this is usually because you didn't record the approprate format. This means that qemu will no longer open the backing image. This is the manifestation of the above problem. -blockdev {"driver":"file","filename":"/home/joj/lib/vm/windows/windows7-Office2013.qcow2","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"} -blockdev {"node-name":"libvirt-1-format","read-only":false,"driver":"qcow2","file":"libvirt-1-storage","backing":"libvirt-2-format"}+}
-device [-virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1-] {+virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=libvirt-1-format,id=virtio-disk0,bootindex=1+}
As you can see, 5.10.0 is expanding the backing file chain, not leaving that to qemu. Under 5.10.0, I get an Operating System not Found error, under 5.9.0, with the shorter -drive argument, everything works as desired.
Can you perhaps advise me how I either fix the domain spec [1] to get libvirt to use a true equivalent to the old qemu invocation as I don't think this is a qemu problem, or how I can turn the new-style JSON parametrization off?
So as a first thing you should create the images properly. This means that when you are creating the overlay image, you have to use the '-F' parameter of qemu-img create. (Note that snapshots/overlays created by libvirt always do this). The -F parameter specifies the format of the backing image specified by -b: qemu-img create -f qcow2 -F qcow2 -b windows7-taintable.qcow2 windows7-Office2013.qcow2 For fixing the image you can use the qemu-img rebase: qemu-img rebase -f qcow2 -F qcow2 -b /home/joj/lib/vm/windows/windows7-taintable.qcow2 windows7-Office2013.qcow2 Please use full paths for the -b argument unless you want relative relationships between images. A second option is to specify the image chain in the XML. This works starting from libvirt 5.10 : <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/home/joj/lib/vm/windows/windows7-Office2013.qcow2'/> <target dev='vda' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> <backingStore type='file'> <format type='qcow2'/> <source file='/home/joj/lib/vm/windows/windows7-taintable.qcow2'/> <backingStore type='file'> <format type='qcow2'/> <source file='/home/joj/lib/vm/windows/windows7-base.qcow2'/> <backingStore/> </backingStore> </backingStore> </disk> Note that this does not fix the images so they will work only with the above XML. As a last possibility you can disable the 'blockdev' feature in /etc/libvirt/qemu.conf. I will not be more specific on how to do it as it's mostly for debugging purposes and could break stuff in the future. Lastly this is more-or-less tracked in: https://bugzilla.redhat.com/show_bug.cgi?id=1588373 I think I should finally fix it and report a proper error rather than silently assuming it might work. With -blockdev it will never work properly anyways.
Other suggestions how to tackle this would also be much appreciated.
--- [1] https://0x0.st/z0KM.xml [2] https://0x0.st/z0K1.txt [3] https://0x0.st/z0Kj.txt
_______________________________________________ libvirt-users mailing list libvirt-users@redhat.com https://www.redhat.com/mailman/listinfo/libvirt-users
Hello Peter, thank you for replying in such detail. On Tue, Dec 17, 2019 at 11:22:52AM +0100, Peter Krempa wrote:
So this output was obtained by 'file'. That unfortunately doesn't give enough information. Please run 'qemu-img info' on those files and look for the 'backing file format:'field.
That field is important, because otherwise we'd fall back to image format detection which was deemed unsafe and libvirt no longer allows it.
You're right. My backing files are missing the 'backing file format' field: image: windows7-base.qcow2 file format: qcow2 virtual size: 50 GiB (53687091200 bytes) disk size: 7.37 GiB cluster_size: 65536 Format specific information: compat: 1.1 lazy refcounts: false refcount bits: 16 corrupt: false image: windows7-Office2013.qcow2 file format: qcow2 virtual size: 75 GiB (80530636800 bytes) disk size: 50.4 GiB cluster_size: 65536 backing file: windows7-taintable.qcow2 Format specific information: compat: 1.1 lazy refcounts: false refcount bits: 16 corrupt: false image: windows7-taintable.qcow2 file format: qcow2 virtual size: 50 GiB (53687091200 bytes) disk size: 7.88 GiB cluster_size: 65536 backing file: windows7-base.qcow2 Format specific information: compat: 1.1 lazy refcounts: false refcount bits: 16 corrupt: false
The root of the problem is that the images mentioned above will not have the format section because -F argument for qemu-img was not used when creating them. This would force libvirt to use format detection which is insecure and thus libvirt does not do that and assumes the image is raw.
Understood. So in order to load an image or backing file, qemu/libvirt expects the format to be explicitly defined whenever a 'next image' is being referenced.
This in normal circumstances causes problems right away, because when running any more advanced security model than unix permissions (selinux apparmor) libvirt would not allow qemu to access the backing file. You don't seem to be running it so it was accidentally working for you until now.
Correct. I'm using neither selinux nor apparmor.
(...) So as a first thing you should create the images properly. that when you are creating the overlay image, you have to use the '-F' parameter of qemu-img create. (...)
Fixed the issue successfully using qemu-img rebase.
(...) (Note that snapshots/overlays created by libvirt always do this). (...)
It's probably clear by now but these images were created manually, many years ago.
Lastly this is more-or-less tracked in: https://bugzilla.redhat.com/show_bug.cgi?id=1588373
I think I should finally fix it and report a proper error rather than silently assuming it might work. With -blockdev it will never work properly anyways.
I agree that failing in a clear way would indeed help the user, though now that I know what I'm looking for, it's also clear from the -blockdev argument what's going on. Thanks, again, for your time, Jens.
participants (2)
-
Jens John -
Peter Krempa