Re: [libvirt PATCH 5/5] qemu: Implement support for vDPA block devices

On Thu, Aug 03, 2023 at 09:48:01 +0200, Stefano Garzarella wrote: Naively and originally I've simply used it as 'raw' at first and formatted it from the guest OS. Then I've shut-down the VM and started it back reconfiguring the image format as qcow2. This normally works with real-file backed storage, and since the vdpa simulator seems to persist the contents I supposed this would work. Yeah, ideally there will be no format driver in qemu used for these devices (this is not yet the case, I'll need to fix libvirt to stop using the 'raw' driver if not needed). Here I'm more interested whether it is supposed to work, in which case we want to allow using qcow2 as a format in libvirt, or it's not supposed to work and we should forbid it before the user gets a suboptimal error message such as now. With the above qemu storage daemon you should be able to do that by simply dropping the qcow2 format driver and simply exposing a qcow2 formatted image. It similarly works with NBD: I've formatted 2 qcow2 images: # qemu-img create -f qcow2 /root/image1.qcow2 100M # qemu-img create -f qcow2 /root/image2.qcow2 100M And then exported them both via vduse and nbd without interpreting qcow2, thus making the QSD into just a dumb storage device: # qemu-storage-daemon \ --blockdev file,filename=/root/image1.qcow2,cache.direct=on,aio=native,node-name=file1 \ --export vduse-blk,id=vduse0,name=vduse0,num-queues=1,node-name=file1,writable=on \ --blockdev file,filename=/root/image2.qcow2,cache.direct=on,aio=native,node-name=file2 \ --nbd-server addr.type=unix,addr.path=/tmp/nbd.sock \ --export nbd,id=nbd0,node-name=file2,writable=on,name=exportname Now when I start a VM using the NBD export in qcow2 format: <disk type='network' device='disk'> <driver name='qemu' type='qcow2'/> <source protocol='nbd' name='exportname'> <host transport='unix' socket='/tmp/nbd.sock'/> </source> <target dev='vda' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </disk> The VM starts fine, but when using: <disk type='vhostvdpa' device='disk'> <driver name='qemu' type='qcow2' cache='none'/> <source dev='/dev/vhost-vdpa-0'/> <target dev='vda' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </disk> I get: error: internal error: QEMU unexpectedly closed the monitor (vm='vdpa'): 2023-08-07T12:34:21.628520Z qemu-system-x86_64: -blockdev {"node-name":"libvirt-1-format","read-only":false,"cache":{"direct":true,"no-flush":false},"driver":"qcow2","file":"libvirt-1-storage"}: Could not read qcow2 header: Invalid argument What is weird though, that if I attempt to use qemu itself to format the image, with VDPA simulator it appears to work the very first time after the image is formatted, but if I restart the VM it no longer recognizes the image. Since the simulator's content seems to be preserved accross VM restarts when using 'raw' I thought it should work: + virsh start cd Domain 'cd' started + virsh qemu-monitor-command cd --pass-fds 6 add-fd '{"fdset-id": 1337}' {"return":{"fd":37,"fdset-id":1337},"id":"libvirt-441"} + virsh qemu-monitor-command cd blockdev-add '{"driver":"virtio-blk-vhost-vdpa","path":"/dev/fdset/1337","node-name":"testformat","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' {"return":{},"id":"libvirt-442"} + virsh qemu-monitor-command cd blockdev-create '{"options":{"driver":"qcow2","file":"testformat","size":10485760},"job-id":"formatjob"}' {"return":{},"id":"libvirt-443"} + virsh qemu-monitor-command cd query-jobs {"return":[{"current-progress":1,"status":"concluded","total-progress":1,"type":"create","id":"formatjob"}],"id":"libvirt-444"} + sleep 1 + virsh qemu-monitor-command cd query-jobs {"return":[{"current-progress":1,"status":"concluded","total-progress":1,"type":"create","id":"formatjob"}],"id":"libvirt-445"} + sleep 1 + virsh qemu-monitor-command cd job-dismiss '{"id":"formatjob"}' {"return":{},"id":"libvirt-446"} + virsh qemu-monitor-command cd blockdev-add '{"node-name":"testqcow2","read-only":false,"driver":"qcow2","file":"testformat"}' {"return":{},"id":"libvirt-447"} + virsh destroy cd Domain 'cd' destroyed + virsh qemu-monitor-command cd --pass-fds 6 add-fd '{"fdset-id": 1337}' {"return":{"fd":37,"fdset-id":1337},"id":"libvirt-441"} + virsh qemu-monitor-command cd blockdev-add '{"driver":"virtio-blk-vhost-vdpa","path":"/dev/fdset/1337","node-name":"testformat","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' {"return":{},"id":"libvirt-442"} + virsh qemu-monitor-command cd blockdev-add '{"node-name":"testqcow2","read-only":false,"driver":"qcow2","file":"testformat"}' {"id":"libvirt-443","error":{"class":"GenericError","desc":"Could not read qcow2 header: Invalid argument"}} Additionally, even weirder is that if I use the qemu-storage-daemon backed vhost-vdpa device, the formatting job simply gets stuck forever: + virsh start cd Domain 'cd' started + virsh qemu-monitor-command cd blockdev-add '{"driver":"virtio-blk-vhost-vdpa","path":"/dev/vhost-vdpa-0","node-name":"testformat","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' {"return":{},"id":"libvirt-441"} + virsh qemu-monitor-command cd blockdev-create '{"options":{"driver":"qcow2","file":"testformat","size":10485760},"job-id":"formatjob"}' {"return":{},"id":"libvirt-442"} + virsh qemu-monitor-command cd query-jobs {"return":[{"current-progress":0,"status":"running","total-progress":1,"type":"create","id":"formatjob"}],"id":"libvirt-443"} + sleep 1 + virsh qemu-monitor-command cd query-jobs {"return":[{"current-progress":0,"status":"running","total-progress":1,"type":"create","id":"formatjob"}],"id":"libvirt-444"} + sleep 1 + virsh qemu-monitor-command cd job-dismiss '{"id":"formatjob"}' {"id":"libvirt-445","error":{"class":"GenericError","desc":"Job 'formatjob' in state 'running' cannot accept command verb 'dismiss'"}} (Note, it behaves the same when using FD passing, I've just tried because it's very weird) Disclaimer: It's possible I broke my test box but I can't simply reboot it at this point.