How to hot plugin a new vhost-user-blk-pci device to running VM?

On Thu, May 13, 2021 at 15:25:23 +0800, 梁朝军 wrote:
Hi Guy,
Does anyone clear how to hot plugin a new vhost-user-blk-pci device to a running VM?
Before staring vm , I pass the disk through QEMU command line like below.
<qemu:commandline> <qemu:arg value='-object'/> <qemu:arg value='memory-backend-file,id=mem0,size=4G,mem-path=/dev/hugepages,share=on'/> <qemu:arg value='-numa'/> <qemu:arg value='node,memdev=mem0'/> <qemu:arg value='-chardev'/> <qemu:arg value='socket,id=spdk_vhost_blk721ea46a-b306-11eb-a280-525400a98761,path=/var/tmp/vhost.721ea46a-b306-11eb-a280-525400a98761,reconnect=1'/> <qemu:arg value='-device'/> <qemu:arg value='vhost-user-blk-pci,chardev=spdk_vhost_blk721ea46a-b306-11eb-a280-525400a98761,bootindex=1,num-queues=4'/> <qemu:arg value='-chardev'/> <qemu:arg value='socket,id=spdk_vhost_blk2f699c58-d222-4629-9fdc-400c3aadc55e,path=/var/tmp/vhost.2f699c58-d222-4629-9fdc-400c3aadc55e,reconnect=1'/> <qemu:arg value='-device'/> <qemu:arg value='vhost-user-blk-pci,chardev=spdk_vhost_blk2f699c58-d222-4629-9fdc-400c3aadc55e,num-queues=4'/> </qemu:commandline>
But I don**t know how to live add a vhost-user-blk-pci device on running VM even with calling attachDevice API now.
You need to use the proper and supported way to use vhost-user-blk: <disk type='vhostuser' device='disk'> <driver name='qemu' type='raw'/> <source type='unix' path='/tmp/vhost-blk.sock'> <reconnect enabled='yes' timeout='10'/> </source> <target dev='vdf' bus='virtio'/> </disk> That works also with attachDevice.
OS: redhat 7.4 Libvirt version: 3.4
This is obviously way too old for it. You'll need at least libvirt-7.1 for that.

Thanks Peter for your quickly response. Is there any workaround to do that?As you know we must take care the risk of using latest version in product environment. Thanks a lot!
在 2021年5月13日,22:25,Peter Krempa <pkrempa@redhat.com> 写道:
On Thu, May 13, 2021 at 15:25:23 +0800, 梁朝军 wrote:
Hi Guy,
Does anyone clear how to hot plugin a new vhost-user-blk-pci device to a running VM?
Before staring vm , I pass the disk through QEMU command line like below.
<qemu:commandline> <qemu:arg value='-object'/> <qemu:arg value='memory-backend-file,id=mem0,size=4G,mem-path=/dev/hugepages,share=on'/> <qemu:arg value='-numa'/> <qemu:arg value='node,memdev=mem0'/> <qemu:arg value='-chardev'/> <qemu:arg value='socket,id=spdk_vhost_blk721ea46a-b306-11eb-a280-525400a98761,path=/var/tmp/vhost.721ea46a-b306-11eb-a280-525400a98761,reconnect=1'/> <qemu:arg value='-device'/> <qemu:arg value='vhost-user-blk-pci,chardev=spdk_vhost_blk721ea46a-b306-11eb-a280-525400a98761,bootindex=1,num-queues=4'/> <qemu:arg value='-chardev'/> <qemu:arg value='socket,id=spdk_vhost_blk2f699c58-d222-4629-9fdc-400c3aadc55e,path=/var/tmp/vhost.2f699c58-d222-4629-9fdc-400c3aadc55e,reconnect=1'/> <qemu:arg value='-device'/> <qemu:arg value='vhost-user-blk-pci,chardev=spdk_vhost_blk2f699c58-d222-4629-9fdc-400c3aadc55e,num-queues=4'/> </qemu:commandline>
But I don**t know how to live add a vhost-user-blk-pci device on running VM even with calling attachDevice API now.
You need to use the proper and supported way to use vhost-user-blk:
<disk type='vhostuser' device='disk'> <driver name='qemu' type='raw'/> <source type='unix' path='/tmp/vhost-blk.sock'> <reconnect enabled='yes' timeout='10'/> </source> <target dev='vdf' bus='virtio'/> </disk>
That works also with attachDevice.
OS: redhat 7.4 Libvirt version: 3.4
This is obviously way too old for it. You'll need at least libvirt-7.1 for that.

On Thu, May 13, 2021 at 23:11:36 +0800, Liang Chaojun wrote:
Thanks Peter for your quickly response. Is there any workaround to do that?As you know we must take care the risk of using latest version in product environment.
Manual approach is to use 'virsh qemu-monitor-command' or the equivalent to attach the appropriate backends and frontends manually. Obviously that is very far from anything I'd recommend to use in any production environment.

Thanks,I have tried qemu monitor as below. I used chardev_add to add a chardev and used device_add it to running vm. But i often hit a issue and cause the vm crash. qemu-system-x86_64: ../hw/virtio/vhost.c:1566: vhost_dev_get_config: Assertion `hdev->vhost_ops' failed. virsh qemu-monitor-command spdk1 --hmp --cmd "chardev-add socket,id=spdk_vhost_blk0,path=/var/tmp/vhost.0,reconnect=1" virsh qemu-monitor-command spdk1 --hmp --cmd "device_add vhost-user-blk-pci,chardev=spdk_vhost_blk0,num-queues=4"
在 2021年5月14日,00:47,Peter Krempa <pkrempa@redhat.com> 写道:
On Thu, May 13, 2021 at 23:11:36 +0800, Liang Chaojun wrote:
Thanks Peter for your quickly response. Is there any workaround to do that?As you know we must take care the risk of using latest version in product environment.
Manual approach is to use 'virsh qemu-monitor-command' or the equivalent to attach the appropriate backends and frontends manually.
Obviously that is very far from anything I'd recommend to use in any production environment.

On 5/14/21 8:33 AM, Liang Chaojun wrote:
Thanks,I have tried qemu monitor as below. I used chardev_add to add a chardev and used device_add it to running vm. But i often hit a issue and cause the vm crash. qemu-system-x86_64: ../hw/virtio/vhost.c:1566: vhost_dev_get_config: Assertion `hdev->vhost_ops' failed.
virsh qemu-monitor-command spdk1 --hmp --cmd "chardev-add socket,id=spdk_vhost_blk0,path=/var/tmp/vhost.0,reconnect=1" virsh qemu-monitor-command spdk1 --hmp --cmd "device_add vhost-user-blk-pci,chardev=spdk_vhost_blk0,num-queues=4"
Smells like problem described here: https://gitlab.com/qemu-project/qemu/-/commit/bc79c87bcde6587a37347f81332fbb... But that's pretty fresh commit (part of qemu 6.0.0 release). And also the commit it references is not that old (qemu 5.1.0 release). What version of qemu are you running? Michal

Thanks,I have tried qemu monitor as below. I used chardev_add to add a chardev and used device_add it to running vm. But i often hit a issue and cause the vm crash. qemu-system-x86_64: ../hw/virtio/vhost.c:1566: vhost_dev_get_config: Assertion `hdev->vhost_ops' failed. virsh qemu-monitor-command spdk1 --hmp --cmd "chardev-add socket,id=spdk_vhost_blk0,path=/var/tmp/vhost.0,reconnect=1" virsh qemu-monitor-command spdk1 --hmp --cmd "device_add vhost-user-blk-pci,chardev=spdk_vhost_blk0,num-queues=4"
在 2021年5月14日,00:47,Peter Krempa <pkrempa@redhat.com> 写道:
On Thu, May 13, 2021 at 23:11:36 +0800, Liang Chaojun wrote:
Thanks Peter for your quickly response. Is there any workaround to do that?As you know we must take care the risk of using latest version in product environment.
Manual approach is to use 'virsh qemu-monitor-command' or the equivalent to attach the appropriate backends and frontends manually.
Obviously that is very far from anything I'd recommend to use in any production environment.

On Fri, May 14, 2021 at 14:33:37 +0800, Liang Chaojun wrote:
Thanks,I have tried qemu monitor as below. I used chardev_add to add a chardev and used device_add it to running vm. But i often hit a issue and cause the vm crash. qemu-system-x86_64: ../hw/virtio/vhost.c:1566: vhost_dev_get_config: Assertion `hdev->vhost_ops' failed.
virsh qemu-monitor-command spdk1 --hmp --cmd "chardev-add socket,id=spdk_vhost_blk0,path=/var/tmp/vhost.0,reconnect=1" virsh qemu-monitor-command spdk1 --hmp --cmd "device_add vhost-user-blk-pci,chardev=spdk_vhost_blk0,num-queues=4"
As noted, this is not an interface we'd provide support for. Please upgrade both libvirt and qemu and try the supported interface if you want us to deal with any problems you might have.
participants (4)
-
Liang Chaojun
-
Michal Prívozník
-
Peter Krempa
-
梁朝军