[libvirt-users] Reply: Fail in virDomainUpdateDeviceFlags (libvirt-4.0.0 + Qemu-kvm 2.9.0 + Ceph 10.2.10)
Dear Michal After I fix the local libvirt master branch follow your patch, and build rpm for CentOS 7.4. virDomainUpdateDeviceFlags as bellow: ================================================ 2018-02-27 09:27:43.782+0000: 16656: debug : virDomainUpdateDeviceFlags:8326 : dom=0x7f2084000c50, (VM: name=6ec499397d594e f2a64fcfc938f38225, uuid=6ec49939-7d59-4ef2-a64f-cfc938f38225), xml=<disk device="cdrom" type="network"><source name="zstac k/08085a31f8c43f278ed2f649ee166b1f@08085a31f8c43f278ed2f649ee166b1f" protocol="rbd"><host name="10.0.229.181" port="6789" / ></source><auth username="zstack"><secret type="ceph" uuid="9b06bb70-dc13-4338-88fd-b0c72d5ab9e9" /></auth><target bus="ide " dev="hdc" /><readonly /></disk>, flags=0x1 ... 2018-02-27 09:27:43.788+0000: 16656: info : virObjectNew:254 : OBJECT_NEW: obj=0x7f2084003cc0 classname=qemuDomainStorageSourcePrivate 2018-02-27 09:27:43.788+0000: 16656: debug : virConnectOpen:1169 : name=secret:///system 2018-02-27 09:27:43.788+0000: 16656: info : virObjectNew:254 : OBJECT_NEW: obj=0x7f20840008c0 classname=virConnect 2018-02-27 09:27:43.788+0000: 16656: debug : virConfLoadConfig:1576 : Loading config file '/etc/libvirt/libvirt.conf' 2018-02-27 09:27:43.788+0000: 16656: debug : virConfReadFile:752 : filename=/etc/libvirt/libvirt.conf 2018-02-27 09:27:43.788+0000: 16656: debug : virFileClose:110 : Closed fd 28 2018-02-27 09:27:43.788+0000: 16656: debug : virConfGetValueStringList:946 : Get value string list (nil) 0 2018-02-27 09:27:43.788+0000: 16656: debug : virConnectOpenInternal:1033 : Split "secret:///system" to URI components: scheme secret server <null> user <null> port 0 path /system .. 2018-02-27 09:27:43.788+0000: 16656: info : virObjectRef:388 : OBJECT_REF: obj=0x563fa24f2360 2018-02-27 09:27:43.788+0000: 16656: info : virObjectUnref:350 : OBJECT_UNREF: obj=0x563fa24f2360 2018-02-27 09:27:43.788+0000: 16656: debug : virConnectOpenInternal:1098 : driver 7 secret returned SUCCESS 2018-02-27 09:27:43.788+0000: 16656: error : qemuDomainGetSecretAESAlias:729 : invalid argument: encrypted secret alias requires valid source alias 2018-02-27 09:27:43.788+0000: 16656: info : virObjectUnref:350 : OBJECT_UNREF: obj=0x7f20840008c0 2018-02-27 09:27:43.788+0000: 16656: info : virObjectUnref:352 : OBJECT_DISPOSE: obj=0x7f20840008c0 2018-02-27 09:27:43.788+0000: 16656: info : virObjectUnref:350 : OBJECT_UNREF: obj=0x7f2030101cb0 2018-02-27 09:27:43.788+0000: 16656: debug : qemuDomainObjEndJob:5594 : Stopping job: modify (async=none vm=0x7f20300fa6e0 name=6ec499397d594ef2a64fcfc938f38225) ====================================================== But it fails. Best Regards, Star Guo -----邮件原件----- 发件人: Michal Privoznik [mailto:mprivozn@redhat.com] 发送时间: 2018年2月27日 16:53 收件人: Star Guo <starg@ceph.me>; libvirt-users@redhat.com 抄送: John Ferlan <jferlan@redhat.com> 主题: Re: [libvirt-users] Fail in virDomainUpdateDeviceFlags (libvirt-4.0.0 + Qemu-kvm 2.9.0 + Ceph 10.2.10) On 02/27/2018 03:06 AM, Star Guo wrote:
Hello Everyone,
My pc run in CentOS 7.4 and install libvirt-4.0.0 + Qemu-kvm 2.9.0 + Ceph 10.2.10 ALL-in-One.
I use python-sdk with libvirt and run [self.domain.updateDeviceFlags(xml, libvirt.VIR_DOMAIN_AFFECT_LIVE)] on CDROM (I want to change media path). However, I enable libvirt debug log , the log as below:
<snip/>
I see the flow is virDomainUpdateDeviceFlags -> qemuMonitorChangeMedia, but the cephx auth is drop, so make update error. Anybody meet this error?
Yes, this is a libvirt bug. I think this fixes the issue: diff --git i/src/qemu/qemu_driver.c w/src/qemu/qemu_driver.c index 96454c17c..0e5ad9971 100644 --- i/src/qemu/qemu_driver.c +++ w/src/qemu/qemu_driver.c @@ -7842,6 +7842,8 @@ qemuDomainChangeDiskLive(virDomainObjPtr vm, virQEMUDriverPtr driver, bool force) { + virQEMUDriverConfigPtr cfg = virQEMUDriverGetConfig(driver); + qemuDomainObjPrivatePtr priv = vm->privateData; virDomainDiskDefPtr disk = dev->data.disk; virDomainDiskDefPtr orig_disk = NULL; virDomainDeviceDef oldDev = { .type = dev->type }; @@ -7850,6 +7852,9 @@ qemuDomainChangeDiskLive(virDomainObjPtr vm, if (virDomainDiskTranslateSourcePool(disk) < 0) goto cleanup; + if (qemuDomainPrepareDiskSource(disk, priv, cfg) < 0) + goto cleanup; + if (qemuDomainDetermineDiskChain(driver, vm, disk, false, true) < 0) goto cleanup; @@ -7898,6 +7903,7 @@ qemuDomainChangeDiskLive(virDomainObjPtr vm, ret = 0; cleanup: + virObjectUnref(cfg); return ret; } Can you check and confirm? Michal
On 02/27/2018 04:42 AM, Star Guo wrote:
Dear Michal
After I fix the local libvirt master branch follow your patch, and build rpm for CentOS 7.4. virDomainUpdateDeviceFlags as bellow:
================================================ 2018-02-27 09:27:43.782+0000: 16656: debug : virDomainUpdateDeviceFlags:8326 : dom=0x7f2084000c50, (VM: name=6ec499397d594e f2a64fcfc938f38225, uuid=6ec49939-7d59-4ef2-a64f-cfc938f38225), xml=<disk device="cdrom" type="network"><source name="zstac k/08085a31f8c43f278ed2f649ee166b1f@08085a31f8c43f278ed2f649ee166b1f" protocol="rbd"><host name="10.0.229.181" port="6789" / ></source><auth username="zstack"><secret type="ceph" uuid="9b06bb70-dc13-4338-88fd-b0c72d5ab9e9" /></auth><target bus="ide " dev="hdc" /><readonly /></disk>, flags=0x1 ... 2018-02-27 09:27:43.788+0000: 16656: info : virObjectNew:254 : OBJECT_NEW: obj=0x7f2084003cc0 classname=qemuDomainStorageSourcePrivate 2018-02-27 09:27:43.788+0000: 16656: debug : virConnectOpen:1169 : name=secret:///system 2018-02-27 09:27:43.788+0000: 16656: info : virObjectNew:254 : OBJECT_NEW: obj=0x7f20840008c0 classname=virConnect 2018-02-27 09:27:43.788+0000: 16656: debug : virConfLoadConfig:1576 : Loading config file '/etc/libvirt/libvirt.conf' 2018-02-27 09:27:43.788+0000: 16656: debug : virConfReadFile:752 : filename=/etc/libvirt/libvirt.conf 2018-02-27 09:27:43.788+0000: 16656: debug : virFileClose:110 : Closed fd 28 2018-02-27 09:27:43.788+0000: 16656: debug : virConfGetValueStringList:946 : Get value string list (nil) 0 2018-02-27 09:27:43.788+0000: 16656: debug : virConnectOpenInternal:1033 : Split "secret:///system" to URI components: scheme secret server <null> user <null> port 0 path /system .. 2018-02-27 09:27:43.788+0000: 16656: info : virObjectRef:388 : OBJECT_REF: obj=0x563fa24f2360 2018-02-27 09:27:43.788+0000: 16656: info : virObjectUnref:350 : OBJECT_UNREF: obj=0x563fa24f2360 2018-02-27 09:27:43.788+0000: 16656: debug : virConnectOpenInternal:1098 : driver 7 secret returned SUCCESS 2018-02-27 09:27:43.788+0000: 16656: error : qemuDomainGetSecretAESAlias:729 : invalid argument: encrypted secret alias requires valid source alias
This indicates that disk->info.alias is NULL meaning that qemuAssignDeviceDiskAlias needs to be called as well prior to qemuDomainPrepareDiskSource from Michal's patch; however, ...
2018-02-27 09:27:43.788+0000: 16656: info : virObjectUnref:350 : OBJECT_UNREF: obj=0x7f20840008c0 2018-02-27 09:27:43.788+0000: 16656: info : virObjectUnref:352 : OBJECT_DISPOSE: obj=0x7f20840008c0 2018-02-27 09:27:43.788+0000: 16656: info : virObjectUnref:350 : OBJECT_UNREF: obj=0x7f2030101cb0 2018-02-27 09:27:43.788+0000: 16656: debug : qemuDomainObjEndJob:5594 : Stopping job: modify (async=none vm=0x7f20300fa6e0 name=6ec499397d594ef2a64fcfc938f38225)
======================================================
But it fails.
Best Regards, Star Guo
-----邮件原件----- 发件人: Michal Privoznik [mailto:mprivozn@redhat.com] 发送时间: 2018年2月27日 16:53 收件人: Star Guo <starg@ceph.me>; libvirt-users@redhat.com 抄送: John Ferlan <jferlan@redhat.com> 主题: Re: [libvirt-users] Fail in virDomainUpdateDeviceFlags (libvirt-4.0.0 + Qemu-kvm 2.9.0 + Ceph 10.2.10)
On 02/27/2018 03:06 AM, Star Guo wrote:
Hello Everyone,
My pc run in CentOS 7.4 and install libvirt-4.0.0 + Qemu-kvm 2.9.0 + Ceph 10.2.10 ALL-in-One.
I use python-sdk with libvirt and run [self.domain.updateDeviceFlags(xml, libvirt.VIR_DOMAIN_AFFECT_LIVE)] on CDROM (I want to change media path). However, I enable libvirt debug log , the log as below:
<snip/>
I see the flow is virDomainUpdateDeviceFlags -> qemuMonitorChangeMedia, but the cephx auth is drop, so make update error. Anybody meet this error?
Yes, this is a libvirt bug. I think this fixes the issue:
diff --git i/src/qemu/qemu_driver.c w/src/qemu/qemu_driver.c index 96454c17c..0e5ad9971 100644 --- i/src/qemu/qemu_driver.c +++ w/src/qemu/qemu_driver.c @@ -7842,6 +7842,8 @@ qemuDomainChangeDiskLive(virDomainObjPtr vm, virQEMUDriverPtr driver, bool force) { + virQEMUDriverConfigPtr cfg = virQEMUDriverGetConfig(driver); + qemuDomainObjPrivatePtr priv = vm->privateData; virDomainDiskDefPtr disk = dev->data.disk; virDomainDiskDefPtr orig_disk = NULL; virDomainDeviceDef oldDev = { .type = dev->type }; @@ -7850,6 +7852,9 @@ qemuDomainChangeDiskLive(virDomainObjPtr vm, if (virDomainDiskTranslateSourcePool(disk) < 0) goto cleanup;
+ if (qemuDomainPrepareDiskSource(disk, priv, cfg) < 0) + goto cleanup; +
... this wouldn't be right anyway I believe... First of all, there's another path to qemuDomainChangeEjectableMedia via the attach command which wouldn't alter the "new" disk source (even if the disk->info.alias was set prior to this call). Secondly we "know" that the guest was started using some sort of disk source provided without the "tray='open'" being set. For example, if my similar type test using: <disk type='network' device='cdrom'> <driver name='qemu' type='raw'/> <source protocol='iscsi' name='iqn.2013-12.com.example:iscsi-1g-disks/1'> <host name='192.168.122.1' port='3260'/> <auth username='redhat'> <secret type='iscsi' usage='libvirtiscsi'/> </auth> </source> <target dev='hdb' bus='ide'/> <readonly/> </disk> had set "tray='open'" on the <target> element, then (as I found out after lots of debugging) qemuBuildDriveSourceStr would end up creating a CDROM, but the type would be 'file' as the subsequent dumpxml shows when the guest is running: <disk type='file' device='cdrom'> <target dev='hdb' bus='ide' tray='open'/> <readonly/> <alias name='ide0-0-1'/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk> So, back to the original issue and away from the diversion... "Theoretically speaking" the original guest start should have added a secret object for the original disk source using that disk source's alias. For example (using the iSCSI example from above) the following command line is created: -object secret,id=ide0-0-1-secret0,data=AtxGltyV5HoTvGhIaSOEeA==,keyid=masterKey0,iv=4l3Uancr25jF/rD5oTuFaw==,format=base64 -drive file.driver=iscsi,file.portal=192.168.122.1:3260,file.target=iqn.2013-12.com.example:iscsi-1g-disks,file.lun=1,file.transport=tcp,file.user=redhat,file.password-secret=ide0-0-1-secret0,format=raw,if=none,id=drive-ide0-0-1,readonly=on That means the secret object exists and "unlike" the other Attach paths that go through qemuDomainAttachDiskGeneric, we would already have the object and thus "should" be able to use the object if we could... The problem is - I see no way to do that. Both the QEMU "rbd" and "iscsi" driver open code need to have some way to authenticate to the server, but no mechanism in the "change" command is provided. For example, for the rbd example we start in qmp_change calling qmp_blockdev_change_medium calling bdrv_open calling qemu_rbd_open calling rados_connect whereupon the failure generates the "error connecting" message with the EOPNOTSUPP message as seen in the original posting (since snipped out): 2018-02-26 13:09:13.678+0000: 50516: error : qemuMonitorJSONCheckError:392 : internal error: unable to execute QEMU command 'change': error connecting: Operation not supported FWIW: The iSCSI code is a bit better at telling you what went wrong: error: internal error: unable to execute QEMU command 'change': iSCSI: Failed to connect to LUN : Failed to log in to target. Status: Authentication failure(513) But I was only able to get that far by making an adjustment to qemuBuildGeneralSecinfoURI to break on VIR_DOMAIN_SECRET_INFO_TYPE_AES so that the qemuGetDriveSourceString called in ChangeEjectableMedia wouldn't fail with "error: An error occurred, but the cause is unknown". I also read in qapi-schema.json that the "@change" command has the following dubious comment: # Notes: This interface is deprecated, and it is strongly recommended that you # avoid using it. For changing block devices, use # blockdev-change-medium; for changing VNC parameters, use # change-vnc-password. Digging slightly deeper into the @blockdev-change-medium command doesn't appear to show a way to also provide a secret object to use for the open, but I didn't follow that code any further In summary, this is a (very) long and winding way to say, you can't get there from here as far as I can tell. Someone at the very least needs to add a "password-secret" attribute/option (whatever it's called) to the qemu blockdev-change-medium command. Then libvirt needs to support the newer code and figure out the correct magic incantation in order to allow adding a network source change for a cdrom. May I also say/point out that qemuDomainDiskSourceDiffers has a comment that starts "This won't be a network storage..." which seems is not entirely true, but that's a different issue. Might be nice to file a couple of bug reports for feature requests so this isn't entirely lost. John Not the way I thought I'd start my post vacation day ;-)... Nothing like jumping headfirst into the very deep end of a particularly thorny and convoluted problem.
if (qemuDomainDetermineDiskChain(driver, vm, disk, false, true) < 0) goto cleanup;
@@ -7898,6 +7903,7 @@ qemuDomainChangeDiskLive(virDomainObjPtr vm,
ret = 0; cleanup: + virObjectUnref(cfg); return ret; }
Can you check and confirm?
Michal
participants (2)
-
John Ferlan -
Star Guo