[libvirt-users] Reply: Fail in virDomainUpdateDeviceFlags (libvirt-4.0.0 + Qemu-kvm 2.9.0 + Ceph 10.2.10)
by Star Guo
Dear Michal
After I fix the local libvirt master branch follow your patch, and build rpm
for CentOS 7.4. virDomainUpdateDeviceFlags as bellow:
================================================
2018-02-27 09:27:43.782+0000: 16656: debug : virDomainUpdateDeviceFlags:8326
: dom=0x7f2084000c50, (VM: name=6ec499397d594e f2a64fcfc938f38225,
uuid=6ec49939-7d59-4ef2-a64f-cfc938f38225), xml=<disk device="cdrom"
type="network"><source name="zstac
k/08085a31f8c43f278ed2f649ee166b1f@08085a31f8c43f278ed2f649ee166b1f"
protocol="rbd"><host name="10.0.229.181" port="6789" / ></source><auth
username="zstack"><secret type="ceph"
uuid="9b06bb70-dc13-4338-88fd-b0c72d5ab9e9" /></auth><target bus="ide "
dev="hdc" /><readonly /></disk>, flags=0x1
...
2018-02-27 09:27:43.788+0000: 16656: info : virObjectNew:254 : OBJECT_NEW:
obj=0x7f2084003cc0 classname=qemuDomainStorageSourcePrivate
2018-02-27 09:27:43.788+0000: 16656: debug : virConnectOpen:1169 :
name=secret:///system
2018-02-27 09:27:43.788+0000: 16656: info : virObjectNew:254 : OBJECT_NEW:
obj=0x7f20840008c0 classname=virConnect
2018-02-27 09:27:43.788+0000: 16656: debug : virConfLoadConfig:1576 :
Loading config file '/etc/libvirt/libvirt.conf'
2018-02-27 09:27:43.788+0000: 16656: debug : virConfReadFile:752 :
filename=/etc/libvirt/libvirt.conf
2018-02-27 09:27:43.788+0000: 16656: debug : virFileClose:110 : Closed fd 28
2018-02-27 09:27:43.788+0000: 16656: debug : virConfGetValueStringList:946 :
Get value string list (nil) 0
2018-02-27 09:27:43.788+0000: 16656: debug : virConnectOpenInternal:1033 :
Split "secret:///system" to URI components:
scheme secret
server <null>
user <null>
port 0
path /system
..
2018-02-27 09:27:43.788+0000: 16656: info : virObjectRef:388 : OBJECT_REF:
obj=0x563fa24f2360
2018-02-27 09:27:43.788+0000: 16656: info : virObjectUnref:350 :
OBJECT_UNREF: obj=0x563fa24f2360
2018-02-27 09:27:43.788+0000: 16656: debug : virConnectOpenInternal:1098 :
driver 7 secret returned SUCCESS
2018-02-27 09:27:43.788+0000: 16656: error : qemuDomainGetSecretAESAlias:729
: invalid argument: encrypted secret alias requires valid source alias
2018-02-27 09:27:43.788+0000: 16656: info : virObjectUnref:350 :
OBJECT_UNREF: obj=0x7f20840008c0
2018-02-27 09:27:43.788+0000: 16656: info : virObjectUnref:352 :
OBJECT_DISPOSE: obj=0x7f20840008c0
2018-02-27 09:27:43.788+0000: 16656: info : virObjectUnref:350 :
OBJECT_UNREF: obj=0x7f2030101cb0
2018-02-27 09:27:43.788+0000: 16656: debug : qemuDomainObjEndJob:5594 :
Stopping job: modify (async=none vm=0x7f20300fa6e0
name=6ec499397d594ef2a64fcfc938f38225)
======================================================
But it fails.
Best Regards,
Star Guo
-----邮件原件-----
发件人: Michal Privoznik [mailto:mprivozn@redhat.com]
发送时间: 2018年2月27日 16:53
收件人: Star Guo <starg(a)ceph.me>; libvirt-users(a)redhat.com
抄送: John Ferlan <jferlan(a)redhat.com>
主题: Re: [libvirt-users] Fail in virDomainUpdateDeviceFlags (libvirt-4.0.0
+ Qemu-kvm 2.9.0 + Ceph 10.2.10)
On 02/27/2018 03:06 AM, Star Guo wrote:
> Hello Everyone,
>
>
>
> My pc run in CentOS 7.4 and install libvirt-4.0.0 + Qemu-kvm 2.9.0 +
> Ceph
> 10.2.10 ALL-in-One.
>
>
>
> I use python-sdk with libvirt and run
> [self.domain.updateDeviceFlags(xml,
> libvirt.VIR_DOMAIN_AFFECT_LIVE)] on CDROM (I want to change media path).
> However, I enable libvirt debug log , the log as below:
>
> <snip/>
>
> I see the flow is virDomainUpdateDeviceFlags ->
> qemuMonitorChangeMedia, but the cephx auth is drop, so make update error.
Anybody meet this error?
Yes, this is a libvirt bug. I think this fixes the issue:
diff --git i/src/qemu/qemu_driver.c w/src/qemu/qemu_driver.c index
96454c17c..0e5ad9971 100644
--- i/src/qemu/qemu_driver.c
+++ w/src/qemu/qemu_driver.c
@@ -7842,6 +7842,8 @@ qemuDomainChangeDiskLive(virDomainObjPtr vm,
virQEMUDriverPtr driver,
bool force)
{
+ virQEMUDriverConfigPtr cfg = virQEMUDriverGetConfig(driver);
+ qemuDomainObjPrivatePtr priv = vm->privateData;
virDomainDiskDefPtr disk = dev->data.disk;
virDomainDiskDefPtr orig_disk = NULL;
virDomainDeviceDef oldDev = { .type = dev->type }; @@ -7850,6 +7852,9
@@ qemuDomainChangeDiskLive(virDomainObjPtr vm,
if (virDomainDiskTranslateSourcePool(disk) < 0)
goto cleanup;
+ if (qemuDomainPrepareDiskSource(disk, priv, cfg) < 0)
+ goto cleanup;
+
if (qemuDomainDetermineDiskChain(driver, vm, disk, false, true) < 0)
goto cleanup;
@@ -7898,6 +7903,7 @@ qemuDomainChangeDiskLive(virDomainObjPtr vm,
ret = 0;
cleanup:
+ virObjectUnref(cfg);
return ret;
}
Can you check and confirm?
Michal
6 years, 9 months
[libvirt-users] QEMU guest-agent safety in hostile VM?
by procmem
Hi. Is it still considered risky to use the QEMU guest agent in an
untrusted guest? A warning on these lines was written in the manual a
few years back when the feature made its debut. I wanted to know if it
was hardened since.
6 years, 9 months
[libvirt-users] "scripts are not supported on interfaces of type vhostuser" error
by Riccardo Ravaioli
Hi,
I'm having trouble starting a VM with vhostuser interfaces.
I have a simple configuration where a VM running Debian has 1 vhostuser
interface plugged into an OVS switch where a DPDK interface is already
plugged in.
$ ovs-vsctl show:
Bridge "switch1"
Port "switch1"
Interface "switch1"
type: internal
Port "1.switch1"
Interface "1.switch1"
type: dpdk
options: {dpdk-devargs="0000:0b:00.0"}
Port "0.switch1"
Interface "0.vm"
type: dpdkvhostuserclient
options: {vhost-server-path="/opt/oa/vhost/0.vm.sock"}
The relevant excerpt from the XML of my VM is:
<?xml version="1.0" ?>
<domain type="kvm" xmlns:ns0="http://libvirt.org/schemas/domain/qemu/1.0">
<!-- [...] -->
<devices>
<emulator>/opt/oa/bin/qemu-system-x86_64</emulator>
<!-- [...] -->
<interface type="vhostuser">
<script path="/opt/oa/etc/qemu-ifup"/>
<source mode="server" path="/opt/oa/vhost/0.vm.sock" type="unix"/>
<mac address="52:54:00:41:37:08"/>
<model type="virtio"/>
<driver queues="1">
<host csum="off"/>
</driver>
</interface>
<!-- [...] -->
</devices>
<ns0:commandline>
<ns0:arg value="-object"/>
<ns0:arg
value="memory-backend-file,id=mem,size=2G,mem-path=/dev/hugepages,share=on"/>
<ns0:arg value="-numa"/>
<ns0:arg value="node,memdev=mem"/>
<ns0:arg value="-mem-prealloc"/>
</ns0:commandline>
</domain>
Now, if I try to start my VM, I get the following error and the VM is not
started at all:
$ virsh create vm.xml
error: Failed to create domain from vm.xml
error: unsupported configuration: scripts are not supported on interfaces
of type vhostuser
The logs from libvirtd.log say:
2018-02-22 09:18:24.982+0000: 2033: warning :
qemuProcessStartWarnShmem:4539 : Detected vhost-user interface without any
shared memory, the interface might not be operational
2018-02-22 09:18:24.982+0000: 2033: error : qemuBuildHostNetStr:3894 :
unsupported configuration: scripts are not supported on interfaces of type
vhostuser
The logs from qemu simply say:
2018-02-22 09:26:15.857+0000: shutting down, reason=failed
And finally, ovs-vswitchd.log:
2018-02-22T09:18:24.715Z|00328|dpdk|INFO|VHOST_CONFIG: vhost-user client:
socket created, fd: 51
2018-02-22T09:18:24.716Z|00329|netdev_dpdk|INFO|vHost User device '0.vm'
created in 'client' mode, using client socket '/opt/oa/vhost/0.vm.sock'
2018-02-22T09:18:24.718Z|00330|dpdk|WARN|VHOST_CONFIG: failed to connect to
/opt/oa/vhost/0.vm.sock: No such file or directory
2018-02-22T09:18:24.718Z|00331|dpdk|INFO|VHOST_CONFIG:
/opt/oa/vhost/0.vm.sock: reconnecting...
2018-02-22T09:18:24.718Z|00332|bridge|INFO|bridge switch1: added interface
0.vm on port 5
Am I missing something on the openvswitch or on the libvirt side?
It looks like openvswitch can't find /opt/oa/vhost/0.vm.sock, but isn't
either openvswitch or libvirt in charge of creating it?
Then, I'm not too sure about the error messages in libvirtd.log...
My software versions are: libvirt 3.10.0, qemu 2.10.2, openvswitch 2.8.1
and DPDK 17.11.
Thanks a lot!
Riccardo
6 years, 10 months
[libvirt-users] How to use rtc_timeoffset (XL toolkit option) in Libvirt and Xen/libxl hypervisor?
by Volo M.
Hello techs,
We used XL toolkit for Xen hypervisor for starting Windows hvm VMs and the
option rtc_timeoffset worked perfectly with XL. It helped us to set correct
Timezone time/offset for Windows HVM VMs from inside virtual_machine xen
config.
For example:
Below option WORKS perfectly with XL toolkit.
[root@hw12xen ~]# cat goillltawdlssq.xl.cfg | grep offset
rtc_timeoffset = "7200" # +2 hours to UTC time (or +/-3600=+/-1 hour to
UTC time)
[root@hw12xen ~]#
Now we try to migrate all the Windows hvm VMs to using Libvirt and Xen
4.6.6 (or 4.4).
But as we see it's impossible to make VMs using correct time/offset set in
XML config like below:
below Libvirt option DOESN'T WORK as we expect...
[root@hw12xen ~]# cat goillltawdlssq.var.xml | grep offset
<clock offset='variable' adjustment='7200' />
[root@hw12xen ~]#
Can you please explain if there's something we're doing wrong way or it's
just a bug like described here:
https://bugzilla.redhat.com/show_bug.cgi?id=1212191 ?
Thanks in advance.
6 years, 10 months
[libvirt-users] what is the practical limit on snapshot generations?
by W Kern
I've googled around on this and haven't found much so I'll ask here.
I have a client who routinely does live snapshots on snapshot on
snapshot ( just keep going recursively)
I've told him to roll them up once in a while, but he feels its
unnecessary and only plans a rollup during a major maintenance period,
where he can take the whole site down and then watch it grind during the
commit.
Note these are Qcow2 (1.1) images and they are external snaps. Most of
the write activity is logging and medium database writes for an
ecommerce store. It is certainly NOT heavy write i/o.
Isn't there a cost (speed, resources) etc to all those generations. Any
safety issues? Is there an optimium # of generations where if you exceed
that you get diminishing returns.
At one point are you shooting yourself in the foot, or is this really no
big deal aside from two hours of downtime while the 'actual' commit occurs.
-wk
6 years, 10 months
[libvirt-users] Ask question
by 李杰
Dear
Recently,I am interested in libvirt,but I found the libvirt does't support revert and delete the external snapshot which the raw disk based guest.My libvirt and qemu release are both 3.2.0.So I have a question,the community how to solve it,if have any plans about it.Can you tell me more about it?Thank you very much.
Best Regards
Rambo
6 years, 10 months
[libvirt-users] kvm/libvirt on CentOS7 w/Windows 10 Pro guest
by Benjammin2068
Hey all,
New to list, so I apologize if this has been asked a bunch already...
Is there something I'm missing with Windows 10 as a guest that keeps Windows Updates from nuking the boot process?
I just did an orderly shutdown and windows updated itself <I forgot to disable in time> only to reboot to the diagnostics screen which couldn't repair.
going to command prompt and doing the usual "bootrec /fixmbr, /fixboot and /RebuildBcd" didn't help.
This has happened a few times. I can't believe how fragile Win10pro is while running in a VM.
(and it's happened on a couple machines I've been experimenting with -- both running same OS, but different hardware.)
I just saw the FAQ about the libvirt repo for the virtio drivers for windows.... I need to go read more on it...
but in the meantime, is there any other smoking gun I'm not aware of? (after lots of google searching)
Thanks,
-Ben
6 years, 10 months
[libvirt-users] Nested KVM: L0 guest produces kernel BUG on wakeup from managed save (while a nested VM is running)
by Florian Haas
Hi everyone,
I hope this is the correct list to discuss this issue; please feel
free to redirect me otherwise.
I have a nested virtualization setup that looks as follows:
- Host: Ubuntu 16.04, kernel 4.4.0 (an OpenStack Nova compute node)
- L0 guest: openSUSE Leap 42.3, kernel 4.4.104-39-default
- Nested guest: SLES 12, kernel 3.12.28-4-default
The nested guest is configured with "<type arch='x86_64'
machine='pc-i440fx-1.4'>hvm</type>".
This is working just beautifully, except when the L0 guest wakes up
from managed save (openstack server resume in OpenStack parlance).
Then, in the L0 guest we immediately see this:
[Tue Feb 6 07:00:37 2018] ------------[ cut here ]------------
[Tue Feb 6 07:00:37 2018] kernel BUG at ../arch/x86/kvm/x86.c:328!
[Tue Feb 6 07:00:37 2018] invalid opcode: 0000 [#1] SMP
[Tue Feb 6 07:00:37 2018] Modules linked in: fuse vhost_net vhost
macvtap macvlan xt_CHECKSUM iptable_mangle ipt_MASQUERADE
nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat
nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT
nf_reject_ipv4 xt_tcpudp tun br_netfilter bridge stp llc
ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter
ip_tables x_tables vboxpci(O) vboxnetadp(O) vboxnetflt(O) af_packet
iscsi_ibft iscsi_boot_sysfs vboxdrv(O) kvm_intel kvm irqbypass
crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel
hid_generic usbhid jitterentropy_rng drbg ansi_cprng ppdev parport_pc
floppy parport joydev aesni_intel processor button aes_x86_64
virtio_balloon virtio_net lrw gf128mul glue_helper pcspkr serio_raw
ablk_helper cryptd i2c_piix4 ext4 crc16 jbd2 mbcache ata_generic
[Tue Feb 6 07:00:37 2018] virtio_blk ata_piix ahci libahci cirrus(O)
drm_kms_helper(O) syscopyarea sysfillrect sysimgblt fb_sys_fops ttm(O)
drm(O) virtio_pci virtio_ring virtio uhci_hcd ehci_hcd usbcore
usb_common libata sg dm_multipath dm_mod scsi_dh_rdac scsi_dh_emc
scsi_dh_alua scsi_mod autofs4
[Tue Feb 6 07:00:37 2018] CPU: 2 PID: 2041 Comm: CPU 0/KVM Tainted: G
W O 4.4.104-39-default #1
[Tue Feb 6 07:00:37 2018] Hardware name: OpenStack Foundation
OpenStack Nova, BIOS 1.10.1-1ubuntu1~cloud0 04/01/2014
[Tue Feb 6 07:00:37 2018] task: ffff880037108d80 ti: ffff88042e964000
task.ti: ffff88042e964000
[Tue Feb 6 07:00:37 2018] RIP: 0010:[<ffffffffa04f20e5>]
[<ffffffffa04f20e5>] kvm_spurious_fault+0x5/0x10 [kvm]
[Tue Feb 6 07:00:37 2018] RSP: 0018:ffff88042e967d70 EFLAGS: 00010246
[Tue Feb 6 07:00:37 2018] RAX: 0000000000000000 RBX: ffff88042c4f0040
RCX: 0000000000000000
[Tue Feb 6 07:00:37 2018] RDX: 0000000000006820 RSI: 0000000000000282
RDI: ffff88042c4f0040
[Tue Feb 6 07:00:37 2018] RBP: ffff88042c4f00d8 R08: ffff88042e964000
R09: 0000000000000002
[Tue Feb 6 07:00:37 2018] R10: 0000000000000004 R11: 0000000000000000
R12: 0000000000000001
[Tue Feb 6 07:00:37 2018] R13: 0000021d34fbb21d R14: 0000000000000001
R15: 000055d2157cf840
[Tue Feb 6 07:00:37 2018] FS: 00007f7c52b96700(0000)
GS:ffff88043fd00000(0000) knlGS:0000000000000000
[Tue Feb 6 07:00:37 2018] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[Tue Feb 6 07:00:37 2018] CR2: 00007f823b15f000 CR3: 0000000429334000
CR4: 0000000000362670
[Tue Feb 6 07:00:37 2018] DR0: 0000000000000000 DR1: 0000000000000000
DR2: 0000000000000000
[Tue Feb 6 07:00:37 2018] DR3: 0000000000000000 DR6: 00000000fffe0ff0
DR7: 0000000000000400
[Tue Feb 6 07:00:37 2018] Stack:
[Tue Feb 6 07:00:37 2018] ffffffffa07939b1 ffffffffa0787875
ffffffffa0503a60 ffff88042c4f0040
[Tue Feb 6 07:00:37 2018] ffffffffa04e5ede ffff88042c4f0040
ffffffffa04e6f0f ffff880037108d80
[Tue Feb 6 07:00:37 2018] ffff88042c4f00e0 ffff88042c4f00e0
ffff88042c4f0040 ffff88042e968000
[Tue Feb 6 07:00:37 2018] Call Trace:
[Tue Feb 6 07:00:37 2018] [<ffffffffa07939b1>]
intel_pmu_set_msr+0xfc1/0x2341 [kvm_intel]
[Tue Feb 6 07:00:37 2018] DWARF2 unwinder stuck at
intel_pmu_set_msr+0xfc1/0x2341 [kvm_intel]
[Tue Feb 6 07:00:37 2018] Leftover inexact backtrace:
[Tue Feb 6 07:00:37 2018] [<ffffffffa0787875>] ?
vmx_interrupt_allowed+0x15/0x30 [kvm_intel]
[Tue Feb 6 07:00:37 2018] [<ffffffffa0503a60>] ?
kvm_arch_vcpu_runnable+0xa0/0xd0 [kvm]
[Tue Feb 6 07:00:37 2018] [<ffffffffa04e5ede>] ?
kvm_vcpu_check_block+0xe/0x60 [kvm]
[Tue Feb 6 07:00:37 2018] [<ffffffffa04e6f0f>] ?
kvm_vcpu_block+0x8f/0x310 [kvm]
[Tue Feb 6 07:00:37 2018] [<ffffffffa0503c17>] ?
kvm_arch_vcpu_ioctl_run+0x187/0x400 [kvm]
[Tue Feb 6 07:00:37 2018] [<ffffffffa04ea6d9>] ?
kvm_vcpu_ioctl+0x359/0x680 [kvm]
[Tue Feb 6 07:00:37 2018] [<ffffffff81016689>] ? __switch_to+0x1c9/0x460
[Tue Feb 6 07:00:37 2018] [<ffffffff81224f02>] ? do_vfs_ioctl+0x322/0x5d0
[Tue Feb 6 07:00:37 2018] [<ffffffff811362ef>] ?
__audit_syscall_entry+0xaf/0x100
[Tue Feb 6 07:00:37 2018] [<ffffffff8100383b>] ?
syscall_trace_enter_phase1+0x15b/0x170
[Tue Feb 6 07:00:37 2018] [<ffffffff81225224>] ? SyS_ioctl+0x74/0x80
[Tue Feb 6 07:00:37 2018] [<ffffffff81634a02>] ?
entry_SYSCALL_64_fastpath+0x16/0xae
[Tue Feb 6 07:00:37 2018] Code: d7 fe ff ff 8b 2d 04 6e 06 00 e9 c2
fe ff ff 48 89 f2 e9 65 ff ff ff 0f 1f 44 00 00 66 2e 0f 1f 84 00 00
00 00 00 0f 1f 44 00 00 <0f> 0b 66 0f 1f 84 00 00 00 00 00 0f 1f 44 00
00 55 89 ff 48 89
[Tue Feb 6 07:00:37 2018] RIP [<ffffffffa04f20e5>]
kvm_spurious_fault+0x5/0x10 [kvm]
[Tue Feb 6 07:00:37 2018] RSP <ffff88042e967d70>
[Tue Feb 6 07:00:37 2018] ---[ end trace e15c567f77920049 ]---
We only hit this kernel bug if we have a nested VM running. The exact
same setup, sent into managed save after shutting down the nested VM,
wakes up just fine.
Now I am aware of https://bugzilla.redhat.com/show_bug.cgi?id=1076294,
which talks about live migration — but I think the same considerations
apply.
I am also aware of
https://fedoraproject.org/wiki/How_to_enable_nested_virtualization_in_KVM,
which strongly suggests to use host-passthrough or host-model. I have
tried both, to no avail. The stack trace persists. I have also tried
running a 4.15 kernel in the L0 guest, from
https://kernel.opensuse.org/packages/stable, but again, the stack
trace persists.
What does fix things, of course, is to switch from the nested guest
from KVM to Qemu — but that also makes things significantly slower.
So I'm wondering: is there someone reading this who does run nested
KVM and has managed to successfully live-migrate or managed-save? If
so, would you be able to share a working host kernel / L0 guest kernel
/ nested guest kernel combination, or any other hints for tuning the
L0 guest to support managed save and live migration?
I'd be extraordinarily grateful for any suggestions. Thanks!
Cheers,
Florian
6 years, 10 months
