May 2018 - Users - libvirt List Archives

Re: [libvirt-users] VM I/O performance drops dramatically during storage migration with drive-mirror
by Kashyap Chamarthy 30 May '18

30 May '18

Cc the QEMU Block Layer mailing list (qemu-block(a)nongnu.org) who might have more insights here; and wrap long lines. On Mon, May 28, 2018 at 06:07:51PM +0800, Chunguang Li wrote: > Hi, everyone. > > Recently I am doing some tests on the VM storage+memory migration with > KVM/QEMU/libvirt. I use the following migrate command through virsh: > "virsh migrate --live --copy-storage-all --verbose vm1 > qemu+ssh://192.168.1.91/system tcp://192.168.1.91". I have checked the > libvirt debug output, and make sure that the drive-mirror + NBD > migration method is used. > > Inside the VM, I use an I/O benchmark (Iometer) to generate an oltp > workload. I record the I/O performance (IOPS) before/during/after > migration. When the migration begins, the IOPS dropped by 30%-40%. > This is reasonable, because the migration I/O competes with the > workload I/O. However, during almost the last period of migration > (which is 66s in my case), the IOPS dropped dramatically, from about > 170 to less than 10. I also show the figure of this experiment in the > attachment of this email. [The attachment should arrive on the 'libvirt-users' list archives; but it's not there yet -- https://www.redhat.com/archives/libvirt-users/2018-May/thread.html] > I want to figure out what results in this period with very low IOPS. > First, I added some printf()s in the QEMU code, and knew that, this > period occurs just before the memory migration phase. (BTW, the memory > migration is very fast, which is just about 5s.) So I think this > period should be the last phase of the "drive-mirror" process of QEMU. > So then I tried to read the code of "drive-mirror" in QEMU, but failed > to understand it very well. > > Does anybody know what may lead to this period with very low IOPS? > Thank you very much. > > Some details of this experiment: The VM disk image file is 30GB > (format = raw,cache=none,aio=native), and Iometer operates on an 10GB > file inside the VM. The oltp workload consists of 33% writes and 67% > reads (8KB request size, all random). The VM memory size is 4GB, most > of which should be zero pages, so the memory migration is very fast. > > -- > Chunguang Li, Ph.D. Candidate > Wuhan National Laboratory for Optoelectronics (WNLO) > Huazhong University of Science & Technology (HUST) > Wuhan, Hubei Prov., China -- /kashyap

3 4

[libvirt-users] Using libvirt to access Ceph RBDs with Xen
by thg 30 May '18

30 May '18

Hi everybody, my background: I'm doing Xen since 10++ years, many years with DRBD for high availability, since some time I'm using preferable GlusterFS with FUSE as replicated storage, where I place the image-files for the vms. In my current project we started (successfully) with Xen/GlusterFS too, but because the provider, where we placed the servers, uses widely CEPH, we decided to switch too. Unfortunately I'm new to CEPH, but with help of a technician, we have running a 3 node CEPH-cluster now, that seems to work fine. Hardware: - Xeons, 24 Cores, 256 GB RAM, 2x 240 GB system-SSDs RAID1, 4x 1,92 TB data-SSDs (no RAID) Software we are using: - CentOS 7.5.1804 - Kernel: 4.9.86-30.el7 @centos-virt-xen-48 - Xen: 4.8.3-5.el7 @centos-virt-xen-48 - libvirt-xen: 4.1.0-2.xen48.el7 @centos-virt-xen-48 - Ceph: 2:12.2.5-0.el7 @Ceph CEPH itself works, Xen too and even libvirt (I'm quite new in this too, normally I use the XL-tools to manage my VMs) ;-) I can manually map an RBD and start a PV-VM via libvirt, after modifying the XML-config and defined the VM with libvirt. That is what happens: # virsh define xml/testvm3.xml Domain testvm3 defined from xml/testvm3.xml # virsh start --console testvm3 error: Failed to start domain testvm3 error: internal error: libxenlight failed to create new domain 'testvm3' xml/testvm3.xml: -> ... <devices> <disk type='network' device='disk'> <source protocol='rbd' name='devel-pool/testvm3.rbd'> <host name="10.20.30.1" port="6789"/> <host name="10.20.30.2" port="6789"/> <host name="10.20.30.3" port="6789"/> </source> <auth username='libvirt'> <secret type='ceph' uuid='07f3a0fe-9178-xxxxxxxxx'/> </auth> <target dev='xvda' bus='xen'/> </disk> ... Logs: /var/log/libvirt/libxl/libxl-driver.log: -> ... 2018-05-27 11:24:36.666+0000: libxl: libxl_bootloader.c:634:bootloader_finished: bootloader failed - consult logfile /var/log/xen/bootloader.14.log 2018-05-27 11:24:36.666+0000: libxl: libxl_exec.c:118:libxl_report_child_exitstatus: bootloader [183216] exited with error status 1 2018-05-27 11:24:36.667+0000: libxl: libxl_create.c:1259:domcreate_rebuild_done: cannot (re-)build domain: -3 /var/log/xen/bootloader.14.log: -> Traceback (most recent call last): File "/usr/lib64/xen/bin/pygrub", line 896, in <module> part_offs = get_partition_offsets(file) File "/usr/lib64/xen/bin/pygrub", line 113, in get_partition_offsets image_type = identify_disk_image(file) File "/usr/lib64/xen/bin/pygrub", line 56, in identify_disk_image fd = os.open(file, os.O_RDONLY) OSError: [Errno 2] No such file or directory: 'rbd:devel-pool/testvm3.rbd:id=libvirt:key=AQBThwFbGFRYFxxxxxxxxxxxxxxx==:auth_supported=cephx\\;none:mon_host=10.20.30.1\\:6789\\;10.20.30.2\\:6789\\;10.20.30.3\\:6789' So the Xen-Python-script, that searches for the kernel/bootloader of the VM, does not find the imagefile. But if I grab the above output and use it with qemu, it does find the blockdevice: # qemu-img info "rbd:devel-pool/testvm3.rbd:id=libvirt:key=AQBThwFbGFRYFxxxxxxxxxxxxxxx==:auth_supported=cephx\\;none:mon_host=10.20.30.1\\:6789\\;10.20.30.2\\:6789\\;10.20.30.3\\:6789" -> image: rbd:devel-pool/testvm3.rbd:id=libvirt:key=AQBThwFbGFRYFxxxxxxxxxxxxxxx==:auth_supported=cephx\;none:mon_host=10.20.30.1\:6789\;10.20.30.2\:6789\;10.20.30.3\:6789 file format: raw virtual size: 16G (17179869184 bytes) disk size: unavailable Thus it seems, that CEPH-auth works and the VM-XML-config is interpreted correct by libvirt. But the next step fails, when libvirt should pass the blockdevice to Xen? Anybody out there who uses a similar configuration? Any hints how to debug this? Unfortunately I do not have anymore logs to look at. Thank you very much, -- Kind regards, thg

2 1

[libvirt-users] virsh migration job unexpectedly failed
by cokabug 26 May '18

26 May '18

Hi , when i have run virsh migrate command , the migration progress fiailed at [61 %]. command : virsh migrate --live --undefinesource --p2p --copy-storage-inc --verbose --persistent instance-00001959 qemu+tcp://host292/system result : Migration: [ 61 %]error: operation failed: migration job: unexpectedly failed os/kernel version: centos 6.7 / kernel 2.6.32 qemu-kvm version : 0.12.1.2 libvirt version : 0.10.2-54 the libvirt.log on target host: 2018-05-22 11:45:38.603+0000: 32314: warning : qemuDomainObjTaint:1459 : Domain id=30 name='instance-00001959' uuid=41975d30-2490-473e-be7b-9d2d08bf66d2 is tainted: high-privileges 2018-05-22 11:45:38.883+0000: 32314: warning : qemuDomainObjEnterMonitorInternal:1062 : This thread seems to be the async job owner; entering monitor without asking for a nested job is dangerous 2018-05-22 11:45:38.902+0000: 32314: warning : qemuDomainObjEnterMonitorInternal:1062 : This thread seems to be the async job owner; entering monitor without asking for a nested job is dangerous 2018-05-22 11:45:38.903+0000: 32314: warning : qemuDomainObjEnterMonitorInternal:1062 : This thread seems to be the async job owner; entering monitor without asking for a nested job is dangerous 2018-05-22 11:45:39.090+0000: 32314: warning : qemuDomainObjEnterMonitorInternal:1062 : This thread seems to be the async job owner; entering monitor without asking for a nested job is dangerous 2018-05-22 11:45:39.090+0000: 32314: warning : qemuDomainObjEnterMonitorInternal:1062 : This thread seems to be the async job owner; entering monitor without asking for a nested job is dangerous 2018-05-22 11:45:39.091+0000: 32314: warning : qemuDomainObjEnterMonitorInternal:1062 : This thread seems to be the async job owner; entering monitor without asking for a nested job is dangerous 2018-05-22 11:45:41.841+0000: 32310: error : qemuMonitorIO:615 : internal error End of file from monitor the qmeu-kvminstance-00001959.log: 2018-05-22 11:45:38.603+0000: starting up LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin HOME=/home/ning.zhao USER=root LOGNAME=root QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -name instance-00001959 -S -M rhel6.6.0 -cpu Broadwell,+abm,+pdpe1gb,+rdrand,+f16c,+osxsave,+dca,+pdcm,+xtpr,+tm2,+est,+smx,+vmx,+ds_cpl,+m$ nitor,+dtes64,+pbe,+tm,+ht,+ss,+acpi,+ds,+vme -enable-kvm -m 4096 -realtime mlock=off -smp 4,sockets=4,cores=1,threads=1 -uuid 41975d30-2490-473e-be7b-9d2d08bf66d2 -smbios type=1,manufacturer=Red Hat Inc.,product=OpenStack Nova,version=2013.1.2-4.el6,serial=fcbe6c27-c9$ d-e811-ac36-a281f9ebc503,uuid=41975d30-2490-473e-be7b-9d2d08bf66d2 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-00001959.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -n$ -kvm-pit-reinjection -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/nova/instances/41975d30-2490-473e-be7b-9d2d08bf66d2/disk,if=none,id=drive-virtio-disk0,format=qcow2,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4$ drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=28,id=hostnet0,vhost=on,vhostfd=29 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:c0:8f:6b,bus=pci.0,addr=0x3 -chardev file,id=charserial0,path=/var/lib/nova/instances/41975d30-2490-473e-b$ 7b-9d2d08bf66d2/console.log -device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -device usb-tablet,id=input0 -vnc 127.0.0.1:2 -k en-us -vga cirrus -incoming tcp: 0.0.0.0:49152 -device virtio-bal$ oon-pci,id=balloon0,bus=pci.0,addr=0x5 -msg timestamp=on Domain id=30 is tainted: high-privileges char device redirected to /dev/pts/2 Receiving block device images qemu: warning: error while loading state for instance 0x0 of device 'block' load of migration failed 2018-05-22 11:45:41.841+0000: shutting down i checked the log and disk file and they have right permissions, has anyone know about this ,i have google many times ,but i cant find the right debug way to resolve it. thanks

2 2

[libvirt-users] Create qcow2 v3 volumes via libvirt
by Gionatan Danti 25 May '18

25 May '18

Hi all, on a fully patched CentOS 7.4 x86-64, I see the following behavior: - when creating a new volumes using vol-create-as, the resulting file is a qcow2 version 2 (compat=0.10) file. Example: [root@gdanti-lenovo vmimages]# virsh vol-create-as default zzz.qcow2 8589934592 --format=qcow2 --backing-vol /mnt/vmimages/centos6.img Vol zzz.qcow2 created [root@gdanti-lenovo vmimages]# file zzz.qcow2 zzz.qcow2: QEMU QCOW Image (v2), has backing file (path /mnt/vmimages/centos6.img), 8589934592 bytes [root@gdanti-lenovo vmimages]# qemu-img info zzz.qcow2 image: zzz.qcow2 file format: qcow2 virtual size: 8.0G (8589934592 bytes) disk size: 196K cluster_size: 65536 backing file: /mnt/vmimages/centos6.img backing file format: raw Format specific information: compat: 0.10 refcount bits: 16 - when creating a snapshot, the resulting file is a qcow2 version 3 (comapt=1.1) file. Example: [root@gdanti-lenovo vmimages]# virsh snapshot-create-as centos6left --disk-only --no-metadata snap.qcow2 Domain snapshot snap.qcow2 created [root@gdanti-lenovo vmimages]# file centos6left.snap.qcow2 centos6left.snap.qcow2: QEMU QCOW Image (v3), has backing file (path /mnt/vmimages/centos6left.qcow2), 8589934592 bytes [root@gdanti-lenovo vmimages]# qemu-img info centos6left.snap.qcow2 image: centos6left.snap.qcow2 file format: qcow2 virtual size: 8.0G (8589934592 bytes) disk size: 196K cluster_size: 65536 backing file: /mnt/vmimages/centos6left.qcow2 backing file format: qcow2 Format specific information: compat: 1.1 lazy refcounts: false refcount bits: 16 corrupt: false From what I know, this is a deliberate decision: compat=1.1 requires relatively recent qemu version, and creating a new volume play on the "safe side" of compatibility. It is possible to create a new volume using qcow2 version 3 (compat=1.1) format *using libvirt/virsh* (I know I can do that via qemu-img)? Any drawback on using version 3 format? Thanks. -- Danti Gionatan Supporto Tecnico Assyoma S.r.l. - www.assyoma.it email: g.danti(a)assyoma.it - info(a)assyoma.it GPG public key ID: FF5F32A8

6 17

[libvirt-users] Unable to connect VMs to a bridge over bonded network on Debian 9 (works fine on Centos 7.4)
by Thing 23 May '18

23 May '18

Hi, Sorry of this is the wrong list, trying to find some help. I am trying to set up 2 KVM hosts using Debian 9 (connecting to a 3 node Centos 7.4 gluster cluster) To get the bandwidth I have bonded 2 NICS each end. However while the network works fine for the Debian VM host the VM guests cannot connect/get an IP using DHCP connecting to br0. (Same setup on the Centos end works fine however) On the Debian host if I connect to br1 which is a bridge to a NIC port it works fine. My problem then seems to be at the bonding layer for libvirt/kvm ? Here is my interfaces config, I cannot see the problem. The VM guest is on the gluster cluster and boots fine just no networking. any ideas what I am doing wrong pls? or a better group/list to go ask in? auto eno1 iface eno1 inet manual #iface eno1 inet static bond-master bond0 #address 192.168.1.62 #netmask 255.255.255.0 #gateway 192.168.1.1 #dns-nameservers 192.168.1.241 192.168.1.104 # mtu 9000 auto enp1s0f1 iface enp1s0f1 inet manual bond-master bond0 # mtu 9000 auto bond0 iface bond0 inet static bond-miimon 100 bond-mode 6 bond-updelay 200 bond-xmit_hash_policy layer3+4 bond-lacp-rate 1 # mtu 9000 # address 192.168.1.62 # netmask 255.255.255.0 # gateway 192.168.1.1 # dns-nameservers 192.168.1.241 192.168.1.104 slaves eno1 enp1s0f1 auto br0 iface br0 inet static address 192.168.1.62 netmask 255.255.255.0 gateway 192.168.1.1 dns-nameservers 192.168.1.241 192.168.1.104 bridge_ports bond0 #bridge_ports eno1 enp1s0f1 bridge_stp on bridge_fd 0.0 # mtu 9000 #post-up ifconfig bond0 mtu 9000 #post-up ip link set dev bond0 mtu 9000 #auto bond1 #iface bond1 inet static # address 10.100.200.62 # netmask 255.255.255.0 # bond_slaves enp3s0f1 # bond_primary enp3s0f1 # mtu 9000 auto br1 iface br1 inet static address 10.100.200.62 netmask 255.255.255.0 gateway 10.100.200.1 bridge_ports enp3s0f1 bridge_stp on bridge_fd 0.0 root@kvm02:/etc/network#

1 0

[libvirt-users] Attempting to use Secure Boot for Windows 10
by frizop f 23 May '18

23 May '18

Anybody have a guide on using secure boot? I managed to set */usr/share/edk2/ovmf/OVMF_CODE.secboot.fd* as the* <loader readonly='yes' type='pflash'>*, but when I change this windows only shows me a black screen. I've also tried to fiddle with the following two settings: *<type arch='x86_64' machine='pc-q35-2.10'>* I'm running fedora 27. Any advice on how to implement secure boot for a VM under KVM/Virt-Manager/qemu/etc etc. I found this https://en.opensuse.org/openSUSE:UEFI_Secure_boot_using_qemu-kvm but it doesn't seem to match up with what I'm seeing on a few different aspects, (obviously of which, is I'm not using SuSE) Anyway, any help at all on this would be greatly appreciated. Thanks! -frizop

1 0

[libvirt-users] Performance of guest
by Allence 22 May '18

22 May '18

I recently used Linux in VMware, but I discovered that kvm can replace it, so I created the kvm environment and then started a window on the host system. But when I use it, I have a problem. My guest's performance is not very good. For example, when I use Windows without kvm, the “lol” fps is 160, but my guest is only 135. why? I only have one guest what is the reason? What can we do to promote my performance? There are my some settings for me. timer: hypervclock cpu: 8 socket(intel@ core i7 7700k) memory: 16G GPU: 1060 network: virtio my environment: libvirt: 4.2 qemu: 2.12 windows: windows 7 host: ubuntu server 1604 If you need my detailed settings, please let me know.

2 2

[libvirt-users] Guest's CPU frequency cannot be similar to host system
by Allence 19 May '18

19 May '18

My resource: kernel: 4.16.8 host system: Ubuntu 16.04 Llbvirt : 4.2 Qemu : 2.12 Cpu : intel@core i7 6700 guest system: windows7 professional My problem: In my host, Cpu frequency is 3800~3900 MHz(4 socket and 2 thread) because i turn on “Intel Turbo Boost”. And silent frequency is 3.40 GHz. Then i boot my guest, But in Guest my cpu frequency is 3408 MHz. why it's not 3800 MHz? And here is my guest xml: <cpu mode='host-passthrough' check='none'> <topology sockets='1' cores='8' threads='1'/> </cpu>

2 2

[libvirt-users] changing pci addr of SCSI storage controller: Red Hat, Inc Virtio block device (rev 01)
by daggs 13 May '18

13 May '18

Greetings, when execute lspci inside my vm linux guest, I see this: :/# lspci | grep -i scsi 03:00.0 SCSI storage controller: Red Hat, Inc Virtio block device (rev 01) I don't see any such address in my xml file, is it possible to move that controller to another pci addr? Thanks, Dagg.

2 3

[libvirt-users] e1000 network interface takes a long time to set the link ready
by Ihar Hrachyshka 12 May '18

12 May '18

Hi, In kubevirt, we discovered [1] that whenever e1000 is used for vNIC, link on the interface becomes ready several seconds after 'ifup' is executed, which for some buggy images like cirros may slow down boot process for up to 1 minute [2]. If we switch from e1000 to virtio, the link is brought up and ready almost immediately. For the record, I am using the following versions: - L0 kernel: 4.16.5-200.fc27.x86_64 #1 SMP - libvirt: 3.7.0-4.fc27 - guest kernel: 4.4.0-28-generic #47-Ubuntu Is there something specific about e1000 that makes it initialize the link too slowly on libvirt or guest side? [1] https://github.com/kubevirt/kubevirt/issues/936 [2] https://bugs.launchpad.net/cirros/+bug/1768955 Ihar

4 6