- Users - libvirt List Archives

[libvirt-users] Lots of threads and increased IO load
by Gergely Horváth 13 Nov '13

13 Nov '13

Hello guys, We have a lot of small computers on top of two servers running QEMU/KVM virtualisation with libvirt. In this case, most of the VMs are not doing much work, some of them barely touches any "hardware" resources. I have two problems, they might be connected. When I do a bigger IO job on one of the VMs (like downloading files to local disk) all other VMs' IO load increases. Even our LDAP server's IO load is triggering an alert in our monitoring system - though that server barely does any IO throughout the day. The big IO job can NOT be considered big in the terms of the host computers IO capabilities, as it has two really fast SSD drives in hardware RAID 0 configuration. It can easily do hundreds of megabytes per second random write. The file copy I was running barely touched 50 MB/s. So this should not be a problem for the host's disks and I also do not understand why other computers IO load is increased. The second problem came when I looked at `iotop`, to see how much IO read and write is going on. The computer, that was doing the copying easily had like 20-30 threads writing pieces to the disk. Some of the threads are obviously "processor emulators" - they been running since the birth of the VM, but what are these other threads? TL;DR; Two questions than, to sum up everything: * How can I control and maybe decrease the number of threads spawned for a virtual machine? * How can I know what is the default IO caching for a VM, and what caching mechanism/policy should I set to minimise IO latency (i.e. I do not really care if the data is written to the disk later than the guest thinks) Thank you. P.S: We have been running hypertable servers the whole time, and even those did not generate such mess with IO... -- Üdvözlettel / Best regards Horváth Gergely | gergely.horvath(a)inepex.com IneTrack - Nyomkövetés egyszerűen | Inepex Kft. Ügyfélszolgálat: support(a)inetrack.hu | +36 30 825 7646 | support.inetrack.hu Web: www.inetrack.hu | nyomkovetes-blog.hu | facebook.com/inetrack Inepex - The White Label GPS fleet-tracking platform | www.inepex.com

2 3

[libvirt-users] Problem with bridged network configuration
by Bob Doolittle 13 Nov '13

13 Nov '13

Hi, When I initially boot my system, it fails to bring up the network. Once booted, if I run "systemctl restart network.service", or even stop, wait a while, then start, it starts up fine and all is well. I'm running Fedora 19 (updated). I followed the instructions on this page to set up a bridged network: http://wiki.libvirt.org/page/Networking I note however that I did *not* have any bridge (for NAT use) configured initially by default on my system, and followed the directions on this page to create it: https://fedoraproject.org/wiki/Getting_started_with_virtualization I have disabled the NetworkManager service, and created network initscripts for my em1 and br0 interfaces. I notice from the systemctl status output, that the network failure is logged, and only *after* that the br0 interface comes up. So I wonder if there is some ordering problem in spite of the fact that I have specified DELAY=0 for br0. I have included output from the following commands: systemctl status network.service (initially, when failed to come up) brctl show ip link ip address Also attached are the network initscripts for my em1 and br0 interfaces. Any guidance would be much appreciated! Thanks, Bob

3 3

[libvirt-users] Turning off polling
by Timothy Ehlers 12 Nov '13

12 Nov '13

Is it possible to turn off polling, we have 50~60 kvm instances per machine and have found that when libvirtd is running each KVM pid has 20% additional cpu usage. I turned on debug logging for 10 minutes and it seems to be 99% of this: virEventPollDispatchHandle But i cannot find a way to disable it. -- Tim Ehlers

2 1

[libvirt-users] InfiniBand Passthrough not working
by Michael Kruse 12 Nov '13

12 Nov '13

1 0

[libvirt-users] newer libvirt version issues
by Franky Van Liedekerke 12 Nov '13

12 Nov '13

Hi, since we're running more than 20 hosts per KVM server, we needed to update libvirt to at least 1.1.2 for virtlockd to be able to cope with this (due to an old hardcoded limit that was in there before). But where 1.1.0 compiles and runs just fine on a fully-patched CentOS 6.4 server, newer version have all kinds of issues: Every version of libvirt >= 1.1.2 crashes with a segfault on up-to-date CentOS 6.4 servers (see below). Also, versions of libvirt >= 1.1.3 need a change in the spec file to be able to compile: apparently the %doc entries that are different with 1.1.2 prevent rpmbuild to succeed on CentOS servers. Using the %doc lines as per the 1.1.2 spec file results in a working rpmbuild. The 1.1.4 version has issues with a symbol not found after compiling and letting it run: error : virDriverLoadModule:78 : failed to load module /usr/lib64/libvirt/connection-driver/libvirt_driver_network.so /usr/lib64/libvirt/connection-driver/libvirt_driver_network.so: undefined symbol: virNetworkList error : virDriverLoadModule:78 : failed to load module /usr/lib64/libvirt/connection-driver/libvirt_driver_storage.so /usr/lib64/libvirt/connection-driver/libvirt_driver_storage.so: undefined symbol: virAsprintf error : virDriverLoadModule:78 : failed to load module /usr/lib64/libvirt/connection-driver/libvirt_driver_nodedev.so /usr/lib64/libvirt/connection-driver/libvirt_driver_nodedev.so: undefined symbol: virNodeDeviceList 31677: error : virDriverLoadModule:78 : failed to load module /usr/lib64/libvirt/connection-driver/libvirt_driver_secret.so /usr/lib64/libvirt/connection-driver/libvirt_driver_secret.so: undefined symbol: virAsprintf 31677: error : virDriverLoadModule:78 : failed to load module /usr/lib64/libvirt/connection-driver/libvirt_driver_nwfilter.so /usr/lib64/libvirt/connection-driver/libvirt_driver_nwfilter.so: undefined symbol: virAsprintf 31677: error : virDriverLoadModule:78 : failed to load module /usr/lib64/libvirt/connection-driver/libvirt_driver_interface.so /usr/lib64/libvirt/connection-driver/libvirt_driver_interface.so: undefined symbol: virAsprintf 31677: error : virDriverLoadModule:78 : failed to load module /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so: undefined symbol: networkAllocateActualDevice 31677: error : virDriverLoadModule:78 : failed to load module /usr/lib64/libvirt/connection-driver/libvirt_driver_lxc.so /usr/lib64/libvirt/connection-driver/libvirt_driver_lxc.so: undefined symbol: networkAllocateActualDevice And every time libvirt segfaults, there's this in the logs: debug : virJSONValueToString:1133 : result={"id":"libvirt-6","error":{"class":"CommandNotFound","desc":"The command qom-list has not been found","data":{"name":"qom-list"}}} debug : virEventPollRunOnce:627 : EVENT_POLL_RUN: nhandles=32 timeout=1684 9479: debug : qemuMonitorJSONCheckError:341 : unable to execute QEMU command {"execute":"qom-list","arguments":{"path":"/"},"id":"libvirt-6"}: {"id":"libvirt-6","error":{"class":"CommandNotFound","desc":"The command qom-list has not been found","data":{"name":"qom-list"}}} 9479: error : qemuMonitorJSONCheckError:352 : internal error: unable to execute QEMU command 'qom-list': The command qom-list has not been found Any insights on any of these issues? Franky

1 0

[libvirt-users] Unable to shutdown/reboot a SLES 10 SP4 VM
by Padma Narwankar 11 Nov '13

11 Nov '13

Hello, I am unable to do a graceful shutdown or reboot of SLES 10 SP4 virtual machines (Kernel version: 2.6.16.60-0.85.1-default) with virsh shutdown or virsh reboot commands. (e.g virsh shutdown "vmname" ). On issuing these commands there is no impact on the VM. Also unable to perform the shutdown or reboot operations from the menu options available on the console when accessing the VM via the virt-manager. The virsh shutdown/reboot commands and the shutdown/reboot menu operations work fine for SLES 11 SP2 VMs. I've verified that the acpi daemon is running with the SLES 10 SP4 VM linux-kb8x:~ # ps aux | grep acpi root 8 0.0 0.0 0 0 ? S< 16:27 0:00 [kacpid] root 9 0.0 0.0 0 0 ? S< 16:27 0:00 [kacpi_notify] root 1830 0.0 0.0 2688 528 ? Ss 16:27 0:00 /sbin/acpid Is there any other process that needs to running for these operations to work or is there a known issue with SLES 10 SP4? Thanks, Padmaja

2 1

[libvirt-users] Error in snapshot create in libvirt 1.1.4
by Georg Meier 11 Nov '13

11 Nov '13

Hi list, i currently have the following issue: I did a test upgrade to libvirt 1.1.4 and now i cannot create new snapshots which can be reverted. Following Situation: Virtualization: kvm Disktype: qcow2 qemu XML: <cpu mode='host-passthrough'> </cpu> Creating of snapshot: virsh snapshot-create-as <guest> <name> <description> --atomic Now i craete a snapshot and as soon as i revert i get the following error: error : virCPUDefParseXML:367 : XML error: Non-empty feature list specified without CPU model I think the error is inside the snapshot xml (virsh snapshot-dumpxml): <cpu mode='host-passthrough'> <feature policy='require' name='pdpe1gb'/> <feature policy='require' name='osxsave'/> <feature policy='require' name='dca'/> <feature policy='require' name='pcid'/> <feature policy='require' name='pdcm'/> <feature policy='require' name='xtpr'/> <feature policy='require' name='tm2'/> <feature policy='require' name='est'/> <feature policy='require' name='smx'/> <feature policy='require' name='vmx'/> <feature policy='require' name='ds_cpl'/> <feature policy='require' name='monitor'/> <feature policy='require' name='dtes64'/> <feature policy='require' name='pbe'/> <feature policy='require' name='tm'/> <feature policy='require' name='ht'/> <feature policy='require' name='ss'/> <feature policy='require' name='acpi'/> <feature policy='require' name='ds'/> <feature policy='require' name='vme'/> </cpu> In older libvirt versions < 1.0.0 the xml looks the following way (virsh snapshot-dumpxml): <cpu mode='host-passthrough' match='minimum'> <model>SandyBridge</model> <vendor>Intel</vendor> <feature policy='require' name='pdpe1gb'/> <feature policy='require' name='osxsave'/> <feature policy='require' name='dca'/> <feature policy='require' name='pcid'/> <feature policy='require' name='pdcm'/> <feature policy='require' name='xtpr'/> <feature policy='require' name='tm2'/> <feature policy='require' name='est'/> <feature policy='require' name='smx'/> <feature policy='require' name='vmx'/> <feature policy='require' name='ds_cpl'/> <feature policy='require' name='monitor'/> <feature policy='require' name='dtes64'/> <feature policy='require' name='pbe'/> <feature policy='require' name='tm'/> <feature policy='require' name='ht'/> <feature policy='require' name='ss'/> <feature policy='require' name='acpi'/> <feature policy='require' name='ds'/> <feature policy='require' name='vme'/> </cpu> So for what ever reason the model and vendor tag are not copied inside the snapshot metadata xml when using cpu mode cpu-passthrough. virsh dumpxml with --update-cpu flag reports the correct cpu node with model and vendor tags. After a stop of libvirtd and loooking inside the xml file the cpu tag looks the following: <cpu mode='host-passthrough' match='minimum'> <model>SandyBridge</model> <vendor>Intel</vendor> <feature policy='require' name='pdpe1gb'/> <feature policy='require' name='osxsave'/> <feature policy='require' name='dca'/> <feature policy='require' name='pcid'/> <feature policy='require' name='pdcm'/> <feature policy='require' name='xtpr'/> <feature policy='require' name='tm2'/> <feature policy='require' name='est'/> <feature policy='require' name='smx'/> <feature policy='require' name='vmx'/> <feature policy='require' name='ds_cpl'/> <feature policy='require' name='monitor'/> <feature policy='require' name='dtes64'/> <feature policy='require' name='pbe'/> <feature policy='require' name='tm'/> <feature policy='require' name='ht'/> <feature policy='require' name='ss'/> <feature policy='require' name='acpi'/> <feature policy='require' name='ds'/> <feature policy='require' name='vme'/> </cpu> After the start of libvirtd virsh reports the cpu tag with missing vendor and model. Does anybody else has this problem? Best regards Georg

1 0

[libvirt-users] virt-install not spawning VM
by Sudarshan Pathalam (supathal) 08 Nov '13

08 Nov '13

Hello, In trying girt-install with below parameters I get the following error, -bash: virt-install --connect qemu:///system --name vm1 --ram 2048 --vcpus: No such file or directory What am I doing wrong? Thanks in advance. host:~# qemu-img create -f qcow2 /var/lib/libvirt/images/vm1.qcow2 20480 Formatting '/var/lib/libvirt/images/vm1.qcow2', fmt=qcow2 size=20480 encryption=off cluster_size=65536 host:~# ls /var/lib/libvirt/images/ ubuntu-12.04.3-server-amd64.iso vm1.qcow2 host:~# virt-install \ > --connect qemu:///system \ > --name vm1 \ > --ram 2048 \ > --vcpus 2 \ > --disk path=/var/lib/libvirt/images/vm1.qcow2,size=20 \ > --cdrom /var/lib/libvirt/images/ubuntu-12.04.3-server-amd64.iso \ > --graphics vnc \ > --virt-type kvm \ > --os-type linux\ > --network=br1 \ > --hvm -bash: virt-install --connect qemu:///system --name vm1 --ram 2048 --vcpus: No such file or directory

2 1

[libvirt-users] RBD images locking
by NEVEU Stephane 08 Nov '13

08 Nov '13

Hi, One short question : Is there any chance to see locks on rbd/images in the next release ? Thank you :)

4 7

[libvirt-users] Calling 'virsh' from within a hook script - avoid deadlock?
by TSADOK, Shlomi (Shlomi) 07 Nov '13

07 Nov '13

I'm trying to create a hook that attaches several virtio-scsi disks to a starting vm, and it seems like the hook enters a deadlock while attempting to do so. Is there any workaround around this? Any better way, forking virsh execution somehow? Could anyone explain why this is creating a deadlock in the first place? Isn't libvirt multithreaded? Thanks for the help! Shlomi ### hook script: #!/bin/bash enum_scsi() { scsi_devices=`cd /dev;ls sd[^a]*` } gen_hba_xml() { cat > ./hba.xml << EOF <controller type='scsi' model='virtio-scsi'/> EOF } passthrough_start() { # create virtio-scsi HBA gen_hba_xml virsh attach-device --persistent $domain_name ./hba.xml rm -f ./hba.xml # attache scsi devices to guest enum_scsi for sd in ${scsi_devices[@]} do virsh attach-disk --persistent $domain_name /dev/${sd} ${sd} done } domain_name=$1 domain_event=$2 case $domain_event in start) passthrough_start ;; *) echo "qemu hook called with unexpected options $*" >&2 ;; esac ### starce libvirt: [root@localhost ~]# strace -f -p 14630 Process 14630 attached with 11 threads - interrupt to quit [pid 14640] futex(0xc34ebc, FUTEX_WAIT_PRIVATE, 2303, NULL <unfinished ...> [pid 14638] futex(0xc34ebc, FUTEX_WAIT_PRIVATE, 2303, NULL <unfinished ...> [pid 14637] futex(0xc34ebc, FUTEX_WAIT_PRIVATE, 2303, NULL <unfinished ...> [pid 14636] futex(0xc34ebc, FUTEX_WAIT_PRIVATE, 2303, NULL <unfinished ...> [pid 14635] futex(0x7fcc240ccd80, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...> [pid 14639] futex(0xc34ebc, FUTEX_WAIT_PRIVATE, 2303, NULL <unfinished ...> [pid 14634] futex(0x7fcc240ccd80, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...> [pid 14631] restart_syscall(<... resuming interrupted call ...> <unfinished ...> [pid 14633] futex(0x7fcc240ccd80, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...> [pid 14630] restart_syscall(<... resuming interrupted call ...> <unfinished ...> [pid 14632] futex(0xc34e24, FUTEX_WAIT_PRIVATE, 3091, NULL^C <unfinished ...> ... [pid 14630] poll([{fd=7, events=POLLIN}, {fd=9, events=POLLIN}, {fd=12, events=POLLIN}, {fd=13, events=POLLIN}, {fd=6, events=POLLIN}, {fd=14, events=POLLIN}, {fd=16, events=POLLIN}, {fd=18, events=POLLIN|POLLOUT}, {fd=24, events=POLLIN}], 9, 1927) = 1 ([{fd=18, revents=POLLOUT}]) [pid 14630] gettid() = 14630 [pid 14630] gettid() = 14630 [pid 14630] gettid() = 14630 [pid 14630] gettid() = 14630 [pid 14630] gettid() = 14630 [pid 14630] gettid() = 14630 [pid 14630] gettid() = 14630 [pid 14630] gettid() = 14630 [pid 14630] gettid() = 14630 [pid 14630] gettid() = 14630 [pid 14630] gettid() = 14630 [pid 14630] gettid() = 14630 [pid 14630] write(18, "\0\0\0\34keep\0\0\0\1\0\0\0\1\0\0\0\2\0\0\0\0\0\0\0\0", 28) = 28 ... ### strace bash: [root@localhost ~]# strace -f -p 14783 Process 14783 attached - interrupt to quit wait4(-1, ...

3 6