- Users - libvirt List Archives

[libvirt-users] vm running slowly in powerful host
by Lentes, Bernd 17 Feb '17

17 Feb '17

Hi, i have a vm which has a poor performance. E.g. top needs seconds to refresh its output on the console. Same with netstat. The guest is hosting a MySQL DB with a webfrontend, its response is poor too. I'm looking for the culprit. Following top in the guest i get these hints: Memory is free enough, system is not swapping. System has 8GB RAM and two cpu's. Cpu 0 is struggling with a lot of software interrupts, between 50% and 80%. Cpu1 is often waiting for IO (wa), between 0% and 20%. No application is consuming much cpu time. Here is an example: top - 11:19:18 up 18:19, 11 users, load average: 1.44, 0.94, 0.66 Tasks: 95 total, 1 running, 94 sleeping, 0 stopped, 0 zombie Cpu0 : 0.0%us, 0.0%sy, 0.0%ni, 20.0%id, 0.0%wa, 0.0%hi, 80.0%si, 0.0%st Cpu1 : 1.9%us, 13.8%sy, 0.0%ni, 73.8%id, 10.5%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 7995216k total, 6385176k used, 1610040k free, 177772k buffers Swap: 2104472k total, 0k used, 2104472k free, 5940884k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 6470 root 16 0 12844 1464 804 S 12 0.0 2:17.13 screen 6022 root 15 0 41032 3052 2340 S 3 0.0 1:10.99 sshd 8322 root 0 -20 10460 4976 2268 S 3 0.1 19:20.38 atop 10806 root 16 0 5540 1216 880 R 0 0.0 0:00.51 top 126 root 15 0 0 0 0 S 0 0.0 0:23.33 pdflush 3531 postgres 15 0 68616 1600 792 S 0 0.0 0:41.24 postmaster The host in which the guest runs has 96GB RAM and 8 cores. It does not seem to do much: top - 11:21:19 up 15 days, 15:53, 14 users, load average: 1.40, 1.39, 1.40 Tasks: 221 total, 2 running, 219 sleeping, 0 stopped, 0 zombie Cpu0 : 15.9%us, 2.7%sy, 0.0%ni, 81.4%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu1 : 5.0%us, 3.0%sy, 0.0%ni, 92.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu2 : 2.0%us, 0.3%sy, 0.0%ni, 97.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu3 : 0.3%us, 1.0%sy, 0.0%ni, 98.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu4 : 1.3%us, 0.3%sy, 0.0%ni, 98.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu5 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu6 : 0.3%us, 0.0%sy, 0.0%ni, 99.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu7 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 96738M total, 13466M used, 83272M free, 3M buffers Swap: 2046M total, 0M used, 2046M free, 3887M cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 21765 root 20 0 105m 15m 4244 S 5 0.0 0:00.15 crm 3180 root 20 0 8572m 8.0g 8392 S 3 8.4 62:25.73 qemu-kvm 8529 hacluste 10 -10 90820 14m 9400 S 0 0.0 29:52.48 cib 21329 root 20 0 9040 1364 940 R 0 0.0 0:00.16 top 28439 root 20 0 0 0 0 S 0 0.0 0:04.51 kworker/4:2 1 root 20 0 10560 828 692 S 0 0.0 0:07.67 init 2 root 20 0 0 0 0 S 0 0.0 0:00.28 kthreadd 3 root 20 0 0 0 0 S 0 0.0 3:03.23 ksoftirqd/0 6 root RT 0 0 0 0 S 0 0.0 0:05.02 migration/0 7 root RT 0 0 0 0 S 0 0.0 0:02.82 watchdog/0 8 root RT 0 0 0 0 S 0 0.0 0:05.18 migration/1 I think the host is not the problem. The vm resides on a SAN which is attached via FC. The whole system is a two node cluster. The vm resides in a raw partition without a FS, which i read should be good for the performance. It runs on the other node slow too. Inside the vm i have logical volumes (it was a physical system i migrated to a vm). The partitions are formatted with reiserfs (The system is already some years old, at that time reiserfs was popular ...). I use iostat on the guest: This is a typical snapshot: Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util vda 0.00 3.05 0.00 2.05 0.00 20.40 19.90 0.09 44.59 31.22 6.40 dm-0 0.00 0.00 0.00 4.55 0.00 18.20 8.00 0.24 52.31 7.74 3.52 dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-2 0.00 0.00 0.00 0.10 0.00 0.40 8.00 0.01 92.00 56.00 0.56 dm-3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-5 0.00 0.00 0.00 0.35 0.00 1.40 8.00 0.03 90.29 65.71 2.30 dm-6 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 vda has several partitions, one for /, one for swap, and two physical volumes for LVM. Following "man iostat", the columns await and svctm seem to be important. Man says: await The average time (in milliseconds) for I/O requests issued to the device to be served. This includes the time spent by the requests in queue and the time spent servicing them. svctm The average service time (in milliseconds) for I/O requests that were issued to the device. It seems system is waiting a long time for IO. Although the amount of transfered data is small. I have some suspicions: - the lvm setup in the guest - some hardware - cache mode for the disk is "none". Otherwise i can't do a live migration. What do you think ? How can i find out from where the high si comes ? Network and disk are virtio devices (which should be fast): vm58820-4:~ # lsmod|grep -i virt virtio_balloon 22788 0 virtio_net 30464 0 virtio_pci 27264 0 virtio_ring 21376 1 virtio_pci virtio_blk 25224 5 virtio 22916 4 virtio_balloon,virtio_net,virtio_pci,virtio_blk That's the config of the guest: <domain type='kvm'> <name>mausdb_vm</name> <uuid>f08c2f32-fe35-137a-0e9d-fa7485d57974</uuid> <memory unit='KiB'>8198144</memory> <currentMemory unit='KiB'>8197376</currentMemory> <vcpu placement='static'>2</vcpu> <os> <type arch='x86_64' machine='pc-i440fx-1.4'>hvm</type> <boot dev='cdrom'/> <bootmenu enable='yes'/> </os> <features> <acpi/> <apic/> <pae/> </features> <clock offset='utc'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/bin/qemu-kvm</emulator> <disk type='block' device='disk'> <driver name='qemu' type='raw' cache='none'/> <source dev='/dev/vg_cluster_01/lv_cluster_01'/> <target dev='vda' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </disk> <controller type='usb' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> </controller> <controller type='pci' index='0' model='pci-root'/> <controller type='ide' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <interface type='bridge'> <mac address='52:54:00:37:92:01'/> <source bridge='br0'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <graphics type='vnc' port='-1' autoport='yes' listen='127.0.0.1'> <listen type='address' address='127.0.0.1'/> </graphics> <video> <model type='cirrus' vram='9216' heads='1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </video> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </memballoon> </devices> <seclabel type='none'/> </domain> Host OS is SLES 11 SP4, guest os is SLES 10 SP4. Both 64bit. Thanks for any hint. Bernd -- Bernd Lentes Systemadministration institute of developmental genetics Gebäude 35.34 - Raum 208 HelmholtzZentrum München bernd.lentes(a)helmholtz-muenchen.de phone: +49 (0)89 3187 1241 fax: +49 (0)89 3187 2294 Erst wenn man sich auf etwas festlegt kann man Unrecht haben Scott Adams Helmholtz Zentrum Muenchen Deutsches Forschungszentrum fuer Gesundheit und Umwelt (GmbH) Ingolstaedter Landstr. 1 85764 Neuherberg www.helmholtz-muenchen.de Aufsichtsratsvorsitzende: MinDir'in Baerbel Brumme-Bothe Geschaeftsfuehrer: Prof. Dr. Guenther Wess, Heinrich Bassler, Dr. Alfons Enhsen Registergericht: Amtsgericht Muenchen HRB 6466 USt-IdNr: DE 129521671

1 0

[libvirt-users] provisioning with vagrant-libvirt leaves .img file only readable by root
by Hans-Christoph Steiner 16 Feb '17

16 Feb '17

I'm using libvirt on Debian/stretch (testing) with vagrant and the vagrant-libvirt plugin. When I import a vagrant box (jessie64.box), the resulting file permssions let anyone in the "kvm" group read the img. But when I build upon that box, only root can read it: -rw------- 1 root root 20198785024 Sep 19 18:19 buildserver_default.img -rwxr--r-- 1 libvirt-qemu kvm 2148663296 Sep 5 22:55 jessie64_vagrant_box_image_0.img How can I control those file permissions as a regular user in the libvirtd group? I need to read that image in order to use qemu-img to rebase and create a new vagrant box. The current `vagrant package` command only works with VirtualBox VMs, but its easy to make a .box if you have read access to the libvirt QEMU .img file. /etc/libvirt/storage/default.xml says: <target> <path>/var/lib/libvirt/images</path> <permissions> <mode>0755</mode> </permissions> </target> You can find the context for this work here: https://gitlab.com/fdroid/fdroidserver/issues/238 .hc -- PGP fingerprint: EE66 20C7 136B 0D2C 456C 0A4D E9E2 8DEA 00AA 5556 https://pgp.mit.edu/pks/lookup?op=vindex&search=0xE9E28DEA00AA5556

2 1

[libvirt-users] high memory guest issues - virsh start and QEMU_JOB_WAIT_TIME
by Blair Bethwaite 15 Feb '17

15 Feb '17

Hi all, In IRC last night Dan helpfully confirmed my analysis of an issue we are seeing attempting to launch high memory KVM guests backed by hugepages... In this case the guests have 240GB of memory allocated from two host NUMA nodes to two guest NUMA nodes. The trouble is that allocating the hugepage backed qemu process seems to take longer than the 30s QEMU_JOB_WAIT_TIME and so libvirt then most unhelpfully kills the barely spawned guest. Dan said there was currently no workaround available so I'm now looking at building a custom libvirt which sets QEMU_JOB_WAIT_TIME=60s. I have two related questions: 1) will this change have any untoward side-effects? 2) if not, then is there any reason not to change it in master until a better solution comes along (or possibly better, alter qemuDomainObjBeginJobInternal to give a domain start job a little longer compared to other jobs)? -- Cheers, ~Blairo

3 7

[libvirt-users] Error in libvirt-GUI
by abhishek jain 15 Feb '17

15 Feb '17

Hi, I am using libvirt --> 0.10.2 and virt-manager --> 0.9.0 version (RHEL 6). Very rare I get following error message dialogue box. (Image is attached for the error dialogue box ) Error saysError polling connection:'qemu+ssh......' Internal error client socket is closed TraceBack (most recent calls)engine.py: 440 conn.tickconnection.py: 1433 self.hostinfo() = self.vmm.getinfo() Is it a problem with libvirt or with my environment Thanks in advance. RegardsAbhishek

2 3

[libvirt-users] Hyper-V support in Windows client
by masaeedu＠gmail.com 13 Feb '17

13 Feb '17

Hi there, I’m trying to connect to Hyper-V on Windows using libvirt. It seems neither the libvirt binaries bundled with Virt Viewer, nor the libvirt binary in the mingw64-libvirt Rawhide RPM have support for Hyper-V. Is it likely that just downloading the source RPM and trying to build –with-hyperv would just work? Thanks, Asad

1 0

[libvirt-users] How to achieve higher priority of a process in a lxc on CentOS machine
by WANG Cheng D 09 Feb '17

09 Feb '17

Dear all, I use libvirt lxc to host a real-time application with priority of -50. The native OS is fedora21 and lxc is also fedora21. Everything works fine. Linux "top" command shows the priority is -50). Then I use Openstack to manage lxc. The native compute node OS is CentOS7.2. and the lxc is fedora. After the real-time application is started in the lxc. top command on the native machine sees the real-time application with priority of 20, which is not what I expect. If lxc is Ubuntu, the priority of real-time application is also 20. Can anybody help me on this? Cheng Wang

1 0

[libvirt-users] Trouble moving OVMF guest to new host
by Bashe, Joe 09 Feb '17

09 Feb '17

Hello, I recently had to reinstall my operating system on my computer. I made a backup of the entire partition beforehand onto an external drive. Now I am trying to import a VM from that backup onto the newly installed system. What I've done so far: - copied over the qcow2 disk image - copied over the XML config file - copied over the OVMF files under /usr/share/edk2.git/ovmf-x64/ I am getting this error whenever I try to boot the VM: Error starting domain: operation failed: unable to find any master var store for loader: /usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd The file /usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd does exist. Where would the "master var store" be located? Thank you, Joe -- Joseph Bashe Technical Director Bashe Development +1 (323) 999-1731

4 4

[libvirt-users] accessing USB as storage device through lxc container.
by ravi mh 08 Feb '17

08 Feb '17

Hi all, I am not able to access USB as storage device in the lxc container. Having tried to pass the usb device with the product and vendor id, not able to see the device mounted in the lxc file system. However, they are seen as char devices in the container at the location. There is no issue of ACL, as the capabilities restrictions have been dropped. Has anyone successfully mounted the storage device in the libvirt. Having gone through the libvirt documentation, couldnt find further information of enabling the usb as storage device. Any pointers towards that would be useful. Host OS: IR800-GOS-1:~# lsusb Bus 001 Device 003: ID 8644:800b Bus 001 Device 001: ID 1d6b:0002 Bus 002 Device 001: ID 1d6b:0001 IR800-GOS-1:~# Lxc app container: root@ir800-lxc:/mnt/usb# ls -la /dev/bus/usb/001/003 *crwx-*----- 1 root root 189, 2 Feb 3 01:44 /dev/bus/usb/001/003 ----------libvirt xml snippet ------------ <hostdev mode='subsystem' type='usb' managed='yes'> <source> <vendor id='0x8644'/> <product id='0x800b'/> </source> </hostdev> ----------------------------- while changing the mode from subsystem to capabilities, it issues error for validating against the schema. ---------------while changing the hostdev mode='capabilities'---------- virsh # edit n01_1 *error: XML document failed to validate against schema: Unable to validate doc against /usr/share/libvirt/schemas/domain.rng* Extra element devices in interleave Element domain failed to validate content Failed. Try again? [y,n,i,f,?]: error: XML document failed to validate against schema: Unable to validate doc against /usr/share/libvirt/schemas/domain.rng -------------------------------------------- Regards, *Ravi*

2 3

[libvirt-users] Real time threads don't work in libvirt containers under CentOS 7.3
by Peter Steele 07 Feb '17

07 Feb '17

We've been using libvirt based containers under CentOS 7 and everything has been working fine. One application we run in our containers is ctdb, which uses SCHED_FIFO (real time) threads. This has been working without problems until our recent upgrade to CentOS 7.3. For some reason, ctdb is no longer able to create real time threads, and I've tried a simple program myself that confirms this. The same program works fine on the hypervisor so I know the kernel supports real time. Does anyone know what may have changed in CentOS 7.3 that breaks real time threads in libvirt containers? This is the simple test program I created to verify the real time threads are failing. This same program works in a libvirt container in CentOS 7.2. #include <stdio.h> #include <string.h> #include <pthread.h> pthread_t test_thread; void *test(void *arg) { printf("Starting thread\n"); sleep(1); printf("Thread complete\n"); return 0; } int main(int argc, char *argv[]) { int rc; printf("Starting main\n"); struct sched_param tsparam; pthread_attr_t tattr; memset(&tsparam, 0, sizeof(tsparam)); pthread_attr_init(&tattr); pthread_attr_setinheritsched(&tattr, PTHREAD_EXPLICIT_SCHED); pthread_attr_setschedpolicy(&tattr, SCHED_FIFO); tsparam.sched_priority = sched_get_priority_max(SCHED_FIFO) - 7; pthread_attr_setschedparam(&tattr, &tsparam); if ((rc = pthread_create(&test_thread, &tattr, test, NULL)) != 0) { printf("Unable to start rt thread\n"); } return 0; }

2 2

[libvirt-users] Communicating between libvirt contains on separate hosts
by Peter Steele 04 Feb '17

04 Feb '17

I have a group of CentOS 7.2 servers running as VMs under VMware's ESXi 6.0. These all reside on the same subnet and we have no problem communicating between the different virtual servers. In addition, each of these servers run a number of libvirt-based LXC containers, also based on CentOS 7.2. The hosts can communicate with their containers without issues and the containers on a given server can communicate with each other. However, containers hosted on two different servers cannot communicate with each other or with other servers. If we duplicate this setup using KVM based VMs instead of ESXi VMs everything works fine--there is no problem communicating between LXC containers regardless of which VM hosts them. I assume this is not a libvirt issue but if anyone has encountered this issue in running libvirt containers under a VMware environment and has found a solution, I'd appreciate hearing from you. Thanks very much, Peter

1 0