November 2017 - Users - libvirt List Archives

[libvirt-users] dramatic performance slowdown due to THP allocation failure with full pagecache
by Blair Bethwaite 15 Nov '17

15 Nov '17

Hi all, This is not really a libvirt issue but I'm hoping some of the smart folks here will know more about this problem... We have noticed when running some HPC applications on our OpenStack (libvirt+KVM) cloud that the same application occasionally performs much worse (4-5x slowdown) than normal. We can reproduce this quite easily by filling pagecache (i.e. dd-ing a single large file to /dev/null) before running the application. The problem seems to be that the kernel is not freeing (or has some trouble freeing) the non-dirty and (presumably immediately) reclaimable pagecache in order to allocate THPs for the application. This behaviour is also observable on regular bare-metal, but the slowdown is only 10-20% there - the nested paging of the guest really makes THP allocation important there (1). Both current CentOS and Ubuntu guests have the issue. Some more environmental context: the VMs are tuned (CPU pinning, NUMA topology, hugepage backing), so we normally see no difference between host and guest performance. After talking to many other colleagues in the HPC space it seems common that people setup their batch schedulers to put drop_caches between jobs, which can help to workaround this and other similar issues, but no-one has been able to explain why this is happening. Hopefully someone who knows about the guts of the Linux MM can shed some light...? (1) a related and possibly dumb question: in the case of a high-performance KVM where the guest is hugepage backed and pinned in host memory anyway, why do we still have a table based resolution for guest physical to host virtual address translation - couldn't this just be done by offset? -- Cheers, ~Blairo

2 5

[libvirt-users] Help：Can libvirt restore several xen snapshots more faster at same time?
by Chenjia (C) 13 Nov '17

13 Nov '17

Dear libvirt expert, we find a blocker issue in our project. Would you please offer us some feedback? Our project have a requirement: restore as much as xen snapshot at same time. we have done the following test(our environment is cpu E52620, 256G memory, SSD hard disk. The xen snapshot is Win7 OS with 1G memory): 1) Save 40 xen snapshots to disk, then restore these snapshot at sametime, we found the restore speed is too slow, and the bottleneck is the IO speed. 2) we save all the 40 xen snapshot to ramdisk, then we got a good performance, but the space is too big, about 40G, it is too expensive . Our question is : 1) each our xen snapshot is almost same except IP address(Win7 OS with 1G memory), does there have some way to reduce the 40 snapshots space? (we find the ‘virsh resotre’ cmd have an argument ‘�xml’ , but it seems not work in source code) 2) Do you have some suggestion to improve the performance of restore as much as xen snapshot at same time? Best regards, wish your reply.

1 0

[libvirt-users] KVM guest incorrect CPU frequency
by xmbbb 13 Nov '17

13 Nov '17

Hello, I am facing an issue where the detected CPU frequencies in KVM guests are incorrect. I am using the new Core i9 7900x with base clock 3.3GHz with "host-passthrough". Both Win10 and Linux guests incorrectly detect 3.45GHz for some reason. The workaround I have found is to temporarily switch to use HPET clocksource in the HOST before starting the VM. This makes the VMs correctly detect a 3.3GHz CPU. Then I can safely switch back to TSC in the VM HOST. This mostly does not impact much for day to day use. The only reproducible bug I can find with an incorrect CPU frequency is playing Bink videos that come with PC games. If I play any Bink video cut scene from the 2013 Tomb Raider game with the incorrect CPU frequency, the video plays a little slower than intended which results in audio cut out once in a while so that the video can "catch up" to the audio. Anyone having experiences with similar issues?

1 0

[libvirt-users] how to add mount options for root filesystem inside lxc container
by mxs kolo 13 Nov '17

13 Nov '17

Hi all We use libvirt 3.0.0 + lxc, disk for container described with <filesystem/> tag, example: <filesystem type='block' accessmode='passthrough'> <source dev='/dev/data/lvm-part1'/> <target dir='/'/> </filesystem> Now we start use ssd disk and think how to provide additional options for mount root FS in container: "discard" and "nobarrier" options for XFS. All level below XFS configured for support trim command ("options raid0 devices_discard_performance=Y" and "issue_discards = 1") I try set options inside container in /etc/fstab, it's not worked: /dev/root / rootfs discard,nobarrier,defaults 0 0 or /dev/root / rootfs discard,nobarrier 0 0 or /dev/root / rootfs remount,discard,nobarrier,defaults 0 0 or /dev/root / rootfs remount,discard,nobarrier 0 0 I found 2 solutions: 1) use 2 lvm partitions - one for base OS system, and second - for data. Second lvs attach as block device /dev/vdX inside container. Now we can mount /dev/vd? with desirable options inside container 2) more tricky variant - before container start, mount her root filesystem on hardware node with desirable options. Then after container started, her filesystem would be mounted with same options as on hardware node. But probably exists another way ? Has <filesystem/> support for mount options ? b.r. Maxim Kozin

1 0

[libvirt-users] using LVM thin pool LVs as a storage for libvirt guest
by Jan Hutař 07 Nov '17

07 Nov '17

Hello, as usual, I'm few years behind trends so I have learned about LVM thin volumes recently and I especially like that your volumes can be "sparse" - that you can have 1TB thin volume on 250GB VG/thin pool. Is it somehow possible to use that with libvirt? I have found this post from 2014: https://www.redhat.com/archives/libvirt-users/2014-August/msg00010.html which says you should be able to create "sparse" volume with `virsh vol-create ...` or that libvirt should be able to see thin volumes you create yourself, but neither of that works for me (libvirt-3.2.1-6.fc26.x86_64). This way I try to create new: # vgs storage VG #PV #LV #SN Attr VSize VFree storage 1 1 0 wz--n- 267.93g 0 # lvs storage -a LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert [lvol0_pmspare] storage ewi------- 68.00m lvol1 storage twi-aotz-- 267.80g 0.00 0.44 [lvol1_tdata] storage Twi-ao---- 267.80g [lvol1_tmeta] storage ewi-ao---- 68.00m # virsh pool-dumpxml storage <pool type='logical'> <name>storage</name> <uuid>f523aed2-a7e4-4dc2-88db-0193a7337704</uuid> <capacity unit='bytes'>287687311360</capacity> <allocation unit='bytes'>287687311360</allocation> <available unit='bytes'>0</available> <source> <device path='/dev/nvme0n1p3'/> <name>storage</name> <format type='lvm2'/> </source> <target> <path>/dev/storage</path> </target> </pool> # cat /tmp/big.xml <volume> <name>big</name> <capacity>1073741824</capacity> <allocation>1048576</allocation> <target> <path>/dev/storage/big</path> </target> </volume> # virsh vol-create storage /tmp/big.xml error: Failed to create vol from /tmp/big.xml error: internal error: Child process (/usr/sbin/lvcreate --name big -L 1024K --type snapshot --virtualsize 1048576K storage) unexpected exit status 5: Volume group "storage" has insufficient free space (0 extents): 1 required. When I create thin volume manually, I do not see it: # lvcreate -n big -V 500G --thinpool storage/lvol1 Using default stripesize 64.00 KiB. WARNING: Sum of all thin volume sizes (500.00 GiB) exceeds the size of thin pool storage/lvol1 and the size of whole volume group (267.93 GiB)! For thin pool auto extension activation/thin_pool_autoextend_threshold should be below 100. Logical volume "big" created. # lvs storage -a LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert big storage Vwi-a-tz-- 500.00g lvol1 0.00 [lvol0_pmspare] storage ewi------- 68.00m lvol1 storage twi-aotz-- 267.80g 0.00 0.45 [lvol1_tdata] storage Twi-ao---- 267.80g [lvol1_tmeta] storage ewi-ao---- 68.00m # virsh vol-list storage Name Path ------------------------------------------------------------------------------ Do I understand the concept incorrectly, or is there something else to configure? At the end I want to get max IO performance with possibility to "overcommit disk space". I know I can use storage "dir", but thought there might be something faster? Thank you very much for a response, Jan -- Jan Hutar Systems Management QA jhutar(a)redhat.com Red Hat, Inc.

3 4

[libvirt-users] this function is not supported by the connection driver: virInterfaceDefineXML
by netsurfed 07 Nov '17

07 Nov '17

Hi all: I'm facing problems trying to define a bridge on Ubuntu 16.04, using libvirt 3.4.0. When I try to do: root@ubuntu-180:/tmp# virsh iface-define iface-testbr1.xml I get the following error: error: Failed to define interface from iface-testbr1.xml error: this function is not supported by the connection driver: virInterfaceDefineXML What can i do to make this function works normally? THX. The xml file as: <interface type='bridge' name='testbr1'> <mtu size='1500'/> <bridge stp='on' delay='1500'> <interface type='ethernet' name='enp9s0'> <link speed='1000' state='up'/> <mac address='0c:c4:7a:c1:63:17'/> </interface> </bridge> </interface> root@ubuntu-180:/tmp# uname -a Linux ubuntu-180 4.4.0-62-generic #83-Ubuntu SMP Wed Jan 18 14:10:15 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

2 1

Re: [libvirt-users] Live Migration Issue
by Hans Knecht 07 Nov '17

07 Nov '17

Sorry, ctrl entered. I'm hoping someone here can help me figure out what's going on here. I've got two CentOS 7 hosts with shared storage that I'm trying to do live migrations between and I'm running into an error with VMs that were originally created on a CentOS 6 host and then moved the CentOS 7 hosts. The CentOS 6 VM runs fine on the host, and can do non-live migrations without any issues, however when attempting to live migrate the VM it fails. The VM itself is CentOS 7. Live migration does work when using VMs created using CentOS 7. Error message: "libvirtd virKeepAliveTimerInternal connection closed due to keep alive timeout. Cannot acquire state change lock (held by remoteDispatchDomainMigratePrepare3params)" Surrounding error messages here: Origin Host: https://pastebin.com/fLN6DRiq Recipient Host: https://pastebin.com/xR6mnaW2 The command being run is: Virsh --connect qemu:///system migrate --live vmname qemu+SSH://host2/system Firewall is turned off, SELinux is turned off, CentOS 7 CPUs are identical, both hosts fully updated. I've tried: Converting the VM from QCOW2 to Raw Converting the VM from QCOW2 to QCOW2 to try to get it to regenerate the XML/inject CentOS 7 Drivers Installed new drivers Updated all the packages in the VM. Removed all the Devices from the VM, except the HDDs. Hans Knecht - DevOps Network Administrator Kinsale Insurance Company I 2221 Edward Holland Drive I Suite 600 I Richmond, VA 23230 (804) 289-1300 x5102 office I (804) 673-5697 fax I <mailto:hans.knecht@kinsaleins.com> hans.knecht(a)kinsaleins.com I www.kinsaleins.com From: Hans Knecht [mailto:hans.knecht@kinsaleins.com] Sent: Monday, November 06, 2017 8:38 AM To: 'libvirt-users(a)redhat.com' <libvirt-users(a)redhat.com> Subject: Live Migration Issue I'm hoping someone here can help me figure out what's going on here. I've got two CentOS 7 hosts with shared storage that I'm trying to do live migrations between and I'm running into an error with VMs that were originally created on a CentOS 6 host and then moved the CentOS 7 hosts. The CentOS 6 VM runs fine on the host, and can do non-live migrations without any issues, however when attempting to live migrate the VM it fails with the following error message: Hans Knecht - DevOps Network Administrator Kinsale Insurance Company I 2221 Edward Holland Drive I Suite 600 I Richmond, VA 23230 (804) 289-1300 x5102 office I (804) 673-5697 fax I <mailto:hans.knecht@kinsaleins.com> hans.knecht(a)kinsaleins.com I www.kinsaleins.com <http://www.kinsaleins.com/> This communication constitutes an electronic communication within the meaning of the Electronic Communications Privacy Act, 19 USC 2510, and its disclosure is strictly limited to the recipient intended by the sender of this message. This communication may contain confidential and privileged material for the sole use of the intended recipient and the disclosure to anyone other than the intended recipient does not constitute a loss of the confidential or privileged nature of the communication. If you are not the intended recipient, please contact the sender by return electronic mail and delete all copies of this communication. Please check for any defects or viruses before opening or using any attachments. There is no warranty that this e-mail is virus free or without error. No Kinsale Company is liable if an attachment is altered without its written consent. Thank you.

2 1

[libvirt-users] Has libvirt guest pagecache level ?
by mxs kolo 06 Nov '17

06 Nov '17

Greetings Has libvirt dedicated page cache area for guest ? If not - what is the difference between cache='none' and cache='directsync' ? >The optional cache attribute controls the cache mechanism, possible >values are "default", "none", "writethrough", "writeback", "directsync" >(like "writethrough", but it bypasses the host page cache) and >"unsafe" (host may cache all disk io, and sync requests from guest >are ignored). As I understand 'unsafe' - always remove flags O_SYNC|O_DIRECT fron guest level. What about other cache options - how it work ? Would be glad see any schema or description, if You have. b.r. Maxim Kozin

1 0

[libvirt-users] Live Migration Issue
by Hans Knecht 06 Nov '17

06 Nov '17

I'm hoping someone here can help me figure out what's going on here. I've got two CentOS 7 hosts with shared storage that I'm trying to do live migrations between and I'm running into an error with VMs that were originally created on a CentOS 6 host and then moved the CentOS 7 hosts. The CentOS 6 VM runs fine on the host, and can do non-live migrations without any issues, however when attempting to live migrate the VM it fails with the following error message: Hans Knecht - DevOps Network Administrator Kinsale Insurance Company I 2221 Edward Holland Drive I Suite 600 I Richmond, VA 23230 (804) 289-1300 x5102 office I (804) 673-5697 fax I <mailto:hans.knecht@kinsaleins.com> hans.knecht(a)kinsaleins.com I www.kinsaleins.com <http://www.kinsaleins.com/> This communication constitutes an electronic communication within the meaning of the Electronic Communications Privacy Act, 19 USC 2510, and its disclosure is strictly limited to the recipient intended by the sender of this message. This communication may contain confidential and privileged material for the sole use of the intended recipient and the disclosure to anyone other than the intended recipient does not constitute a loss of the confidential or privileged nature of the communication. If you are not the intended recipient, please contact the sender by return electronic mail and delete all copies of this communication. Please check for any defects or viruses before opening or using any attachments. There is no warranty that this e-mail is virus free or without error. No Kinsale Company is liable if an attachment is altered without its written consent. Thank you.

1 0

[libvirt-users] What global_quota option mean? What's its logic?
by Volo M. 03 Nov '17

03 Nov '17

Hi techs, Could you please explain or refer me to documentation where I could find a description of global_quota schedinfo option. Someone told me that it could mean a cpu limitation in percents like: global_quota : 25000 (does it mean cpu limit % set to 25 ?) Is it true? What other parameters mean in below output? I didn't find any related documentation at libvirt.org. Could you help me to describe me or refer to some description what below schedinfo options mean? Thanks. [root@cpkvm ~]# virsh list Id Name State ---------------------------------------------------- 3 kvmtestvm running [root@cpkvm ~]# virsh schedinfo 3 Scheduler : posix cpu_shares : 154 vcpu_period : 100000 vcpu_quota : -1 emulator_period: 100000 emulator_quota : -1 global_period : 100000 global_quota : -1 iothread_period: 100000 iothread_quota : -1 [root@cpkvm ~]# virsh schedinfo 3 --set global_quota=25 Scheduler : posix error: invalid argument: value of 'global_quota' is out of range [1000, 18446744073709551] [1000, 18446744073709551] - what this range mean? [root@cpkvm ~]# [root@cpkvm ~]# virsh schedinfo 3 --set global_quota=25000 Scheduler : posix cpu_shares : 154 vcpu_period : 100000 vcpu_quota : 12500 emulator_period: 100000 emulator_quota : -1 global_period : 100000 global_quota : 25000 iothread_period: 100000 iothread_quota : 25000 [root@cpkvm ~]#

3 2