[libvirt-users] dramatic performance slowdown due to THP allocation failure with full pagecache
by Blair Bethwaite
Hi all,
This is not really a libvirt issue but I'm hoping some of the smart folks
here will know more about this problem...
We have noticed when running some HPC applications on our OpenStack
(libvirt+KVM) cloud that the same application occasionally performs much
worse (4-5x slowdown) than normal. We can reproduce this quite easily by
filling pagecache (i.e. dd-ing a single large file to /dev/null) before
running the application. The problem seems to be that the kernel is not
freeing (or has some trouble freeing) the non-dirty and (presumably
immediately) reclaimable pagecache in order to allocate THPs for the
application.
This behaviour is also observable on regular bare-metal, but the slowdown
is only 10-20% there - the nested paging of the guest really makes THP
allocation important there (1). Both current CentOS and Ubuntu guests have
the issue. Some more environmental context: the VMs are tuned (CPU pinning,
NUMA topology, hugepage backing), so we normally see no difference between
host and guest performance.
After talking to many other colleagues in the HPC space it seems common
that people setup their batch schedulers to put drop_caches between jobs,
which can help to workaround this and other similar issues, but no-one has
been able to explain why this is happening. Hopefully someone who knows
about the guts of the Linux MM can shed some light...?
(1) a related and possibly dumb question: in the case of a high-performance
KVM where the guest is hugepage backed and pinned in host memory anyway,
why do we still have a table based resolution for guest physical to host
virtual address translation - couldn't this just be done by offset?
--
Cheers,
~Blairo
6 years, 8 months
[libvirt-users] Help：Can libvirt restore several xen snapshots more faster at same time?
by Chenjia (C)
Dear libvirt expert,
we find a blocker issue in our project. Would you please offer us some feedback?
Our project have a requirement: restore as much as xen snapshot at same time. we have done the following test(our environment is cpu E52620, 256G memory, SSD hard disk. The xen snapshot is Win7 OS with 1G memory):
1) Save 40 xen snapshots to disk, then restore these snapshot at sametime, we found the restore speed is too slow, and the bottleneck is the IO speed.
2) we save all the 40 xen snapshot to ramdisk, then we got a good performance, but the space is too big, about 40G, it is too expensive .
Our question is :
1) each our xen snapshot is almost same except IP address(Win7 OS with 1G memory), does there have some way to reduce the 40 snapshots space? (we find the ‘virsh resotre’ cmd have an argument ‘�xml’ , but it seems not work in source code)
2) Do you have some suggestion to improve the performance of restore as much as xen snapshot at same time?
Best regards, wish your reply.
6 years, 8 months
[libvirt-users] KVM guest incorrect CPU frequency
by xmbbb
Hello,
I am facing an issue where the detected CPU frequencies in KVM guests are incorrect.
I am using the new Core i9 7900x with base clock 3.3GHz with "host-passthrough". Both Win10 and Linux guests incorrectly detect 3.45GHz for some reason.
The workaround I have found is to temporarily switch to use HPET clocksource in the HOST before starting the VM. This makes the VMs correctly detect a 3.3GHz CPU. Then I can safely switch back to TSC in the VM HOST.
This mostly does not impact much for day to day use. The only reproducible bug I can find with an incorrect CPU frequency is playing Bink videos that come with PC games. If I play any Bink video cut scene from the 2013 Tomb Raider game with the incorrect CPU frequency, the video plays a little slower than intended which results in audio cut out once in a while so that the video can "catch up" to the audio.
Anyone having experiences with similar issues?
6 years, 8 months
[libvirt-users] how to add mount options for root filesystem inside lxc container
by mxs kolo
Hi all
We use libvirt 3.0.0 + lxc, disk for container described with
<filesystem/> tag, example:
<filesystem type='block' accessmode='passthrough'>
<source dev='/dev/data/lvm-part1'/>
<target dir='/'/>
</filesystem>
Now we start use ssd disk and think how to provide additional options
for mount root FS in container: "discard" and "nobarrier" options for
XFS.
All level below XFS configured for support trim command ("options
raid0 devices_discard_performance=Y" and "issue_discards = 1")
I try set options inside container in /etc/fstab, it's not worked:
/dev/root / rootfs discard,nobarrier,defaults 0 0
or
/dev/root / rootfs discard,nobarrier 0 0
or
/dev/root / rootfs remount,discard,nobarrier,defaults
0 0
or
/dev/root / rootfs remount,discard,nobarrier 0 0
I found 2 solutions:
1) use 2 lvm partitions - one for base OS system, and second - for
data. Second lvs attach as block device /dev/vdX inside container.
Now we can mount /dev/vd? with desirable options inside container
2) more tricky variant - before container start, mount her root
filesystem on hardware node with desirable options. Then after
container started, her filesystem would be mounted with same options
as on hardware node.
But probably exists another way ? Has <filesystem/> support for mount options ?
b.r.
Maxim Kozin
6 years, 8 months
[libvirt-users] using LVM thin pool LVs as a storage for libvirt guest
by Jan Hutař
Hello,
as usual, I'm few years behind trends so I have learned about LVM thin
volumes recently and I especially like that your volumes can be "sparse"
- that you can have 1TB thin volume on 250GB VG/thin pool.
Is it somehow possible to use that with libvirt?
I have found this post from 2014:
https://www.redhat.com/archives/libvirt-users/2014-August/msg00010.html
which says you should be able to create "sparse" volume with `virsh
vol-create ...` or that libvirt should be able to see thin volumes you
create yourself, but neither of that works for me
(libvirt-3.2.1-6.fc26.x86_64).
This way I try to create new:
# vgs storage
VG #PV #LV #SN Attr VSize VFree
storage 1 1 0 wz--n- 267.93g 0
# lvs storage -a
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
[lvol0_pmspare] storage ewi------- 68.00m
lvol1 storage twi-aotz-- 267.80g 0.00 0.44
[lvol1_tdata] storage Twi-ao---- 267.80g
[lvol1_tmeta] storage ewi-ao---- 68.00m
# virsh pool-dumpxml storage
<pool type='logical'>
<name>storage</name>
<uuid>f523aed2-a7e4-4dc2-88db-0193a7337704</uuid>
<capacity unit='bytes'>287687311360</capacity>
<allocation unit='bytes'>287687311360</allocation>
<available unit='bytes'>0</available>
<source>
<device path='/dev/nvme0n1p3'/>
<name>storage</name>
<format type='lvm2'/>
</source>
<target>
<path>/dev/storage</path>
</target>
</pool>
# cat /tmp/big.xml
<volume>
<name>big</name>
<capacity>1073741824</capacity>
<allocation>1048576</allocation>
<target>
<path>/dev/storage/big</path>
</target>
</volume>
# virsh vol-create storage /tmp/big.xml
error: Failed to create vol from /tmp/big.xml
error: internal error: Child process (/usr/sbin/lvcreate --name big -L 1024K --type snapshot --virtualsize 1048576K storage) unexpected exit status 5: Volume group "storage" has insufficient free space (0 extents): 1 required.
When I create thin volume manually, I do not see it:
# lvcreate -n big -V 500G --thinpool storage/lvol1
Using default stripesize 64.00 KiB.
WARNING: Sum of all thin volume sizes (500.00 GiB) exceeds the size of thin pool storage/lvol1 and the size of whole volume group (267.93 GiB)!
For thin pool auto extension activation/thin_pool_autoextend_threshold should be below 100.
Logical volume "big" created.
# lvs storage -a
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
big storage Vwi-a-tz-- 500.00g lvol1 0.00
[lvol0_pmspare] storage ewi------- 68.00m
lvol1 storage twi-aotz-- 267.80g 0.00 0.45
[lvol1_tdata] storage Twi-ao---- 267.80g
[lvol1_tmeta] storage ewi-ao---- 68.00m
# virsh vol-list storage
Name Path
------------------------------------------------------------------------------
Do I understand the concept incorrectly, or is there something else to
configure?
At the end I want to get max IO performance with possibility to
"overcommit disk space". I know I can use storage "dir", but thought
there might be something faster?
Thank you very much for a response,
Jan
--
Jan Hutar Systems Management QA
jhutar(a)redhat.com Red Hat, Inc.
6 years, 8 months
[libvirt-users] this function is not supported by the connection driver: virInterfaceDefineXML
by netsurfed
Hi all:
I'm facing problems trying to define a bridge on Ubuntu 16.04, using libvirt 3.4.0.
When I try to do:
root@ubuntu-180:/tmp# virsh iface-define iface-testbr1.xml
I get the following error:
error: Failed to define interface from iface-testbr1.xml
error: this function is not supported by the connection driver: virInterfaceDefineXML
What can i do to make this function works normally? THX.
The xml file as:
<interface type='bridge' name='testbr1'>
<mtu size='1500'/>
<bridge stp='on' delay='1500'>
<interface type='ethernet' name='enp9s0'>
<link speed='1000' state='up'/>
<mac address='0c:c4:7a:c1:63:17'/>
</interface>
</bridge>
</interface>
root@ubuntu-180:/tmp# uname -a
Linux ubuntu-180 4.4.0-62-generic #83-Ubuntu SMP Wed Jan 18 14:10:15 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
6 years, 8 months
Re: [libvirt-users] Live Migration Issue
by Hans Knecht
Sorry, ctrl entered.
I'm hoping someone here can help me figure out what's going on here.
I've got two CentOS 7 hosts with shared storage that I'm trying to do live
migrations between and I'm running into an error with VMs that were
originally created on a CentOS 6 host and then moved the CentOS 7 hosts.
The CentOS 6 VM runs fine on the host, and can do non-live migrations
without any issues, however when attempting to live migrate the VM it
fails. The VM itself is CentOS 7. Live migration does work when using VMs
created using CentOS 7.
Error message:
"libvirtd virKeepAliveTimerInternal connection closed due to keep alive
timeout.
Cannot acquire state change lock (held by
remoteDispatchDomainMigratePrepare3params)"
Surrounding error messages here:
Origin Host: https://pastebin.com/fLN6DRiq
Recipient Host: https://pastebin.com/xR6mnaW2
The command being run is:
Virsh --connect qemu:///system migrate --live vmname
qemu+SSH://host2/system
Firewall is turned off, SELinux is turned off, CentOS 7 CPUs are
identical, both hosts fully updated.
I've tried:
Converting the VM from QCOW2 to Raw
Converting the VM from QCOW2 to QCOW2 to try to get it to regenerate the
XML/inject CentOS 7 Drivers
Installed new drivers
Updated all the packages in the VM.
Removed all the Devices from the VM, except the HDDs.
Hans Knecht - DevOps Network Administrator
Kinsale Insurance Company I 2221 Edward Holland Drive I Suite 600 I
Richmond, VA 23230
(804) 289-1300 x5102 office I (804) 673-5697 fax I
<mailto:hans.knecht@kinsaleins.com> hans.knecht(a)kinsaleins.com I
www.kinsaleins.com
From: Hans Knecht [mailto:hans.knecht@kinsaleins.com]
Sent: Monday, November 06, 2017 8:38 AM
To: 'libvirt-users(a)redhat.com' <libvirt-users(a)redhat.com>
Subject: Live Migration Issue
I'm hoping someone here can help me figure out what's going on here.
I've got two CentOS 7 hosts with shared storage that I'm trying to do live
migrations between and I'm running into an error with VMs that were
originally created on a CentOS 6 host and then moved the CentOS 7 hosts.
The CentOS 6 VM runs fine on the host, and can do non-live migrations
without any issues, however when attempting to live migrate the VM it
fails with the following error message:
Hans Knecht - DevOps Network Administrator
Kinsale Insurance Company I 2221 Edward Holland Drive I Suite 600 I
Richmond, VA 23230
(804) 289-1300 x5102 office I (804) 673-5697 fax I
<mailto:hans.knecht@kinsaleins.com> hans.knecht(a)kinsaleins.com I
www.kinsaleins.com
<http://www.kinsaleins.com/>
This communication constitutes an electronic communication within the
meaning of the Electronic Communications Privacy Act, 19 USC 2510, and its
disclosure is strictly limited to the recipient intended by the sender of
this message. This communication may contain confidential and privileged
material for the sole use of the intended recipient and the disclosure to
anyone other than the intended recipient does not constitute a loss of the
confidential or privileged nature of the communication. If you are not the
intended recipient, please contact the sender by return electronic mail
and delete all copies of this communication. Please check for any defects
or viruses before opening or using any attachments. There is no warranty
that this e-mail is virus free or without error. No Kinsale Company is
liable if an attachment is altered without its written consent. Thank you.
6 years, 8 months
[libvirt-users] Has libvirt guest pagecache level ?
by mxs kolo
Greetings
Has libvirt dedicated page cache area for guest ?
If not - what is the difference between
cache='none' and cache='directsync' ?
>The optional cache attribute controls the cache mechanism, possible >values are "default", "none", "writethrough", "writeback", "directsync" >(like "writethrough", but it bypasses the host page cache) and >"unsafe" (host may cache all disk io, and sync requests from guest >are ignored).
As I understand 'unsafe' - always remove flags O_SYNC|O_DIRECT fron
guest level.
What about other cache options - how it work ?
Would be glad see any schema or description, if You have.
b.r.
Maxim Kozin
6 years, 8 months
[libvirt-users] Live Migration Issue
by Hans Knecht
I'm hoping someone here can help me figure out what's going on here.
I've got two CentOS 7 hosts with shared storage that I'm trying to do live
migrations between and I'm running into an error with VMs that were
originally created on a CentOS 6 host and then moved the CentOS 7 hosts.
The CentOS 6 VM runs fine on the host, and can do non-live migrations
without any issues, however when attempting to live migrate the VM it
fails with the following error message:
Hans Knecht - DevOps Network Administrator
Kinsale Insurance Company I 2221 Edward Holland Drive I Suite 600 I
Richmond, VA 23230
(804) 289-1300 x5102 office I (804) 673-5697 fax I
<mailto:hans.knecht@kinsaleins.com> hans.knecht(a)kinsaleins.com I
www.kinsaleins.com
<http://www.kinsaleins.com/>
This communication constitutes an electronic communication within the
meaning of the Electronic Communications Privacy Act, 19 USC 2510, and its
disclosure is strictly limited to the recipient intended by the sender of
this message. This communication may contain confidential and privileged
material for the sole use of the intended recipient and the disclosure to
anyone other than the intended recipient does not constitute a loss of the
confidential or privileged nature of the communication. If you are not the
intended recipient, please contact the sender by return electronic mail
and delete all copies of this communication. Please check for any defects
or viruses before opening or using any attachments. There is no warranty
that this e-mail is virus free or without error. No Kinsale Company is
liable if an attachment is altered without its written consent. Thank you.
6 years, 8 months
[libvirt-users] What global_quota option mean? What's its logic?
by Volo M.
Hi techs,
Could you please explain or refer me to documentation where I could find a
description of global_quota schedinfo option. Someone told me that it could
mean a cpu limitation in percents like:
global_quota : 25000 (does it mean cpu limit % set to 25 ?)
Is it true?
What other parameters mean in below output?
I didn't find any related documentation at libvirt.org.
Could you help me to describe me or refer to some description what below
schedinfo options mean? Thanks.
[root@cpkvm ~]# virsh list
Id Name State
----------------------------------------------------
3 kvmtestvm running
[root@cpkvm ~]# virsh schedinfo 3
Scheduler : posix
cpu_shares : 154
vcpu_period : 100000
vcpu_quota : -1
emulator_period: 100000
emulator_quota : -1
global_period : 100000
global_quota : -1
iothread_period: 100000
iothread_quota : -1
[root@cpkvm ~]# virsh schedinfo 3 --set global_quota=25
Scheduler : posix
error: invalid argument: value of 'global_quota' is out of range [1000,
18446744073709551]
[1000, 18446744073709551] - what this range mean?
[root@cpkvm ~]#
[root@cpkvm ~]# virsh schedinfo 3 --set global_quota=25000
Scheduler : posix
cpu_shares : 154
vcpu_period : 100000
vcpu_quota : 12500
emulator_period: 100000
emulator_quota : -1
global_period : 100000
global_quota : 25000
iothread_period: 100000
iothread_quota : 25000
[root@cpkvm ~]#
6 years, 8 months
