Re: [libvirt-users] VM I/O performance drops dramatically during storage migration with drive-mirror
by Kashyap Chamarthy
Cc the QEMU Block Layer mailing list (qemu-block(a)nongnu.org), who might
have more insights here; and wrap long lines.
On Mon, May 28, 2018 at 06:07:51PM +0800, Chunguang Li wrote:
> Hi, everyone.
>
> Recently I am doing some tests on the VM storage+memory migration with
> KVM/QEMU/libvirt. I use the following migrate command through virsh:
> "virsh migrate --live --copy-storage-all --verbose vm1
> qemu+ssh://192.168.1.91/system tcp://192.168.1.91". I have checked the
> libvirt debug output, and make sure that the drive-mirror + NBD
> migration method is used.
>
> Inside the VM, I use an I/O benchmark (Iometer) to generate an oltp
> workload. I record the I/O performance (IOPS) before/during/after
> migration. When the migration begins, the IOPS dropped by 30%-40%.
> This is reasonable, because the migration I/O competes with the
> workload I/O. However, during almost the last period of migration
> (which is 66s in my case), the IOPS dropped dramatically, from about
> 170 to less than 10. I also show the figure of this experiment in the
> attachment of this email.
[The attachment should arrive on the 'libvirt-users' list archives; but
it's not there yet --
https://www.redhat.com/archives/libvirt-users/2018-May/thread.html]
> I want to figure out what results in this period with very low IOPS.
> First, I added some printf()s in the QEMU code, and knew that, this
> period occurs just before the memory migration phase. (BTW, the memory
> migration is very fast, which is just about 5s.) So I think this
> period should be the last phase of the "drive-mirror" process of QEMU.
> So then I tried to read the code of "drive-mirror" in QEMU, but failed
> to understand it very well.
>
> Does anybody know what may lead to this period with very low IOPS?
> Thank you very much.
>
> Some details of this experiment: The VM disk image file is 30GB
> (format = raw,cache=none,aio=native), and Iometer operates on an 10GB
> file inside the VM. The oltp workload consists of 33% writes and 67%
> reads (8KB request size, all random). The VM memory size is 4GB, most
> of which should be zero pages, so the memory migration is very fast.
>
> --
> Chunguang Li, Ph.D. Candidate
> Wuhan National Laboratory for Optoelectronics (WNLO)
> Huazhong University of Science & Technology (HUST)
> Wuhan, Hubei Prov., China
--
/kashyap
6 years, 5 months
[libvirt-users] Using libvirt to access Ceph RBDs with Xen
by thg
Hi everybody,
my background: I'm doing Xen since 10++ years, many years with DRBD for
high availability, since some time I'm using preferable GlusterFS with
FUSE as replicated storage, where I place the image-files for the vms.
In my current project we started (successfully) with Xen/GlusterFS too,
but because the provider, where we placed the servers, uses widely CEPH,
we decided to switch too.
Unfortunately I'm new to CEPH, but with help of a technician, we have
running a 3 node CEPH-cluster now, that seems to work fine.
Hardware:
- Xeons, 24 Cores, 256 GB RAM,
2x 240 GB system-SSDs RAID1, 4x 1,92 TB data-SSDs (no RAID)
Software we are using:
- CentOS 7.5.1804
- Kernel: 4.9.86-30.el7 @centos-virt-xen-48
- Xen: 4.8.3-5.el7 @centos-virt-xen-48
- libvirt-xen: 4.1.0-2.xen48.el7 @centos-virt-xen-48
- Ceph: 2:12.2.5-0.el7 @Ceph
CEPH itself works, Xen too and even libvirt (I'm quite new in this too,
normally I use the XL-tools to manage my VMs) ;-) I can manually map an
RBD and start a PV-VM via libvirt, after modifying the XML-config and
defined the VM with libvirt.
That is what happens:
# virsh define xml/testvm3.xml
Domain testvm3 defined from xml/testvm3.xml
# virsh start --console testvm3
error: Failed to start domain testvm3
error: internal error: libxenlight failed to create new domain 'testvm3'
xml/testvm3.xml:
-> ...
<devices>
<disk type='network' device='disk'>
<source protocol='rbd' name='devel-pool/testvm3.rbd'>
<host name="10.20.30.1" port="6789"/>
<host name="10.20.30.2" port="6789"/>
<host name="10.20.30.3" port="6789"/>
</source>
<auth username='libvirt'>
<secret type='ceph' uuid='07f3a0fe-9178-xxxxxxxxx'/>
</auth>
<target dev='xvda' bus='xen'/>
</disk>
...
Logs:
/var/log/libvirt/libxl/libxl-driver.log:
-> ...
2018-05-27 11:24:36.666+0000: libxl:
libxl_bootloader.c:634:bootloader_finished: bootloader failed - consult
logfile /var/log/xen/bootloader.14.log
2018-05-27 11:24:36.666+0000: libxl:
libxl_exec.c:118:libxl_report_child_exitstatus: bootloader [183216]
exited with error status 1
2018-05-27 11:24:36.667+0000: libxl:
libxl_create.c:1259:domcreate_rebuild_done: cannot (re-)build domain: -3
/var/log/xen/bootloader.14.log:
->
Traceback (most recent call last):
File "/usr/lib64/xen/bin/pygrub", line 896, in <module>
part_offs = get_partition_offsets(file)
File "/usr/lib64/xen/bin/pygrub", line 113, in get_partition_offsets
image_type = identify_disk_image(file)
File "/usr/lib64/xen/bin/pygrub", line 56, in identify_disk_image
fd = os.open(file, os.O_RDONLY)
OSError: [Errno 2] No such file or directory:
'rbd:devel-pool/testvm3.rbd:id=libvirt:key=AQBThwFbGFRYFxxxxxxxxxxxxxxx==:auth_supported=cephx\\;none:mon_host=10.20.30.1\\:6789\\;10.20.30.2\\:6789\\;10.20.30.3\\:6789'
So the Xen-Python-script, that searches for the kernel/bootloader of the
VM, does not find the imagefile. But if I grab the above output and use
it with qemu, it does find the blockdevice:
# qemu-img info
"rbd:devel-pool/testvm3.rbd:id=libvirt:key=AQBThwFbGFRYFxxxxxxxxxxxxxxx==:auth_supported=cephx\\;none:mon_host=10.20.30.1\\:6789\\;10.20.30.2\\:6789\\;10.20.30.3\\:6789"
->
image:
rbd:devel-pool/testvm3.rbd:id=libvirt:key=AQBThwFbGFRYFxxxxxxxxxxxxxxx==:auth_supported=cephx\;none:mon_host=10.20.30.1\:6789\;10.20.30.2\:6789\;10.20.30.3\:6789
file format: raw
virtual size: 16G (17179869184 bytes)
disk size: unavailable
Thus it seems, that CEPH-auth works and the VM-XML-config is interpreted
correct by libvirt. But the next step fails, when libvirt should pass
the blockdevice to Xen?
Anybody out there who uses a similar configuration? Any hints how to
debug this? Unfortunately I do not have anymore logs to look at.
Thank you very much,
--
Kind regards,
thg
6 years, 5 months
[libvirt-users] virsh migration job unexpectedly failed
by cokabug
Hi ,
when i have run virsh migrate command , the migration progress fiailed
at [61 %].
command :
virsh migrate --live --undefinesource --p2p --copy-storage-inc
--verbose --persistent instance-00001959 qemu+tcp://host292/system
result :
Migration: [ 61 %]error: operation failed: migration job: unexpectedly
failed
os/kernel version: centos 6.7 / kernel 2.6.32
qemu-kvm version : 0.12.1.2
libvirt version : 0.10.2-54
the libvirt.log on target host:
2018-05-22 11:45:38.603+0000: 32314: warning : qemuDomainObjTaint:1459
: Domain id=30 name='instance-00001959'
uuid=41975d30-2490-473e-be7b-9d2d08bf66d2 is tainted: high-privileges
2018-05-22 11:45:38.883+0000: 32314: warning :
qemuDomainObjEnterMonitorInternal:1062 : This thread seems to be the async
job owner; entering monitor without asking for a nested job is dangerous
2018-05-22 11:45:38.902+0000: 32314: warning :
qemuDomainObjEnterMonitorInternal:1062 : This thread seems to be the async
job owner; entering monitor without asking for a nested job is dangerous
2018-05-22 11:45:38.903+0000: 32314: warning :
qemuDomainObjEnterMonitorInternal:1062 : This thread seems to be the async
job owner; entering monitor without asking for a nested job is dangerous
2018-05-22 11:45:39.090+0000: 32314: warning :
qemuDomainObjEnterMonitorInternal:1062 : This thread seems to be the async
job owner; entering monitor without asking for a nested job is dangerous
2018-05-22 11:45:39.090+0000: 32314: warning :
qemuDomainObjEnterMonitorInternal:1062 : This thread seems to be the async
job owner; entering monitor without asking for a nested job is dangerous
2018-05-22 11:45:39.091+0000: 32314: warning :
qemuDomainObjEnterMonitorInternal:1062 : This thread seems to be the async
job owner; entering monitor without asking for a nested job is dangerous
2018-05-22 11:45:41.841+0000: 32310: error : qemuMonitorIO:615 : internal
error End of file from monitor
the qmeu-kvminstance-00001959.log:
2018-05-22 11:45:38.603+0000: starting up
LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin HOME=/home/ning.zhao USER=root
LOGNAME=root QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -name
instance-00001959 -S -M rhel6.6.0 -cpu
Broadwell,+abm,+pdpe1gb,+rdrand,+f16c,+osxsave,+dca,+pdcm,+xtpr,+tm2,+est,+smx,+vmx,+ds_cpl,+m$
nitor,+dtes64,+pbe,+tm,+ht,+ss,+acpi,+ds,+vme -enable-kvm -m 4096 -realtime
mlock=off -smp 4,sockets=4,cores=1,threads=1 -uuid
41975d30-2490-473e-be7b-9d2d08bf66d2 -smbios type=1,manufacturer=Red Hat
Inc.,product=OpenStack Nova,version=2013.1.2-4.el6,serial=fcbe6c27-c9$
d-e811-ac36-a281f9ebc503,uuid=41975d30-2490-473e-be7b-9d2d08bf66d2
-nodefconfig -nodefaults -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-00001959.monitor,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc
base=utc,driftfix=slew -n$
-kvm-pit-reinjection -no-shutdown -device
piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive
file=/var/lib/nova/instances/41975d30-2490-473e-be7b-9d2d08bf66d2/disk,if=none,id=drive-virtio-disk0,format=qcow2,cache=none
-device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4$
drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev
tap,fd=28,id=hostnet0,vhost=on,vhostfd=29 -device
virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:c0:8f:6b,bus=pci.0,addr=0x3
-chardev
file,id=charserial0,path=/var/lib/nova/instances/41975d30-2490-473e-b$
7b-9d2d08bf66d2/console.log -device
isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1
-device isa-serial,chardev=charserial1,id=serial1 -device
usb-tablet,id=input0 -vnc 127.0.0.1:2 -k en-us -vga cirrus -incoming tcp:
0.0.0.0:49152 -device virtio-bal$
oon-pci,id=balloon0,bus=pci.0,addr=0x5 -msg timestamp=on
Domain id=30 is tainted: high-privileges
char device redirected to /dev/pts/2
Receiving block device images
qemu: warning: error while loading state for instance 0x0 of device 'block'
load of migration failed
2018-05-22 11:45:41.841+0000: shutting down
i checked the log and disk file and they have right permissions, has
anyone know about this ,i have google many times ,but i cant find the right
debug way to resolve it.
thanks
6 years, 6 months
[libvirt-users] Create qcow2 v3 volumes via libvirt
by Gionatan Danti
Hi all,
on a fully patched CentOS 7.4 x86-64, I see the following behavior:
- when creating a new volumes using vol-create-as, the resulting file is
a qcow2 version 2 (compat=0.10) file. Example:
[root@gdanti-lenovo vmimages]# virsh vol-create-as default zzz.qcow2
8589934592 --format=qcow2 --backing-vol /mnt/vmimages/centos6.img
Vol zzz.qcow2 created
[root@gdanti-lenovo vmimages]# file zzz.qcow2
zzz.qcow2: QEMU QCOW Image (v2), has backing file (path
/mnt/vmimages/centos6.img), 8589934592 bytes
[root@gdanti-lenovo vmimages]# qemu-img info zzz.qcow2
image: zzz.qcow2
file format: qcow2
virtual size: 8.0G (8589934592 bytes)
disk size: 196K
cluster_size: 65536
backing file: /mnt/vmimages/centos6.img
backing file format: raw
Format specific information:
compat: 0.10
refcount bits: 16
- when creating a snapshot, the resulting file is a qcow2 version 3
(comapt=1.1) file. Example:
[root@gdanti-lenovo vmimages]# virsh snapshot-create-as centos6left
--disk-only --no-metadata snap.qcow2
Domain snapshot snap.qcow2 created
[root@gdanti-lenovo vmimages]# file centos6left.snap.qcow2
centos6left.snap.qcow2: QEMU QCOW Image (v3), has backing file (path
/mnt/vmimages/centos6left.qcow2), 8589934592 bytes
[root@gdanti-lenovo vmimages]# qemu-img info centos6left.snap.qcow2
image: centos6left.snap.qcow2
file format: qcow2
virtual size: 8.0G (8589934592 bytes)
disk size: 196K
cluster_size: 65536
backing file: /mnt/vmimages/centos6left.qcow2
backing file format: qcow2
Format specific information:
compat: 1.1
lazy refcounts: false
refcount bits: 16
corrupt: false
From what I know, this is a deliberate decision: compat=1.1 requires
relatively recent qemu version, and creating a new volume play on the
"safe side" of compatibility.
It is possible to create a new volume using qcow2 version 3 (compat=1.1)
format *using libvirt/virsh* (I know I can do that via qemu-img)? Any
drawback on using version 3 format?
Thanks.
--
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: g.danti(a)assyoma.it - info(a)assyoma.it
GPG public key ID: FF5F32A8
6 years, 6 months
[libvirt-users] Unable to connect VMs to a bridge over bonded network on Debian 9 (works fine on Centos 7.4)
by Thing
Hi,
Sorry of this is the wrong list, trying to find some help.
I am trying to set up 2 KVM hosts using Debian 9 (connecting to a 3 node
Centos 7.4 gluster cluster) To get the bandwidth I have bonded 2 NICS each
end. However while the network works fine for the Debian VM host the VM
guests cannot connect/get an IP using DHCP connecting to br0. (Same setup
on the Centos end works fine however)
On the Debian host if I connect to br1 which is a bridge to a NIC port it
works fine. My problem then seems to be at the bonding layer for
libvirt/kvm ?
Here is my interfaces config, I cannot see the problem. The VM guest is on
the gluster cluster and boots fine just no networking.
any ideas what I am doing wrong pls? or a better group/list to go ask in?
auto eno1
iface eno1 inet manual
#iface eno1 inet static
bond-master bond0
#address 192.168.1.62
#netmask 255.255.255.0
#gateway 192.168.1.1
#dns-nameservers 192.168.1.241 192.168.1.104
# mtu 9000
auto enp1s0f1
iface enp1s0f1 inet manual
bond-master bond0
# mtu 9000
auto bond0
iface bond0 inet static
bond-miimon 100
bond-mode 6
bond-updelay 200
bond-xmit_hash_policy layer3+4
bond-lacp-rate 1
# mtu 9000
# address 192.168.1.62
# netmask 255.255.255.0
# gateway 192.168.1.1
# dns-nameservers 192.168.1.241 192.168.1.104
slaves eno1 enp1s0f1
auto br0
iface br0 inet static
address 192.168.1.62
netmask 255.255.255.0
gateway 192.168.1.1
dns-nameservers 192.168.1.241 192.168.1.104
bridge_ports bond0
#bridge_ports eno1 enp1s0f1
bridge_stp on
bridge_fd 0.0
# mtu 9000
#post-up ifconfig bond0 mtu 9000
#post-up ip link set dev bond0 mtu 9000
#auto bond1
#iface bond1 inet static
# address 10.100.200.62
# netmask 255.255.255.0
# bond_slaves enp3s0f1
# bond_primary enp3s0f1
# mtu 9000
auto br1
iface br1 inet static
address 10.100.200.62
netmask 255.255.255.0
gateway 10.100.200.1
bridge_ports enp3s0f1
bridge_stp on
bridge_fd 0.0
root@kvm02:/etc/network#
6 years, 6 months
[libvirt-users] Attempting to use Secure Boot for Windows 10
by frizop f
Anybody have a guide on using secure boot?
I managed to set */usr/share/edk2/ovmf/OVMF_CODE.secboot.fd* as the*
<loader readonly='yes' type='pflash'>*, but when I change this windows only
shows me a black screen.
I've also tried to fiddle with the following two settings: *<type
arch='x86_64' machine='pc-q35-2.10'>*
I'm running fedora 27.
Any advice on how to implement secure boot for a VM under
KVM/Virt-Manager/qemu/etc etc.
I found this
https://en.opensuse.org/openSUSE:UEFI_Secure_boot_using_qemu-kvm but it
doesn't seem to match up with what I'm seeing on a few different aspects,
(obviously of which, is I'm not using SuSE)
Anyway, any help at all on this would be greatly appreciated.
Thanks!
-frizop
6 years, 6 months
[libvirt-users] Performance of guest
by Allence
I recently used Linux in VMware, but I discovered that kvm can replace it, so I created the kvm environment and then started a window on the host system.
But when I use it, I have a problem. My guest's performance is not very good.
For example, when I use Windows without kvm, the “lol” fps is 160, but my guest is only 135.
why? I only have one guest
what is the reason?
What can we do to promote my performance?
There are my some settings for me.
timer: hypervclock
cpu: 8 socket(intel@ core i7 7700k)
memory: 16G
GPU: 1060
network: virtio
my environment:
libvirt: 4.2
qemu: 2.12
windows: windows 7
host: ubuntu server 1604
If you need my detailed settings, please let me know.
6 years, 6 months
[libvirt-users] Guest's CPU frequency cannot be similar to host system
by Allence
My resource:
kernel: 4.16.8
host system: Ubuntu 16.04
Llbvirt : 4.2
Qemu : 2.12
Cpu : intel@core i7 6700
guest system: windows7 professional
My problem:
In my host, Cpu frequency is 3800~3900 MHz(4 socket and 2 thread) because i turn on “Intel Turbo Boost”. And silent frequency is 3.40 GHz.
Then i boot my guest, But in Guest my cpu frequency is 3408 MHz. why it's not 3800 MHz?
And here is my guest xml:
<cpu mode='host-passthrough' check='none'>
<topology sockets='1' cores='8' threads='1'/>
</cpu>
6 years, 6 months
