8:54 a.m.
Hi,
i have a vm which has a poor performance.
E.g. top needs seconds to refresh its output on the console. Same with netstat.
The guest is hosting a MySQL DB with a webfrontend, its response is poor too.
I'm looking for the culprit.
Following top in the guest i get these hints:
Memory is free enough, system is not swapping.
System has 8GB RAM and two cpu's.
Cpu 0 is struggling with a lot of software interrupts, between 50% and 80%.
Cpu1 is often waiting for IO (wa), between 0% and 20%.
No application is consuming much cpu time.
Here is an example:
top - 11:19:18 up 18:19, 11 users, load average: 1.44, 0.94, 0.66
Tasks: 95 total, 1 running, 94 sleeping, 0 stopped, 0 zombie
Cpu0 : 0.0%us, 0.0%sy, 0.0%ni, 20.0%id, 0.0%wa, 0.0%hi, 80.0%si, 0.0%st
Cpu1 : 1.9%us, 13.8%sy, 0.0%ni, 73.8%id, 10.5%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 7995216k total, 6385176k used, 1610040k free, 177772k buffers
Swap: 2104472k total, 0k used, 2104472k free, 5940884k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6470 root 16 0 12844 1464 804 S 12 0.0 2:17.13 screen
6022 root 15 0 41032 3052 2340 S 3 0.0 1:10.99 sshd
8322 root 0 -20 10460 4976 2268 S 3 0.1 19:20.38 atop
10806 root 16 0 5540 1216 880 R 0 0.0 0:00.51 top
126 root 15 0 0 0 0 S 0 0.0 0:23.33 pdflush
3531 postgres 15 0 68616 1600 792 S 0 0.0 0:41.24 postmaster
The host in which the guest runs has 96GB RAM and 8 cores.
It does not seem to do much:
top - 11:21:19 up 15 days, 15:53, 14 users, load average: 1.40, 1.39, 1.40
Tasks: 221 total, 2 running, 219 sleeping, 0 stopped, 0 zombie
Cpu0 : 15.9%us, 2.7%sy, 0.0%ni, 81.4%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu1 : 5.0%us, 3.0%sy, 0.0%ni, 92.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu2 : 2.0%us, 0.3%sy, 0.0%ni, 97.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu3 : 0.3%us, 1.0%sy, 0.0%ni, 98.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu4 : 1.3%us, 0.3%sy, 0.0%ni, 98.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu5 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu6 : 0.3%us, 0.0%sy, 0.0%ni, 99.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu7 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 96738M total, 13466M used, 83272M free, 3M buffers
Swap: 2046M total, 0M used, 2046M free, 3887M cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
21765 root 20 0 105m 15m 4244 S 5 0.0 0:00.15 crm
3180 root 20 0 8572m 8.0g 8392 S 3 8.4 62:25.73 qemu-kvm
8529 hacluste 10 -10 90820 14m 9400 S 0 0.0 29:52.48 cib
21329 root 20 0 9040 1364 940 R 0 0.0 0:00.16 top
28439 root 20 0 0 0 0 S 0 0.0 0:04.51 kworker/4:2
1 root 20 0 10560 828 692 S 0 0.0 0:07.67 init
2 root 20 0 0 0 0 S 0 0.0 0:00.28 kthreadd
3 root 20 0 0 0 0 S 0 0.0 3:03.23 ksoftirqd/0
6 root RT 0 0 0 0 S 0 0.0 0:05.02 migration/0
7 root RT 0 0 0 0 S 0 0.0 0:02.82 watchdog/0
8 root RT 0 0 0 0 S 0 0.0 0:05.18 migration/1
I think the host is not the problem.
The vm resides on a SAN which is attached via FC. The whole system is a two node cluster.
The vm resides in a raw partition without a FS, which i read should be good for the
performance.
It runs on the other node slow too. Inside the vm i have logical volumes
(it was a physical system i migrated to a vm). The partitions are formatted with reiserfs
(The system is already some years old, at that time reiserfs was popular ...).
I use iostat on the guest:
This is a typical snapshot:
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz
await svctm %util
vda 0.00 3.05 0.00 2.05 0.00 20.40 19.90 0.09
44.59 31.22 6.40
dm-0 0.00 0.00 0.00 4.55 0.00 18.20 8.00 0.24
52.31 7.74 3.52
dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00
dm-2 0.00 0.00 0.00 0.10 0.00 0.40 8.00 0.01
92.00 56.00 0.56
dm-3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00
dm-4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00
dm-5 0.00 0.00 0.00 0.35 0.00 1.40 8.00 0.03
90.29 65.71 2.30
dm-6 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00
vda has several partitions, one for /, one for swap, and two physical volumes for LVM.
Following "man iostat", the columns await and svctm seem to be important. Man
says:
await
The average time (in milliseconds) for I/O requests issued to the device to be served.
This includes the time spent by the requests in queue and the time spent servicing them.
svctm
The average service time (in milliseconds) for I/O requests that were issued to the
device.
It seems system is waiting a long time for IO. Although the amount of transfered data is
small.
I have some suspicions:
- the lvm setup in the guest
- some hardware
- cache mode for the disk is "none". Otherwise i can't do a live migration.
What do you think ? How can i find out from where the high si comes ?
Network and disk are virtio devices (which should be fast):
vm58820-4:~ # lsmod|grep -i virt
virtio_balloon 22788 0
virtio_net 30464 0
virtio_pci 27264 0
virtio_ring 21376 1 virtio_pci
virtio_blk 25224 5
virtio 22916 4 virtio_balloon,virtio_net,virtio_pci,virtio_blk
That's the config of the guest:
<domain type='kvm'>
<name>mausdb_vm</name>
<uuid>f08c2f32-fe35-137a-0e9d-fa7485d57974</uuid>
<memory unit='KiB'>8198144</memory>
<currentMemory unit='KiB'>8197376</currentMemory>
<vcpu placement='static'>2</vcpu>
<os>
<type arch='x86_64' machine='pc-i440fx-1.4'>hvm</type>
<boot dev='cdrom'/>
<bootmenu enable='yes'/>
</os>
<features>
<acpi/>
<apic/>
<pae/>
</features>
<clock offset='utc'/>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<devices>
<emulator>/usr/bin/qemu-kvm</emulator>
<disk type='block' device='disk'>
<driver name='qemu' type='raw' cache='none'/>
<source dev='/dev/vg_cluster_01/lv_cluster_01'/>
<target dev='vda' bus='virtio'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x04' function='0x0'/>
</disk>
<controller type='usb' index='0'>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x01' function='0x2'/>
</controller>
<controller type='pci' index='0' model='pci-root'/>
<controller type='ide' index='0'>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x01' function='0x1'/>
</controller>
<interface type='bridge'>
<mac address='52:54:00:37:92:01'/>
<source bridge='br0'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x03' function='0x0'/>
</interface>
<input type='mouse' bus='ps2'/>
<input type='keyboard' bus='ps2'/>
<graphics type='vnc' port='-1' autoport='yes'
listen='127.0.0.1'>
<listen type='address' address='127.0.0.1'/>
</graphics>
<video>
<model type='cirrus' vram='9216' heads='1'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x02' function='0x0'/>
</video>
<memballoon model='virtio'>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x05' function='0x0'/>
</memballoon>
</devices>
<seclabel type='none'/>
</domain>
Host OS is SLES 11 SP4, guest os is SLES 10 SP4. Both 64bit.
Thanks for any hint.
Bernd
--
Bernd Lentes
Systemadministration
institute of developmental genetics
Gebäude 35.34 - Raum 208
HelmholtzZentrum München
bernd.lentes(a)helmholtz-muenchen.de
phone: +49 (0)89 3187 1241
fax: +49 (0)89 3187 2294
Erst wenn man sich auf etwas festlegt kann man Unrecht haben
Scott Adams
Helmholtz Zentrum Muenchen
Deutsches Forschungszentrum fuer Gesundheit und Umwelt (GmbH)
Ingolstaedter Landstr. 1
85764 Neuherberg
www.helmholtz-muenchen.de
Aufsichtsratsvorsitzende: MinDir'in Baerbel Brumme-Bothe
Geschaeftsfuehrer: Prof. Dr. Guenther Wess, Heinrich Bassler, Dr. Alfons Enhsen
Registergericht: Amtsgericht Muenchen HRB 6466
USt-IdNr: DE 129521671