[libvirt] cgroup limits only affect kvm guest under certain conditions
Hi I am playing with cgroups and try to limit block io for guests. The proof of concept is: # mkdir /dev/cgroup/blkio # mount -t cgroup -o blkio blkio /dev/cgroup/blkio/ # cd blkio/ # mkdir test # cd test/ # ls -l /dev/vdisks/kirk lrwxrwxrwx 1 root root 7 2011-01-06 13:46 /dev/vdisks/kirk -> ../dm-5 # ls -l /dev/dm-5 brw-rw---- 1 root disk 253, 5 2011-01-06 13:36 /dev/dm-5 # echo "253:5 1048576" > blkio.throttle.write_bps_device # echo $$ > tasks # dd if=/dev/zero of=/dev/dm-5 bs=1M count=20 20+0 records in 20+0 records out 20971520 bytes (21 MB) copied, 20.0223 s, 1.0 MB/s So limit applies to the dd child of my shell. Now I assign /dev/dm-5 (/dev/vdisks/kirk) to a vm and echo the qemu-kvm pid into tasks. Limits are not applied, the guest can happily use max io bandwidth. However, if I start the guest manually like # qemu-kvm <options like libvirt creates> & echo $! > /dev/cgroup/blkio/test/tasks The limits _are_ applied. So, this looks like some sort of race condition to me. I tried to get information on this on the kernel mailing list [1], but either noone read it or at least no one replied. Maybe someone here can shed some light and maybe even fix the issue, if it is an issue. Need more information? Please ask for it. I don't know what else to supply at this point. My current lab is built by: OpenSuSE 11.3 64bit Vanilla Kernel 2.6.37 libvirt 0.8.7 qemu-kvm 0.13.0 vm is started like this: /usr/bin/qemu-kvm -M pc-0.12 -enable-kvm -m 2048 -smp 2,sockets=2,cores=1,threads=1 -name cliff -uuid a8247e1e-e3d2-d0fc-c5e5-47a173c3e460 -nodefconfig -nodefaults -chardev socket,id=monitor,path=/var/lib/libvirt/qemu/cliff.monitor,server,nowait -mon chardev=monitor,mode=readline -rtc base=utc -boot c -device lsi,id=scsi0,bus=pci.0,addr=0x6 -drive file=/dev/vdisks/cliff,if=none,id=drive-virtio-disk0,boot=on,format=raw -device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0 -drive file=/root/openSUSE-11.3-NET-x86_64.iso,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -drive file=/dev/vdisks/jason,if=none,id=drive-virtio-disk1,format=raw -device virtio-blk-pci,bus=pci.0,addr=0x7,drive=drive-virtio-disk1,id=virtio-disk1 -drive file=/dev/vdisks/rob,if=none,id=drive-ide0-0-0,format=raw -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -drive file=/dev/vdisks/james,if=none,id=drive-scsi0-0-0,format=raw -device scsi-disk,bus=scsi0.0,scsi-id=0,drive=drive-scsi0-0-0,id=scsi0-0-0 -netdev tap,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:37:84:e0,bus=pci.0,addr=0x5 -usb -vnc 127.0.0.1:0 -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3 Regards Dominik [1] http://help.lockergnome.com/linux/race-condition-net_cls-found-qemu-kvm-envi...
On Thu, Jan 06, 2011 at 02:15:37PM +0100, Dominik Klein wrote:
Hi
I am playing with cgroups and try to limit block io for guests.
The proof of concept is:
# mkdir /dev/cgroup/blkio # mount -t cgroup -o blkio blkio /dev/cgroup/blkio/ # cd blkio/ # mkdir test # cd test/ # ls -l /dev/vdisks/kirk lrwxrwxrwx 1 root root 7 2011-01-06 13:46 /dev/vdisks/kirk -> ../dm-5 # ls -l /dev/dm-5 brw-rw---- 1 root disk 253, 5 2011-01-06 13:36 /dev/dm-5 # echo "253:5 1048576" > blkio.throttle.write_bps_device # echo $$ > tasks # dd if=/dev/zero of=/dev/dm-5 bs=1M count=20 20+0 records in 20+0 records out 20971520 bytes (21 MB) copied, 20.0223 s, 1.0 MB/s
So limit applies to the dd child of my shell.
Now I assign /dev/dm-5 (/dev/vdisks/kirk) to a vm and echo the qemu-kvm pid into tasks. Limits are not applied, the guest can happily use max io bandwidth.
Did you just echo the main qemu-kvm PID, or did you also add the PIDs of every thread too ? From this description of the problem, I'd guess you've only confined the main process thread and thus the I/O & VCPU threads are not confined. Daniel
Did you just echo the main qemu-kvm PID, or did you also add the PIDs of every thread too ? From this description of the problem, I'd guess you've only confined the main process thread and thus the I/O & VCPU threads are not confined.
That was indeed correct. I was mislead by the fact that no child _processes_ were shown in ps. Once I added /proc/<qemu-pid>/tasks/* to tasks, it worked even when libvirt started the process. Thank you very much! Dominik
On Thu, 06 Jan 2011 14:15:37 +0100 Dominik Klein <dk@in-telegence.net> wrote:
Hi
I am playing with cgroups and try to limit block io for guests.
The proof of concept is:
# mkdir /dev/cgroup/blkio # mount -t cgroup -o blkio blkio /dev/cgroup/blkio/ # cd blkio/ # mkdir test # cd test/ # ls -l /dev/vdisks/kirk lrwxrwxrwx 1 root root 7 2011-01-06 13:46 /dev/vdisks/kirk -> ../dm-5 # ls -l /dev/dm-5 brw-rw---- 1 root disk 253, 5 2011-01-06 13:36 /dev/dm-5 # echo "253:5 1048576" > blkio.throttle.write_bps_device # echo $$ > tasks # dd if=/dev/zero of=/dev/dm-5 bs=1M count=20 20+0 records in 20+0 records out 20971520 bytes (21 MB) copied, 20.0223 s, 1.0 MB/s
So limit applies to the dd child of my shell.
Now I assign /dev/dm-5 (/dev/vdisks/kirk) to a vm and echo the qemu-kvm pid into tasks. Limits are not applied, the guest can happily use max io bandwidth.
qemu consists of several threads. Cgroup works per thread now. Could you double check "all threads for qemu" are in a cgroup ? I think you have to write all thread-ID to "tasks" file when you move qemu after starting it. Thanks, -Kame
qemu consists of several threads. Cgroup works per thread now. Could you double check "all threads for qemu" are in a cgroup ? I think you have to write all thread-ID to "tasks" file when you move qemu after starting it.
As stated yesterday: Yes, that was the problem. Works as expected now, thanks. Dominik
participants (3)
-
Daniel P. Berrange -
Dominik Klein -
KAMEZAWA Hiroyuki