7:15 a.m.
Hi
I am playing with cgroups and try to limit block io for guests.
The proof of concept is:
# mkdir /dev/cgroup/blkio
# mount -t cgroup -o blkio blkio /dev/cgroup/blkio/
# cd blkio/
# mkdir test
# cd test/
# ls -l /dev/vdisks/kirk
lrwxrwxrwx 1 root root 7 2011-01-06 13:46 /dev/vdisks/kirk -> ../dm-5
# ls -l /dev/dm-5
brw-rw---- 1 root disk 253, 5 2011-01-06 13:36 /dev/dm-5
# echo "253:5 1048576" > blkio.throttle.write_bps_device
# echo $$ > tasks
# dd if=/dev/zero of=/dev/dm-5 bs=1M count=20
20+0 records in
20+0 records out
20971520 bytes (21 MB) copied, 20.0223 s, 1.0 MB/s
So limit applies to the dd child of my shell.
Now I assign /dev/dm-5 (/dev/vdisks/kirk) to a vm and echo the qemu-kvm
pid into tasks. Limits are not applied, the guest can happily use max io
bandwidth.
However, if I start the guest manually like
# qemu-kvm <options like libvirt creates> & echo $! >
/dev/cgroup/blkio/test/tasks
The limits _are_ applied.
So, this looks like some sort of race condition to me.
I tried to get information on this on the kernel mailing list [1], but
either noone read it or at least no one replied. Maybe someone here can
shed some light and maybe even fix the issue, if it is an issue.
Need more information? Please ask for it. I don't know what else to
supply at this point.
My current lab is built by:
OpenSuSE 11.3 64bit
Vanilla Kernel 2.6.37
libvirt 0.8.7
qemu-kvm 0.13.0
vm is started like this:
/usr/bin/qemu-kvm -M pc-0.12 -enable-kvm -m 2048 -smp
2,sockets=2,cores=1,threads=1 -name cliff -uuid
a8247e1e-e3d2-d0fc-c5e5-47a173c3e460 -nodefconfig -nodefaults -chardev
socket,id=monitor,path=/var/lib/libvirt/qemu/cliff.monitor,server,nowait
-mon chardev=monitor,mode=readline -rtc base=utc -boot c -device
lsi,id=scsi0,bus=pci.0,addr=0x6 -drive
file=/dev/vdisks/cliff,if=none,id=drive-virtio-disk0,boot=on,format=raw
-device
virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0
-drive
file=/root/openSUSE-11.3-NET-x86_64.iso,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw
-device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0
-drive file=/dev/vdisks/jason,if=none,id=drive-virtio-disk1,format=raw
-device
virtio-blk-pci,bus=pci.0,addr=0x7,drive=drive-virtio-disk1,id=virtio-disk1
-drive file=/dev/vdisks/rob,if=none,id=drive-ide0-0-0,format=raw -device
ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -drive
file=/dev/vdisks/james,if=none,id=drive-scsi0-0-0,format=raw -device
scsi-disk,bus=scsi0.0,scsi-id=0,drive=drive-scsi0-0-0,id=scsi0-0-0
-netdev tap,id=hostnet0 -device
virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:37:84:e0,bus=pci.0,addr=0x5
-usb -vnc 127.0.0.1:0 -vga cirrus -device
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3
Regards
Dominik
[1]
http://help.lockergnome.com/linux/race-condition-net_cls-found-qemu-kvm-e...
7:36 a.m.
On Thu, Jan 06, 2011 at 02:15:37PM +0100, Dominik Klein wrote:
Hi
I am playing with cgroups and try to limit block io for guests.
The proof of concept is:
# mkdir /dev/cgroup/blkio
# mount -t cgroup -o blkio blkio /dev/cgroup/blkio/
# cd blkio/
# mkdir test
# cd test/
# ls -l /dev/vdisks/kirk
lrwxrwxrwx 1 root root 7 2011-01-06 13:46 /dev/vdisks/kirk -> ../dm-5
# ls -l /dev/dm-5
brw-rw---- 1 root disk 253, 5 2011-01-06 13:36 /dev/dm-5
# echo "253:5 1048576" > blkio.throttle.write_bps_device
# echo $$ > tasks
# dd if=/dev/zero of=/dev/dm-5 bs=1M count=20
20+0 records in
20+0 records out
20971520 bytes (21 MB) copied, 20.0223 s, 1.0 MB/s
So limit applies to the dd child of my shell.
Now I assign /dev/dm-5 (/dev/vdisks/kirk) to a vm and echo the qemu-kvm
pid into tasks. Limits are not applied, the guest can happily use max io
bandwidth.
Did you just echo the main qemu-kvm PID, or did you also
add the PIDs of every thread too ? From this description
of the problem, I'd guess you've only confined the main
process thread and thus the I/O & VCPU threads are not
confined.
Daniel
7:47 a.m.
Did you just echo the main qemu-kvm PID, or did you also
add the PIDs of every thread too ? From this description
of the problem, I'd guess you've only confined the main
process thread and thus the I/O & VCPU threads are not
confined.
That was indeed correct. I was mislead by the fact that no child
_processes_ were shown in ps.
Once I added /proc/<qemu-pid>/tasks/* to tasks, it worked even when
libvirt started the process.
Thank you very much!
Dominik
12:07 a.m.
On Thu, 06 Jan 2011 14:15:37 +0100
Dominik Klein <dk(a)in-telegence.net> wrote:
Hi
I am playing with cgroups and try to limit block io for guests.
The proof of concept is:
# mkdir /dev/cgroup/blkio
# mount -t cgroup -o blkio blkio /dev/cgroup/blkio/
# cd blkio/
# mkdir test
# cd test/
# ls -l /dev/vdisks/kirk
lrwxrwxrwx 1 root root 7 2011-01-06 13:46 /dev/vdisks/kirk -> ../dm-5
# ls -l /dev/dm-5
brw-rw---- 1 root disk 253, 5 2011-01-06 13:36 /dev/dm-5
# echo "253:5 1048576" > blkio.throttle.write_bps_device
# echo $$ > tasks
# dd if=/dev/zero of=/dev/dm-5 bs=1M count=20
20+0 records in
20+0 records out
20971520 bytes (21 MB) copied, 20.0223 s, 1.0 MB/s
So limit applies to the dd child of my shell.
Now I assign /dev/dm-5 (/dev/vdisks/kirk) to a vm and echo the qemu-kvm
pid into tasks. Limits are not applied, the guest can happily use max io
bandwidth.
qemu consists of several threads. Cgroup works per thread now.
Could you double check "all threads for qemu" are in a cgroup ?
I think you have to write all thread-ID to "tasks" file when you
move qemu after starting it.
Thanks,
-Kame
1:17 a.m.
qemu consists of several threads. Cgroup works per thread now.
Could you double check "all threads for qemu" are in a cgroup ?
I think you have to write all thread-ID to "tasks" file when you
move qemu after starting it.
As stated yesterday: Yes, that was the problem. Works as expected now,
thanks.
Dominik
5068
days inactive
5069
days old
4 comments
3 participants
participants (3)
-
Daniel P. Berrange -
Dominik Klein -
KAMEZAWA Hiroyuki