7:29 a.m.
Hi
to be as verbose and clear as possible, I will write every command I use.
This sets everything up for the test:
# setup start
importantvms="kernel1 kernel2"
notimportantvms="kernel3 kernel4 kernel5 kernel6 kernel7 kernel8"
for vm in $importantvms $notimportantvms; do
virsh domstate $vm|grep -q running || virsh start $vm;
done
echo 1 > /sys/block/sdb/queue/iosched/group_isolation
mount -t cgroup -o blkio none /mnt
cd /mnt
mkdir important
mkdir notimportant
for vm in $notimportantvms; do
cd /proc/$(pgrep -f "qemu-kvm.*$vm")/task
for task in *; do
/bin/echo $task > /mnt/notimportant/tasks
done
done
for vm in $importantvms; do
cd /proc/$(pgrep -f "qemu-kvm.*$vm")/task
for task in *; do
/bin/echo $task > /mnt/important/tasks
done
done
#ls -lH /dev/vdisks/kernel[3-8]
#brw-rw---- 1 root root 254, 25 Feb 22 13:42 kernel3
#brw-rw---- 1 root root 254, 26 Feb 22 13:42 kernel4
#brw-rw---- 1 root root 254, 27 Feb 22 13:42 kernel5
#brw-rw---- 1 root root 254, 28 Feb 22 13:42 kernel6
#brw-rw---- 1 root root 254, 29 Feb 22 13:43 kernel7
#brw-rw---- 1 root root 254, 30 Feb 22 13:43 kernel8
for i in $(seq 25 30); do
/bin/echo 254:$i 10000000 >
/mnt/notimportant/blkio.throttle.write_bps_device
done
# setup complete
Hm..., this sounds bad. If you have put a limit of ~10Mb/s then no
"bo" is bad. That would explain that why your box is not responding
and you need to do power reset.
- I am assuming that you have not put any throttling limits on root group.
Is your system root also on /dev/sdb or on a separate disk altogether.
No throttling on root. Correct.
system root is on sda
vms are on sdb
- This sounds like a bug in throttling logic. To narrow it down can
you
start running "deadline" on end device. If it still happens, it is more
or less in throttling layer.
cat /sys/block/sdb/queue/scheduler
noop deadline [cfq]
echo deadline > /sys/block/sdb/queue/scheduler
cat /sys/block/sdb/queue/scheduler
noop [deadline] cfq
This changes things:
vmstat before test (nothing going on):
r b swpd free buff cache si so bi bo in cs us sy
id wa
1 0 0 130231696 17968 69424 0 0 0 0 16573 32834
0 0 100 0
vmstat during test while all 8 vms are writing (2 unthrottled, 6 at 10M):
r b swpd free buff cache si so bi bo in cs us sy
id wa
2 9 0 126257984 17968 69424 0 0 8 164462 20114 36751
5 4 52 39
vmstat during test when only throttled vms are still dd'ing
2 21 0 124518928 17976 69424 0 0 0 63876 17410 33670
1 1 59 39
Load is 12-ish during the test and the results are as expected:
Throttled VMs write with approx 10M, unthrottled get like 80 each, which
sums up to about the 200M max capacity of the device for direct io.
- We can also try to remove dm layers and just create partitions on
/dev/sdb and export as virtio disks to virtual machines and take
dm layer out of picture and see if it still happens.
Did that. Also happens when scheduler is cfq. Also does not happen when
scheduler is deadline. So it does not seem to be a dm issue.
- In one of the mails you mentioned that with 1 virutal machine
throttling
READs and WRITEs is working for you. So it looks like 1 virtual machine
does not hang but once you launch 8 virtual machines it hangs. Can we
try increasing the number of vitual machines gragually and confirm that
it happens only if some certain number of virtual machines are launched.
Back on cfq here.
1 throttled vm: works, load ~4
2 throttled vms: works, load ~6
3 throttled vms: works, load ~9
4 throttled vms: works, load ~12
6 throttled vms: works, load ~20
The number of blocked threads increases with the number of vms dd'ing.
At the beginning of each test, the blocked threads number goes really
high (4 vms 160, 6 vms 220), but then drops significantly and stays low.
So it seems that when only throttled vms are running, the problem does
not occur.
1 throttled + 1 unthrottled vm: works, load ~5
2 throttled + 1 unthrottled vm: boom
Constantly 144 blocked threads, bo=0, load increasing to 144. System
needs power reset.
So, thinking about what I did in the initial setup, I re-tested without the
for vm in $importantvms; do
cd /proc/$(pgrep -f "qemu-kvm.*$vm")/task
for task in *; do
/bin/echo $task > /mnt/important/tasks
done
done
since I don't do anything with that "important" cgroup (yet) anyway. It
did not make a difference though.
- Can you also paste me the rules you have put on important and
non-important
groups. Somehow I suspect that some of the rule has gone horribly bad
in the sense that it is very low and effectively no virtual machine
is making any progress.
See setup commands in the beginning of this email.
- How long does it take to reach in this locked state where bo=0.
It goes there "instantly", right after the dd commands start.
- you can also try to redirect blktrace output to blkparse and
redirect
it to standard output and see capture some output by copying pasting
last messages.
I hope this is what you meant:
blktrace -d /dev/sdb -o - | blkparse -i -
Output is in attachment.
Regards
Dominik