8:33 p.m.
Hi~
It seems that libvirt is not cpu hotplug aware.
Please refer to the following problem.
1. At first, we have 2 cpus.
# cat /cgroup/cpuset/cpuset.cpus
0-1
# cat /cgroup/cpuset/libvirt/qemu/cpuset.cpus
0-1
2. And we have a vm1 with following configuration.
<cputune>
<vcpupin vcpu='0' cpuset='1'/>
<hypervisorpin cpuset='1'/>
</cputune>
3. Offline cpu1.
# echo 0 > /sys/devices/system/cpu/cpu1/online
# cat /sys/devices/system/cpu/cpu1/online
0
# cat /cgroup/cpuset/cpuset.cpus
0
# cat /cgroup/cpuset/libvirt/qemu/cpuset.cpus
0
4. Online cpu1.
# echo 1 > /sys/devices/system/cpu/cpu1/online
# cat /sys/devices/system/cpu/cpu1/online
1
# cat /cgroup/cpuset/cpuset.cpus
0-1
# cat /cgroup/cpuset/libvirt/cpuset.cpus
0
# cat /cgroup/cpuset/libvirt/qemu/cpuset.cpus
0
# cat /cgroup/cpuset/libvirt/lxc/cpuset.cpus
0
Here，cgroup updated cpuset.cpus，but not for libvirt directory，and also qemu and lxc
directory.
vm1 cannot be started again.
# virsh start vm1
error: Failed to start domain vm1
error: Unable to set cpuset.cpus: Permission denied
And libvird gave the following errors.
2012-07-17 07:30:22.478+0000: 3118: error : qemuSetupCgroupVcpuPin:498 : Unable to set
cpuset.cpus: Permission denied
I am trying to use netlink socket with NETLINK_KOBJECT_UEVENT protocol to
listen to cpu hotplug events. But I met a little problem here.
virNetlinkEventServiceStart() only create a global variable server, and
create a NETLINK_ROUTE netlink socket for it.
So if I want to create another different netlink socket, such as NETLINK_KOBJECT_UEVENT,
what should I do ?
Shall we make the server global variable a global array ?
(It seems there are a lot of work to do.)
Thanks. :)
--
Best Regards,
Tang chen
8:44 p.m.
On 07/17/2012 07:33 PM, tangchen wrote:
Hi~
It seems that libvirt is not cpu hotplug aware.
Portions of libvirt are aware of host hotplug issues, but you are
correct that there are still lingering bugs (I just found one today in
nodeinfo.c).
I think this is related to (if not the same as) this known kernel bug:
https://bugzilla.redhat.com/show_bug.cgi?id=714271
Basically, when the kernel suspends and then resumes, it is not properly
restoring descendant cgroup information. Hot unplug of a host cpu is
more or less a subset of suspending.
Interesting approach. Is it also possible to use inotify?
I'm not sure here, having not really coded much with netlink sockets
myself. If you have a patch pending, then post it and we can discuss
whether it can be improved.
--
Eric Blake eblake(a)redhat.com +1-919-301-3266
Libvirt virtualization library http://libvirt.org
10:01 p.m.
Hi～
On 07/18/2012 09:44 AM, Eric Blake wrote:
Could you give me some more info about hotplug support in libvirt ?
I tried to find out, but I didn't see any.
Yes, hot unplug cpu function in kernel is not usable now.
But cgroup doesn't know what value a file used to be. As a result,
it cannot recover any value in subdirectories.
I have tried inotify. But since /sys is a special file system, it seems that
inotify failed to keep an eye on file systems like /proc, /cgroup or /sys.
But netlink works fine.
OK.:)
I'll try, but I am still wondering if anybody else could give me some advice.
Thank. :)
--
Best Regards,
Tang chen
4:06 a.m.
On 07/18/2012 03:44 AM, Eric Blake wrote:
That really depends on the architecture, on s390 with it's multi-level
virtualization, it's common to hot unplug host (the host being a 1st level
guest) CPUs during lower utilization periods. If these are part of a KVM
guest's CPU set, then they are gone for good...
I was actually considering a workaround in libvirt for what I believe to be a
kernel misbehavior, however it is possible to deconfigure the cpuset controller
in the host (i.e. removing it from /etc/cgconfig.conf). The CPU pinning will
still work, only that it using the "legacy" taskset mechanism. All other
cgroup-related functionality continues to work as well.
One issue with monitoring the online state of cpus is that libvirtd might not
be running all the time (crash, update) and thus can miss hotplug events.
--
Mit freundlichen Grüßen/Kind Regards
Viktor Mihajlovski
IBM Deutschland Research & Development GmbH
Vorsitzender des Aufsichtsrats: Martin Jetter
Geschäftsführung: Dirk Wittkopp
Sitz der Gesellschaft: Böblingen
Registergericht: Amtsgericht Stuttgart, HRB 243294
5:53 a.m.
Hi Viktor:
On 07/18/2012 05:06 PM, Viktor Mihajlovski wrote:
I don't think remove cpuset controller from cgroup is a good idea.
Maybe other apps will need it.
So I am considering capture the hotplug event (which is uevent) with netlink.
And change the behavior of cgroup in libvirt.
How do you think ?
Thanks. :)
--
Best Regards,
Tang chen
6:47 a.m.
On 07/18/2012 12:53 PM, tangchen wrote:
Maybe ... but a workaround in libvirt will not solve the other apps' issues ;-)
I still would maintain this has to be fixed in the kernel.
Looking forward to see a patch...
Thanks. :)
--
Mit freundlichen Grüßen/Kind Regards
Viktor Mihajlovski
IBM Deutschland Research & Development GmbH
Vorsitzender des Aufsichtsrats: Martin Jetter
Geschäftsführung: Dirk Wittkopp
Sitz der Gesellschaft: Böblingen
Registergericht: Amtsgericht Stuttgart, HRB 243294
1:50 p.m.
On 07/18/2012 05:17 PM, Viktor Mihajlovski wrote:
Not too long ago, there was a long thread in the Linux kernel mailing list (see below)
where this issue was discussed in detail. And the conclusion (in short) was that, for
suspend/resume, its the kernel's responsibility to restore the cpusets to the same
state as to how it was before suspend. (Note that, suspend/resume invokes CPU hotplug
internally, and this is hidden from the user).
However, in the regular case of CPU hotplug, the user himself initiates CPU hotplug.
So it is expected that the user be aware of its consequences and take appropriate
actions. In short, there is nothing to fix in the kernel, for the case of regular
CPU hotplug. It is an expected behaviour.
[Note that, technically the kernel can be "fixed" for this case as well. But
this
would break kernel semantics for CPU hotplug. Hence we have chosen not to do it.
The suspend/resume case was inevitable (as long as it depends on CPU hotplug).]
And the suspend/resume case has been fixed, in the 3.6 merge window by this commit:
commit d35be8bab9b0ce44bed4b9453f86ebf64062721e
Author: Srivatsa S. Bhat <srivatsa.bhat(a)linux.vnet.ibm.com>
Date: Thu May 24 19:46:26 2012 +0530
CPU hotplug, cpusets, suspend: Don't modify cpusets during suspend/resume
Links to the earlier discussions:
--------------------------------
[1]. http://thread.gmane.org/gmane.linux.documentation/4805
[2]. http://thread.gmane.org/gmane.linux.kernel/1296339
[3]. http://article.gmane.org/gmane.linux.kernel/1298967
http://thread.gmane.org/gmane.linux.kernel/1298967/focus=1300380
[4]. http://thread.gmane.org/gmane.linux.kernel/1302893
(this is the version which went upstream in the 3.6 merge window)
Regards,
Srivatsa S. Bhat
4615
days inactive
4647
days old
6 comments
4 participants
participants (4)
-
Eric Blake -
Srivatsa S. Bhat -
tangchen -
Viktor Mihajlovski