[libvirt] virsh "cpu-stats" not working !!!
Hi All,
Can you please guide us in solving this issue.
Version used : libvirt-1.0.6 # rpm -qa | egrep "libvirt|qemu" qemu-kvm-0.14.1-1_WR4.3.x86_64_XXX.x86_64 libvirt-1.0.6-1_WR4.3.x86_64_XXX.2.x86_64
Problem :
# virsh cpu-stats CLA-0 error: Failed to retrieve CPU statistics for domain 'CLA-0' *error: internal error error accessing cgroup cpuacct for vcpu*
Related info :
# cat /proc/cgroups #subsys_name hierarchy num_cgroups enabled cpuset 3 4 1 ns 0 1 1 cpu 1 6 1 cpuacct 7 3 1 memory 2 3 1 freezer 0 1 1 net_cls 0 1 1 blkio 0 1 1
# cat /proc/mounts | grep cpu cpu /dev/cgroup_cpu cgroup rw,relatime,cpu 0 0 cpuset /dev/cgroup_cpuset cgroup rw,relatime,cpuset 0 0 cpuacct /dev/cgroup_cpuacct cgroup rw,relatime,cpuacct 0 0
# cat /proc/self/cgroup 7:cpuacct:/ 3:cpuset:/ 2:memory:/ 1:cpu:/
virsh # vcpucount CLA-0 maximum config 3 maximum live 3 current config 3 current live 3
virsh # vcpuinfo CLA-0 VCPU: 0 CPU: 0 State: running CPU time: 5.7s CPU Affinity: yyyyyyyy
virsh # vcpupin CLA-0 --current VCPU: CPU Affinity ---------------------------------- 0: 0-7 1: 0-7 2: 0-7
# cat /dev/cgroup_cpuacct/cpuacct.usage_percpu 532996776861 117928641102 125174572624 167825543709 155234053284 97350309867 122372010702 110640812251 # cat /dev/cgroup_cpuacct/cpuacct.usage 1431288376064 # cat /dev/cgroup_cpuacct/cpuacct.stat user 12148 system 81469
# cat /dev/cgroup_cpuacct/*machine*/cpuacct.usage_percpu 0 0 0 0 0 0 0 0 # cat /dev/cgroup_cpuacct/*machine/*cpuacct.usage 0
# chkconfig --list | grep cg cgroups_mount 0:off 1:on 2:on 3:on 4:on 5:on 6:off
We have also found that there is no "*cgroup*" directory created under * "/sys/fs/".* We have tried mounting the same manually using the below commands but we were not successful.
mkdir -p /dev/cgroup_cpuacct mount -t cgroup -o cpuacct cpuacct /dev/cgroup_cpuacct
(or)
mkdir -p /sys/fs/cgroup mount -t cgroup -o cgroup cgroup /sys/fs/cgroup
Please find the the domain info used for the creation of VM in the attachment. Please let us know, how to proceed futher.
Your inputs would be of great help to us.
Thanks and Regards, Shree Duth Awasthi.
On 08/21/2013 03:55 AM, SHREE DUTH AWASTHI wrote: [putting your name in ALL CAPS looks like you are shouting]
Hi All,
Can you please guide us in solving this issue.
Version used : libvirt-1.0.6 # rpm -qa | egrep "libvirt|qemu" qemu-kvm-0.14.1-1_WR4.3.x86_64_XXX.x86_64 libvirt-1.0.6-1_WR4.3.x86_64_XXX.2.x86_64
Problem :
# virsh cpu-stats CLA-0 error: Failed to retrieve CPU statistics for domain 'CLA-0' *error: internal error error accessing cgroup cpuacct for vcpu*
Are you sure cpuacct was mounted prior to libvirtd starting?
Related info :
# cat /proc/cgroups #subsys_name hierarchy num_cgroups enabled cpuset 3 4 1 ns 0 1 1 cpu 1 6 1 cpuacct 7 3 1
The fact that it has a rather high hierarchy number (7) makes me think you mounted it after the fact.
We have also found that there is no "*cgroup*" directory created under * "/sys/fs/".* We have tried mounting the same manually using the below commands but we were not successful.
Cgroups are typically mounted by systemd in newer systems, but the fact that you are trying to mount manually says that you aren't using systemd. Libvirt won't mount cgroups; it only uses what was mounted at the time it was started. -- Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Hi Eric, Thanks for your valuable time and comments.
Are you sure cpuacct was mounted prior to libvirtd starting? Yes, we are mounting cpuacct before starting the libvirtd daemon.
Libvirt won't mount cgroups; Yes, Agree. it only uses what was mounted at the time it was started. But we are mounting it well before it is started.
We are facing the issue only while executing the "cpu-stats <domain name>" command. One more observation is that : While starting the domain, we are observing the below error. "2013-08-22 12:45:44.326+0000: 17554: warning : qemuSetupCgroupForVcpu:953 : Unable to get vcpus' pids." This may be the root cause of virsh cpu-stats not working. ( seems to be because of CPU pinning issue ) Because of this the VCPU directories are not created for the corresponding domain in "/dev/cgroup_<controller>/machine/<domain>.libvirt-qemu/" and hence the "cpu-stats" command is not working. Please let us know your view on this. We have collected the debug logs, please find the attachment for the same.. Thanks and Regards, Shree Duth Awasthi. On Wed, Aug 21, 2013 at 9:35 PM, Eric Blake <eblake@redhat.com> wrote:
On 08/21/2013 03:55 AM, SHREE DUTH AWASTHI wrote:
[putting your name in ALL CAPS looks like you are shouting]
Hi All,
Can you please guide us in solving this issue.
Version used : libvirt-1.0.6 # rpm -qa | egrep "libvirt|qemu" qemu-kvm-0.14.1-1_WR4.3.x86_64_XXX.x86_64 libvirt-1.0.6-1_WR4.3.x86_64_XXX.2.x86_64
Problem :
# virsh cpu-stats CLA-0 error: Failed to retrieve CPU statistics for domain 'CLA-0' *error: internal error error accessing cgroup cpuacct for vcpu*
Are you sure cpuacct was mounted prior to libvirtd starting?
Related info :
# cat /proc/cgroups #subsys_name hierarchy num_cgroups enabled cpuset 3 4 1 ns 0 1 1 cpu 1 6 1 cpuacct 7 3 1
The fact that it has a rather high hierarchy number (7) makes me think you mounted it after the fact.
We have also found that there is no "*cgroup*" directory created under * "/sys/fs/".* We have tried mounting the same manually using the below commands but we were not successful.
Cgroups are typically mounted by systemd in newer systems, but the fact that you are trying to mount manually says that you aren't using systemd. Libvirt won't mount cgroups; it only uses what was mounted at the time it was started.
-- Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Hi Again Eric, Sorry for spamming. Adding to the below mail. The cgroups are mounted at "/dev" as mentioned below # ls /dev/ | grep cgroup cgroup_cpu cgroup_cpuacct cgroup_cpuset cgroup_mem On a liter note, " On 08/21/2013 03:55 AM, SHREE DUTH AWASTHI wrote: " is coming by default while I try to reply. Anyway, I will explicitly modify it from now. Thanks and Regards, Shree Duth Awasthi. On Fri, Aug 23, 2013 at 9:16 AM, Shree Duth Awasthi < shreeduth.awasthi@gmail.com> wrote:
Hi Eric,
Thanks for your valuable time and comments.
Are you sure cpuacct was mounted prior to libvirtd starting? Yes, we are mounting cpuacct before starting the libvirtd daemon.
Libvirt won't mount cgroups; Yes, Agree.
it only uses what was mounted at the time it was started. But we are mounting it well before it is started.
We are facing the issue only while executing the "cpu-stats <domain name>" command.
One more observation is that : While starting the domain, we are observing the below error.
"2013-08-22 12:45:44.326+0000: 17554: warning : qemuSetupCgroupForVcpu:953 : Unable to get vcpus' pids."
This may be the root cause of virsh cpu-stats not working. ( seems to be because of CPU pinning issue )
Because of this the VCPU directories are not created for the corresponding domain in "/dev/cgroup_<controller>/machine/<domain>.libvirt-qemu/" and hence the "cpu-stats" command is not working. Please let us know your view on this.
We have collected the debug logs, please find the attachment for the same..
Thanks and Regards, Shree Duth Awasthi.
On Wed, Aug 21, 2013 at 9:35 PM, Eric Blake <eblake@redhat.com> wrote:
On 08/21/2013 03:55 AM, SHREE DUTH AWASTHI wrote:
[putting your name in ALL CAPS looks like you are shouting]
Hi All,
Can you please guide us in solving this issue.
Version used : libvirt-1.0.6 # rpm -qa | egrep "libvirt|qemu" qemu-kvm-0.14.1-1_WR4.3.x86_64_XXX.x86_64 libvirt-1.0.6-1_WR4.3.x86_64_XXX.2.x86_64
Problem :
# virsh cpu-stats CLA-0 error: Failed to retrieve CPU statistics for domain 'CLA-0' *error: internal error error accessing cgroup cpuacct for vcpu*
Are you sure cpuacct was mounted prior to libvirtd starting?
Related info :
# cat /proc/cgroups #subsys_name hierarchy num_cgroups enabled cpuset 3 4 1 ns 0 1
1
cpu 1 6 1 cpuacct 7 3 1
The fact that it has a rather high hierarchy number (7) makes me think you mounted it after the fact.
"/sys/fs/".* We have tried mounting the same manually using the below commands but we were not successful.
We have also found that there is no "*cgroup*" directory created under
Cgroups are typically mounted by systemd in newer systems, but the fact that you are trying to mount manually says that you aren't using systemd. Libvirt won't mount cgroups; it only uses what was mounted at the time it was started.
-- Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
participants (2)
-
Eric Blake -
SHREE DUTH AWASTHI