[libvirt-users] Information needed regarding the libvirt version libvirt-1.0.6-1
Hi, We have upgraded the libvirt version from libvirt-1.0.5 to libvirt-1.0.6-1 on our linux machine. After this upgrade "virsh cpu-stats <domain>" command is not working on our HardWare. *virsh # cpu-stats test error: Failed to retrieve CPU statistics for domain 'test' error: internal error error accessing cgroup cpuacct for vcpu* ** *# cat /proc/cgroups #subsys_name hierarchy num_cgroups enabled cpuset 3 4 1 ns 0 1 1 cpu 1 6 1 cpuacct 7 3 1 memory 2 3 1 freezer 0 1 1 net_cls 0 1 1 blkio 0 1 1 * *# chkconfig --list | grep cg cgroups_mount 0:off 1:on 2:on 3:on 4:on 5:on 6:off* We have also found that there is no "cgroup" directory created by kenel under "/sys/fs/". We suspect that the 'cgroup' controllers are not mounted properly and hence the issue is observed. While using the libvirt-1.0.5 version below mentioned mount commands were sufficient to mount cgroup controllers *mkdir -p /dev/cgroup_cpuacct mount -t cgroup -o cpuacct cpuacct /dev/cgroup_cpuacct* As cgroups hieracrhy inside libvirtd has changed in libvirt-1.0.6-1, we feel that the above commands are now not sufficient. We feel we have to do some more steps get the 'cgroups' mounted properly on our hardware. Since we are new to libvirt and can you please help us by providing some pointers to fix the above issue and get cpu-stats working on our HardWare. Since we have to fix this on top priority, can you please reply as soon as possible. Please let us know if you need any further information from our side. Thanks in advance. BR Abhinay
On Wed, Aug 21, 2013 at 09:06:21AM +0200, arun abhinay wrote:
Hi,
We have upgraded the libvirt version from libvirt-1.0.5 to libvirt-1.0.6-1 on our linux machine. After this upgrade "virsh cpu-stats <domain>" command is not working on our HardWare.
*virsh # cpu-stats test error: Failed to retrieve CPU statistics for domain 'test' error: internal error error accessing cgroup cpuacct for vcpu* ** *# cat /proc/cgroups #subsys_name hierarchy num_cgroups enabled cpuset 3 4 1 ns 0 1 1 cpu 1 6 1 cpuacct 7 3 1 memory 2 3 1 freezer 0 1 1 net_cls 0 1 1 blkio 0 1 1 * *# chkconfig --list | grep cg cgroups_mount 0:off 1:on 2:on 3:on 4:on 5:on 6:off*
We have also found that there is no "cgroup" directory created by kenel under "/sys/fs/". We suspect that the 'cgroup' controllers are not mounted properly and hence the issue is observed. While using the libvirt-1.0.5 version below mentioned mount commands were sufficient to mount cgroup controllers
*mkdir -p /dev/cgroup_cpuacct mount -t cgroup -o cpuacct cpuacct /dev/cgroup_cpuacct*
As cgroups hieracrhy inside libvirtd has changed in libvirt-1.0.6-1, we
Actually it changed in version 1.0.5
feel that the above commands are now not sufficient. We feel we have to do some more steps get the 'cgroups' mounted properly on our hardware.
Since we are new to libvirt and can you please help us by providing some pointers to fix the above issue and get cpu-stats working on our HardWare. Since we have to fix this on top priority, can you please reply as soon as possible. Please let us know if you need any further information from our side.
Libvirt will never attempt to mount any cgroups controllers. It delegates this task to the base OS. Historically the 'cgconfig' init script would be used. On distros which use systemd though, systemd will take care of this task. There's a bit more info here http://libvirt.org/cgroups.html Basically our recommendation is to mount all the controllers individually at /sys/fs/cgroup/<controllername> Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
Hi Daniel, Thanks a lot for your reply. We have mounted all the cgoup controllers at "/dev/cgroup_<controller>" as mentioned below. ls /dev/ | grep cgroup cgroup_cpu cgroup_cpuacct cgroup_cpuset cgroup_mem On further debugging we dound that the "VCPU" directories are not being created for our domains and hence the comamnd "cpu-stats <domain>" is failing. Below is the snippet of logs collected with debug logs enabled for libvirt. 2013-08-21 15:03:35.162+0000: 1273: error : virCgroupMakeGroup:708 : Make group /machine/anusha.libvirt-qemu/vcpu0 2013-08-21 15:03:35.162+0000: 1273: error : virCgroupMakeGroup:729 : Makecontroller /dev/cgroup_cpu/machine/anusha.libvirt-qemu/vcpu0/ 2013-08-21 15:03:35.162+0000: 1273: error : virCgroupMakeGroup:745 : Failed to create controller cpu for group Where anusha --> is the domain which was created for libvirt. Can you please let us know if it is enough to mount the controllers mentioned above or we have missed mounting of any other controller. Your inputs would be of great help to us. Thanking you in advance. BR Abhinay On Wed, Aug 21, 2013 at 1:39 PM, Daniel P. Berrange <berrange@redhat.com>wrote:
Hi,
We have upgraded the libvirt version from libvirt-1.0.5 to
On Wed, Aug 21, 2013 at 09:06:21AM +0200, arun abhinay wrote: libvirt-1.0.6-1
on our linux machine. After this upgrade "virsh cpu-stats <domain>" command is not working on our HardWare.
*virsh # cpu-stats test error: Failed to retrieve CPU statistics for domain 'test' error: internal error error accessing cgroup cpuacct for vcpu* ** *# cat /proc/cgroups #subsys_name hierarchy num_cgroups enabled cpuset 3 4 1 ns 0 1 1 cpu 1 6 1 cpuacct 7 3 1 memory 2 3 1 freezer 0 1 1 net_cls 0 1 1 blkio 0 1 1 * *# chkconfig --list | grep cg cgroups_mount 0:off 1:on 2:on 3:on 4:on 5:on 6:off*
We have also found that there is no "cgroup" directory created by kenel under "/sys/fs/". We suspect that the 'cgroup' controllers are not mounted properly and hence the issue is observed. While using the libvirt-1.0.5 version below mentioned mount commands were sufficient to mount cgroup controllers
*mkdir -p /dev/cgroup_cpuacct mount -t cgroup -o cpuacct cpuacct /dev/cgroup_cpuacct*
As cgroups hieracrhy inside libvirtd has changed in libvirt-1.0.6-1, we
Actually it changed in version 1.0.5
feel that the above commands are now not sufficient. We feel we have to do some more steps get the 'cgroups' mounted properly on our hardware.
Since we are new to libvirt and can you please help us by providing some pointers to fix the above issue and get cpu-stats working on our HardWare. Since we have to fix this on top priority, can you please reply as soon as possible. Please let us know if you need any further information from our side.
Libvirt will never attempt to mount any cgroups controllers. It delegates this task to the base OS. Historically the 'cgconfig' init script would be used. On distros which use systemd though, systemd will take care of this task. There's a bit more info here
http://libvirt.org/cgroups.html
Basically our recommendation is to mount all the controllers individually at /sys/fs/cgroup/<controllername>
Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/:| |: http://libvirt.org -o- http://virt-manager.org:| |: http://autobuild.org -o- http://search.cpan.org/~danberr/:| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc:|
On Thu, Aug 22, 2013 at 01:35:29PM +0200, arun abhinay wrote:
Hi Daniel,
Thanks a lot for your reply.
We have mounted all the cgoup controllers at "/dev/cgroup_<controller>" as mentioned below. ls /dev/ | grep cgroup cgroup_cpu cgroup_cpuacct cgroup_cpuset cgroup_mem
On further debugging we dound that the "VCPU" directories are not being created for our domains and hence the comamnd "cpu-stats <domain>" is failing. Below is the snippet of logs collected with debug logs enabled for libvirt.
2013-08-21 15:03:35.162+0000: 1273: error : virCgroupMakeGroup:708 : Make group /machine/anusha.libvirt-qemu/vcpu0 2013-08-21 15:03:35.162+0000: 1273: error : virCgroupMakeGroup:729 : Makecontroller /dev/cgroup_cpu/machine/anusha.libvirt-qemu/vcpu0/ 2013-08-21 15:03:35.162+0000: 1273: error : virCgroupMakeGroup:745 : Failed to create controller cpu for group
Is that a debug log from when you started the guest, or from when you run the cpu-stats command. We need a full log from when you started the guest to try & diagnose this. Run libvirtd with LIBVIRT_LOG_FILTERS="1:cgroup" LIBVIRT_LOG_OUTPUTS=1:file:/tmp/libvirt.log /usr/sbin/libvirtd and start the guest, and then upload the log somewhere we can see it Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
Hi Daniel, Thanks for your immidiate reply. The starting of virtual guest was ok on our machine. We are facing this issue while executing the "cpu-stats <domain name>" command. I have collected the debug logs as you suggested and attaching the same. <<libvirt_testing_starting_VG.log >> --> Log collected while starting the virtual guest <<libvirt_testing_cpu_stats.log >> ---> Log collected while executing cpu-stats comamnd <<CLA_new-0.xml >> ---> Domian.xml used for configuration of virtual guest Can you please check and let us know your opinion. Thanks Abhinay On Thu, Aug 22, 2013 at 1:43 PM, Daniel P. Berrange <berrange@redhat.com>wrote:
Hi Daniel,
Thanks a lot for your reply.
We have mounted all the cgoup controllers at "/dev/cgroup_<controller>" as mentioned below. ls /dev/ | grep cgroup cgroup_cpu cgroup_cpuacct cgroup_cpuset cgroup_mem
On further debugging we dound that the "VCPU" directories are not being created for our domains and hence the comamnd "cpu-stats <domain>" is failing. Below is the snippet of logs collected with debug logs enabled for
On Thu, Aug 22, 2013 at 01:35:29PM +0200, arun abhinay wrote: libvirt.
2013-08-21 15:03:35.162+0000: 1273: error : virCgroupMakeGroup:708 : Make group /machine/anusha.libvirt-qemu/vcpu0 2013-08-21 15:03:35.162+0000: 1273: error : virCgroupMakeGroup:729 : Makecontroller /dev/cgroup_cpu/machine/anusha.libvirt-qemu/vcpu0/ 2013-08-21 15:03:35.162+0000: 1273: error : virCgroupMakeGroup:745 :
Failed
to create controller cpu for group
Is that a debug log from when you started the guest, or from when you run the cpu-stats command. We need a full log from when you started the guest to try & diagnose this.
Run libvirtd with
LIBVIRT_LOG_FILTERS="1:cgroup" LIBVIRT_LOG_OUTPUTS=1:file:/tmp/libvirt.log /usr/sbin/libvirtd
and start the guest, and then upload the log somewhere we can see it
Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/:| |: http://libvirt.org -o- http://virt-manager.org:| |: http://autobuild.org -o- http://search.cpan.org/~danberr/:| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc:|
On Thu, Aug 22, 2013 at 03:04:41PM +0200, arun abhinay wrote:
Hi Daniel,
Thanks for your immidiate reply. The starting of virtual guest was ok on our machine. We are facing this issue while executing the "cpu-stats <domain name>" command. I have collected the debug logs as you suggested and attaching the same.
The cpu-stats command relies on the vCPU cgroups existing. These are created when the guest starts. So the fact that cpu-stats can't find the cgroups shows that something failed during startup. Your log file shows this: 2013-08-22 12:45:44.326+0000: 17554: warning : qemuSetupCgroupForVcpu:953 : Unable to get vcpus' pids.^M which is the root cause of your problems. If libvirt can't find the vcpu PIDs, this suggests that your guests are running using QEMU TCG instead of KVM. Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
Hi Daniel, Thanks a lot for providing us your pointers. From the domain.xml(attached in my previous mail) which we have used for defining virtual guest i can see below lines <emulator>/usr/local/bin/qemu-system-x86_64</emulator> We were using same domain.xml in eralier libvirt version and the execution of cpu_stats command was successful. Now if we have to get "cpu_stats" command working using QEMU emulator on libvirt can you suggest what can we do. I have found below links on google while checking for the vcpu pid's error https://www.redhat.com/archives/libvir-list/2013-June/msg00214.htm<https://www.redhat.com/archives/libvir-list/2013-June/msg00214.html> <https://www.redhat.com/archives/libvir-list/2013-June/msg00385.html> https://www.redhat.com/archives/libvir-list/2012-August/msg01405.html https://www.redhat.com/archives/libvir-list/2012-August/msg01413.html Can you let us know if any of the above patches fix our issue. Thanks Abhinay On Thu, Aug 22, 2013 at 3:19 PM, Daniel P. Berrange <berrange@redhat.c om> wrote:
On Thu, Aug 22, 2013 at 03:04:41PM +0200, arun abhinay wrote:
Hi Daniel,
Thanks for your immidiate reply. The starting of virtual guest was ok on our machine. We are facing this issue while executing the "cpu-stats <domain name>" command. I have collected the debug logs as you suggested and attaching the same.
The cpu-stats command relies on the vCPU cgroups existing. These are created when the guest starts. So the fact that cpu-stats can't find the cgroups shows that something failed during startup.
Your log file shows this:
2013-08-22 12:45:44.326+0000: 17554: warning : qemuSetupCgroupForVcpu:953 : Unable to get vcpus' pids.^M
which is the root cause of your problems.
If libvirt can't find the vcpu PIDs, this suggests that your guests are running using QEMU TCG instead of KVM.
Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/:| |: http://libvirt.org -o- http://virt-manager.org:| |: http://autobuild.org -o- http://search.cpan.org/~danberr/:| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc:|
Hi Daniel, Adding to trailing mail. We are currently using below qemu version on our Hardware qemu-kvm-0.14.1-1 Will there be any change to test with the newer version qemu. Thanks Abhinay On Thu, Aug 22, 2013 at 5:44 PM, arun abhinay <arun.abhinay@gmail.com>wrote:
Hi Daniel,
Thanks a lot for providing us your pointers. From the domain.xml(attached in my previous mail) which we have used for defining virtual guest i can see below lines <emulator>/usr/local/bin/qemu-system-x86_64</emulator>
We were using same domain.xml in eralier libvirt version and the execution of cpu_stats command was successful. Now if we have to get "cpu_stats" command working using QEMU emulator on libvirt can you suggest what can we do.
I have found below links on google while checking for the vcpu pid's error https://www.redhat.com/archives/libvir-list/2013-June/msg00214.htm<https://www.redhat.com/archives/libvir-list/2013-June/msg00214.html> <https://www.redhat.com/archives/libvir-list/2013-June/msg00385.html> https://www.redhat.com/archives/libvir-list/2012-August/msg01405.html https://www.redhat.com/archives/libvir-list/2012-August/msg01413.html
Can you let us know if any of the above patches fix our issue.
Thanks Abhinay On Thu, Aug 22, 2013 at 3:19 PM, Daniel P. Berrange <berrange@redhat.c om> wrote:
On Thu, Aug 22, 2013 at 03:04:41PM +0200, arun abhinay wrote:
Hi Daniel,
Thanks for your immidiate reply. The starting of virtual guest was ok on our machine. We are facing this issue while executing the "cpu-stats <domain name>" command. I have collected the debug logs as you suggested and attaching the same.
The cpu-stats command relies on the vCPU cgroups existing. These are created when the guest starts. So the fact that cpu-stats can't find the cgroups shows that something failed during startup.
Your log file shows this:
2013-08-22 12:45:44.326+0000: 17554: warning : qemuSetupCgroupForVcpu:953 : Unable to get vcpus' pids.^M
which is the root cause of your problems.
If libvirt can't find the vcpu PIDs, this suggests that your guests are running using QEMU TCG instead of KVM.
Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
participants (2)
-
arun abhinay -
Daniel P. Berrange