On Wed, Aug 21, 2013 at 09:06:21AM +0200, arun abhinay wrote: > Hi, > > We have upgraded the libvirt version from libvirt-1.0.5 to libvirt-1.0.6-1 > on our linux machine. After this upgrade "virsh cpu-stats <domain>" command > is not working on our HardWare. > > *virsh # cpu-stats test > error: Failed to retrieve CPU statistics for domain 'test' > error: internal error error accessing cgroup cpuacct for vcpu* > ** > *# cat /proc/cgroups > #subsys_name hierarchy num_cgroups enabled > cpuset 3 4 1 > ns 0 1 1 > cpu 1 6 1 > cpuacct 7 3 1 > memory 2 3 1 > freezer 0 1 1 > net_cls 0 1 1 > blkio 0 1 1 > * > *# chkconfig --list | grep cg > cgroups_mount 0:off 1:on 2:on 3:on 4:on 5:on 6:off* > > We have also found that there is no "cgroup" directory created by > kenel under "/sys/fs/". We suspect that the 'cgroup' controllers are not > mounted properly and hence the issue is observed. While using the > libvirt-1.0.5 version below mentioned mount commands were sufficient to > mount cgroup controllers > > *mkdir -p /dev/cgroup_cpuacct > mount -t cgroup -o cpuacct cpuacct /dev/cgroup_cpuacct* > > As cgroups hieracrhy inside libvirtd has changed in libvirt-1.0.6-1, we Actually it changed in version 1.0.5 > feel that the above commands are now not sufficient. We feel we have to do > some more steps get the 'cgroups' mounted properly on our hardware. > > Since we are new to libvirt and can you please help us by providing some > pointers to fix the above issue and get cpu-stats working on our HardWare. > Since we have to fix this on top priority, can you please reply as soon as > possible. Please let us know if you need any further information from our > side. Libvirt will never attempt to mount any cgroups controllers. It delegates this task to the base OS. Historically the 'cgconfig' init script would be used. On distros which use systemd though, systemd will take care of this task. There's a bit more info here http://libvirt.org/cgroups.html Basically our recommendation is to mount all the controllers individually at /sys/fs/cgroup/<controllername> Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/:| |: http://libvirt.org -o- http://virt-manager.org:| |: http://autobuild.org -o- http://search.cpan.org/~danberr/:| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc:|

On Thu, Aug 22, 2013 at 01:35:29PM +0200, arun abhinay wrote: > Hi Daniel, > > Thanks a lot for your reply. > > We have mounted all the cgoup controllers at "/dev/cgroup_<controller>" as > mentioned below. > ls /dev/ | grep cgroup > cgroup_cpu cgroup_cpuacct cgroup_cpuset cgroup_mem > > On further debugging we dound that the "VCPU" directories are not being > created for our domains and hence the comamnd "cpu-stats <domain>" is > failing. > Below is the snippet of logs collected with debug logs enabled for libvirt. > > 2013-08-21 15:03:35.162+0000: 1273: error : virCgroupMakeGroup:708 : Make > group /machine/anusha.libvirt-qemu/vcpu0 > 2013-08-21 15:03:35.162+0000: 1273: error : virCgroupMakeGroup:729 : > Makecontroller /dev/cgroup_cpu/machine/anusha.libvirt-qemu/vcpu0/ > 2013-08-21 15:03:35.162+0000: 1273: error : virCgroupMakeGroup:745 : Failed > to create controller cpu for group Is that a debug log from when you started the guest, or from when you run the cpu-stats command. We need a full log from when you started the guest to try & diagnose this. Run libvirtd with LIBVIRT_LOG_FILTERS="1:cgroup" LIBVIRT_LOG_OUTPUTS=1:file:/tmp/libvirt.log /usr/sbin/libvirtd and start the guest, and then upload the log somewhere we can see it Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/:| |: http://libvirt.org -o- http://virt-manager.org:| |: http://autobuild.org -o- http://search.cpan.org/~danberr/:| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc:|

On Thu, Aug 22, 2013 at 03:04:41PM +0200, arun abhinay wrote: > Hi Daniel, > > Thanks for your immidiate reply. The starting of virtual guest was ok on > our machine. We are facing this issue while executing the "cpu-stats > <domain name>" command. > I have collected the debug logs as you suggested and attaching the same. The cpu-stats command relies on the vCPU cgroups existing. These are created when the guest starts. So the fact that cpu-stats can't find the cgroups shows that something failed during startup. Your log file shows this: 2013-08-22 12:45:44.326+0000: 17554: warning : qemuSetupCgroupForVcpu:953 : Unable to get vcpus' pids.^M which is the root cause of your problems. If libvirt can't find the vcpu PIDs, this suggests that your guests are running using QEMU TCG instead of KVM. Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/:| |: http://libvirt.org -o- http://virt-manager.org:| |: http://autobuild.org -o- http://search.cpan.org/~danberr/:| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc:|