[libvirt] can't remove cgroup files of the domain, thanks!

HI all: I am now working with libvirt 1.0.0 and come up a problem when I try to destroy a domain, The libvirtd just waiting rmdir command to return for hours, but it seem to never to return .. The stack is : 0x00007fa752261287 in rmdir () from /lib/x86_64-linux-gnu/libc.so.6 (gdb) bt #0 0x00007fa752261287 in rmdir () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007fa7533f44ad in virCgroupRemoveRecursively (grppath=0x138dca0 "/sys/fs/cgroup/libvirt/qemu/win7_gl//emulator") at util/cgroup.c:728 #2 0x00007fa7533f443a in virCgroupRemoveRecursively (grppath=0x138c5b0 "/sys/fs/cgroup/libvirt/qemu/win7_gl/") at util/cgroup.c:720 #3 0x00007fa7533f45f4 in virCgroupRemove (group=0x138c8d0) at util/cgroup.c:773 #4 0x00000000004d74ba in qemuRemoveCgroup (driver=0x7fa744010170, vm=0x7fa744005eb0, quiet=0) at qemu/qemu_cgroup.c:760 #5 0x00000000004c9813 in qemuProcessStop (driver=0x7fa744010170, vm=0x7fa744005eb0, reason=VIR_DOMAIN_SHUTOFF_SHUTDOWN, flags=0) at qemu/qemu_process.c:4159 #6 0x00000000004becc4 in qemuProcessHandleMonitorEOF (mon=0x7fa72c008e30, vm=0x7fa744005eb0) at qemu/qemu_process.c:312 #7 0x00000000004a0e14 in qemuMonitorIO (watch=332, fd=22, events=0, opaque=0x7fa72c008e30) at qemu/qemu_monitor.c:641 #8 0x00007fa7533f85b7 in virEventPollDispatchHandles (nfds=11, fds=0x13949e0) at util/event_poll.c:485 #9 0x00007fa7533f8dd8 in virEventPollRunOnce () at util/event_poll.c:632 #10 0x00007fa7533f6fd8 in virEventRunDefaultImpl () at util/event.c:247 #11 0x00007fa75351cbb9 in virNetServerRun (srv=0x1389ea0) at rpc/virnetserver.c:1004 #12 0x00000000004222ac in main (argc=2, argv=0x7fff95b34878) at libvirtd.c:1354 (gdb) root@vmc-22:~# lsof | grep cgroup libvirtd 2435 root 17r DIR 0,19 0 326488 /sys/fs/cgroup/libvirt/qemu/win7_gl root@vmc-22:~# root@vmc-22:~# root@vmc-22:~# lsattr /sys/fs/cgroup/libvirt/qemu/win7_gl lsattr: Inappropriate ioctl for device While reading flags on /sys/fs/cgroup/libvirt/qemu/win7_gl/emulator lsattr: Inappropriate ioctl for device While reading flags on /sys/fs/cgroup/libvirt/qemu/win7_gl/vcpu1 lsattr: Inappropriate ioctl for device While reading flags on /sys/fs/cgroup/libvirt/qemu/win7_gl/vcpu0 lsattr: Inappropriate ioctl for device While reading flags on /sys/fs/cgroup/libvirt/qemu/win7_gl/cpuset.memory_spread_slab lsattr: Inappropriate ioctl for device While reading flags on /sys/fs/cgroup/libvirt/qemu/win7_gl/cpuset.memory_spread_page lsattr: Inappropriate ioctl for device While reading flags on /sys/fs/cgroup/libvirt/qemu/win7_gl/cpuset.memory_pressure lsattr: Inappropriate ioctl for device While reading flags on /sys/fs/cgroup/libvirt/qemu/win7_gl/cpuset.memory_migrate lsattr: Inappropriate ioctl for device While reading flags on /sys/fs/cgroup/libvirt/qemu/win7_gl/cpuset.sched_relax_domain_level lsattr: Inappropriate ioctl for device While reading flags on /sys/fs/cgroup/libvirt/qemu/win7_gl/cpuset.sched_load_balance lsattr: Inappropriate ioctl for device While reading flags on /sys/fs/cgroup/libvirt/qemu/win7_gl/cpuset.mem_hardwall lsattr: Inappropriate ioctl for device While reading flags on /sys/fs/cgroup/libvirt/qemu/win7_gl/cpuset.mem_exclusive lsattr: Inappropriate ioctl for device While reading flags on /sys/fs/cgroup/libvirt/qemu/win7_gl/cpuset.cpu_exclusive lsattr: Inappropriate ioctl for device While reading flags on /sys/fs/cgroup/libvirt/qemu/win7_gl/cpuset.mems lsattr: Inappropriate ioctl for device While reading flags on /sys/fs/cgroup/libvirt/qemu/win7_gl/cpuset.cpus lsattr: Inappropriate ioctl for device While reading flags on /sys/fs/cgroup/libvirt/qemu/win7_gl/cpu.rt_period_us lsattr: Inappropriate ioctl for device While reading flags on /sys/fs/cgroup/libvirt/qemu/win7_gl/cpu.rt_runtime_us lsattr: Inappropriate ioctl for device While reading flags on /sys/fs/cgroup/libvirt/qemu/win7_gl/cpu.stat lsattr: Inappropriate ioctl for device While reading flags on /sys/fs/cgroup/libvirt/qemu/win7_gl/cpu.cfs_period_us lsattr: Inappropriate ioctl for device While reading flags on /sys/fs/cgroup/libvirt/qemu/win7_gl/cpu.cfs_quota_us lsattr: Inappropriate ioctl for device While reading flags on /sys/fs/cgroup/libvirt/qemu/win7_gl/cpu.shares lsattr: Inappropriate ioctl for device While reading flags on /sys/fs/cgroup/libvirt/qemu/win7_gl/cpuacct.stat lsattr: Inappropriate ioctl for device While reading flags on /sys/fs/cgroup/libvirt/qemu/win7_gl/cpuacct.usage_percpu lsattr: Inappropriate ioctl for device While reading flags on /sys/fs/cgroup/libvirt/qemu/win7_gl/cpuacct.usage lsattr: Inappropriate ioctl for device While reading flags on /sys/fs/cgroup/libvirt/qemu/win7_gl/memory.memsw.failcnt lsattr: Inappropriate ioctl for device While reading flags on /sys/fs/cgroup/libvirt/qemu/win7_gl/memory.memsw.limit_in_bytes lsattr: Inappropriate ioctl for device While reading flags on /sys/fs/cgroup/libvirt/qemu/win7_gl/memory.memsw.max_usage_in_bytes lsattr: Inappropriate ioctl for device While reading flags on /sys/fs/cgroup/libvirt/qemu/win7_gl/memory.memsw.usage_in_bytes lsattr: Inappropriate ioctl for device While reading flags on /sys/fs/cgroup/libvirt/qemu/win7_gl/memory.numa_stat lsattr: Inappropriate ioctl for device While reading flags on /sys/fs/cgroup/libvirt/qemu/win7_gl/memory.oom_control lsattr: Inappropriate ioctl for device While reading flags on /sys/fs/cgroup/libvirt/qemu/win7_gl/memory.move_charge_at_immigrate lsattr: Inappropriate ioctl for device While reading flags on /sys/fs/cgroup/libvirt/qemu/win7_gl/memory.swappiness lsattr: Inappropriate ioctl for device While reading flags on /sys/fs/cgroup/libvirt/qemu/win7_gl/memory.use_hierarchy lsattr: Inappropriate ioctl for device While reading flags on /sys/fs/cgroup/libvirt/qemu/win7_gl/memory.force_empty lsattr: Inappropriate ioctl for device While reading flags on /sys/fs/cgroup/libvirt/qemu/win7_gl/memory.stat lsattr: Inappropriate ioctl for device While reading flags on /sys/fs/cgroup/libvirt/qemu/win7_gl/memory.failcnt lsattr: Inappropriate ioctl for device While reading flags on /sys/fs/cgroup/libvirt/qemu/win7_gl/memory.soft_limit_in_bytes lsattr: Inappropriate ioctl for device While reading flags on /sys/fs/cgroup/libvirt/qemu/win7_gl/memory.limit_in_bytes lsattr: Inappropriate ioctl for device While reading flags on /sys/fs/cgroup/libvirt/qemu/win7_gl/memory.max_usage_in_bytes lsattr: Inappropriate ioctl for device While reading flags on /sys/fs/cgroup/libvirt/qemu/win7_gl/memory.usage_in_bytes lsattr: Inappropriate ioctl for device While reading flags on /sys/fs/cgroup/libvirt/qemu/win7_gl/devices.list lsattr: Inappropriate ioctl for device While reading flags on /sys/fs/cgroup/libvirt/qemu/win7_gl/devices.deny lsattr: Inappropriate ioctl for device While reading flags on /sys/fs/cgroup/libvirt/qemu/win7_gl/devices.allow lsattr: Inappropriate ioctl for device While reading flags on /sys/fs/cgroup/libvirt/qemu/win7_gl/freezer.state lsattr: Inappropriate ioctl for device While reading flags on /sys/fs/cgroup/libvirt/qemu/win7_gl/blkio.throttle.io_serviced lsattr: Inappropriate ioctl for device While reading flags on /sys/fs/cgroup/libvirt/qemu/win7_gl/blkio.throttle.io_service_bytes lsattr: Inappropriate ioctl for device While reading flags on /sys/fs/cgroup/libvirt/qemu/win7_gl/blkio.throttle.write_iops_device lsattr: Inappropriate ioctl for device While reading flags on /sys/fs/cgroup/libvirt/qemu/win7_gl/blkio.throttle.read_iops_device lsattr: Inappropriate ioctl for device While reading flags on /sys/fs/cgroup/libvirt/qemu/win7_gl/blkio.throttle.write_bps_device lsattr: Inappropriate ioctl for device While reading flags on /sys/fs/cgroup/libvirt/qemu/win7_gl/blkio.throttle.read_bps_device lsattr: Inappropriate ioctl for device While reading flags on /sys/fs/cgroup/libvirt/qemu/win7_gl/blkio.reset_stats lsattr: Inappropriate ioctl for device While reading flags on /sys/fs/cgroup/libvirt/qemu/win7_gl/blkio.io_queued lsattr: Inappropriate ioctl for device While reading flags on /sys/fs/cgroup/libvirt/qemu/win7_gl/blkio.io_merged lsattr: Inappropriate ioctl for device While reading flags on /sys/fs/cgroup/libvirt/qemu/win7_gl/blkio.io_wait_time lsattr: Inappropriate ioctl for device While reading flags on /sys/fs/cgroup/libvirt/qemu/win7_gl/blkio.io_service_time lsattr: Inappropriate ioctl for device While reading flags on /sys/fs/cgroup/libvirt/qemu/win7_gl/blkio.io_serviced lsattr: Inappropriate ioctl for device While reading flags on /sys/fs/cgroup/libvirt/qemu/win7_gl/blkio.io_service_bytes lsattr: Inappropriate ioctl for device While reading flags on /sys/fs/cgroup/libvirt/qemu/win7_gl/blkio.sectors lsattr: Inappropriate ioctl for device While reading flags on /sys/fs/cgroup/libvirt/qemu/win7_gl/blkio.time lsattr: Inappropriate ioctl for device While reading flags on /sys/fs/cgroup/libvirt/qemu/win7_gl/blkio.weight lsattr: Inappropriate ioctl for device While reading flags on /sys/fs/cgroup/libvirt/qemu/win7_gl/blkio.weight_device lsattr: Inappropriate ioctl for device While reading flags on /sys/fs/cgroup/libvirt/qemu/win7_gl/cgroup.clone_children lsattr: Inappropriate ioctl for device While reading flags on /sys/fs/cgroup/libvirt/qemu/win7_gl/cgroup.event_control lsattr: Inappropriate ioctl for device While reading flags on /sys/fs/cgroup/libvirt/qemu/win7_gl/notify_on_release lsattr: Inappropriate ioctl for device While reading flags on /sys/fs/cgroup/libvirt/qemu/win7_gl/cgroup.procs lsattr: Inappropriate ioctl for device While reading flags on /sys/fs/cgroup/libvirt/qemu/win7_gl/tasks root@vmc-22:~# when I use command "service libvirt-bin stop", I can stop libvirtd; but rmdir command still never return: root@vmc-22:~# rmdir /sys/fs/cgroup/libvirt/qemu/win7_gl//emulator ^C root@vmc-22:~# Is someone know this problem? How can I solve this problem? Thank you very much! ------------------------------------------------------------------------------------------------------------------------------------- ???????????????????????????????????????? ???????????????????????????????????????? ???????????????????????????????????????? ??? This e-mail and its attachments contain confidential information from H3C, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it!