[libvirt] simple LXC/libvirt busybox container (Unable to get cgroup)
i'm trying to get even the simplest busybox container with libvirt+LXC with very limited success. I feel l am missing something supremely simple for me to be hung on this for weeks. i dont see anything interesting in domain log, but getting this error from "LIBVIRT_DEBUG=1 libvirtd": 05:27:56.113: error : lxcDomainGetInfo:462 : internal error Unable to get cgroup for arch-nano 05:27:56.113: debug : virDomainFree:2004 : domain=0x81d8e68 05:27:56.113: debug : virUnrefDomain:422 : unref domain 0x81d8e68 arch-nano 1 05:27:56.113: debug : virReleaseDomain:376 : release domain 0x81d8e68 arch-nano 05:27:56.113: debug : virReleaseDomain:392 : unref connection 0x81dc0f0 2 05:27:56.113: debug : remoteSerializeError:141 : prog=536903814 ver=1 proc=16 type=1 serial=4, msg=internal error Unable to get cgroup for arch-nano i've been using this root filesystem layout: [root@PHS-001 arch-nano]# tree . |-- bin | |-- cat -> ../sbin/busybox | |-- chdir -> ../sbin/busybox | |-- chmod -> ../sbin/busybox | |-- ls -> ../sbin/busybox | |-- rm -> ../sbin/busybox | |-- sh -> ../sbin/busybox | `-- vi -> ../sbin/busybox |-- dev | `-- pts |-- etc |-- proc |-- sbin | |-- busybox | `-- init -> busybox `-- sys all folders besides /bin and /sbin were created by libvirt. i tried using the /sbin/init script previously suggested: #!/sbin/busybox sh but i get a similar results either way (script/symlink): 8173 ? Ss 0:00 /usr/lib/libvirt-git/libvirt_lxc --name arch-nano --console 11 --background 8175 pts/0 Ss+ 0:00 init 8177 ? Ss 0:00 init 8181 ? Zs 0:00 [init] <defunct> 8182 ? Zs 0:00 [init] <defunct> 8183 ? Zs 0:00 [init] <defunct> "busybox init" doc says that without an /etc/inittab, sh will be started on /dev/tty2... im am using this config: <domain type='lxc'> <name>arch-nano</name> <memory>500000</memory> <os> <type>exe</type> <init>/sbin/init</init> </os> <devices> <filesystem type='mount'> <source dir='/vps/dom/arch-nano'/> <target dir='/'/> </filesystem> </devices> </domain> on this config i tried removing the console/serial section altogether, but i attempted many configurations of serial/console, including changing the <target> to "2", in attempt to match busybox default. [root@PHS-001 arch-nano]# virsh -c lxc:/// console arch-nano error: Unable to get domain status error: internal error Unable to get cgroup for arch-nano [root@PHS-001 arch-nano]# mount | grep cgroup none on /cgroup type cgroup (rw) if anyone can please point out what i am doing wrong to not be able to move the root and get a console, i'd greatly appreciate it. ive been really stuck on this; i'd rather not write a bunch of scripts/wrappers for lxc-* tools when libvirt does it all splendidly already! libvirt 0.7.4+ kernel 2.6.32 (am i missing a CONFIG_*?) thanks for your time
On Thu, Dec 10, 2009 at 5:22 PM, Tony Risinger <sweetsinsemilla@gmail.com> wrote:
i'm trying to get even the simplest busybox container with libvirt+LXC with very limited success. I feel l am missing something supremely simple for me to be hung on this for weeks.
i dont see anything interesting in domain log, but getting this error from "LIBVIRT_DEBUG=1 libvirtd":
05:27:56.113: error : lxcDomainGetInfo:462 : internal error Unable to get cgroup for arch-nano 05:27:56.113: debug : virDomainFree:2004 : domain=0x81d8e68 05:27:56.113: debug : virUnrefDomain:422 : unref domain 0x81d8e68 arch-nano 1 05:27:56.113: debug : virReleaseDomain:376 : release domain 0x81d8e68 arch-nano 05:27:56.113: debug : virReleaseDomain:392 : unref connection 0x81dc0f0 2 05:27:56.113: debug : remoteSerializeError:141 : prog=536903814 ver=1 proc=16 type=1 serial=4, msg=internal error Unable to get cgroup for arch-nano
i've been using this root filesystem layout:
[root@PHS-001 arch-nano]# tree . |-- bin | |-- cat -> ../sbin/busybox | |-- chdir -> ../sbin/busybox | |-- chmod -> ../sbin/busybox | |-- ls -> ../sbin/busybox | |-- rm -> ../sbin/busybox | |-- sh -> ../sbin/busybox | `-- vi -> ../sbin/busybox |-- dev | `-- pts |-- etc |-- proc |-- sbin | |-- busybox | `-- init -> busybox `-- sys
all folders besides /bin and /sbin were created by libvirt. i tried using the /sbin/init script previously suggested:
#!/sbin/busybox sh
but i get a similar results either way (script/symlink):
8173 ? Ss 0:00 /usr/lib/libvirt-git/libvirt_lxc --name arch-nano --console 11 --background 8175 pts/0 Ss+ 0:00 init 8177 ? Ss 0:00 init 8181 ? Zs 0:00 [init] <defunct> 8182 ? Zs 0:00 [init] <defunct> 8183 ? Zs 0:00 [init] <defunct>
"busybox init" doc says that without an /etc/inittab, sh will be started on /dev/tty2... im am using this config:
<domain type='lxc'> <name>arch-nano</name> <memory>500000</memory> <os> <type>exe</type> <init>/sbin/init</init> </os> <devices> <filesystem type='mount'> <source dir='/vps/dom/arch-nano'/> <target dir='/'/> </filesystem> </devices> </domain>
on this config i tried removing the console/serial section altogether, but i attempted many configurations of serial/console, including changing the <target> to "2", in attempt to match busybox default.
[root@PHS-001 arch-nano]# virsh -c lxc:/// console arch-nano error: Unable to get domain status error: internal error Unable to get cgroup for arch-nano
[root@PHS-001 arch-nano]# mount | grep cgroup none on /cgroup type cgroup (rw)
if anyone can please point out what i am doing wrong to not be able to move the root and get a console, i'd greatly appreciate it. ive been really stuck on this; i'd rather not write a bunch of scripts/wrappers for lxc-* tools when libvirt does it all splendidly already!
I'm successfully using lxc via lbivirt. I doubt ns subsystem of cgroups because I usually disable it that is the difference between my configuration and yours and if I enable it I also get the same error. So could you try without ns? like: mount -t cgroup -o memory,devices,cpu,cpuacct none /cgroup I think libvirt lxc still contains a problem around ns subsystem. ozaki-r
libvirt 0.7.4+ kernel 2.6.32 (am i missing a CONFIG_*?)
thanks for your time
-- Libvir-list mailing list Libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
On Thu, Dec 10, 2009 at 02:22:37AM -0600, Tony Risinger wrote:
i'm trying to get even the simplest busybox container with libvirt+LXC with very limited success. I feel l am missing something supremely simple for me to be hung on this for weeks.
i dont see anything interesting in domain log, but getting this error from "LIBVIRT_DEBUG=1 libvirtd":
05:27:56.113: error : lxcDomainGetInfo:462 : internal error Unable to get cgroup for arch-nano 05:27:56.113: debug : virDomainFree:2004 : domain=0x81d8e68 05:27:56.113: debug : virUnrefDomain:422 : unref domain 0x81d8e68 arch-nano 1 05:27:56.113: debug : virReleaseDomain:376 : release domain 0x81d8e68 arch-nano 05:27:56.113: debug : virReleaseDomain:392 : unref connection 0x81dc0f0 2 05:27:56.113: debug : remoteSerializeError:141 : prog=536903814 ver=1 proc=16 type=1 serial=4, msg=internal error Unable to get cgroup for arch-nano
i've been using this root filesystem layout:
[root@PHS-001 arch-nano]# tree . |-- bin | |-- cat -> ../sbin/busybox | |-- chdir -> ../sbin/busybox | |-- chmod -> ../sbin/busybox | |-- ls -> ../sbin/busybox | |-- rm -> ../sbin/busybox | |-- sh -> ../sbin/busybox | `-- vi -> ../sbin/busybox |-- dev | `-- pts |-- etc |-- proc |-- sbin | |-- busybox | `-- init -> busybox `-- sys
all folders besides /bin and /sbin were created by libvirt. i tried using the /sbin/init script previously suggested:
#!/sbin/busybox sh
Sorry, my suggestion was wrong. I forgot that if you have #!/sbin/busybox it will attempt to execute the command matching the name of the script. So it will in fact try to run 'init', rather than 'sh'. Just make the libvirt XML point directly to /bin/sh instead and it should work. I even tested it this time :-)
but i get a similar results either way (script/symlink):
8173 ? Ss 0:00 /usr/lib/libvirt-git/libvirt_lxc --name arch-nano --console 11 --background 8175 pts/0 Ss+ 0:00 init 8177 ? Ss 0:00 init 8181 ? Zs 0:00 [init] <defunct> 8182 ? Zs 0:00 [init] <defunct> 8183 ? Zs 0:00 [init] <defunct>
Yeah this is what I see too, when i have /sbin/init - changing it to /bin/sh works Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
On Thu, Dec 10, 2009 at 9:03 PM, Daniel P. Berrange <berrange@redhat.com> wrote:
On Thu, Dec 10, 2009 at 02:22:37AM -0600, Tony Risinger wrote:
i'm trying to get even the simplest busybox container with libvirt+LXC with very limited success. I feel l am missing something supremely simple for me to be hung on this for weeks.
i dont see anything interesting in domain log, but getting this error from "LIBVIRT_DEBUG=1 libvirtd":
05:27:56.113: error : lxcDomainGetInfo:462 : internal error Unable to get cgroup for arch-nano 05:27:56.113: debug : virDomainFree:2004 : domain=0x81d8e68 05:27:56.113: debug : virUnrefDomain:422 : unref domain 0x81d8e68 arch-nano 1 05:27:56.113: debug : virReleaseDomain:376 : release domain 0x81d8e68 arch-nano 05:27:56.113: debug : virReleaseDomain:392 : unref connection 0x81dc0f0 2 05:27:56.113: debug : remoteSerializeError:141 : prog=536903814 ver=1 proc=16 type=1 serial=4, msg=internal error Unable to get cgroup for arch-nano
i've been using this root filesystem layout:
[root@PHS-001 arch-nano]# tree . |-- bin | |-- cat -> ../sbin/busybox | |-- chdir -> ../sbin/busybox | |-- chmod -> ../sbin/busybox | |-- ls -> ../sbin/busybox | |-- rm -> ../sbin/busybox | |-- sh -> ../sbin/busybox | `-- vi -> ../sbin/busybox |-- dev | `-- pts |-- etc |-- proc |-- sbin | |-- busybox | `-- init -> busybox `-- sys
all folders besides /bin and /sbin were created by libvirt. i tried using the /sbin/init script previously suggested:
#!/sbin/busybox sh
Sorry, my suggestion was wrong. I forgot that if you have #!/sbin/busybox it will attempt to execute the command matching the name of the script. So it will in fact try to run 'init', rather than 'sh'.
Just make the libvirt XML point directly to /bin/sh instead and it should work. I even tested it this time :-)
Hem, I still have a problem with ns subsystem enabled. Yes, I can launch a container however the cgroup hierarchy is wrong from libvirtd expecting like: /: libvirtd --daemon /5345: /usr/libexec/libvirt_lxc --name Daniel, could you confirm how about your cgroup hierarchy? ozaki-r
but i get a similar results either way (script/symlink):
8173 ? Ss 0:00 /usr/lib/libvirt-git/libvirt_lxc --name arch-nano --console 11 --background 8175 pts/0 Ss+ 0:00 init 8177 ? Ss 0:00 init 8181 ? Zs 0:00 [init] <defunct> 8182 ? Zs 0:00 [init] <defunct> 8183 ? Zs 0:00 [init] <defunct>
Yeah this is what I see too, when i have /sbin/init - changing it to /bin/sh works
Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
-- Libvir-list mailing list Libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
On Thu, Dec 10, 2009 at 09:26:39PM +0900, Ryota Ozaki wrote:
On Thu, Dec 10, 2009 at 9:03 PM, Daniel P. Berrange <berrange@redhat.com> wrote:
On Thu, Dec 10, 2009 at 02:22:37AM -0600, Tony Risinger wrote:
i'm trying to get even the simplest busybox container with libvirt+LXC with very limited success. I feel l am missing something supremely simple for me to be hung on this for weeks.
i dont see anything interesting in domain log, but getting this error from "LIBVIRT_DEBUG=1 libvirtd":
05:27:56.113: error : lxcDomainGetInfo:462 : internal error Unable to get cgroup for arch-nano 05:27:56.113: debug : virDomainFree:2004 : domain=0x81d8e68 05:27:56.113: debug : virUnrefDomain:422 : unref domain 0x81d8e68 arch-nano 1 05:27:56.113: debug : virReleaseDomain:376 : release domain 0x81d8e68 arch-nano 05:27:56.113: debug : virReleaseDomain:392 : unref connection 0x81dc0f0 2 05:27:56.113: debug : remoteSerializeError:141 : prog=536903814 ver=1 proc=16 type=1 serial=4, msg=internal error Unable to get cgroup for arch-nano
i've been using this root filesystem layout:
[root@PHS-001 arch-nano]# tree . |-- bin | |-- cat -> ../sbin/busybox | |-- chdir -> ../sbin/busybox | |-- chmod -> ../sbin/busybox | |-- ls -> ../sbin/busybox | |-- rm -> ../sbin/busybox | |-- sh -> ../sbin/busybox | `-- vi -> ../sbin/busybox |-- dev | `-- pts |-- etc |-- proc |-- sbin | |-- busybox | `-- init -> busybox `-- sys
all folders besides /bin and /sbin were created by libvirt. i tried using the /sbin/init script previously suggested:
#!/sbin/busybox sh
Sorry, my suggestion was wrong. I forgot that if you have #!/sbin/busybox it will attempt to execute the command matching the name of the script. So it will in fact try to run 'init', rather than 'sh'.
Just make the libvirt XML point directly to /bin/sh instead and it should work. I even tested it this time :-)
Hem, I still have a problem with ns subsystem enabled. Yes, I can launch a container however the cgroup hierarchy is wrong from libvirtd expecting like:
/: libvirtd --daemon /5345: /usr/libexec/libvirt_lxc --name
Daniel, could you confirm how about your cgroup hierarchy?
What you do mean by 'ns' subsystem ? # grep cgroup /proc/mounts cgroup /dev/cgroups/cpu cgroup rw,relatime,cpuacct,cpu 0 0 cgroup /dev/cgroups/memory cgroup rw,relatime,memory 0 0 cgroup /dev/cgroups/devices cgroup rw,relatime,devices 0 0 # cat /proc/`pgrep libvirtd`/cgroup 32:devices:/sysdefault 16:memory:/sysdefault 12:cpuacct,cpu:/sysdefault # cat /proc/`pgrep libvirt_lxc`/cgroup 32:devices:/sysdefault/libvirt/lxc/vm1 16:memory:/sysdefault/libvirt/lxc/vm1 12:cpuacct,cpu:/sysdefault/libvirt/lxc/vm1 And the process inside the contanier is PID 12309 # cat /proc/12309/cgroup 32:devices:/sysdefault/libvirt/lxc/vm1 16:memory:/sysdefault/libvirt/lxc/vm1 12:cpuacct,cpu:/sysdefault/libvirt/lxc/vm1 Which all appears to be correct to me This is on a Fedora 12 host 2.6.31.6-145.fc12.i686.PAE with CONFIG_UTS_NS=y CONFIG_IPC_NS=y CONFIG_USER_NS=y CONFIG_PID_NS=y CONFIG_NET_NS=y CONFIG_CGROUP_SCHED=y CONFIG_CGROUPS=y # CONFIG_CGROUP_DEBUG is not set CONFIG_CGROUP_NS=y CONFIG_CGROUP_FREEZER=y CONFIG_CGROUP_DEVICE=y CONFIG_CGROUP_CPUACCT=y CONFIG_CGROUP_MEM_RES_CTLR=y CONFIG_CGROUP_MEM_RES_CTLR_SWAP=y CONFIG_NET_CLS_CGROUP=y Regards, Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
On Thu, Dec 10, 2009 at 9:36 PM, Daniel P. Berrange <berrange@redhat.com> wrote:
On Thu, Dec 10, 2009 at 09:26:39PM +0900, Ryota Ozaki wrote:
On Thu, Dec 10, 2009 at 9:03 PM, Daniel P. Berrange <berrange@redhat.com> wrote:
On Thu, Dec 10, 2009 at 02:22:37AM -0600, Tony Risinger wrote:
i'm trying to get even the simplest busybox container with libvirt+LXC with very limited success. I feel l am missing something supremely simple for me to be hung on this for weeks.
i dont see anything interesting in domain log, but getting this error from "LIBVIRT_DEBUG=1 libvirtd":
05:27:56.113: error : lxcDomainGetInfo:462 : internal error Unable to get cgroup for arch-nano 05:27:56.113: debug : virDomainFree:2004 : domain=0x81d8e68 05:27:56.113: debug : virUnrefDomain:422 : unref domain 0x81d8e68 arch-nano 1 05:27:56.113: debug : virReleaseDomain:376 : release domain 0x81d8e68 arch-nano 05:27:56.113: debug : virReleaseDomain:392 : unref connection 0x81dc0f0 2 05:27:56.113: debug : remoteSerializeError:141 : prog=536903814 ver=1 proc=16 type=1 serial=4, msg=internal error Unable to get cgroup for arch-nano
i've been using this root filesystem layout:
[root@PHS-001 arch-nano]# tree . |-- bin | |-- cat -> ../sbin/busybox | |-- chdir -> ../sbin/busybox | |-- chmod -> ../sbin/busybox | |-- ls -> ../sbin/busybox | |-- rm -> ../sbin/busybox | |-- sh -> ../sbin/busybox | `-- vi -> ../sbin/busybox |-- dev | `-- pts |-- etc |-- proc |-- sbin | |-- busybox | `-- init -> busybox `-- sys
all folders besides /bin and /sbin were created by libvirt. i tried using the /sbin/init script previously suggested:
#!/sbin/busybox sh
Sorry, my suggestion was wrong. I forgot that if you have #!/sbin/busybox it will attempt to execute the command matching the name of the script. So it will in fact try to run 'init', rather than 'sh'.
Just make the libvirt XML point directly to /bin/sh instead and it should work. I even tested it this time :-)
Hem, I still have a problem with ns subsystem enabled. Yes, I can launch a container however the cgroup hierarchy is wrong from libvirtd expecting like:
/: libvirtd --daemon /5345: /usr/libexec/libvirt_lxc --name
Daniel, could you confirm how about your cgroup hierarchy?
What you do mean by 'ns' subsystem ?
'ns' is one of functions of cgroups like such as devices, memory, cpu, etc. and it is enabled if you mount cgroup without any options that Tony is doing.
# grep cgroup /proc/mounts cgroup /dev/cgroups/cpu cgroup rw,relatime,cpuacct,cpu 0 0 cgroup /dev/cgroups/memory cgroup rw,relatime,memory 0 0 cgroup /dev/cgroups/devices cgroup rw,relatime,devices 0 0
Oh, you don't enable 'ns', so yes, things go fine in your environment.
# cat /proc/`pgrep libvirtd`/cgroup 32:devices:/sysdefault 16:memory:/sysdefault 12:cpuacct,cpu:/sysdefault
# cat /proc/`pgrep libvirt_lxc`/cgroup 32:devices:/sysdefault/libvirt/lxc/vm1 16:memory:/sysdefault/libvirt/lxc/vm1 12:cpuacct,cpu:/sysdefault/libvirt/lxc/vm1
And the process inside the contanier is PID 12309
# cat /proc/12309/cgroup 32:devices:/sysdefault/libvirt/lxc/vm1 16:memory:/sysdefault/libvirt/lxc/vm1 12:cpuacct,cpu:/sysdefault/libvirt/lxc/vm1
Which all appears to be correct to me
This is on a Fedora 12 host 2.6.31.6-145.fc12.i686.PAE with
CONFIG_UTS_NS=y CONFIG_IPC_NS=y CONFIG_USER_NS=y CONFIG_PID_NS=y CONFIG_NET_NS=y CONFIG_CGROUP_SCHED=y CONFIG_CGROUPS=y # CONFIG_CGROUP_DEBUG is not set CONFIG_CGROUP_NS=y
This is the function I'm mentioning. ozaki-r
CONFIG_CGROUP_FREEZER=y CONFIG_CGROUP_DEVICE=y CONFIG_CGROUP_CPUACCT=y CONFIG_CGROUP_MEM_RES_CTLR=y CONFIG_CGROUP_MEM_RES_CTLR_SWAP=y CONFIG_NET_CLS_CGROUP=y
Regards, Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
On Thu, Dec 10, 2009 at 09:45:03PM +0900, Ryota Ozaki wrote:
On Thu, Dec 10, 2009 at 9:36 PM, Daniel P. Berrange <berrange@redhat.com> wrote:
What you do mean by 'ns' subsystem ?
'ns' is one of functions of cgroups like such as devices, memory, cpu, etc. and it is enabled if you mount cgroup without any options that Tony is doing.
# grep cgroup /proc/mounts cgroup /dev/cgroups/cpu cgroup rw,relatime,cpuacct,cpu 0 0 cgroup /dev/cgroups/memory cgroup rw,relatime,memory 0 0 cgroup /dev/cgroups/devices cgroup rw,relatime,devices 0 0
Oh, you don't enable 'ns', so yes, things go fine in your environment.
I added the 'ns' controller as another mount poiint, and weirdly everything still worked. It was only when i rebooted and mounted everything at the same mount point that it stopped working. I'll investigate why this is & try and come up with a fix for 'ns' Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
participants (3)
-
Daniel P. Berrange -
Ryota Ozaki -
Tony Risinger