[libvirt] the <filesystem> tag (WAS: simple LXC/libvirt busybox container (Unable to get cgroup))
wow! i was starting to think i would never get past this problem! [root@PHS-001 ~]# echo $VIRSH_DEFAULT_CONNECT_URI lxc:/// [root@PHS-001 ~]# virsh create /vps/def/exec/sys/arch-nano.xml Domain arch-nano created from /vps/def/exec/sys/arch-nano.xml [root@PHS-001 ~]# virsh console arch-nano Connected to domain arch-nano Escape character is ^] # [root@PHS-001 ~]# grep cgroup /proc/mounts none /cgroup cgroup rw,relatime,devices,memory,cpuacct,cpu 0 0 everything seems to be working fine now after disabling "ns" from cgroup mount; thank you very much Ryota and Daniel for you time and assistance. i must have sunk 40+ hours into this, but i learned a tremendous amount, so nothing is lost. :-) this is somewhat of a digression, and i can submit a new thread if need be, but my last remaining question is about the <filesystem> tag/root mounting. i have not found any mention of the <filesystem> tag in the docs or how to work with it... i want to have libvirt mount a specified device (a btrfs subvolume) and use it for the root filesystem of the created LXC container. is there a way to do this? perhaps <source dev="/dev/sdb" options="subvol....."/>, or do i need to use the storage XML? i didnt see a way to use a storage/device definition as the rootfs. im eager to get working with python bindings, just trying to see it all work with XML first. thanks++ On Thu, Dec 10, 2009 at 6:45 AM, Ryota Ozaki <ozaki.ryota@gmail.com> wrote:
On Thu, Dec 10, 2009 at 9:36 PM, Daniel P. Berrange <berrange@redhat.com> wrote:
On Thu, Dec 10, 2009 at 09:26:39PM +0900, Ryota Ozaki wrote:
On Thu, Dec 10, 2009 at 9:03 PM, Daniel P. Berrange <berrange@redhat.com> wrote:
On Thu, Dec 10, 2009 at 02:22:37AM -0600, Tony Risinger wrote:
i'm trying to get even the simplest busybox container with libvirt+LXC with very limited success. I feel l am missing something supremely simple for me to be hung on this for weeks.
i dont see anything interesting in domain log, but getting this error from "LIBVIRT_DEBUG=1 libvirtd":
05:27:56.113: error : lxcDomainGetInfo:462 : internal error Unable to get cgroup for arch-nano 05:27:56.113: debug : virDomainFree:2004 : domain=0x81d8e68 05:27:56.113: debug : virUnrefDomain:422 : unref domain 0x81d8e68 arch-nano 1 05:27:56.113: debug : virReleaseDomain:376 : release domain 0x81d8e68 arch-nano 05:27:56.113: debug : virReleaseDomain:392 : unref connection 0x81dc0f0 2 05:27:56.113: debug : remoteSerializeError:141 : prog=536903814 ver=1 proc=16 type=1 serial=4, msg=internal error Unable to get cgroup for arch-nano
i've been using this root filesystem layout:
[root@PHS-001 arch-nano]# tree . |-- bin | |-- cat -> ../sbin/busybox | |-- chdir -> ../sbin/busybox | |-- chmod -> ../sbin/busybox | |-- ls -> ../sbin/busybox | |-- rm -> ../sbin/busybox | |-- sh -> ../sbin/busybox | `-- vi -> ../sbin/busybox |-- dev | `-- pts |-- etc |-- proc |-- sbin | |-- busybox | `-- init -> busybox `-- sys
all folders besides /bin and /sbin were created by libvirt. i tried using the /sbin/init script previously suggested:
#!/sbin/busybox sh
Sorry, my suggestion was wrong. I forgot that if you have #!/sbin/busybox it will attempt to execute the command matching the name of the script. So it will in fact try to run 'init', rather than 'sh'.
Just make the libvirt XML point directly to /bin/sh instead and it should work. I even tested it this time :-)
Hem, I still have a problem with ns subsystem enabled. Yes, I can launch a container however the cgroup hierarchy is wrong from libvirtd expecting like:
/: libvirtd --daemon /5345: /usr/libexec/libvirt_lxc --name
Daniel, could you confirm how about your cgroup hierarchy?
What you do mean by 'ns' subsystem ?
'ns' is one of functions of cgroups like such as devices, memory, cpu, etc. and it is enabled if you mount cgroup without any options that Tony is doing.
# grep cgroup /proc/mounts cgroup /dev/cgroups/cpu cgroup rw,relatime,cpuacct,cpu 0 0 cgroup /dev/cgroups/memory cgroup rw,relatime,memory 0 0 cgroup /dev/cgroups/devices cgroup rw,relatime,devices 0 0
Oh, you don't enable 'ns', so yes, things go fine in your environment.
# cat /proc/`pgrep libvirtd`/cgroup 32:devices:/sysdefault 16:memory:/sysdefault 12:cpuacct,cpu:/sysdefault
# cat /proc/`pgrep libvirt_lxc`/cgroup 32:devices:/sysdefault/libvirt/lxc/vm1 16:memory:/sysdefault/libvirt/lxc/vm1 12:cpuacct,cpu:/sysdefault/libvirt/lxc/vm1
And the process inside the contanier is PID 12309
# cat /proc/12309/cgroup 32:devices:/sysdefault/libvirt/lxc/vm1 16:memory:/sysdefault/libvirt/lxc/vm1 12:cpuacct,cpu:/sysdefault/libvirt/lxc/vm1
Which all appears to be correct to me
This is on a Fedora 12 host 2.6.31.6-145.fc12.i686.PAE with
CONFIG_UTS_NS=y CONFIG_IPC_NS=y CONFIG_USER_NS=y CONFIG_PID_NS=y CONFIG_NET_NS=y CONFIG_CGROUP_SCHED=y CONFIG_CGROUPS=y # CONFIG_CGROUP_DEBUG is not set CONFIG_CGROUP_NS=y
This is the function I'm mentioning.
ozaki-r
CONFIG_CGROUP_FREEZER=y CONFIG_CGROUP_DEVICE=y CONFIG_CGROUP_CPUACCT=y CONFIG_CGROUP_MEM_RES_CTLR=y CONFIG_CGROUP_MEM_RES_CTLR_SWAP=y CONFIG_NET_CLS_CGROUP=y
Regards, Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
On Thu, Dec 10, 2009 at 05:44:30PM -0600, Tony Risinger wrote:
wow! i was starting to think i would never get past this problem!
[root@PHS-001 ~]# echo $VIRSH_DEFAULT_CONNECT_URI lxc:///
[root@PHS-001 ~]# virsh create /vps/def/exec/sys/arch-nano.xml Domain arch-nano created from /vps/def/exec/sys/arch-nano.xml
[root@PHS-001 ~]# virsh console arch-nano Connected to domain arch-nano Escape character is ^] #
[root@PHS-001 ~]# grep cgroup /proc/mounts none /cgroup cgroup rw,relatime,devices,memory,cpuacct,cpu 0 0
everything seems to be working fine now after disabling "ns" from cgroup mount; thank you very much Ryota and Daniel for you time and assistance. i must have sunk 40+ hours into this, but i learned a tremendous amount, so nothing is lost. :-)
this is somewhat of a digression, and i can submit a new thread if need be, but my last remaining question is about the <filesystem> tag/root mounting. i have not found any mention of the <filesystem> tag in the docs or how to work with it... i want to have libvirt mount a specified device (a btrfs subvolume) and use it for the root filesystem of the created LXC container. is there a way to do this? perhaps <source dev="/dev/sdb" options="subvol....."/>, or do i need to use the storage XML? i didnt see a way to use a storage/device definition as the rootfs.
The plan is that the <filesystem> tag will eventually support 3 types of sources, a directory (which gets bind mounted into the root), a file (loopback device + mounted), or block device (directly mounted). Currently though we only support the first option. So if you have a block device, you should first mount it in your host OS, and then let LXC bind the mount point into the container. Regards, Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
That all sounds great. wanted to say libvirt has been very useful to my personal projects as well as the former company i worked for; thanks to everyone invloved, its a great product! on a final note, any chance there is a way to get libvirt to trigger an arbitrary script (like the network scripts) that could handle this mount for me? didnt see anything like that in docs but i figured i'd ask anyway. thanks again for all the great work. On Fri, Dec 11, 2009 at 4:20 AM, Daniel P. Berrange <berrange@redhat.com> wrote:
On Thu, Dec 10, 2009 at 05:44:30PM -0600, Tony Risinger wrote:
wow! i was starting to think i would never get past this problem!
[root@PHS-001 ~]# echo $VIRSH_DEFAULT_CONNECT_URI lxc:///
[root@PHS-001 ~]# virsh create /vps/def/exec/sys/arch-nano.xml Domain arch-nano created from /vps/def/exec/sys/arch-nano.xml
[root@PHS-001 ~]# virsh console arch-nano Connected to domain arch-nano Escape character is ^] #
[root@PHS-001 ~]# grep cgroup /proc/mounts none /cgroup cgroup rw,relatime,devices,memory,cpuacct,cpu 0 0
everything seems to be working fine now after disabling "ns" from cgroup mount; thank you very much Ryota and Daniel for you time and assistance. i must have sunk 40+ hours into this, but i learned a tremendous amount, so nothing is lost. :-)
this is somewhat of a digression, and i can submit a new thread if need be, but my last remaining question is about the <filesystem> tag/root mounting. i have not found any mention of the <filesystem> tag in the docs or how to work with it... i want to have libvirt mount a specified device (a btrfs subvolume) and use it for the root filesystem of the created LXC container. is there a way to do this? perhaps <source dev="/dev/sdb" options="subvol....."/>, or do i need to use the storage XML? i didnt see a way to use a storage/device definition as the rootfs.
The plan is that the <filesystem> tag will eventually support 3 types of sources, a directory (which gets bind mounted into the root), a file (loopback device + mounted), or block device (directly mounted). Currently though we only support the first option. So if you have a block device, you should first mount it in your host OS, and then let LXC bind the mount point into the container.
Regards, Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
participants (2)
-
Daniel P. Berrange -
Tony Risinger