[libvirt]: Not able to login to container when cgroup is enabled
Hi All, On F12 i was playing with libvirt support for container. I am seeing two issue over here: 1> Not able to create a container when start cgconfig services. 2> When i stop cgconfig, then able to create a container, but seeing few error messages to libvirtd daemon. 3> After creation of container through libvirt, i am not able to touch any file or directory. Attached: vm1.xml , through which i created container. I referred following link: http://libvirt.org/drvlxc.html http://openvz.org/pipermail/devel/2008-September/014314.html Detailed Bug Report: When i create a container without starting "cgconfig" service, i am able to create but getting some error message in libvirtd daemon: [root@mx3950 ~]# libvirtd 14:47:24.556: warning : qemudStartup:565 : Unable to create cgroup for driver: No such device or address 14:47:24.647: warning : lxcStartup:1462 : Unable to create cgroup for driver: No such device or address libvir: Linux Container error : failed to connect to client socket: No such file or directory libvir: Linux Container error : failed to connect to client socket: No such file or directory libvir: Linux Container error : failed to connect to client socket: No such file or directory But when i start cgconfig service, then i am not able to open console for my container. Below is the detailed steps which i executed. [root@mx3950 ~]# /etc/init.d/cgconfig stop Stopping cgconfig service: [ OK ] [root@mx3950 ~]# virsh -c lxc:/// define vm1.xml error: Failed to define domain from vm1.xml error: operation failed: domain 'vm1' is already defined with uuid ef558ab3-54ea-5b95-205d-8c0096ae9a4f [root@mx3950 ~]# virsh -c lxc:/// start vm1 Domain vm1 started [root@mx3950 ~]# virsh -c lxc:/// console vm1 Connected to domain vm1 Escape character is ^] sh-4.0# ps -ef UID PID PPID C STIME TTY TIME CMD root 1 0 0 15:23 pts/4 00:00:00 /bin/sh root 4 1 0 15:23 pts/4 00:00:00 ps -ef sh-4.0# touch rishi touch: cannot touch `rishi': Permission denied sh-4.0# exit exit [root@mx3950 ~]# /etc/init.d/cgconfig start Starting cgconfig service: [ OK ] [root@mx3950 ~]# virsh -c lxc:/// start vm1 Domain vm1 started [root@mx3950 ~]# virsh -c lxc:/// console vm1 Connected to domain vm1 Escape character is ^] 15:24:12.869: error : vshRunConsole:77 : unable to open tty /dev/pts/3: No such file or directory [root@mx3950 ~]# Package List: [root@mx3950 ~]# rpm -qa | grep -e libcgroup -e libvirt libcgroup-0.34-2.fc12.x86_64 libvirt-0.7.1-1.fc12.x86_64 libvirt-client-0.7.1-1.fc12.x86_64 libvirt-python-0.7.1-1.fc12.x86_64 [root@mx3950 ~]# uname -a Linux mx3950.in.ibm.com 2.6.31-14.fc12.x86_64 #1 SMP Tue Sep 15 03:48:57 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux
Hi Dan, can you help me here ? -Rishi Rishikesh wrote:
Hi All,
On F12 i was playing with libvirt support for container. I am seeing two issue over here: 1> Not able to create a container when start cgconfig services. 2> When i stop cgconfig, then able to create a container, but seeing few error messages to libvirtd daemon. 3> After creation of container through libvirt, i am not able to touch any file or directory.
Attached: vm1.xml , through which i created container. I referred following link:
http://libvirt.org/drvlxc.html http://openvz.org/pipermail/devel/2008-September/014314.html
Detailed Bug Report: When i create a container without starting "cgconfig" service, i am able to create but getting some error message in libvirtd daemon:
[root@mx3950 ~]# libvirtd 14:47:24.556: warning : qemudStartup:565 : Unable to create cgroup for driver: No such device or address 14:47:24.647: warning : lxcStartup:1462 : Unable to create cgroup for driver: No such device or address libvir: Linux Container error : failed to connect to client socket: No such file or directory libvir: Linux Container error : failed to connect to client socket: No such file or directory libvir: Linux Container error : failed to connect to client socket: No such file or directory
But when i start cgconfig service, then i am not able to open console for my container. Below is the detailed steps which i executed.
[root@mx3950 ~]# /etc/init.d/cgconfig stop Stopping cgconfig service: [ OK ] [root@mx3950 ~]# virsh -c lxc:/// define vm1.xml error: Failed to define domain from vm1.xml error: operation failed: domain 'vm1' is already defined with uuid ef558ab3-54ea-5b95-205d-8c0096ae9a4f
[root@mx3950 ~]# virsh -c lxc:/// start vm1 Domain vm1 started
[root@mx3950 ~]# virsh -c lxc:/// console vm1 Connected to domain vm1 Escape character is ^] sh-4.0# ps -ef UID PID PPID C STIME TTY TIME CMD root 1 0 0 15:23 pts/4 00:00:00 /bin/sh root 4 1 0 15:23 pts/4 00:00:00 ps -ef sh-4.0# touch rishi touch: cannot touch `rishi': Permission denied sh-4.0# exit exit
[root@mx3950 ~]# /etc/init.d/cgconfig start Starting cgconfig service: [ OK ] [root@mx3950 ~]# virsh -c lxc:/// start vm1 Domain vm1 started
[root@mx3950 ~]# virsh -c lxc:/// console vm1 Connected to domain vm1 Escape character is ^] 15:24:12.869: error : vshRunConsole:77 : unable to open tty /dev/pts/3: No such file or directory
[root@mx3950 ~]#
Package List: [root@mx3950 ~]# rpm -qa | grep -e libcgroup -e libvirt libcgroup-0.34-2.fc12.x86_64 libvirt-0.7.1-1.fc12.x86_64 libvirt-client-0.7.1-1.fc12.x86_64 libvirt-python-0.7.1-1.fc12.x86_64 [root@mx3950 ~]# uname -a Linux mx3950.in.ibm.com 2.6.31-14.fc12.x86_64 #1 SMP Tue Sep 15 03:48:57 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux
------------------------------------------------------------------------
-- Libvir-list mailing list Libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
R> can you help me here ? Probably not, I'm not familiar with cgconfig nor the latest cgroup support in libvirt. R> 14:47:24.556: warning : qemudStartup:565 : Unable to create cgroup R> for driver: No such device or address My guess is that this means you don't have the cgroup filesystem mounted, or it's mounted in a way that prevents libvirt from doing its work. IIRC, there are things you can do to the system that require a reboot to undo. If cgconfig is setting up the cgroups in such a way, you might need to reboot between stopping cgconfig and trying libvirt by itself. R> But when i start cgconfig service, then i am not able to open R> console for my container. Again, knowing nothing of cgconfig, I can only speculate that it's doing something to prevent accessing the console pty. I would push on the libcgroup folks to help figure out what it's doing (either by default or with your config) to prevent that and then go from there. -- Dan Smith IBM Linux Technology Center email: danms@us.ibm.com
On Wed, Oct 07, 2009 at 06:57:48AM -0700, Dan Smith wrote:
R> can you help me here ?
Probably not, I'm not familiar with cgconfig nor the latest cgroup support in libvirt.
R> 14:47:24.556: warning : qemudStartup:565 : Unable to create cgroup R> for driver: No such device or address
My guess is that this means you don't have the cgroup filesystem mounted, or it's mounted in a way that prevents libvirt from doing its work. IIRC, there are things you can do to the system that require a reboot to undo. If cgconfig is setting up the cgroups in such a way, you might need to reboot between stopping cgconfig and trying libvirt by itself.
Or the cgroups are being setup in funny fashion. Rishi, what does your config file look like?
R> But when i start cgconfig service, then i am not able to open R> console for my container.
Again, knowing nothing of cgconfig, I can only speculate that it's doing something to prevent accessing the console pty.
cgconfig does nothing apart from setup the cgroup filesystem as mentioned in the configuration files.
I would push on the libcgroup folks to help figure out what it's doing (either by default or with your config) to prevent that and then go from there.
Would appreciate any help from the libvirt folks who have worked on the cgroup support in libvirt. thanks, -- regards, Dhaval
On Wed, Oct 07, 2009 at 08:12:02PM +0530, Dhaval Giani wrote:
On Wed, Oct 07, 2009 at 06:57:48AM -0700, Dan Smith wrote:
R> can you help me here ?
Probably not, I'm not familiar with cgconfig nor the latest cgroup support in libvirt.
R> 14:47:24.556: warning : qemudStartup:565 : Unable to create cgroup R> for driver: No such device or address
My guess is that this means you don't have the cgroup filesystem mounted, or it's mounted in a way that prevents libvirt from doing its work. IIRC, there are things you can do to the system that require a reboot to undo. If cgconfig is setting up the cgroups in such a way, you might need to reboot between stopping cgconfig and trying libvirt by itself.
Or the cgroups are being setup in funny fashion. Rishi, what does your config file look like?
R> But when i start cgconfig service, then i am not able to open R> console for my container.
Again, knowing nothing of cgconfig, I can only speculate that it's doing something to prevent accessing the console pty.
cgconfig does nothing apart from setup the cgroup filesystem as mentioned in the configuration files.
I would push on the libcgroup folks to help figure out what it's doing (either by default or with your config) to prevent that and then go from there.
Would appreciate any help from the libvirt folks who have worked on the cgroup support in libvirt.
The most likely suspect is that the cgroups controllers were mounted or unmounted after libvirtd started. A restart of libvirtd ought to resolve the problem if this is indeed the case Daniel; -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
On Wed, Oct 07, 2009 at 03:53:56PM +0100, Daniel P. Berrange wrote:
On Wed, Oct 07, 2009 at 08:12:02PM +0530, Dhaval Giani wrote:
On Wed, Oct 07, 2009 at 06:57:48AM -0700, Dan Smith wrote:
R> can you help me here ?
Probably not, I'm not familiar with cgconfig nor the latest cgroup support in libvirt.
R> 14:47:24.556: warning : qemudStartup:565 : Unable to create cgroup R> for driver: No such device or address
My guess is that this means you don't have the cgroup filesystem mounted, or it's mounted in a way that prevents libvirt from doing its work. IIRC, there are things you can do to the system that require a reboot to undo. If cgconfig is setting up the cgroups in such a way, you might need to reboot between stopping cgconfig and trying libvirt by itself.
Or the cgroups are being setup in funny fashion. Rishi, what does your config file look like?
R> But when i start cgconfig service, then i am not able to open R> console for my container.
Again, knowing nothing of cgconfig, I can only speculate that it's doing something to prevent accessing the console pty.
cgconfig does nothing apart from setup the cgroup filesystem as mentioned in the configuration files.
I would push on the libcgroup folks to help figure out what it's doing (either by default or with your config) to prevent that and then go from there.
Would appreciate any help from the libvirt folks who have worked on the cgroup support in libvirt.
The most likely suspect is that the cgroups controllers were mounted or unmounted after libvirtd started. A restart of libvirtd ought to resolve the problem if this is indeed the case
If Rishi is using the cgconfig service to start it, at least in Fedora I am aware of it starting early in the boot sequence. Rishi could you confirm this please? thanks, -- regards, Dhaval
Hi daniel, Sorry for late response, was busy with some other high priority task. Daniel P. Berrange wrote:
On Wed, Oct 07, 2009 at 08:12:02PM +0530, Dhaval Giani wrote:
The most likely suspect is that the cgroups controllers were mounted or unmounted after libvirtd started. A restart of libvirtd ought to resolve the problem if this is indeed the case
Actually not , restarting libvirt daemon did not help me. I had controllers mounted as per /proc/mount . And i am really seeing an issue after starting cgconfig services on my system. Shell :1 virsh # start vm1 Domain vm1 started virsh # list --all Id Name State ---------------------------------- - vm1 shut off virsh # Shell 2: [root@mls41 ~]# service libvirtd restart Stopping libvirtd daemon: [ OK ] Starting libvirtd daemon: [ OK ] [root@mls41 ~]# cat /proc/mounts | grep cgroup cgroup /cgroup cgroup rw,relatime,cpuacct,cpu 0 0 [root@mls41 ~]# Let me know if you need some more info. -Rishi
Daniel;
Rishikesh wrote:
Hi daniel,
Sorry for late response, was busy with some other high priority task.
Daniel P. Berrange wrote:
On Wed, Oct 07, 2009 at 08:12:02PM +0530, Dhaval Giani wrote:
The most likely suspect is that the cgroups controllers were mounted or unmounted after libvirtd started. A restart of libvirtd ought to resolve the problem if this is indeed the case
Ok, Here i tried one more scenario, i think this is what you meant: 1> Stop libvirtd 2> Run cgroup " /etc/init.d/cgconfig start" 3> Start libvirtd 4> And then try to create a container. Though above steps are also failing. But now i am getting some different failure. I am not seeing the earlier warning messages. You can see below on shell 1. My previous post was following this steps: 1> Run cgroup once again 2> service libvirtd restart 3> Create a container Here are the different error which i am getting: Shell:1 [root@mhs21a ~]# libvirtd libvir: Linux Container error : failed to connect to client socket: No such file or directory 11:17:36.743: error : lxcVmStart:1007 : Failed to read pid file /var/run/libvirt/lxc/vm1.pid: No such file or directory libvir: Linux Container error : Failed to read pid file /var/run/libvirt/lxc/vm1.pid: No such file or directory Shell 2: [root@mhs21a ~]# vim /etc/cgconfig.conf [root@mhs21a ~]# /etc/init.d/cgconfig start Starting cgconfig service: [ OK ] [root@mhs21a ~]# virsh -c lxc:/// Welcome to virsh, the virtualization interactive terminal. Type: 'help' for help with commands 'quit' to quit virsh # list --all Id Name State ---------------------------------- - vm1 shut off virsh # start vm1 error: Failed to start domain vm1 error: Failed to read pid file /var/run/libvirt/lxc/vm1.pid: No such file or directory virsh # list --all Id Name State ---------------------------------- - vm1 shut off virsh # So now we need to deal with two issue: - One we need to decide , ( is it neccessary to stop libvirt before creating cgroup ?) - How container can be started using cgroup ? -Rishi
Actually not , restarting libvirt daemon did not help me.
I had controllers mounted as per /proc/mount . And i am really seeing an issue after starting cgconfig services on my system.
Shell :1 virsh # start vm1 Domain vm1 started
virsh # list --all Id Name State ---------------------------------- - vm1 shut off
virsh #
Shell 2: [root@mls41 ~]# service libvirtd restart Stopping libvirtd daemon: [ OK ] Starting libvirtd daemon: [ OK ] [root@mls41 ~]# cat /proc/mounts | grep cgroup cgroup /cgroup cgroup rw,relatime,cpuacct,cpu 0 0 [root@mls41 ~]#
Let me know if you need some more info.
-Rishi
Daniel;
------------------------------------------------------------------------------ Come build with us! The BlackBerry(R) Developer Conference in SF, CA is the only developer event you need to attend this year. Jumpstart your developing skills, take BlackBerry mobile applications to market and stay ahead of the curve. Join us from November 9 - 12, 2009. Register now! http://p.sf.net/sfu/devconference _______________________________________________ Libcg-devel mailing list Libcg-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/libcg-devel
Rishikesh wrote:
Rishikesh wrote:
Hi daniel,
Sorry for late response, was busy with some other high priority task.
Daniel P. Berrange wrote:
On Wed, Oct 07, 2009 at 08:12:02PM +0530, Dhaval Giani wrote:
The most likely suspect is that the cgroups controllers were mounted or unmounted after libvirtd started. A restart of libvirtd ought to resolve the problem if this is indeed the case
Ok, Here i tried one more scenario, i think this is what you meant:
1> Stop libvirtd 2> Run cgroup " /etc/init.d/cgconfig start" 3> Start libvirtd 4> And then try to create a container.
Though above steps are also failing. But now i am getting some different failure. I am not seeing the earlier warning messages. You can see below on shell 1.
My previous post was following this steps: 1> Run cgroup once again 2> service libvirtd restart 3> Create a container
Here are the different error which i am getting: Shell:1
[root@mhs21a ~]# libvirtd libvir: Linux Container error : failed to connect to client socket: No such file or directory 11:17:36.743: error : lxcVmStart:1007 : Failed to read pid file /var/run/libvirt/lxc/vm1.pid: No such file or directory libvir: Linux Container error : Failed to read pid file /var/run/libvirt/lxc/vm1.pid: No such file or directory
Shell 2:
[root@mhs21a ~]# vim /etc/cgconfig.conf [root@mhs21a ~]# /etc/init.d/cgconfig start Starting cgconfig service: [ OK ] [root@mhs21a ~]# virsh -c lxc:/// Welcome to virsh, the virtualization interactive terminal.
Type: 'help' for help with commands 'quit' to quit
virsh # list --all Id Name State ---------------------------------- - vm1 shut off
virsh # start vm1 error: Failed to start domain vm1 error: Failed to read pid file /var/run/libvirt/lxc/vm1.pid: No such file or directory
virsh # list --all Id Name State ---------------------------------- - vm1 shut off
virsh #
So now we need to deal with two issue: - One we need to decide , ( is it neccessary to stop libvirt before creating cgroup ?) - How container can be started using cgroup ?
-Rishi
Cgroup config file & controllers mount shows like this : [root@mhs21a ~]# cat /proc/mounts | grep cgroup cgroup /cgroups/cpu cgroup rw,relatime,cpu 0 0 cgroup /cgroups/cpuacct cgroup rw,relatime,cpuacct 0 0 cgroup /cgroups/memory cgroup rw,relatime,memory 0 0 [root@mhs21a ~]# cat /etc/cgconfig. cat: /etc/cgconfig.: No such file or directory [root@mhs21a ~]# cat /etc/cgconfig.conf # # Copyright IBM Corporation. 2007 # group test1/fruits/mango { perm { task { uid = test; gid = test; } admin { uid = root; gid = root; } } cpu { cpu.shares = 1024; } } mount { cpu = /cgroups/cpu; cpuacct = /cgroups/cpuacct; memory = /cgroups/memory; } [root@mhs21a ~]#
Actually not , restarting libvirt daemon did not help me.
I had controllers mounted as per /proc/mount . And i am really seeing an issue after starting cgconfig services on my system.
Shell :1 virsh # start vm1 Domain vm1 started
virsh # list --all Id Name State ---------------------------------- - vm1 shut off
virsh #
Shell 2: [root@mls41 ~]# service libvirtd restart Stopping libvirtd daemon: [ OK ] Starting libvirtd daemon: [ OK ] [root@mls41 ~]# cat /proc/mounts | grep cgroup cgroup /cgroup cgroup rw,relatime,cpuacct,cpu 0 0 [root@mls41 ~]#
Let me know if you need some more info.
-Rishi
Daniel;
------------------------------------------------------------------------------ Come build with us! The BlackBerry(R) Developer Conference in SF, CA is the only developer event you need to attend this year. Jumpstart your developing skills, take BlackBerry mobile applications to market and stay ahead of the curve. Join us from November 9 - 12, 2009. Register now! http://p.sf.net/sfu/devconference _______________________________________________ Libcg-devel mailing list Libcg-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/libcg-devel
------------------------------------------------------------------------------ Come build with us! The BlackBerry(R) Developer Conference in SF, CA is the only developer event you need to attend this year. Jumpstart your developing skills, take BlackBerry mobile applications to market and stay ahead of the curve. Join us from November 9 - 12, 2009. Register now! http://p.sf.net/sfu/devconference _______________________________________________ Libcg-devel mailing list Libcg-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/libcg-devel
Hi Rishikesh, On Tue, Oct 13, 2009 at 3:09 PM, Rishikesh <risrajak@linux.vnet.ibm.com> wrote:
Rishikesh wrote:
Cgroup config file & controllers mount shows like this : [root@mhs21a ~]# cat /proc/mounts | grep cgroup cgroup /cgroups/cpu cgroup rw,relatime,cpu 0 0 cgroup /cgroups/cpuacct cgroup rw,relatime,cpuacct 0 0 cgroup /cgroups/memory cgroup rw,relatime,memory 0 0
AFAIW, libvirt requires "device" subsystem of cgroups as well. Without it, an assistant subprocess of libvirtd will die soon.
virsh # start vm1 error: Failed to start domain vm1 error: Failed to read pid file /var/run/libvirt/lxc/vm1.pid: No such file or directory
This error looks that happens to me. ozaki-r
On Wed, Oct 14, 2009 at 06:07:26AM +0900, Ryota Ozaki wrote:
Hi Rishikesh,
On Tue, Oct 13, 2009 at 3:09 PM, Rishikesh <risrajak@linux.vnet.ibm.com> wrote:
Rishikesh wrote:
Cgroup config file & controllers mount shows like this : [root@mhs21a ~]# cat /proc/mounts | grep cgroup cgroup /cgroups/cpu cgroup rw,relatime,cpu 0 0 cgroup /cgroups/cpuacct cgroup rw,relatime,cpuacct 0 0 cgroup /cgroups/memory cgroup rw,relatime,memory 0 0
AFAIW, libvirt requires "device" subsystem of cgroups as well. Without it, an assistant subprocess of libvirtd will die soon.
So rishi, could add a devices line as well to the cgconfig configuratoin file, and test it out? -- regards, Dhaval
Dhaval Giani wrote:
On Wed, Oct 14, 2009 at 06:07:26AM +0900, Ryota Ozaki wrote:
Hi Rishikesh,
On Tue, Oct 13, 2009 at 3:09 PM, Rishikesh <risrajak@linux.vnet.ibm.com> wrote:
Rishikesh wrote:
Cgroup config file & controllers mount shows like this : [root@mhs21a ~]# cat /proc/mounts | grep cgroup cgroup /cgroups/cpu cgroup rw,relatime,cpu 0 0 cgroup /cgroups/cpuacct cgroup rw,relatime,cpuacct 0 0 cgroup /cgroups/memory cgroup rw,relatime,memory 0 0
AFAIW, libvirt requires "device" subsystem of cgroups as well. Without it, an assistant subprocess of libvirtd will die soon.
So rishi, could add a devices line as well to the cgconfig configuratoin file, and test it out?
Cool !!! I am able to run container, once mounted "device" susbsystem too. Thanks Ryota. [root@mhs21a ~]# cat /proc/mounts | grep cgroup cgroup /cgroups/cpu cgroup rw,relatime,cpu 0 0 cgroup /cgroups/cpuacct cgroup rw,relatime,cpuacct 0 0 cgroup /cgroups/memory cgroup rw,relatime,memory 0 0 cgroup /cgroups/devices cgroup rw,relatime,devices 0 0 [root@mhs21a ~]# /etc/init.d/cgconfig stop Stopping cgconfig service: [ OK ] [root@mhs21a ~]# /etc/init.d/cgconfig start Starting cgconfig service: [ OK ] [root@mhs21a ~]# virsh -c lxc:/// Welcome to virsh, the virtualization interactive terminal. Type: 'help' for help with commands 'quit' to quit virsh # list --all Id Name State ---------------------------------- - vm1 shut off virsh # start vm1 Domain vm1 started virsh # list --all Id Name State ---------------------------------- 1525 vm1 running virsh # console vm1 Connected to domain vm1 Escape character is ^] sh-4.0# ps -ef UID PID PPID C STIME TTY TIME CMD root 1 0 0 10:27 pts/3 00:00:00 /bin/sh root 4 1 0 10:27 pts/3 00:00:00 ps -ef sh-4.0# So, Are we fixing the flow of libvirt & cgroup start mechanism ? Because user need not have to stop "libvirtd" , if he wants to use cgroup ? Please suggest. -Rishi
On Wed, Oct 14, 2009 at 10:44:49AM +0530, Rishikesh wrote:
Dhaval Giani wrote:
On Wed, Oct 14, 2009 at 06:07:26AM +0900, Ryota Ozaki wrote:
Hi Rishikesh,
On Tue, Oct 13, 2009 at 3:09 PM, Rishikesh <risrajak@linux.vnet.ibm.com> wrote:
Rishikesh wrote:
Cgroup config file & controllers mount shows like this : [root@mhs21a ~]# cat /proc/mounts | grep cgroup cgroup /cgroups/cpu cgroup rw,relatime,cpu 0 0 cgroup /cgroups/cpuacct cgroup rw,relatime,cpuacct 0 0 cgroup /cgroups/memory cgroup rw,relatime,memory 0 0
AFAIW, libvirt requires "device" subsystem of cgroups as well. Without it, an assistant subprocess of libvirtd will die soon.
So rishi, could add a devices line as well to the cgconfig configuratoin file, and test it out?
Cool !!! I am able to run container, once mounted "device" susbsystem too. Thanks Ryota.
[root@mhs21a ~]# cat /proc/mounts | grep cgroup cgroup /cgroups/cpu cgroup rw,relatime,cpu 0 0 cgroup /cgroups/cpuacct cgroup rw,relatime,cpuacct 0 0 cgroup /cgroups/memory cgroup rw,relatime,memory 0 0 cgroup /cgroups/devices cgroup rw,relatime,devices 0 0 [root@mhs21a ~]# /etc/init.d/cgconfig stop Stopping cgconfig service: [ OK ] [root@mhs21a ~]# /etc/init.d/cgconfig start Starting cgconfig service: [ OK ] [root@mhs21a ~]# virsh -c lxc:/// Welcome to virsh, the virtualization interactive terminal.
Type: 'help' for help with commands 'quit' to quit
virsh # list --all Id Name State ---------------------------------- - vm1 shut off
virsh # start vm1 Domain vm1 started
virsh # list --all Id Name State ---------------------------------- 1525 vm1 running
virsh # console vm1 Connected to domain vm1 Escape character is ^] sh-4.0# ps -ef UID PID PPID C STIME TTY TIME CMD root 1 0 0 10:27 pts/3 00:00:00 /bin/sh root 4 1 0 10:27 pts/3 00:00:00 ps -ef sh-4.0#
So, Are we fixing the flow of libvirt & cgroup start mechanism ? Because user need not have to stop "libvirtd" , if he wants to use cgroup ?
We can certainly improve that - can you file a bug against libvirt with this info Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
On Wed, Oct 14, 2009 at 06:07:26AM +0900, Ryota Ozaki wrote:
Hi Rishikesh,
On Tue, Oct 13, 2009 at 3:09 PM, Rishikesh <risrajak@linux.vnet.ibm.com> wrote:
Rishikesh wrote:
Cgroup config file & controllers mount shows like this : [root@mhs21a ~]# cat /proc/mounts | grep cgroup cgroup /cgroups/cpu cgroup rw,relatime,cpu 0 0 cgroup /cgroups/cpuacct cgroup rw,relatime,cpuacct 0 0 cgroup /cgroups/memory cgroup rw,relatime,memory 0 0
AFAIW, libvirt requires "device" subsystem of cgroups as well. Without it, an assistant subprocess of libvirtd will die soon.
Ahh, thanks for remmebering that - I had completely forgotten. We should add a check to libvirt so that it gives back a useful error message if the device controller is missing. Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
Dhaval Giani wrote:
On Wed, Oct 07, 2009 at 06:57:48AM -0700, Dan Smith wrote:
R> can you help me here ?
Probably not, I'm not familiar with cgconfig nor the latest cgroup support in libvirt.
R> 14:47:24.556: warning : qemudStartup:565 : Unable to create cgroup R> for driver: No such device or address
My guess is that this means you don't have the cgroup filesystem mounted, or it's mounted in a way that prevents libvirt from doing its work. IIRC, there are things you can do to the system that require a reboot to undo. If cgconfig is setting up the cgroups in such a way, you might need to reboot between stopping cgconfig and trying libvirt by itself.
Or the cgroups are being setup in funny fashion. Rishi, what does your config file look like?
I am attaching the config file.
R> But when i start cgconfig service, then i am not able to open R> console for my container.
Again, knowing nothing of cgconfig, I can only speculate that it's doing something to prevent accessing the console pty.
cgconfig does nothing apart from setup the cgroup filesystem as mentioned in the configuration files.
I would push on the libcgroup folks to help figure out what it's doing (either by default or with your config) to prevent that and then go from there.
Would appreciate any help from the libvirt folks who have worked on the cgroup support in libvirt.
thanks,
group default { perm { admin { uid = root; git = root; } task { uid = root; gid = root; } } cpu { cpu.shares = 1024; cpu.rt_runtime_us = 920000; } } group test { perm { admin { uid = test; gid = test; } task { uid = test; gid = test; } } cpu { cpu.shares = 4096; cpu.rt_runtime_us = 30000; } } mount { cpu = /cgroup; cpuacct = /cgroup; }
On Wed, Oct 07, 2009 at 08:39:09PM +0530, Rishikesh wrote:
Dhaval Giani wrote:
On Wed, Oct 07, 2009 at 06:57:48AM -0700, Dan Smith wrote:
R> can you help me here ?
Probably not, I'm not familiar with cgconfig nor the latest cgroup support in libvirt.
R> 14:47:24.556: warning : qemudStartup:565 : Unable to create cgroup R> for driver: No such device or address
My guess is that this means you don't have the cgroup filesystem mounted, or it's mounted in a way that prevents libvirt from doing its work. IIRC, there are things you can do to the system that require a reboot to undo. If cgconfig is setting up the cgroups in such a way, you might need to reboot between stopping cgconfig and trying libvirt by itself.
Or the cgroups are being setup in funny fashion. Rishi, what does your config file look like?
I am attaching the config file.
OK, thanks rishi! So cpu and cpuacct subsystems are getting mounted at /cgroup and there are two groups created at /cgroup/default and /cgroup/test/. The perm section defines the user/group who own all the files apart from tasks and the task section defines the owner for the task file. Are there more subsystems that should be mounted? And is there a specific mount point that should be used? Thanks, [leaving the conf file in for folks to see]
group default { perm { admin { uid = root; git = root; } task { uid = root; gid = root; } } cpu { cpu.shares = 1024; cpu.rt_runtime_us = 920000; } }
group test { perm { admin { uid = test; gid = test; } task { uid = test; gid = test; } } cpu { cpu.shares = 4096; cpu.rt_runtime_us = 30000; } }
mount { cpu = /cgroup; cpuacct = /cgroup; }
-- regards, Dhaval
On Wed, Oct 07, 2009 at 09:18:18AM +0530, Rishikesh wrote:
Rishikesh wrote:
On F12 i was playing with libvirt support for container. I am seeing two issue over here: 1> Not able to create a container when start cgconfig services. 2> When i stop cgconfig, then able to create a container, but seeing few error messages to libvirtd daemon. 3> After creation of container through libvirt, i am not able to touch any file or directory.
Attached: vm1.xml , through which i created container. I referred following link:
Detailed Bug Report: When i create a container without starting "cgconfig" service, i am able to create but getting some error message in libvirtd daemon:
[root@mx3950 ~]# libvirtd 14:47:24.556: warning : qemudStartup:565 : Unable to create cgroup for driver: No such device or address 14:47:24.647: warning : lxcStartup:1462 : Unable to create cgroup for driver: No such device or address
When libvirtd starts up, it checks /proc/mounts to discover where all the cgroups controllers are living. It also checks if its own process was placed inside the root vs a child, etc. If it doesn't find any controllers mounted at all, then it'll disable its cgroup support & log a warning - those are the 2 messages you are seeing here. This is harmless - it just means some features of libvirt will be disabled, the rest should still work
But when i start cgconfig service, then i am not able to open console for my container. Below is the detailed steps which i executed.
[root@mx3950 ~]# /etc/init.d/cgconfig start Starting cgconfig service: [ OK ]
I'm assuming that 'cgconfig' mounts or unmounts cgroups controllers. The libvirt daemon only checks where they are all mounted when it initially starts up. So it won't be at all happy if you mount/unmount cgroups while it is running. If you restarts libvirtd after this it would pick up the changes
[root@mx3950 ~]# virsh -c lxc:/// start vm1 Domain vm1 started
[root@mx3950 ~]# virsh -c lxc:/// console vm1 Connected to domain vm1 Escape character is ^] 15:24:12.869: error : vshRunConsole:77 : unable to open tty /dev/pts/3: No such file or directory
I'm rather surprised if this error is relating to the use of cgroups - the private devpts instance support is not anything todo with cgroups really. What cgroups controllers do you actually have active & mounted ? And if you restart libvirtd after stop/start of cgconfig, does it all work as expected ? Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
I'm assuming that 'cgconfig' mounts or unmounts cgroups controllers. The libvirt daemon only checks where they are all mounted when it initially starts up. So it won't be at all happy if you mount/unmount cgroups while it is running. If you restarts libvirtd after this it would pick up the changes
hmm. This does not seem to be a good idea. I can envisage many scenarios where folks would want to have cgroups mounted after logging into the system (whether they use libcgroup or not). In such a situation you would expect libvirt to be restarted? thanks. -- regards, Dhaval
participants (5)
-
Dan Smith -
Daniel P. Berrange -
Dhaval Giani -
Rishikesh -
Ryota Ozaki