[libvirt] inability to open local read-only connection
I have been experimenting with using libvirt (0.3.3) on a variety of systems (RHEL, CentOS and Oracle VM). I have run into an issue when I try to open a local read-only connection to the hypervisor that is failing only on Oracle VM server release 2.2.0. I have created a root owned setuid executable that is effectively running as root, but even so, still cannot open the local read-only connection of the hypervisor. It only works if I run it directly as root. This is not an option. I do not understand why it works as is on my RHEL and CentOS machines, but not my Oracle machine. It would seem as thought it is not checking if the effective uid is root, just the uid. Has anyone run into a similar issue or have any suggestions of what I might try to fix this issue or can tell me that this is a defect that needs (is) fixed?? Thanks.
On Thu, Feb 25, 2010 at 02:34:02PM -0600, Tavares, John wrote:
I have been experimenting with using libvirt (0.3.3) on a variety of systems (RHEL, CentOS and Oracle VM). I have run into an issue when I try to open a local read-only connection to the hypervisor that is failing only on Oracle VM server release 2.2.0. I have created a root owned setuid executable that is effectively running as root, but even so, still cannot open the local read-only connection of the hypervisor. It only works if I run it directly as root. This is not an option. I do not understand why it works as is on my RHEL and CentOS machines, but not my Oracle machine. It would seem as thought it is not checking if the effective uid is root, just the uid.
A readonly connection essentially just goes to a separate UNIX socket, which has more relaxed permissions (mode 0777, instead of 0700). The kernel does the permissions checking when attempting to open it, so it should be using the effective ID.
Has anyone run into a similar issue or have any suggestions of what I might try to fix this issue or can tell me that this is a defect that needs (is) fixed??
I'd suggest trying to strace your process & see where it fails, and/or run with LIBVIRT_DEBUG=1 environemnt variable set. Regards, Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
Thanks for your suggestions Daniel! I will rerun my executable thru strace with LIBVIRT_DEBUG=1 to see what happens and will send the results if need be. John -----Original Message----- From: Daniel P. Berrange [mailto:berrange@redhat.com] Sent: Monday, March 01, 2010 7:58 AM To: Tavares, John Cc: libvir-list@redhat.com Subject: Re: [libvirt] inability to open local read-only connection On Thu, Feb 25, 2010 at 02:34:02PM -0600, Tavares, John wrote:
I have been experimenting with using libvirt (0.3.3) on a variety of systems (RHEL, CentOS and Oracle VM). I have run into an issue when I try to open a local read-only connection to the hypervisor that is failing only on Oracle VM server release 2.2.0. I have created a root owned setuid executable that is effectively running as root, but even so, still cannot open the local read-only connection of the hypervisor. It only works if I run it directly as root. This is not an option. I do not understand why it works as is on my RHEL and CentOS machines, but not my Oracle machine. It would seem as thought it is not checking if the effective uid is root, just the uid.
A readonly connection essentially just goes to a separate UNIX socket, which has more relaxed permissions (mode 0777, instead of 0700). The kernel does the permissions checking when attempting to open it, so it should be using the effective ID.
Has anyone run into a similar issue or have any suggestions of what I might try to fix this issue or can tell me that this is a defect that needs (is) fixed??
I'd suggest trying to strace your process & see where it fails, and/or run with LIBVIRT_DEBUG=1 environemnt variable set. Regards, Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
Hi Daniel, Sorry it took so long to back to this. I just recently ran into the same problem on RHEL 5.5 where I am running into problems using the root owned setuid executable to get domain information. Basically all that I needed to do to show this was to modify the example program dominfo/info1.c that ships with the libvirt source and added a function that attempts to get any information about that domain. In this case, I chose the virDomainGetSchedulerType() function. Here is the output of 3 separate runs, the first will be as a non-root user, the second is from root and the last is from the same executable but just root owned setuid and run as a normal user: $ ll info1 -r-xr-xr-x 1 perform perform 9326 Jul 12 16:50 info1 $ whoami perform $ ./info1 Domains 0: 2 CPUs $ su Password: # ./info1 Domains 0: 2 CPUs Scheduler = credit # chown 0:0 info1;chmod 6555 info1 # ll info1 -r-sr-sr-x 1 root root 9326 Jul 12 16:50 info1 # exit $ whoami perform $ ./info1 Domains 0: 2 CPUs libvir: Xen Daemon error : this function is not supported by the hypervisor: unsupported in xendConfigVersion < 4 $ virsh version error: failed to connect to the hypervisor $ su root -c "virsh version" Password: Compiled against library: libvir 0.6.3 Using library: libvir 0.6.3 Using API: Xen 3.0.1 Running hypervisor: Xen 3.1.0 As you can see, even virsh needs to be run as root in order to connect to the hypervisor. To help, I have attached both the source code and the executable I was using. Please let me know if anything else is required to get this problem resolved. Thanks. John -----Original Message----- From: Daniel P. Berrange [mailto:berrange@redhat.com] Sent: Monday, March 01, 2010 7:58 AM To: Tavares, John Cc: libvir-list@redhat.com Subject: Re: [libvirt] inability to open local read-only connection On Thu, Feb 25, 2010 at 02:34:02PM -0600, Tavares, John wrote:
I have been experimenting with using libvirt (0.3.3) on a variety of systems (RHEL, CentOS and Oracle VM). I have run into an issue when I try to open a local read-only connection to the hypervisor that is failing only on Oracle VM server release 2.2.0. I have created a root owned setuid executable that is effectively running as root, but even so, still cannot open the local read-only connection of the hypervisor. It only works if I run it directly as root. This is not an option. I do not understand why it works as is on my RHEL and CentOS machines, but not my Oracle machine. It would seem as thought it is not checking if the effective uid is root, just the uid.
A readonly connection essentially just goes to a separate UNIX socket, which has more relaxed permissions (mode 0777, instead of 0700). The kernel does the permissions checking when attempting to open it, so it should be using the effective ID.
Has anyone run into a similar issue or have any suggestions of what I might try to fix this issue or can tell me that this is a defect that needs (is) fixed??
I'd suggest trying to strace your process & see where it fails, and/or run with LIBVIRT_DEBUG=1 environemnt variable set. Regards, Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
On Mon, Jul 12, 2010 at 06:38:27AM -0500, Tavares, John wrote:
Hi Daniel,
Sorry it took so long to back to this. I just recently ran into the same problem on RHEL 5.5 where I am running into problems using the root owned setuid executable to get domain information. Basically all that I needed to do to show this was to modify the example program dominfo/info1.c that ships with the libvirt source and added a function that attempts to get any information about that domain. In this case, I chose the virDomainGetSchedulerType() function. Here is the output of 3 separate runs, the first will be as a non-root user, the second is from root and the last is from the same executable but just root owned setuid and run as a normal user:
I've figured out what is going on here. Originally libvirt provided a setuid proxy for accessing the Xen driver as non-root. This is /usr/libexec/libvirt_proxy. This only supports a small sub-set of API calls and is only for the Xen hypervisor. Later we added a general purpose daemon (/usr/sbin/libvirtd) that runs as root and works for all APIs and all hypervisors. This daemon replaces the setuid proxy, but for reasons of backwards compatability in RHEL5 the proxy is still the default for Xen. The virDomainGetSchedulerType() API is an example of one that is supported by libvirtd, but not by the Xen proxy. To force use of the libvirtd, give it an explicit URI, xen+unix:/// will make it connect to Xen, over libvirtd's UNIX domain socket.
$ ll info1 -r-xr-xr-x 1 perform perform 9326 Jul 12 16:50 info1 $ whoami perform $ ./info1 Domains 0: 2 CPUs
This doesn't print schedular info because it is not supported by the Xen proxy.
$ su Password: # ./info1 Domains 0: 2 CPUs Scheduler = credit # chown 0:0 info1;chmod 6555 info1 # ll info1 -r-sr-sr-x 1 root root 9326 Jul 12 16:50 info1 # exit $ whoami perform $ ./info1 Domains 0: 2 CPUs libvir: Xen Daemon error : this function is not supported by the hypervisor: unsupported in xendConfigVersion < 4
Not entirely sure why you get this error message here, but I strongly recommend against running this setuid regardless.
$ virsh version error: failed to connect to the hypervisor
You need to pass the --readonly flag to virsh $ whoami berrange $ virsh --readonly version Compiled against library: libvir 0.6.3 Using library: libvir 0.6.3 Using API: Xen 3.0.1 Running hypervisor: Xen 3.1.0 And to make it use the libvirtd instead of the proxy give it a URI: $ virsh -c xen+unix:/// --readonly schedinfo Domain-0 Scheduler : credit weight : 256 cap : 0
$ su root -c "virsh version" Password: Compiled against library: libvir 0.6.3 Using library: libvir 0.6.3 Using API: Xen 3.0.1 Running hypervisor: Xen 3.1.0
Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
Hi Daniel, Thank you for looking into this for me and providing such a detailed explanation, I really appreciate it. As a related question, will this work for RHEL6 for KVM as well along with the virsh CLI?? John -----Original Message----- From: Daniel P. Berrange [mailto:berrange@redhat.com] Sent: Friday, August 20, 2010 6:30 AM To: Tavares, John Cc: libvir-list@redhat.com; Tomen Tse; Betley, Greg; Chen, Jianjiun Subject: Re: [libvirt] inability to open local read-only connection On Mon, Jul 12, 2010 at 06:38:27AM -0500, Tavares, John wrote:
Hi Daniel,
Sorry it took so long to back to this. I just recently ran into the same problem on RHEL 5.5 where I am running into problems using the root owned setuid executable to get domain information. Basically all that I needed to do to show this was to modify the example program dominfo/info1.c that ships with the libvirt source and added a function that attempts to get any information about that domain. In this case, I chose the virDomainGetSchedulerType() function. Here is the output of 3 separate runs, the first will be as a non-root user, the second is from root and the last is from the same executable but just root owned setuid and run as a normal user:
I've figured out what is going on here. Originally libvirt provided a setuid proxy for accessing the Xen driver as non-root. This is /usr/libexec/libvirt_proxy. This only supports a small sub-set of API calls and is only for the Xen hypervisor. Later we added a general purpose daemon (/usr/sbin/libvirtd) that runs as root and works for all APIs and all hypervisors. This daemon replaces the setuid proxy, but for reasons of backwards compatability in RHEL5 the proxy is still the default for Xen. The virDomainGetSchedulerType() API is an example of one that is supported by libvirtd, but not by the Xen proxy. To force use of the libvirtd, give it an explicit URI, xen+unix:/// will make it connect to Xen, over libvirtd's UNIX domain socket.
$ ll info1 -r-xr-xr-x 1 perform perform 9326 Jul 12 16:50 info1 $ whoami perform $ ./info1 Domains 0: 2 CPUs
This doesn't print schedular info because it is not supported by the Xen proxy.
$ su Password: # ./info1 Domains 0: 2 CPUs Scheduler = credit # chown 0:0 info1;chmod 6555 info1 # ll info1 -r-sr-sr-x 1 root root 9326 Jul 12 16:50 info1 # exit $ whoami perform $ ./info1 Domains 0: 2 CPUs libvir: Xen Daemon error : this function is not supported by the hypervisor: unsupported in xendConfigVersion < 4
Not entirely sure why you get this error message here, but I strongly recommend against running this setuid regardless.
$ virsh version error: failed to connect to the hypervisor
You need to pass the --readonly flag to virsh $ whoami berrange $ virsh --readonly version Compiled against library: libvir 0.6.3 Using library: libvir 0.6.3 Using API: Xen 3.0.1 Running hypervisor: Xen 3.1.0 And to make it use the libvirtd instead of the proxy give it a URI: $ virsh -c xen+unix:/// --readonly schedinfo Domain-0 Scheduler : credit weight : 256 cap : 0
$ su root -c "virsh version" Password: Compiled against library: libvir 0.6.3 Using library: libvir 0.6.3 Using API: Xen 3.0.1 Running hypervisor: Xen 3.1.0
Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
On Fri, Aug 20, 2010 at 07:03:05AM -0500, Tavares, John wrote:
Hi Daniel,
Thank you for looking into this for me and providing such a detailed explanation, I really appreciate it.
As a related question, will this work for RHEL6 for KVM as well along with the virsh CLI??
Yes, just change the URI to 'qemu:///system'. No need for the +unix bit here because QEMU always goes via libvirtd. You have KVM in RHEL5.4 or later too that this should work with... Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
Thanks! This is good to know. John -----Original Message----- From: Daniel P. Berrange [mailto:berrange@redhat.com] Sent: Friday, August 20, 2010 8:12 AM To: Tavares, John Cc: libvir-list@redhat.com; Tomen Tse; Betley, Greg; Chen, Jianjiun Subject: Re: [libvirt] inability to open local read-only connection On Fri, Aug 20, 2010 at 07:03:05AM -0500, Tavares, John wrote:
Hi Daniel,
Thank you for looking into this for me and providing such a detailed explanation, I really appreciate it.
As a related question, will this work for RHEL6 for KVM as well along with the virsh CLI??
Yes, just change the URI to 'qemu:///system'. No need for the +unix bit here because QEMU always goes via libvirtd. You have KVM in RHEL5.4 or later too that this should work with... Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
If you don't mind, I have a few other questions related to this that I'd like to ask before I forget about them. 1) Is there any technical reason for not shipping the 32-bit version of the libvirt shared library for 64-bit systems?? 2) Is it possible via libvirt to get that some additional information about the guests, such as: a) uptime b) hostname c) real operating system Thanks again. John -----Original Message----- From: Daniel P. Berrange [mailto:berrange@redhat.com] Sent: Friday, August 20, 2010 8:12 AM To: Tavares, John Cc: libvir-list@redhat.com; Tomen Tse; Betley, Greg; Chen, Jianjiun Subject: Re: [libvirt] inability to open local read-only connection On Fri, Aug 20, 2010 at 07:03:05AM -0500, Tavares, John wrote:
Hi Daniel,
Thank you for looking into this for me and providing such a detailed explanation, I really appreciate it.
As a related question, will this work for RHEL6 for KVM as well along with the virsh CLI??
Yes, just change the URI to 'qemu:///system'. No need for the +unix bit here because QEMU always goes via libvirtd. You have KVM in RHEL5.4 or later too that this should work with... Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
On Fri, Aug 20, 2010 at 07:42:32AM -0500, Tavares, John wrote:
If you don't mind, I have a few other questions related to this that I'd like to ask before I forget about them.
1) Is there any technical reason for not shipping the 32-bit version of the libvirt shared library for 64-bit systems??
The libvirt RPC protocol is arch-invariant, so you can use a 32-bit libvirt library to talk to a 64-bit host, and vica-verca. So you can install the 32bit library if desired, but really 32-bit is dead technology, so its better to just recompile any apps for 64-bit.
2) Is it possible via libvirt to get that some additional information about the guests, such as: a) uptime b) hostname c) real operating system
You can't reliably get any of that information without having an agent running inside the guest OS. Currently libvirt does not concern itself with guest agent services, the focus being primarily on host OS management. There's a small chance this may change in the future, but its too early to say for sure. Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
On 08/20/2010 10:58 PM, Daniel P. Berrange wrote: <snip>
a) uptime <snip> You can't reliably get any of that information without having an agent running inside the guest OS.
We should be able to get the uptime of the guest's qemu process itself though, and report that back "over the wire" to a requesting client, without needing a guest OS agent. Or even keep track of when libvirt launched the guest as wall clock time. Not the same thing as "how long has the guest OS been up?", but potentially useful for management tools. Regards and best wishes, Justin Clift
On Fri, Aug 20, 2010 at 11:15:00PM +1000, Justin Clift wrote:
On 08/20/2010 10:58 PM, Daniel P. Berrange wrote: <snip>
a) uptime <snip> You can't reliably get any of that information without having an agent running inside the guest OS.
We should be able to get the uptime of the guest's qemu process itself though, and report that back "over the wire" to a requesting client, without needing a guest OS agent. Or even keep track of when libvirt launched the guest as wall clock time.
Yep that would be possible, I was just asuming John wanted the OS uptime based on his other data points there.
Not the same thing as "how long has the guest OS been up?", but potentially useful for management tools.
Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
It is available via xm, so my point was if it can, why can't libvirt?? In the end, I had to switch to calling virsh until I can get all of problem resolved. Ideally, virsh to would be able to provide most of the same data as going thru libvirt directly. It might also be nice if it provided that data in a more machine parseable format. -----Original Message----- From: Justin Clift [mailto:jclift@redhat.com] Sent: Friday, August 20, 2010 9:15 AM To: Daniel P. Berrange Cc: Tavares, John; libvir-list@redhat.com Subject: Re: [libvirt] inability to open local read-only connection On 08/20/2010 10:58 PM, Daniel P. Berrange wrote: <snip>
a) uptime <snip> You can't reliably get any of that information without having an agent running inside the guest OS.
We should be able to get the uptime of the guest's qemu process itself though, and report that back "over the wire" to a requesting client, without needing a guest OS agent. Or even keep track of when libvirt launched the guest as wall clock time. Not the same thing as "how long has the guest OS been up?", but potentially useful for management tools. Regards and best wishes, Justin Clift
On Fri, Aug 20, 2010 at 08:49:50AM -0500, Tavares, John wrote:
It is available via xm, so my point was if it can, why can't libvirt??
No particular reason why not, other than nobody has requested it before. NB, 'xm uptime' is giving the uptime of the virtual domain, not the guest OS running inside it. ie if you reboot the guest OS, the 'uptime' doesn't get reset because the virtual domain itself (usually[1]) never stops.
In the end, I had to switch to calling virsh until I can get all of problem resolved.
Ideally, virsh to would be able to provide most of the same data as going thru libvirt directly. It might also be nice if it provided that data in a more machine parseable format.
We do plan to make it available in machine readable formats in the future, but in most cases you'd be better off using one of the APIs directly instead of virsh. There's a choice of C, python, perl, java, ocaml, ruby. It would be significantly more efficient to use the API becasue you would be avoiding the overhead of opening & closing a connection for every single data item collected. Daniel [1] For paravirt guests IIRC Xen may actually destroy & recreate the virtual domain during a reboot, but fullvirt should be transparent. -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
I initially used libvirt directly, but when I started running into the lack of a 32-bit library on 64-bit issue and the inconsistent effective root user behavior issue, I needed to abandon the use of it and turned to virsh as it had no such issues and pay the performance penalty for stability. Maybe for our next release I will go back to it, but it is a little late in the game now for me to go back now. It also looks like the default version that comes with each revision of RHEL is a bit dated. -----Original Message----- From: Daniel P. Berrange [mailto:berrange@redhat.com] Sent: Friday, August 20, 2010 9:58 AM To: Tavares, John Cc: Justin Clift; libvir-list@redhat.com Subject: Re: [libvirt] inability to open local read-only connection On Fri, Aug 20, 2010 at 08:49:50AM -0500, Tavares, John wrote:
It is available via xm, so my point was if it can, why can't libvirt??
No particular reason why not, other than nobody has requested it before. NB, 'xm uptime' is giving the uptime of the virtual domain, not the guest OS running inside it. ie if you reboot the guest OS, the 'uptime' doesn't get reset because the virtual domain itself (usually[1]) never stops.
In the end, I had to switch to calling virsh until I can get all of problem resolved.
Ideally, virsh to would be able to provide most of the same data as going thru libvirt directly. It might also be nice if it provided that data in a more machine parseable format.
We do plan to make it available in machine readable formats in the future, but in most cases you'd be better off using one of the APIs directly instead of virsh. There's a choice of C, python, perl, java, ocaml, ruby. It would be significantly more efficient to use the API becasue you would be avoiding the overhead of opening & closing a connection for every single data item collected. Daniel [1] For paravirt guests IIRC Xen may actually destroy & recreate the virtual domain during a reboot, but fullvirt should be transparent. -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
Should the memory utilization as report from inside a guest match that as reported by the hypervisor?? I think one of my QA engineers may have pointed out a Windows guest where this was not the case. -----Original Message----- From: Tavares, John Sent: Friday, August 20, 2010 10:04 AM To: 'Daniel P. Berrange' Cc: Justin Clift; libvir-list@redhat.com Subject: RE: [libvirt] inability to open local read-only connection I initially used libvirt directly, but when I started running into the lack of a 32-bit library on 64-bit issue and the inconsistent effective root user behavior issue, I needed to abandon the use of it and turned to virsh as it had no such issues and pay the performance penalty for stability. Maybe for our next release I will go back to it, but it is a little late in the game now for me to go back now. It also looks like the default version that comes with each revision of RHEL is a bit dated. -----Original Message----- From: Daniel P. Berrange [mailto:berrange@redhat.com] Sent: Friday, August 20, 2010 9:58 AM To: Tavares, John Cc: Justin Clift; libvir-list@redhat.com Subject: Re: [libvirt] inability to open local read-only connection On Fri, Aug 20, 2010 at 08:49:50AM -0500, Tavares, John wrote:
It is available via xm, so my point was if it can, why can't libvirt??
No particular reason why not, other than nobody has requested it before. NB, 'xm uptime' is giving the uptime of the virtual domain, not the guest OS running inside it. ie if you reboot the guest OS, the 'uptime' doesn't get reset because the virtual domain itself (usually[1]) never stops.
In the end, I had to switch to calling virsh until I can get all of problem resolved.
Ideally, virsh to would be able to provide most of the same data as going thru libvirt directly. It might also be nice if it provided that data in a more machine parseable format.
We do plan to make it available in machine readable formats in the future, but in most cases you'd be better off using one of the APIs directly instead of virsh. There's a choice of C, python, perl, java, ocaml, ruby. It would be significantly more efficient to use the API becasue you would be avoiding the overhead of opening & closing a connection for every single data item collected. Daniel [1] For paravirt guests IIRC Xen may actually destroy & recreate the virtual domain during a reboot, but fullvirt should be transparent. -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
On Fri, Aug 20, 2010 at 09:15:04AM -0500, Tavares, John wrote:
Should the memory utilization as report from inside a guest match that as reported by the hypervisor?? I think one of my QA engineers may have pointed out a Windows guest where this was not the case.
It depends what libvirt APi you are referring to when you talk about memory utilization. virDomainGetInfo returns data about the memory allocated to the guest by the hypervisor. This says nothing about what the guest OS is doing with the memory. eg, the guest has been given 500 MB, but its OS may have only used 200 MB so far. We have a fairly new API virDomainGetMemoryStats that can give info about how much of its allocated memory a guest has actually used, along with stats on swap in/out and major/minor fails. This is only implemented for KVM + virtio though and requires a guest with a pretty new balloon driver to report the data Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
When I was using libvirt directly, I did see that, but could never make it work. This is some of the additional memory/performance information we would be interested in. In the event I stick with using virsh, it is available it as well as I have yet to see it?? -----Original Message----- From: Daniel P. Berrange [mailto:berrange@redhat.com] Sent: Friday, August 20, 2010 10:30 AM To: Tavares, John Cc: Justin Clift; libvir-list@redhat.com Subject: Re: [libvirt] inability to open local read-only connection On Fri, Aug 20, 2010 at 09:15:04AM -0500, Tavares, John wrote:
Should the memory utilization as report from inside a guest match that as reported by the hypervisor?? I think one of my QA engineers may have pointed out a Windows guest where this was not the case.
It depends what libvirt APi you are referring to when you talk about memory utilization. virDomainGetInfo returns data about the memory allocated to the guest by the hypervisor. This says nothing about what the guest OS is doing with the memory. eg, the guest has been given 500 MB, but its OS may have only used 200 MB so far. We have a fairly new API virDomainGetMemoryStats that can give info about how much of its allocated memory a guest has actually used, along with stats on swap in/out and major/minor fails. This is only implemented for KVM + virtio though and requires a guest with a pretty new balloon driver to report the data Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
On Fri, Aug 20, 2010 at 09:34:22AM -0500, Tavares, John wrote:
When I was using libvirt directly, I did see that, but could never make it work. This is some of the additional memory/ performance information we would be interested in. In the event I stick with using virsh, it is available it as well as I have yet to see it??
The 'virsh dommemstat' command is the one you want. It will only work on a RHEL-6 host, with a RHEL6 / F14 guests eg # virsh dommemstat f14x8_64 swap_in 0 swap_out 0 major_fault 534 minor_fault 714956 unused 595656 available 1023720 Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
Thanks again Daniel. All of this information is going to be very handy as we are going to start to look at KVM and RHEL6 soon. -----Original Message----- From: Daniel P. Berrange [mailto:berrange@redhat.com] Sent: Friday, August 20, 2010 10:48 AM To: Tavares, John Cc: Justin Clift; libvir-list@redhat.com Subject: Re: [libvirt] inability to open local read-only connection On Fri, Aug 20, 2010 at 09:34:22AM -0500, Tavares, John wrote:
When I was using libvirt directly, I did see that, but could never make it work. This is some of the additional memory/ performance information we would be interested in. In the event I stick with using virsh, it is available it as well as I have yet to see it??
The 'virsh dommemstat' command is the one you want. It will only work on a RHEL-6 host, with a RHEL6 / F14 guests eg # virsh dommemstat f14x8_64 swap_in 0 swap_out 0 major_fault 534 minor_fault 714956 unused 595656 available 1023720 Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
You may consider 32-bit a dead technology, but we do not. We have one 32-bit application that runs and collects configuration and performance data for all types of Linux based system regardless if it is 32-bit or 64-bit. Until such time as there is no 32-bit systems, I do not see this going away anytime soon. I can understand that libvirt is not too concerned about the internal of the guest, but for our purposes, it is a concern. While having the uptime (which is available from xm) and hostname (a nice to have), the reporting of the real operating system without forcing the user to deploy agents into each and every guest just to obtain this sort of basic information about it. The operating system is of importance to us for use cases like server/workload consolidation where you need to know if the workload from one system can indeed be migrated to another, where the first factor is are they running the same operating systems. This is something to keep in mind. John -----Original Message----- From: Daniel P. Berrange [mailto:berrange@redhat.com] Sent: Friday, August 20, 2010 8:58 AM To: Tavares, John Cc: libvir-list@redhat.com; Tomen Tse; Betley, Greg; Chen, Jianjiun Subject: Re: [libvirt] inability to open local read-only connection On Fri, Aug 20, 2010 at 07:42:32AM -0500, Tavares, John wrote:
If you don't mind, I have a few other questions related to this that I'd like to ask before I forget about them.
1) Is there any technical reason for not shipping the 32-bit version of the libvirt shared library for 64-bit systems??
The libvirt RPC protocol is arch-invariant, so you can use a 32-bit libvirt library to talk to a 64-bit host, and vica-verca. So you can install the 32bit library if desired, but really 32-bit is dead technology, so its better to just recompile any apps for 64-bit.
2) Is it possible via libvirt to get that some additional information about the guests, such as: a) uptime b) hostname c) real operating system
You can't reliably get any of that information without having an agent running inside the guest OS. Currently libvirt does not concern itself with guest agent services, the focus being primarily on host OS management. There's a small chance this may change in the future, but its too early to say for sure. Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
On 08/20/2010 06:42 AM, Tavares, John wrote:
2) Is it possible via libvirt to get that some additional information about the guests, such as: a) uptime b) hostname c) real operating system
hostname and OS of local disk images can be determined using libguestfs (in particular, I think virt-inspector command line tool from the libguestfs suite answers both of those questions). I'm not sure if uptime can also be retrieved with libguestfs, but if not, it is a reasonable feature request. -- Eric Blake eblake@redhat.com +1-801-349-2682 Libvirt virtualization library http://libvirt.org
Hi Daniel, I added some more info to this. I added a call to try to get the max number of vcpus as well. For starters, here is the output of my 3 tests: $ info1 -bash: info1: command not found [perform@pun-rlnx55-bpa01 tavares]$ ./info1 Failed to bind to socket /tmp/livirt_proxy_conn Domains 0: 2 CPUs Failed to get scheduler inforamtion for Domain 0 Failed to get max vcpus for Domain 0 $ su root -c "./info1" Password: Domains 0: 2 CPUs Scheduler = credit Max VCPUs = 2 $ su root -c "chown 0:0 info1;chmod 6555 info1" Password: [perform@pun-rlnx55-bpa01 tavares]$ ./info1 Domains 0: 2 CPUs libvir: Xen Daemon error : this function is not supported by the hypervisor: unsupported in xendConfigVersion < 4 Failed to get scheduler inforamtion for Domain 0 Failed to get max vcpus for Domain 0 I have attached a zipped tar file which now contains the source code, the executable, the strace output from all three tests and the configuration files from /etc/libvirt. Please let me know if there is anything else that I can provide. I will also see if I could do this on the Oracle VM Dom0 (if I can remember which one I used) and run similar tests. Thanks. John -----Original Message----- From: Tavares, John Sent: Monday, July 12, 2010 7:38 AM To: 'Daniel P. Berrange' Cc: libvir-list@redhat.com; 'Tomen Tse'; Betley, Greg; Chen, Jianjiun; Tavares, John Subject: RE: [libvirt] inability to open local read-only connection Importance: High Hi Daniel, Sorry it took so long to back to this. I just recently ran into the same problem on RHEL 5.5 where I am running into problems using the root owned setuid executable to get domain information. Basically all that I needed to do to show this was to modify the example program dominfo/info1.c that ships with the libvirt source and added a function that attempts to get any information about that domain. In this case, I chose the virDomainGetSchedulerType() function. Here is the output of 3 separate runs, the first will be as a non-root user, the second is from root and the last is from the same executable but just root owned setuid and run as a normal user: $ ll info1 -r-xr-xr-x 1 perform perform 9326 Jul 12 16:50 info1 $ whoami perform $ ./info1 Domains 0: 2 CPUs $ su Password: # ./info1 Domains 0: 2 CPUs Scheduler = credit # chown 0:0 info1;chmod 6555 info1 # ll info1 -r-sr-sr-x 1 root root 9326 Jul 12 16:50 info1 # exit $ whoami perform $ ./info1 Domains 0: 2 CPUs libvir: Xen Daemon error : this function is not supported by the hypervisor: unsupported in xendConfigVersion < 4 $ virsh version error: failed to connect to the hypervisor $ su root -c "virsh version" Password: Compiled against library: libvir 0.6.3 Using library: libvir 0.6.3 Using API: Xen 3.0.1 Running hypervisor: Xen 3.1.0 As you can see, even virsh needs to be run as root in order to connect to the hypervisor. To help, I have attached both the source code and the executable I was using. Please let me know if anything else is required to get this problem resolved. Thanks. John -----Original Message----- From: Daniel P. Berrange [mailto:berrange@redhat.com] Sent: Monday, March 01, 2010 7:58 AM To: Tavares, John Cc: libvir-list@redhat.com Subject: Re: [libvirt] inability to open local read-only connection On Thu, Feb 25, 2010 at 02:34:02PM -0600, Tavares, John wrote:
I have been experimenting with using libvirt (0.3.3) on a variety of systems (RHEL, CentOS and Oracle VM). I have run into an issue when I try to open a local read-only connection to the hypervisor that is failing only on Oracle VM server release 2.2.0. I have created a root owned setuid executable that is effectively running as root, but even so, still cannot open the local read-only connection of the hypervisor. It only works if I run it directly as root. This is not an option. I do not understand why it works as is on my RHEL and CentOS machines, but not my Oracle machine. It would seem as thought it is not checking if the effective uid is root, just the uid.
A readonly connection essentially just goes to a separate UNIX socket, which has more relaxed permissions (mode 0777, instead of 0700). The kernel does the permissions checking when attempting to open it, so it should be using the effective ID.
Has anyone run into a similar issue or have any suggestions of what I might try to fix this issue or can tell me that this is a defect that needs (is) fixed??
I'd suggest trying to strace your process & see where it fails, and/or run with LIBVIRT_DEBUG=1 environemnt variable set. Regards, Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
On Mon, Jul 12, 2010 at 03:49:17PM -0500, Tavares, John wrote:
Hi Daniel,
I added some more info to this. I added a call to try to get the max number of vcpus as well. For starters, here is the output of my 3 tests:
$ info1 -bash: info1: command not found [perform@pun-rlnx55-bpa01 tavares]$ ./info1 Failed to bind to socket /tmp/livirt_proxy_conn Domains 0: 2 CPUs Failed to get scheduler inforamtion for Domain 0 Failed to get max vcpus for Domain 0
If you edit your demo program to pass 'xen+unix:///' to virConnectOpenReadonly() this should work as you expect Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
participants (4)
-
Daniel P. Berrange -
Eric Blake -
Justin Clift -
Tavares, John