[libvirt] Some questions about the libvirt ESX driver
I'm playing with VMWare ESX 4.0 and I decided to try driving it from libvirt [git version]. Here are some random questions and issues I encountered, in no particular order. Listing domains works fine, but: (a) I have to specify ?no_verify=1 to get it to ignore the lack of valid https certificate. Is there some documentation on how to set up certificates correctly? [vSphere gives exactly the same error] (b) I have to type the root password (of the ESX server) each time. Can I avoid that? $ sudo ./tools/virsh -c 'esx://192.168.2.121/?no_verify=1' list --all Enter username for 192.168.2.121 [root]: Enter root password for 192.168.2.121: Id Name State ---------------------------------- - TestLinux shut off - TestWin shut off (c) Starting domains doesn't seem to work, and the error message is obscure: $ sudo ./tools/virsh -c 'esx://192.168.2.121/?no_verify=1' start TestLinux Enter username for 192.168.2.121 [root]: Enter root password for 192.168.2.121: error: Failed to start domain TestLinux error: internal error HTTP response code 500 for call to 'PowerOnVM_Task'. Fault: ServerFaultCode - The operation is not allowed in the current state. Is this to do with the "ask questions" API that was discussed on this list before? Specifying auto_answer=0 or auto_answer=1 didn't make any difference. (d) dumpxml XML output looks very good, and consistent with the other drivers. However it produces an odd <source file> attribute: <disk type='file' device='disk'> <driver name='lsilogic'/> <source file='[Storage1] TestLinux/TestLinux.vmdk'/> <target dev='sda' bus='scsi'/> </disk> <disk type='file' device='cdrom'> <source file='[Storage1] Fedora-12-x86_64-Live.iso'/> <target dev='hdc' bus='ide'/> </disk> (e) pool-list, net-list are not supported by the driver. That's a particular problem for me because I was really hoping to use the storage APIs from libvirt to access the VMDK files that contain domains (eg. [Storage1] TestLinux/TestLinux.vmdk in the example above). Is support for storage (particularly) and/or networks on the horizon for this driver? Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://et.redhat.com/~rjones/virt-top
2009/12/30 Richard W.M. Jones <rjones@redhat.com>:
I'm playing with VMWare ESX 4.0 and I decided to try driving it from libvirt [git version]. Here are some random questions and issues I encountered, in no particular order.
Listing domains works fine, but:
(a) I have to specify ?no_verify=1 to get it to ignore the lack of valid https certificate. Is there some documentation on how to set up certificates correctly? [vSphere gives exactly the same error]
The default ESX certificates are autogenerated and self-signed ones. The 'ESX Configuration Guide' [1] suggest to replace them by CA-signed ones. The 'ESX Configuration Guide' [1] contains a section about 'Replace a Default Certificate with a CA-Signed Certificate'. I created new self-signed certificates as described on the libvirt website [2] and copied them to the ESX server servercert.pem -> /etc/vmware/ssl/rui.crt serverkey.pem -> /etc/vmware/ssl/rui.key Then restart the hostd process either with service mgmt-vmware restart or if that doesn't work (like with my ESXi 3.5 on USB flash drive) use /etc/init.d/hostd restart On my Ubuntu box I copied the cacert.pem to /usr/share/ca-certificates/esx-certs/cacert.pem, appended esx-certs/cacert.pem to /etc/ca-certificates.conf and executed update-ca-certificates. After this libcurl can verify the ESX server certificate. libvirt uses explicit paths for its certificates for a remote TLS connection using gnutls. This is possible with libcurl too, using the CURLOPT_CAINFO option. For example the ESX driver could use LIBVIRT_CACERT as the remote driver does, but this would require the ESX server certificates to be signed by the same CA as the libvirtd certificates, because libcurl doesn't fall back to the system CA certificates if verification with the explicitly specified CA fails.
(b) I have to type the root password (of the ESX server) each time. Can I avoid that?
$ sudo ./tools/virsh -c 'esx://192.168.2.121/?no_verify=1' list --all Enter username for 192.168.2.121 [root]: Enter root password for 192.168.2.121: Id Name State ---------------------------------- - TestLinux shut off - TestWin shut off
The ESX driver needs to login to the ESX server. virsh uses the default auth callback that asks you to type in the credentials (see virConnectAuthCallbackDefault in libvirt.c). Currently the only option for automatic handling is policykit, but that doesn't work for the ESX driver because policykit is not meant to provide credentials. A possible way to "automatically" provide the credentials is to pass them in the connection URI like this: esx://<user>:<pass>@hostname. But this is currently not implemented in the ESX driver. Another option would be to add new automatic authentication methods to virConnectAuthCallbackDefault based on VIR_CRED_EXTERNAL and to change the ESX driver to use them.
(c) Starting domains doesn't seem to work, and the error message is obscure:
$ sudo ./tools/virsh -c 'esx://192.168.2.121/?no_verify=1' start TestLinux Enter username for 192.168.2.121 [root]: Enter root password for 192.168.2.121: error: Failed to start domain TestLinux error: internal error HTTP response code 500 for call to 'PowerOnVM_Task'. Fault: ServerFaultCode - The operation is not allowed in the current state.
Is this to do with the "ask questions" API that was discussed on this list before? Specifying auto_answer=0 or auto_answer=1 didn't make any difference.
No, the error message would say that there is a question blocking the task. This message comes directly from the ESX server, as part of the 500er response. I've never seen this error message from the ESX server before. Many errors reported by the ESX server contain a details section, but the deserialization of this details requires many new SOAP types and I haven't implemented this yet. Can you start the domain using the VI client? Maybe the domain really is in a state where it cannot be started. Can you reproduce this error with other domains?
(d) dumpxml XML output looks very good, and consistent with the other drivers. However it produces an odd <source file> attribute:
<disk type='file' device='disk'> <driver name='lsilogic'/> <source file='[Storage1] TestLinux/TestLinux.vmdk'/> <target dev='sda' bus='scsi'/> </disk> <disk type='file' device='cdrom'> <source file='[Storage1] Fedora-12-x86_64-Live.iso'/> <target dev='hdc' bus='ide'/> </disk>
What's wrong with it in your opinion? The <source file> attribute is correct. See the ESX driver website section [3] about this.
(e) pool-list, net-list are not supported by the driver.
That's a particular problem for me because I was really hoping to use the storage APIs from libvirt to access the VMDK files that contain domains (eg. [Storage1] TestLinux/TestLinux.vmdk in the example above). Is support for storage (particularly) and/or networks on the horizon for this driver?
Rich.
I plan to implement the ESX driver as complete as possible, this includes network and storage handling. I've took a look at this some time ago and saw some problems. For examples datastores don't have a UUID, so I may need to make them up and store them somewhere like the IBM Power driver does it for the domain UUIDs, the libvirt network model cannot represent a vSwitch ... I think these drivers are much harder to implemented than the main ESX driver, so don't expect any news on this in the next weeks, but stay tuned :) What do you mean by "to access the VMDK files that contain domains", just list them? What part of the storage driver would that involve? [1] http://www.vmware.com/pdf/vsphere4/r40/vsp_40_esx_server_config.pdf [2] http://www.libvirt.org/remote.html#Remote_certificates [3] http://www.libvirt.org/drvesx.html#xmlspecial Matthias
On Wed, Dec 30, 2009 at 09:29:10PM +0100, Matthias Bolte wrote:
I've never seen this error message from the ESX server before. Many errors reported by the ESX server contain a details section, but the deserialization of this details requires many new SOAP types and I haven't implemented this yet.
Can we get LIBVIRT_DEBUG=1 to just dump the SOAP response? /me checks the source ... It seems like currently debug output is hard-coded into a #define. Having it wired to LIBVIRT_DEBUG would allow people to debug these issues without needing to recompile.
Can you start the domain using the VI client? Maybe the domain really is in a state where it cannot be started.
OK, turns out that the VMWare server was in "maintenance mode".
What's wrong with it in your opinion? The <source file> attribute is correct. See the ESX driver website section [3] about this.
Nothing wrong with it, just a bit unusual. But as you say, it is a valid path according to ESX's unusual path convention.
(e) pool-list, net-list are not supported by the driver. [...] I plan to implement the ESX driver as complete as possible, this includes network and storage handling. I've took a look at this some time ago and saw some problems. For examples datastores don't have a UUID, so I may need to make them up and store them somewhere like the IBM Power driver does it for the domain UUIDs, the libvirt network model cannot represent a vSwitch ... I think these drivers are much harder to implemented than the main ESX driver, so don't expect any news on this in the next weeks, but stay tuned :)
What do you mean by "to access the VMDK files that contain domains", just list them?
It's so that we can inspect the VM's disk, for virt-inspector: http://libguestfs.org/virt-inspector.1.html Ideally we'd like to download all or part of the VM's disk (ie. VMDK file(s)). Thanks for your detailed reply! Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones New in Fedora 11: Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 70 libraries supprt'd http://fedoraproject.org/wiki/MinGW http://www.annexia.org/fedora_mingw
I was starting to use libvirt on a project, and realized that there was nothing in the API for snapshots. As I need to be able to take snapshots without taking down the machine (that and KVM's restore function didn't seem to be working terribly well), I added a few functions so that I could take snapshots, and I also added a screenshot function so I could give users a view of their snapshots in my application. At this point, as I only really use KVM/QEMU, that is the only hypervisor I implemented it for, and I'm sure there will be some problems with my structure in the other drivers. However, I figured I should at least submit the patch, even if it turns out to be of no use to anyone :) At this point for the QEMU driver, it only does snapshots for qcow2 drives, and will report an error if qemu says that no valid drives were found, or if for restoring/deleting the specified snapshot wasn't found. Philip Jameson
On 12/31/2009 07:42 PM, Philip Jameson wrote:
I was starting to use libvirt on a project, and realized that there was nothing in the API for snapshots. As I need to be able to take snapshots without taking down the machine (that and KVM's restore function didn't seem to be working terribly well), I added a few functions so that I could take snapshots, and I also added a screenshot function so I could give users a view of their snapshots in my application. At this point, as I only really use KVM/QEMU, that is the only hypervisor I implemented it for, and I'm sure there will be some problems with my structure in the other drivers. However, I figured I should at least submit the patch, even if it turns out to be of no use to anyone :)
At this point for the QEMU driver, it only does snapshots for qcow2 drives, and will report an error if qemu says that no valid drives were found, or if for restoring/deleting the specified snapshot wasn't found.
Hi Philip, Thanks for the patch, snapshot handling is definitely one of the big missing pieces in libvirt. For this patch to get better attention, I'd recommend making a top level post (for some reason this mail appeared as a reply to an unrelated thread), and split the patch into discreet parts. You can see an example of such a split at http://libvirt.org/api_extension.html. Using git send-email will make this process very easy. It's probably also a good idea to split the screenshot API from the snapshot pieces and submit them separately. FYI, Dan (the primary libvirt architect) did a write up a while ago describing what he thought a snapshotting API should look like: http://www.redhat.com/archives/libvir-list/2009-March/msg00205.html It's pretty similar to yours: it might help to adjust your API entry points to take into account Dan's ideas. Thanks for the contribution! - Cole
2009/12/31 Richard W.M. Jones <rjones@redhat.com>:
On Wed, Dec 30, 2009 at 09:29:10PM +0100, Matthias Bolte wrote:
I've never seen this error message from the ESX server before. Many errors reported by the ESX server contain a details section, but the deserialization of this details requires many new SOAP types and I haven't implemented this yet.
Can we get LIBVIRT_DEBUG=1 to just dump the SOAP response?
/me checks the source ...
It seems like currently debug output is hard-coded into a #define. Having it wired to LIBVIRT_DEBUG would allow people to debug these issues without needing to recompile.
I assume you refer to ESX_VI__CURL__ENABLE_DEBUG_OUTPUT and esxVI_CURL_Debug. I just noticed that this is broken and also doesn't dump the transfered SOAP messages. I don't think that dumping the complete SOAP communication using VIR_DEBUG is useful in general, but I'm going to dump the undeserialized SOAP faults so the message details are available in the debug output.
Can you start the domain using the VI client? Maybe the domain really is in a state where it cannot be started.
OK, turns out that the VMWare server was in "maintenance mode".
Ah. I should add a check to the ESX driver and output a warning or at least an info about the ESX server being in maintenance mode.
What's wrong with it in your opinion? The <source file> attribute is correct. See the ESX driver website section [3] about this.
Nothing wrong with it, just a bit unusual. But as you say, it is a valid path according to ESX's unusual path convention.
(e) pool-list, net-list are not supported by the driver. [...] I plan to implement the ESX driver as complete as possible, this includes network and storage handling. I've took a look at this some time ago and saw some problems. For examples datastores don't have a UUID, so I may need to make them up and store them somewhere like the IBM Power driver does it for the domain UUIDs, the libvirt network model cannot represent a vSwitch ... I think these drivers are much harder to implemented than the main ESX driver, so don't expect any news on this in the next weeks, but stay tuned :)
What do you mean by "to access the VMDK files that contain domains", just list them?
It's so that we can inspect the VM's disk, for virt-inspector:
http://libguestfs.org/virt-inspector.1.html
Ideally we'd like to download all or part of the VM's disk (ie. VMDK file(s)).
Hm, I haven't looked into the storage driver API in detail yet, but maybe it allows report an URL to access a volume. An ESX server exposes its datastores (local and remote) and all files in them over https://<hostname>/folder. You can also put files there using this mechanism.
Thanks for your detailed reply!
Rich.
Matthias
participants (4)
-
Cole Robinson -
Matthias Bolte -
Philip Jameson -
Richard W.M. Jones