On 05/27/2014 03:44 PM, Eric Blake wrote:

On 05/26/2014 08:18 PM, Alvin Starr wrote:

...

I have been trying do some simulations of an openstack environment on my workstation that is running xen and libvirt. I managed to create nested HVM environments under lx but found a number of shortfalls in libxl code.

I'm not a libxl user myself, but can at least answer some things:

...

I have added a nestedhvm as a domain feature and was looking at inspecting the domain configuration when I realized that the persistant data is keept in config files in /var/lib/xen/userdata.....

Libvirt and lx have incompatible file names and are using different config formats. Libvirt keeps the data as XML and xl keeps them as xm config files.

This means that libvirt domains cannot be manged with xl or xl domains managed by libvirt. Part of me thinks that sticking with the XL file format would be nice from the point of view of being able to use xenlight tools once the domain is configured. At the very least it may make sense to keep an XL copy of the config file in a format that xenlight can use. Libvirt is a higher layer than xl. The design decision is that if you use libvirt to manage domains, you should use libvirt syntax (and not xl syntax), because it makes it easier to translate between libxl or qemu as your hypervisor all while being used to the common libvirt framework (in fact, the reason libvirt exists at all is because of xen vs. qemu issues 7 years ago needing a common wrapper). So libxl should NOT learn libvirt XML config; but rather, libvirt should have a translation both into and out of lower-layer config formats. What has driven me to this point is that libvirt does not have enough control over a xen domain to let me run nested domains. So I started doing stuff with libxl and the xl command.

I can understand livirt not being able to work with xl created domains since xl is a lower level tool set but I would expect that xl should be able to work with libvirt created domains and that cannot happen now. I have added a few patches already to get some features enabled but I will need to do some serious surgery to get things like CPUID enabled. I was thinking of domxml-to/from-native and just using that code but it exists as part of the old xm toolchain. So I can use it more or less as it is or hack it into the xl toolchain. I believe that the xm config format is a subset of the xl config format but I am not really sure. For my simple cases they do seem to be interchangeable. I am happy to supply patches back to the community but I would like to make sure that before I commit some real hackery that I am moving in the right direction. That being said what should I use for my software base to generate patches against. Right now I am working against the F20 SRPMs but I know that is wrong for providing usable patches.

Such a command already exists, in the form of 'virsh domxml-from-native' and 'domxml-to-native'; and according to the source code, the libvirt driver for libxl domains has already supported conversions for the "xen-xm" and "xen-sxpr" formats since libvirt 0.9.0. If the xl config format you are talking about is different than xen-xm or xen-sxpr, then it's merely a matter of patching src/libxl/libxl_driver:libxlConnectDomainXML{To,From}Native to understand a third format. Patches welcome :)

-- Alvin Starr || voice: (905)513-7688 Netvel Inc. || Cell: (416)806-0133 alvin@netvel.net ||

Alvin Starr wrote:

On 05/28/2014 02:04 AM, Jim Fehlig wrote:

...

Alvin Starr wrote:

Adding to Eric's comments...

...

I have been trying do some simulations of an openstack environment on my workstation that is running xen and libvirt. I managed to create nested HVM environments under lx but found a number of shortfalls in libxl code. Just to clarify terminology, libxl is the new interface for managing a xen host. xl (not lx) is one client of this interface. The libvirt libxl driver is another. Some times I am a bit dyslexic and xl becomes lx.

Ok. In your writings I'll assume lx == xl.

I guess I could have phrased that better but the short and long was that I was forced to use lx because virsh/libvirt did not have enough control over the domains to do what I needed.

...

...

I have added a nestedhvm as a domain feature and was looking at inspecting the domain configuration when I realized that the persistant data is keept in config files in /var/lib/xen/userdata.....

Libvirt and lx have incompatible file names and are using different config formats. Libvirt keeps the data as XML and xl keeps them as xm config files. Right. They are different libxl clients, and one could argue this is a way to determine that.

...

This means that libvirt domains cannot be manged with xl or xl domains managed by libvirt. Which is correct behavior, right? I.e., one libxl client should not be able to manipulate the domains of another?

I can be convinced I am wrong but my take on this would be that the domains are not owned by the tool used to create them but are owned by the underlying technology(ie. xen).

I think a cloud admin would beg to differ, and would be unhappy if they didn't own the instances they created.

From this perspective all tools should be able to manipulate to some level any xen domain. So either xl needs to be able to read and parse libvirt-xml userdata or libvirt needs to be able to read xl userdata.

I do agree that xl should be the big hammer, and afaik it works as such today. E.g. you can alway 'xl destroy' a domain created by other libxl apps.

By analogy I would use libc/linux as an example. Nobody would try to argue that programs created in one programing language should be excluded from interacting with programs created in another programming language. Just think how life would be if perl programs could not invoke C programs or python programs.

I think a better example of your libc/linux analogy is two processes running on libc/linux. They share the functionality provided by libc/linux, but don't share their data.

Ok. Now that I have flogged that one to death...

...

...

Part of me thinks that sticking with the XL file format would be nice from the point of view of being able to use xenlight tools once the domain is configured. At the very least it may make sense to keep an XL copy of the config file in a format that xenlight can use. Ideally, the libvirt libxl driver should support all the configuration options supported by xl, allowing it to be used in place of xl. I agree here completely.

WRT importing an xl domain to libvirt, I think Eric's suggestion to use domxml-{to,from}-native is the way to go. The libxl driver would need to learn 'xen-xl' (slightly different than xen-xm) and 'xen-json' formats so one could e.g. do

virsh domxml-from-native xen-xl /path/to/dom-cfg.xl > dom-cfg.xml virsh define dom-cfg.xml

or

xl list --long dom > dom-json.cfg virsh domxml-from-native xen-json dom-json.cfg > dom-cfg.xml virsh define dom-cfg.xml

I suppose another option would be to support the equivalent of 'virsh qemu-attach'. The metadata is stored in files like

userdata-d.28.da587c68-da14-11e3-b2d6-37945c264b9d.libvirt-xml or userdata-d.2.62d064fb-3eb4-5b46-3298-47dcb1b6c13f.xl

where the suffix is the persistant data format. It would be possible to use domxml-to/from-native to convert one to the other but my quick try just now kicked out a virsh internal error.

So domxml conversion needs to be fixed.

I guess the question would be. Should there be a new xen-lx format or is extending xen-xm enough?

To avoid confusion in the future, I think a new xen-xl format should be added to the conversion code. Since the two formats are similar, they can share parsing code.

If there is to be a new xen-lx format should it be built into the new libxl code or just added to the xen code?

It should be added to the common code in src/xenxs. xen_xm.c contains the existing xm config parser.

Since the xend/xm interface is being replaced with libxl/xl will the xen driver go away?

I think it will be around for quite some time. Many years will pass before all xm/xend installations disappear :-). Regards, Jim