9:18 p.m.
I have been trying do some simulations of an openstack environment on my
workstation that is running xen and libvirt.
I managed to create nested HVM environments under lx but found a number
of shortfalls in libxl code.
I have added a nestedhvm as a domain feature and was looking at
inspecting the domain configuration when I realized that the persistant
data is keept in config files in /var/lib/xen/userdata.....
Libvirt and lx have incompatible file names and are using different
config formats.
Libvirt keeps the data as XML and xl keeps them as xm config files.
This means that libvirt domains cannot be manged with xl or xl domains
managed by libvirt.
Part of me thinks that sticking with the XL file format would be nice
from the point of view of being able to use xenlight tools once the
domain is configured.
At the very least it may make sense to keep an XL copy of the config
file in a format that xenlight can use.
To achieve my original goal of managing a nested HVM environment from
libvirt I need to get the CPUID flags working but I do believe that it
would be nice to have the various xen and libvirt tools being able to
talk to each other.
Any pearls of wisdom would be greatly appreciated.
--
Alvin Starr || voice: (905)513-7688
Netvel Inc. || Cell: (416)806-0133
alvin(a)netvel.net ||
2:44 p.m.
On 05/26/2014 08:18 PM, Alvin Starr wrote:
I have been trying do some simulations of an openstack environment on
my
workstation that is running xen and libvirt.
I managed to create nested HVM environments under lx but found a number
of shortfalls in libxl code.
I'm not a libxl user myself, but can at least answer some things:
I have added a nestedhvm as a domain feature and was looking at
inspecting the domain configuration when I realized that the persistant
data is keept in config files in /var/lib/xen/userdata.....
Libvirt and lx have incompatible file names and are using different
config formats.
Libvirt keeps the data as XML and xl keeps them as xm config files.
This means that libvirt domains cannot be manged with xl or xl domains
managed by libvirt.
Part of me thinks that sticking with the XL file format would be nice
from the point of view of being able to use xenlight tools once the
domain is configured.
At the very least it may make sense to keep an XL copy of the config
file in a format that xenlight can use.
Libvirt is a higher layer than xl. The design decision is that if you
use libvirt to manage domains, you should use libvirt syntax (and not xl
syntax), because it makes it easier to translate between libxl or qemu
as your hypervisor all while being used to the common libvirt framework
(in fact, the reason libvirt exists at all is because of xen vs. qemu
issues 7 years ago needing a common wrapper). So libxl should NOT learn
libvirt XML config; but rather, libvirt should have a translation both
into and out of lower-layer config formats.
Such a command already exists, in the form of 'virsh domxml-from-native'
and 'domxml-to-native'; and according to the source code, the libvirt
driver for libxl domains has already supported conversions for the
"xen-xm" and "xen-sxpr" formats since libvirt 0.9.0. If the xl
config
format you are talking about is different than xen-xm or xen-sxpr, then
it's merely a matter of patching
src/libxl/libxl_driver:libxlConnectDomainXML{To,From}Native to
understand a third format. Patches welcome :)
--
Eric Blake eblake redhat com +1-919-301-3266
Libvirt virtualization library http://libvirt.org
3:45 p.m.
On 05/27/2014 03:44 PM, Eric Blake wrote:
On 05/26/2014 08:18 PM, Alvin Starr wrote:
> I have been trying do some simulations of an openstack environment on my
> workstation that is running xen and libvirt.
> I managed to create nested HVM environments under lx but found a number
> of shortfalls in libxl code.
>
I'm not a libxl user myself, but can at least answer some things:
> I have added a nestedhvm as a domain feature and was looking at
> inspecting the domain configuration when I realized that the persistant
> data is keept in config files in /var/lib/xen/userdata.....
>
> Libvirt and lx have incompatible file names and are using different
> config formats.
> Libvirt keeps the data as XML and xl keeps them as xm config files.
>
> This means that libvirt domains cannot be manged with xl or xl domains
> managed by libvirt.
> Part of me thinks that sticking with the XL file format would be nice
> from the point of view of being able to use xenlight tools once the
> domain is configured.
> At the very least it may make sense to keep an XL copy of the config
> file in a format that xenlight can use.
Libvirt is a higher layer than xl. The design decision is that if you
use libvirt to manage domains, you should use libvirt syntax (and not xl
syntax), because it makes it easier to translate between libxl or qemu
as your hypervisor all while being used to the common libvirt framework
(in fact, the reason libvirt exists at all is because of xen vs. qemu
issues 7 years ago needing a common wrapper). So libxl should NOT learn
libvirt XML config; but rather, libvirt should have a translation both
into and out of lower-layer config formats.
What has driven me to this point is
that libvirt does not have enough
control over a xen domain to let me run nested domains.
So I started doing stuff with libxl and the xl command.
I can understand livirt not being able to work with xl created domains
since xl is a lower level tool set but I would expect that xl should be
able to work with libvirt created domains and that cannot happen now.
I have added a few patches already to get some features enabled but I
will need to do some serious surgery to get things like CPUID enabled.
I was thinking of domxml-to/from-native and just using that code but it
exists as part of the old xm toolchain.
So I can use it more or less as it is or hack it into the xl toolchain.
I believe that the xm config format is a subset of the xl config format
but I am not really sure.
For my simple cases they do seem to be interchangeable.
I am happy to supply patches back to the community but I would like to
make sure that before I commit some real hackery that I am moving in the
right direction.
That being said what should I use for my software base to generate
patches against.
Right now I am working against the F20 SRPMs but I know that is wrong
for providing usable patches.
Such a command already exists, in the form of 'virsh domxml-from-native'
and 'domxml-to-native'; and according to the source code, the libvirt
driver for libxl domains has already supported conversions for the
"xen-xm" and "xen-sxpr" formats since libvirt 0.9.0. If the xl
config
format you are talking about is different than xen-xm or xen-sxpr, then
it's merely a matter of patching
src/libxl/libxl_driver:libxlConnectDomainXML{To,From}Native to
understand a third format. Patches welcome :)
--
Alvin Starr || voice: (905)513-7688
Netvel Inc. || Cell: (416)806-0133
alvin(a)netvel.net ||
4:44 p.m.
On 05/27/2014 02:45 PM, Alvin Starr wrote:
I am happy to supply patches back to the community but I would like to
make sure that before I commit some real hackery that I am moving in the
right direction.
I'll let other hackers more familiar with the libxl code chime in on
design decisions.
That being said what should I use for my software base to generate
patches against.
Right now I am working against the F20 SRPMs but I know that is wrong
for providing usable patches.
But this answer is easy: http://libvirt.org/hacking.html
--
Eric Blake eblake redhat com +1-919-301-3266
Libvirt virtualization library http://libvirt.org
1:04 a.m.
Alvin Starr wrote:
Adding to Eric's comments...
I have been trying do some simulations of an openstack environment
on
my workstation that is running xen and libvirt.
I managed to create nested HVM environments under lx but found a
number of shortfalls in libxl code.
Just to clarify terminology, libxl is the new interface for managing a
xen host. xl (not lx) is one client of this interface. The libvirt
libxl driver is another.
I have added a nestedhvm as a domain feature and was looking at
inspecting the domain configuration when I realized that the
persistant data is keept in config files in /var/lib/xen/userdata.....
Libvirt and lx have incompatible file names and are using different
config formats.
Libvirt keeps the data as XML and xl keeps them as xm config files.
Right. They are different libxl clients, and one could argue this is a
way to determine that.
This means that libvirt domains cannot be manged with xl or xl domains
managed by libvirt.
Which is correct behavior, right? I.e., one libxl client should not be
able to manipulate the domains of another?
Part of me thinks that sticking with the XL file format would be
nice
from the point of view of being able to use xenlight tools once the
domain is configured.
At the very least it may make sense to keep an XL copy of the config
file in a format that xenlight can use.
Ideally, the libvirt libxl driver should support all the configuration
options supported by xl, allowing it to be used in place of xl.
WRT importing an xl domain to libvirt, I think Eric's suggestion to use
domxml-{to,from}-native is the way to go. The libxl driver would need
to learn 'xen-xl' (slightly different than xen-xm) and 'xen-json'
formats so one could e.g. do
virsh domxml-from-native xen-xl /path/to/dom-cfg.xl > dom-cfg.xml
virsh define dom-cfg.xml
or
xl list --long dom > dom-json.cfg
virsh domxml-from-native xen-json dom-json.cfg > dom-cfg.xml
virsh define dom-cfg.xml
I suppose another option would be to support the equivalent of 'virsh
qemu-attach'.
To achieve my original goal of managing a nested HVM environment
from
libvirt I need to get the CPUID flags working but I do believe that it
would be nice to have the various xen and libvirt tools being able to
talk to each other.
It should be possible to support specifying CPUID flags if that is
exposed through the libxl interface. I think it is just a matter of
mapping libvirt's CPU model options to Xen's CPUID flags
http://libvirt.org/formatdomain.html#elementsCPU
Regards,
Jim
9:11 a.m.
On 05/28/2014 02:04 AM, Jim Fehlig wrote:
Alvin Starr wrote:
Adding to Eric's comments...
> I have been trying do some simulations of an openstack environment on
> my workstation that is running xen and libvirt.
> I managed to create nested HVM environments under lx but found a
> number of shortfalls in libxl code.
Just to clarify terminology, libxl is the new interface for managing a
xen host. xl (not lx) is one client of this interface. The libvirt
libxl driver is another.
Some times I am a bit dyslexic and xl becomes lx.
I guess I could have phrased that better but the short and long was that
I was forced to use lx because virsh/libvirt did not have enough control
over the domains to do what I needed.
> I have added a nestedhvm as a domain feature and was looking at
> inspecting the domain configuration when I realized that the
> persistant data is keept in config files in /var/lib/xen/userdata.....
>
> Libvirt and lx have incompatible file names and are using different
> config formats.
> Libvirt keeps the data as XML and xl keeps them as xm config files.
Right. They are different libxl clients, and one could argue this is a
way to determine that.
> This means that libvirt domains cannot be manged with xl or xl domains
> managed by libvirt.
Which is correct behavior, right? I.e., one libxl client should not be
able to manipulate the domains of another?
I can be convinced I am wrong but my take on this would be that the
domains are not owned by the tool used to create them but are owned by
the underlying technology(ie. xen).
From this perspective all tools should be able to manipulate to some
level any xen domain.
So either xl needs to be able to read and parse libvirt-xml userdata or
libvirt needs to be able to read xl userdata.
By analogy I would use libc/linux as an example.
Nobody would try to argue that programs created in one programing
language should be excluded from interacting with programs created in
another programming language.
Just think how life would be if perl programs could not invoke C
programs or python programs.
Ok. Now that I have flogged that one to death...
> Part of me thinks that sticking with the XL file format would be
nice
> from the point of view of being able to use xenlight tools once the
> domain is configured.
> At the very least it may make sense to keep an XL copy of the config
> file in a format that xenlight can use.
Ideally, the libvirt libxl driver should support all the configuration
options supported by xl, allowing it to be used in place of xl.
I agree here
completely.
WRT importing an xl domain to libvirt, I think Eric's suggestion to use
domxml-{to,from}-native is the way to go. The libxl driver would need
to learn 'xen-xl' (slightly different than xen-xm) and 'xen-json'
formats so one could e.g. do
virsh domxml-from-native xen-xl /path/to/dom-cfg.xl > dom-cfg.xml
virsh define dom-cfg.xml
or
xl list --long dom > dom-json.cfg
virsh domxml-from-native xen-json dom-json.cfg > dom-cfg.xml
virsh define dom-cfg.xml
I suppose another option would be to support the equivalent of 'virsh
qemu-attach'.
The metadata is stored in files like
userdata-d.28.da587c68-da14-11e3-b2d6-37945c264b9d.libvirt-xml
or
userdata-d.2.62d064fb-3eb4-5b46-3298-47dcb1b6c13f.xl
where the suffix is the persistant data format.
It would be possible to use domxml-to/from-native to convert one to the
other but my quick try just now kicked out a virsh internal error.
So domxml conversion needs to be fixed.
I guess the question would be.
Should there be a new xen-lx format or is extending xen-xm enough?
If there is to be a new xen-lx format should it be built into the new
libxl code or just added to the xen code?
Since the xend/xm interface is being replaced with libxl/xl will the xen
driver go away?
> To achieve my original goal of managing a nested HVM environment
from
> libvirt I need to get the CPUID flags working but I do believe that it
> would be nice to have the various xen and libvirt tools being able to
> talk to each other.
It should be possible to support specifying CPUID flags if that is
exposed through the libxl interface. I think it is just a matter of
mapping libvirt's CPU model options to Xen's CPUID flags
http://libvirt.org/formatdomain.html#elementsCPU
It looks to be doable.
Regards,
Jim
--
Alvin Starr || voice: (905)513-7688
Netvel Inc. || Cell: (416)806-0133
alvin(a)netvel.net ||
10:40 a.m.
Alvin Starr wrote:
On 05/28/2014 02:04 AM, Jim Fehlig wrote:
> Alvin Starr wrote:
>
> Adding to Eric's comments...
>
>> I have been trying do some simulations of an openstack environment on
>> my workstation that is running xen and libvirt.
>> I managed to create nested HVM environments under lx but found a
>> number of shortfalls in libxl code.
> Just to clarify terminology, libxl is the new interface for managing a
> xen host. xl (not lx) is one client of this interface. The libvirt
> libxl driver is another.
Some times I am a bit dyslexic and xl becomes lx.
Ok. In your writings I'll assume lx == xl.
I guess I could have phrased that better but the short and long was
that I was forced to use lx because virsh/libvirt did not have enough
control over the domains to do what I needed.
>
>> I have added a nestedhvm as a domain feature and was looking at
>> inspecting the domain configuration when I realized that the
>> persistant data is keept in config files in /var/lib/xen/userdata.....
>>
>> Libvirt and lx have incompatible file names and are using different
>> config formats.
>> Libvirt keeps the data as XML and xl keeps them as xm config files.
> Right. They are different libxl clients, and one could argue this is a
> way to determine that.
>
>> This means that libvirt domains cannot be manged with xl or xl domains
>> managed by libvirt.
> Which is correct behavior, right? I.e., one libxl client should not be
> able to manipulate the domains of another?
I can be convinced I am wrong but my take on this would be that the
domains are not owned by the tool used to create them but are owned by
the underlying technology(ie. xen).
I think a cloud admin would beg to differ, and would be unhappy if they
didn't own the instances they created.
From this perspective all tools should be able to manipulate to some
level any xen domain.
So either xl needs to be able to read and parse libvirt-xml userdata
or libvirt needs to be able to read xl userdata.
I do agree that xl should be the big hammer, and afaik it works as such
today. E.g. you can alway 'xl destroy' a domain created by other libxl
apps.
By analogy I would use libc/linux as an example.
Nobody would try to argue that programs created in one programing
language should be excluded from interacting with programs created in
another programming language.
Just think how life would be if perl programs could not invoke C
programs or python programs.
I think a better example of your libc/linux analogy is two processes
running on libc/linux. They share the functionality provided by
libc/linux, but don't share their data.
Ok. Now that I have flogged that one to death...
>> Part of me thinks that sticking with the XL file format would be nice
>> from the point of view of being able to use xenlight tools once the
>> domain is configured.
>> At the very least it may make sense to keep an XL copy of the config
>> file in a format that xenlight can use.
> Ideally, the libvirt libxl driver should support all the configuration
> options supported by xl, allowing it to be used in place of xl.
I agree here completely.
>
> WRT importing an xl domain to libvirt, I think Eric's suggestion to use
> domxml-{to,from}-native is the way to go. The libxl driver would need
> to learn 'xen-xl' (slightly different than xen-xm) and 'xen-json'
> formats so one could e.g. do
>
> virsh domxml-from-native xen-xl /path/to/dom-cfg.xl > dom-cfg.xml
> virsh define dom-cfg.xml
>
> or
>
> xl list --long dom > dom-json.cfg
> virsh domxml-from-native xen-json dom-json.cfg > dom-cfg.xml
> virsh define dom-cfg.xml
>
> I suppose another option would be to support the equivalent of 'virsh
> qemu-attach'.
The metadata is stored in files like
userdata-d.28.da587c68-da14-11e3-b2d6-37945c264b9d.libvirt-xml
or
userdata-d.2.62d064fb-3eb4-5b46-3298-47dcb1b6c13f.xl
where the suffix is the persistant data format.
It would be possible to use domxml-to/from-native to convert one to
the other but my quick try just now kicked out a virsh internal error.
So domxml conversion needs to be fixed.
I guess the question would be.
Should there be a new xen-lx format or is extending xen-xm enough?
To avoid confusion in the future, I think a new xen-xl format should be
added to the conversion code. Since the two formats are similar, they
can share parsing code.
If there is to be a new xen-lx format should it be built into the
new
libxl code or just added to the xen code?
It should be added to the common code in src/xenxs. xen_xm.c contains
the existing xm config parser.
Since the xend/xm interface is being replaced with libxl/xl will the
xen driver go away?
I think it will be around for quite some time. Many years will pass
before all xm/xend installations disappear :-).
Regards,
Jim
3823
days inactive
3824
days old
6 comments
3 participants
participants (3)
-
Alvin Starr -
Eric Blake -
Jim Fehlig