3:15 a.m.
----- Original Message -----
From: "Laine Stump" <lstump(a)redhat.com>
To: "Oved Ourfalli" <ovedo(a)redhat.com>
Cc: "Ayal Baron" <abaron(a)redhat.com>, "Barak Azulay"
<bazulay(a)redhat.com>, "Shahar Havivi" <shaharh(a)redhat.com>,
"Itamar Heim" <iheim(a)redhat.com>, "Dan Kenigsberg"
<danken(a)redhat.com>
Sent: Thursday, April 28, 2011 10:20:35 AM
Subject: Re: migration of vnlink VMs
Oved,
Would it be okay to repost this message to the thread on libvir-list
so
that other parties can add their thoughts?
Of course. I'm sending my answer to the libvirt list.
On 04/27/2011 09:58 AM, Oved Ourfalli wrote:
> Laine, hello.
>
> We read your proposal for abstraction of guest<--> host network
> connection in libvirt.
>
> You has an open issue there regarding the vepa/vnlink attributes:
> "3) What about the parameters in the<virtualport> element that are
> currently used by vepa/vnlink. Do those belong with the host, or
> with the guest?"
>
> The parameters for the virtualport element should be on the guest,
> and not the host, because a specific interface can run multiple
> profiles,
Are you talking about host interface or guest interface? If you mean
that multiple different profiles can be used when connecting to a
particular switch - as long as there are only a few different
profiles,
rather than each guest having its own unique profile, then it still
seems better to have the port profile live with the network definition
(and just define multiple networks, one for each port profile).
The profile names can be changed regularly, so it looks like it will be better to
put them in the guest level, so that the network host file won't have to be changed on
all hosts once something has changed in the profiles.
Also, you will have a duplication of data, writing all the profile name on all the hosts
that are connected to the vn-link/vepa switch.
> so it will be a mistake to define a profile to be interface
> specific on the host. Moreover, putting it in the guest level will
> enable us in the future (if supported by libvirt/qemu) to migrate
> a vm from a host with vepa/vnlink interfaces, to another host with
> a bridge, for example.
It seems to me like doing exactly the opposite would make it easier to
migrate to a host that used a different kind of switching (from vepa
to
vnlink, or from a bridged interface to vepa, etc), since the port
profile required for a particular host's network would be at the host
waiting to be used.
You are right, but we would want to have the option to prevent
that from happening in case we wouldn't want to allow it.
We can make the ability to migrate between different network types configurable, and we
would like an easy way to tell libvirt - "please allow/don't allow it".
> So, in the networks at the host level you will have:
> <network type='direct'>
> <name>red-network</name>
> <source mode='vepa'>
> <pool>
> <interface>
> <name>eth0</name>
> .....
> </interface>
> <interface>
> <name>eth4</name>
> .....
> </interface>
> <interface>
> <name>eth18</name>
> .....
> </interface>
> </pool>
> </source>
> </network>
>
> And in the guest you will have (for vepa):
> <interface type='network'>
> <source network='red-network'/>
> <virtualport type="802.1Qbg">
> <parameters managerid="11" typeid="1193047"
typeidversion="2"
> instanceid="09b11c53-8b5c-4eeb-8f00-d84eaa0aaa4f"/>
> </virtualport>
> </interface>
>
> Or (for vnlink):
> <interface type='network'>
> <source network='red-network'/>
> <virtualport type="802.1Qbh">
> <parameters profile_name="profile1"/>
> </virtualport>
> </interface>
This illustrates the problem I was wondering about - in your example
it
would not be possible for the guest to migrate from the host using a
vepa switch to the host using a vnlink switch (and it would be
possible
You are right. When trying to migrate between vepa and vnlink there will
be missing attributes in each in case we leave it on the host.
to migrate to a host using a standard bridge only if the virtualport
element was ignored). If the virtualport element lived with the
network
definition of red-network on each host, it could be migrated without
problem.
The only problematic thing would be if any of the attributes within
<parameters> was unique for each guest (I don't know anything about
the
individual attributes, but "instanceid" sounds like it might be
different for each guest).
> Then, when migrating from a vepa/vnlink host to another vepa/vnlink
> host containing red-network, the profile attributes will be
> available at the guest domain xml.
> In case the target host has a red-network, which isn't vepa/vnlink,
> we want to be able to choose whether to make the use of the profile
> attributes optional (i.e., libvirt won't fail in case of migrating
> to a network of another type), or mandatory (i.e., libvirt will fail
> in case of migration to a non-vepa/vnlink network).
>
> We have something similar in CPU flags:
> <cpu match="exact">
> <model>qemu64</model>
> <topology sockets="S" cores="C"
threads="T"/>
> <feature policy="require/optional/disable......"
> name="sse2"/>
> </cpu>
In this analogy, does "CPU flags" == "mode (vepa/vnlink/bridge)" or
does
"CPU flags" == "virtualport parameters" ? It seems like what
you're
wanting can be satisfied by simply not defining "red-network" on the
hosts that don't have the proper networking setup available (maybe
what
you *really* want to call it is "red-vnlink-network").
What I meant to say in that is that we would like to have the ability to say if an
attribute must me used, or not.
The issues you mention are indeed interesting. I'm cc-ing libvirt-list to see what
other people think.
Putting it on the guest will indeed make it problematic to migrate between networks that
need different parameters (vnlink/vepa for example).
Oved
7:45 a.m.
New subject: [libvirt] migration of vnlink VMs
On 04/28/2011 04:15 AM, Oved Ourfalli wrote:
> From: "Laine Stump"<lstump(a)redhat.com
> On 04/27/2011 09:58 AM, Oved Ourfalli wrote:
>> Laine, hello.
>>
>> We read your proposal for abstraction of guest<--> host network
>> connection in libvirt.
>>
>> You has an open issue there regarding the vepa/vnlink attributes:
>> "3) What about the parameters in the<virtualport> element that are
>> currently used by vepa/vnlink. Do those belong with the host, or
>> with the guest?"
>>
>> The parameters for the virtualport element should be on the guest,
>> and not the host, because a specific interface can run multiple
>> profiles,
> Are you talking about host interface or guest interface? If you mean
> that multiple different profiles can be used when connecting to a
> particular switch - as long as there are only a few different
> profiles,
> rather than each guest having its own unique profile, then it still
> seems better to have the port profile live with the network definition
> (and just define multiple networks, one for each port profile).
>
The profile names can be changed regularly, so it looks like it will be better to put
them in the guest level, so that the network host file won't have to be changed on all
hosts once something has changed in the profiles.
Also, you will have a duplication of data, writing all the profile name on all the hosts
that are connected to the vn-link/vepa switch.
But is it potentially the same for many/all guests, or is it necessarily
different for every guest? If it's the former, then do you have more
guests, or more hosts?
>> so it will be a mistake to define a profile to be
interface
>> specific on the host. Moreover, putting it in the guest level will
>> enable us in the future (if supported by libvirt/qemu) to migrate
>> a vm from a host with vepa/vnlink interfaces, to another host with
>> a bridge, for example.
> It seems to me like doing exactly the opposite would make it easier to
> migrate to a host that used a different kind of switching (from vepa
> to
> vnlink, or from a bridged interface to vepa, etc), since the port
> profile required for a particular host's network would be at the host
> waiting to be used.
You are right, but we would want to have the option to prevent that from happening in
case we wouldn't want to allow it.
We can make the ability to migrate between different network types configurable, and we
would like an easy way to tell libvirt - "please allow/don't allow it".
I *think* what you're getting at is this situation:
HostA has a group of interfaces that are connected to a vepa-capable
switch, HostB has a group of interfaces connected to a vnlink-capable
switch. Guest1 is allowed to connect either via a vnlink switch or a
vepa switch, but Guest2 should only use vepa.
In that case, HostA would have a network that had a pool of interfaces
and type "vepa", while HostB would have a pool of interfaces and a type
"vnlink". Guest1 could be accommodated by giving both networks the same
name, or Guest2 could be accommodated by giving each network a different
name (when migrating, if the dest. host doesn't have the desired
network, the migration would fail). However, using just the network
naming, it wouldn't be possible to allow both.
I don't think keeping the virtualport parameters only with the guest
would help (or hurt) this though. What would be needed would be to have
the information about network type *optionally* specified in the guest
interface config (as well as in the network config); if present the
migration would only succeed if the given network on the dest host
matched the given type (and parameters, if any) in the guest config.
>> So, in the networks at the host level you will have:
>> <network type='direct'>
>> <name>red-network</name>
>> <source mode='vepa'>
>> <pool>
>> <interface>
>> <name>eth0</name>
>> .....
>> </interface>
>> <interface>
>> <name>eth4</name>
>> .....
>> </interface>
>> <interface>
>> <name>eth18</name>
>> .....
>> </interface>
>> </pool>
>> </source>
>> </network>
>>
>> And in the guest you will have (for vepa):
>> <interface type='network'>
>> <source network='red-network'/>
>> <virtualport type="802.1Qbg">
>> <parameters managerid="11" typeid="1193047"
typeidversion="2"
>> instanceid="09b11c53-8b5c-4eeb-8f00-d84eaa0aaa4f"/>
>> </virtualport>
>> </interface>
>>
>> Or (for vnlink):
>> <interface type='network'>
>> <source network='red-network'/>
>> <virtualport type="802.1Qbh">
>> <parameters profile_name="profile1"/>
>> </virtualport>
>> </interface>
What would the interface for a 2nd guest of each type look like? Could
it be identical? Or might some parameters change for every single guest?
Perhaps it would be best to have virtualport parameters on both network
and guest interface XML, and merge the two to arrive at what's used (the
network definition could contain all the attributes that would be common
to all guests using that network on that host, and the guest interface
definition would contain extra parameters specific to that host. In the
case of a parameter being specified in both places, if they were not
identical, the migration would fail).
> This illustrates the problem I was wondering about - in your
example
> it
> would not be possible for the guest to migrate from the host using a
> vepa switch to the host using a vnlink switch (and it would be
> possible
You are right. When trying to migrate between vepa and vnlink there will be missing
attributes in each in case we leave it on the host.
(you mean if we leave the config on the *guest*, I guess...)
> to migrate to a host using a standard bridge only if the
virtualport
> element was ignored). If the virtualport element lived with the
> network
> definition of red-network on each host, it could be migrated without
> problem.
>
> The only problematic thing would be if any of the attributes within
> <parameters> was unique for each guest (I don't know anything about
> the
> individual attributes, but "instanceid" sounds like it might be
> different for each guest).
>> Then, when migrating from a vepa/vnlink host to another vepa/vnlink
>> host containing red-network, the profile attributes will be
>> available at the guest domain xml.
>> In case the target host has a red-network, which isn't vepa/vnlink,
>> we want to be able to choose whether to make the use of the profile
>> attributes optional (i.e., libvirt won't fail in case of migrating
>> to a network of another type), or mandatory (i.e., libvirt will fail
>> in case of migration to a non-vepa/vnlink network).
>>
>> We have something similar in CPU flags:
>> <cpu match="exact">
>> <model>qemu64</model>
>> <topology sockets="S" cores="C"
threads="T"/>
>> <feature policy="require/optional/disable......"
>> name="sse2"/>
>> </cpu>
> In this analogy, does "CPU flags" == "mode (vepa/vnlink/bridge)"
or
> does
> "CPU flags" == "virtualport parameters" ? It seems like what
you're
> wanting can be satisfied by simply not defining "red-network" on the
> hosts that don't have the proper networking setup available (maybe
> what
> you *really* want to call it is "red-vnlink-network").
What I meant to say in that is that we would like to have the ability to say if an
attribute must me used, or not.
Sure, it sounds useful. Would what I outlined above be sufficient? (It
would allow you to say "this guest must have a vepa network connection"
or "this guest can have any network connection, as long as it's named
"red-network". It *won't* allow saying "this guest must have vepa or
vnlink, bridge is not allowed, even if the network name is the same".
You could also put most of the config with the host network definition,
but allow, eg instanceid to be specified in the guest config.
The issues you mention are indeed interesting. I'm cc-ing
libvirt-list to see what other people think.
Putting it on the guest will indeed make it problematic to migrate between networks that
need different parameters (vnlink/vepa for example).
8:29 a.m.
New subject: [libvirt] migration of vnlink VMs
See my comments below.
Thank you,
Oved
----- Original Message -----
From: "Laine Stump" <laine(a)laine.org>
To: libvir-list(a)redhat.com
Sent: Friday, April 29, 2011 3:45:50 PM
Subject: Re: [libvirt] migration of vnlink VMs
On 04/28/2011 04:15 AM, Oved Ourfalli wrote:
>> From: "Laine Stump"<lstump(a)redhat.com
>> On 04/27/2011 09:58 AM, Oved Ourfalli wrote:
>>> Laine, hello.
>>>
>>> We read your proposal for abstraction of guest<--> host network
>>> connection in libvirt.
>>>
>>> You has an open issue there regarding the vepa/vnlink attributes:
>>> "3) What about the parameters in the<virtualport> element that
are
>>> currently used by vepa/vnlink. Do those belong with the host, or
>>> with the guest?"
>>>
>>> The parameters for the virtualport element should be on the guest,
>>> and not the host, because a specific interface can run multiple
>>> profiles,
>> Are you talking about host interface or guest interface? If you
>> mean
>> that multiple different profiles can be used when connecting to a
>> particular switch - as long as there are only a few different
>> profiles,
>> rather than each guest having its own unique profile, then it still
>> seems better to have the port profile live with the network
>> definition
>> (and just define multiple networks, one for each port profile).
>>
> The profile names can be changed regularly, so it looks like it will
> be better to put them in the guest level, so that the network host
> file won't have to be changed on all hosts once something has
> changed in the profiles.
>
> Also, you will have a duplication of data, writing all the profile
> name on all the hosts that are connected to the vn-link/vepa switch.
But is it potentially the same for many/all guests, or is it
necessarily
different for every guest? If it's the former, then do you have more
guests, or more hosts?
I guess it will be the same for many guests. There will be some profiles, and each
group of guests will use the same profile, according to its demands.
>>> so it will be a mistake to define a profile to be interface
>>> specific on the host. Moreover, putting it in the guest level
>>> will
>>> enable us in the future (if supported by libvirt/qemu) to
>>> migrate
>>> a vm from a host with vepa/vnlink interfaces, to another host
>>> with
>>> a bridge, for example.
>> It seems to me like doing exactly the opposite would make it easier
>> to
>> migrate to a host that used a different kind of switching (from
>> vepa
>> to
>> vnlink, or from a bridged interface to vepa, etc), since the port
>> profile required for a particular host's network would be at the
>> host
>> waiting to be used.
> You are right, but we would want to have the option to prevent that
> from happening in case we wouldn't want to allow it.
> We can make the ability to migrate between different network types
> configurable, and we would like an easy way to tell libvirt -
> "please allow/don't allow it".
I *think* what you're getting at is this situation:
HostA has a group of interfaces that are connected to a vepa-capable
switch, HostB has a group of interfaces connected to a vnlink-capable
switch. Guest1 is allowed to connect either via a vnlink switch or a
vepa switch, but Guest2 should only use vepa.
In that case, HostA would have a network that had a pool of interfaces
and type "vepa", while HostB would have a pool of interfaces and a
type
"vnlink". Guest1 could be accommodated by giving both networks the
same
name, or Guest2 could be accommodated by giving each network a
different
name (when migrating, if the dest. host doesn't have the desired
network, the migration would fail). However, using just the network
naming, it wouldn't be possible to allow both.
I don't think keeping the virtualport parameters only with the guest
would help (or hurt) this though. What would be needed would be to
have
the information about network type *optionally* specified in the guest
interface config (as well as in the network config); if present the
migration would only succeed if the given network on the dest host
matched the given type (and parameters, if any) in the guest config.
That would be great. It will enable the flexibility we need.
>>> So, in the networks at the host level you will have:
>>> <network type='direct'>
>>> <name>red-network</name>
>>> <source mode='vepa'>
>>> <pool>
>>> <interface>
>>> <name>eth0</name>
>>> .....
>>> </interface>
>>> <interface>
>>> <name>eth4</name>
>>> .....
>>> </interface>
>>> <interface>
>>> <name>eth18</name>
>>> .....
>>> </interface>
>>> </pool>
>>> </source>
>>> </network>
>>>
>>> And in the guest you will have (for vepa):
>>> <interface type='network'>
>>> <source network='red-network'/>
>>> <virtualport type="802.1Qbg">
>>> <parameters managerid="11" typeid="1193047"
typeidversion="2"
>>> instanceid="09b11c53-8b5c-4eeb-8f00-d84eaa0aaa4f"/>
>>> </virtualport>
>>> </interface>
>>>
>>> Or (for vnlink):
>>> <interface type='network'>
>>> <source network='red-network'/>
>>> <virtualport type="802.1Qbh">
>>> <parameters profile_name="profile1"/>
>>> </virtualport>
>>> </interface>
What would the interface for a 2nd guest of each type look like? Could
it be identical? Or might some parameters change for every single
guest?
For vn-link it will be the same, just the profile_name.
As for vepa, the instanceid is vm specific so it should be on the guest (taken from
http://libvirt.org/formatdomain.html):
"managerid - The VSI Manager ID identifies the database containing the VSI type and
instance definitions. This is an integer value and the value 0 is reserved.
typeid - The VSI Type ID identifies a VSI type characterizing the network access. VSI
types are typically managed by network administrator. This is an integer value.
typeidversion - The VSI Type Version allows multiple versions of a VSI Type. This is an
integer value.
instanceid - The VSI Instance ID Identifier is generated when a VSI instance (i.e. a
virtual interface of a virtual machine) is created. This is a globally unique
identifier."
That's what we know on vepa an vn-link now. I guess that when we'll have the
possibility to test these environments we will learn more on them.
Perhaps it would be best to have virtualport parameters on both
network
and guest interface XML, and merge the two to arrive at what's used
(the
network definition could contain all the attributes that would be
common
to all guests using that network on that host, and the guest interface
definition would contain extra parameters specific to that host. In
the
case of a parameter being specified in both places, if they were not
identical, the migration would fail).
Sounds good.
>> This illustrates the problem I was wondering about - in
your
>> example
>> it
>> would not be possible for the guest to migrate from the host using
>> a
>> vepa switch to the host using a vnlink switch (and it would be
>> possible
> You are right. When trying to migrate between vepa and vnlink there
> will be missing attributes in each in case we leave it on the host.
(you mean if we leave the config on the *guest*, I guess...)
>> to migrate to a host using a standard bridge only if the
>> virtualport
>> element was ignored). If the virtualport element lived with the
>> network
>> definition of red-network on each host, it could be migrated
>> without
>> problem.
>>
>> The only problematic thing would be if any of the attributes within
>> <parameters> was unique for each guest (I don't know anything about
>> the
>> individual attributes, but "instanceid" sounds like it might be
>> different for each guest).
>>> Then, when migrating from a vepa/vnlink host to another
>>> vepa/vnlink
>>> host containing red-network, the profile attributes will be
>>> available at the guest domain xml.
>>> In case the target host has a red-network, which isn't
>>> vepa/vnlink,
>>> we want to be able to choose whether to make the use of the
>>> profile
>>> attributes optional (i.e., libvirt won't fail in case of migrating
>>> to a network of another type), or mandatory (i.e., libvirt will
>>> fail
>>> in case of migration to a non-vepa/vnlink network).
>>>
>>> We have something similar in CPU flags:
>>> <cpu match="exact">
>>> <model>qemu64</model>
>>> <topology sockets="S" cores="C"
threads="T"/>
>>> <feature
policy="require/optional/disable......"
>>> name="sse2"/>
>>> </cpu>
>> In this analogy, does "CPU flags" == "mode
(vepa/vnlink/bridge)" or
>> does
>> "CPU flags" == "virtualport parameters" ? It seems like what
you're
>> wanting can be satisfied by simply not defining "red-network" on
>> the
>> hosts that don't have the proper networking setup available (maybe
>> what
>> you *really* want to call it is "red-vnlink-network").
> What I meant to say in that is that we would like to have the
> ability to say if an attribute must me used, or not.
Sure, it sounds useful. Would what I outlined above be sufficient? (It
would allow you to say "this guest must have a vepa network
connection"
or "this guest can have any network connection, as long as it's named
"red-network". It *won't* allow saying "this guest must have vepa or
vnlink, bridge is not allowed, even if the network name is the same".
You could also put most of the config with the host network
definition,
but allow, eg instanceid to be specified in the guest config.
I think this would indeed be enough.
> The issues you mention are indeed interesting. I'm cc-ing
> libvirt-list to see what other people think.
> Putting it on the guest will indeed make it problematic to migrate
> between networks that need different parameters (vnlink/vepa for
> example).
--
libvir-list mailing list
libvir-list(a)redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list
3:12 p.m.
New subject: [libvirt] migration of vnlink VMs
To "cut to the chase", go down to the end of the message, where I
outline the proposed XML changes to support this. If everyone approves
of it, I'll make the RNG based on that description and update the
parser, then worry about filling in the functionality.
On 04/29/2011 09:29 AM, Oved Ourfalli wrote:
See my comments below.
Thank you,
Oved
This is very useful, Oved! Thanks for the replies.
----- Original Message -----
> From: "Laine Stump"<laine(a)laine.org>
> To: libvir-list(a)redhat.com
> Sent: Friday, April 29, 2011 3:45:50 PM
> Subject: Re: [libvirt] migration of vnlink VMs
> On 04/28/2011 04:15 AM, Oved Ourfalli wrote:
>>> From: "Laine Stump"<lstump(a)redhat.com
>>> On 04/27/2011 09:58 AM, Oved Ourfalli wrote:
>>>> Laine, hello.
>>>>
>>>> We read your proposal for abstraction of guest<--> host network
>>>> connection in libvirt.
>>>>
>>>> You has an open issue there regarding the vepa/vnlink attributes:
>>>> "3) What about the parameters in the<virtualport> element
that are
>>>> currently used by vepa/vnlink. Do those belong with the host, or
>>>> with the guest?"
>>>>
>>>> The parameters for the virtualport element should be on the guest,
>>>> and not the host, because a specific interface can run multiple
>>>> profiles,
>>> Are you talking about host interface or guest interface? If you
>>> mean
>>> that multiple different profiles can be used when connecting to a
>>> particular switch - as long as there are only a few different
>>> profiles,
>>> rather than each guest having its own unique profile, then it still
>>> seems better to have the port profile live with the network
>>> definition
>>> (and just define multiple networks, one for each port profile).
>>>
>> The profile names can be changed regularly, so it looks like it will
>> be better to put them in the guest level, so that the network host
>> file won't have to be changed on all hosts once something has
>> changed in the profiles.
>>
>> Also, you will have a duplication of data, writing all the profile
>> name on all the hosts that are connected to the vn-link/vepa switch.
> But is it potentially the same for many/all guests, or is it
> necessarily
> different for every guest? If it's the former, then do you have more
> guests, or more hosts?
>
I guess it will be the same for many guests. There will be some profiles, and each group
of guests will use the same profile, according to its demands.
Except instanceid, as you point out below. Since there is at least one
exception to the "same for all guests", we do need to keep <virtualport>
with the guest, but since many are changed, we should also supply it
with the network. Things are starting to take shape now :-)
>>>> so it will be a mistake to define a profile to be
interface
>>>> specific on the host. Moreover, putting it in the guest level
>>>> will
>>>> enable us in the future (if supported by libvirt/qemu) to
>>>> migrate
>>>> a vm from a host with vepa/vnlink interfaces, to another host
>>>> with
>>>> a bridge, for example.
>>> It seems to me like doing exactly the opposite would make it easier
>>> to
>>> migrate to a host that used a different kind of switching (from
>>> vepa
>>> to
>>> vnlink, or from a bridged interface to vepa, etc), since the port
>>> profile required for a particular host's network would be at the
>>> host
>>> waiting to be used.
>> You are right, but we would want to have the option to prevent that
>> from happening in case we wouldn't want to allow it.
>> We can make the ability to migrate between different network types
>> configurable, and we would like an easy way to tell libvirt -
>> "please allow/don't allow it".
> I *think* what you're getting at is this situation:
>
> HostA has a group of interfaces that are connected to a vepa-capable
> switch, HostB has a group of interfaces connected to a vnlink-capable
> switch. Guest1 is allowed to connect either via a vnlink switch or a
> vepa switch, but Guest2 should only use vepa.
>
> In that case, HostA would have a network that had a pool of interfaces
> and type "vepa", while HostB would have a pool of interfaces and a
> type
> "vnlink". Guest1 could be accommodated by giving both networks the
> same
> name, or Guest2 could be accommodated by giving each network a
> different
> name (when migrating, if the dest. host doesn't have the desired
> network, the migration would fail). However, using just the network
> naming, it wouldn't be possible to allow both.
>
> I don't think keeping the virtualport parameters only with the guest
> would help (or hurt) this though. What would be needed would be to
> have
> the information about network type *optionally* specified in the guest
> interface config (as well as in the network config); if present the
> migration would only succeed if the given network on the dest host
> matched the given type (and parameters, if any) in the guest config.
>
That would be great. It will enable the flexibility we need
>>>> So, in the networks at the host level you will have:
>>>> <network type='direct'>
>>>> <name>red-network</name>
>>>> <source mode='vepa'>
>>>> <pool>
>>>> <interface>
>>>> <name>eth0</name>
>>>> .....
>>>> </interface>
>>>> <interface>
>>>> <name>eth4</name>
>>>> .....
>>>> </interface>
>>>> <interface>
>>>> <name>eth18</name>
>>>> .....
>>>> </interface>
>>>> </pool>
I'm assuming there will never be a case when an interface pool might
need to be used by *both* a vepa network and a vnlink network (doesn't
really make sense - it will be physically connected to a switch that
supports vnlink, or one that supports vepa; by definition it can't be
connected to both (I don't foresee any switch that supports both!). So
defining the pool inside network should work okay. Since each interface
will always have a name (and just one name), I think it makes sense to
make that an attribute rather than subelement.
>>>> </source>
>>>> </network>
>>>>
>>>> And in the guest you will have (for vepa):
>>>> <interface type='network'>
>>>> <source network='red-network'/>
and if you want to force connecting only to vepa or vnlink, you can do:
<source network='red-network' mode='vepa'/>
for example.
>>>> <virtualport type="802.1Qbg">
>>>> <parameters managerid="11" typeid="1193047"
typeidversion="2"
>>>> instanceid="09b11c53-8b5c-4eeb-8f00-d84eaa0aaa4f"/>
>>>> </virtualport>
Likewise, anything that's specified here MUST either match what is in
the <network> definition on the host, or be unspecified on the host. The
final virtualport definition used will be the OR of the two.
>>>> </interface>
>>>>
>>>> Or (for vnlink):
>>>> <interface type='network'>
>>>> <source network='red-network'/>
>>>> <virtualport type="802.1Qbh">
>>>> <parameters profile_name="profile1"/>
>>>> </virtualport>
>>>> </interface>
> What would the interface for a 2nd guest of each type look like? Could
> it be identical? Or might some parameters change for every single
> guest?
For vn-link it will be the same, just the profile_name.
As for vepa, the instanceid is vm specific so it should be on the guest (taken from
http://libvirt.org/formatdomain.html):
"managerid - The VSI Manager ID identifies the database containing the VSI type and
instance definitions. This is an integer value and the value 0 is reserved.
typeid - The VSI Type ID identifies a VSI type characterizing the network access. VSI
types are typically managed by network administrator. This is an integer value.
typeidversion - The VSI Type Version allows multiple versions of a VSI Type. This is an
integer value.
instanceid - The VSI Instance ID Identifier is generated when a VSI instance (i.e. a
virtual interface of a virtual machine) is created. This is a globally unique
identifier."
That's what we know on vepa an vn-link now. I guess that when we'll have the
possibility to test these environments we will learn more on them.
> Perhaps it would be best to have virtualport parameters on both
> network
> and guest interface XML, and merge the two to arrive at what's used
> (the
> network definition could contain all the attributes that would be
> common
> to all guests using that network on that host, and the guest interface
> definition would contain extra parameters specific to that host. In
> the
> case of a parameter being specified in both places, if they were not
> identical, the migration would fail).
>
Sounds good.
Okay, here's a brief description of what I *think* will work. I'll build
up the RNG based on this pseudo-xml:
For the <interface> definition in the guest XML, the main change will be
that <source .. mode='something'> will be valid (but optional) when
interface type='network' - in this case, it will just be used to match
against the source mode of the network on the host. <virtualport> will
also become valid for type='network', and will serve two purposes:
1) if there is a mismatch with the virtualport on the host network, the
migrate/start will fail.
2) It will be ORed with <virtualport> on the host network to arrive at
the virtualport settings actually used.
For example:
<interface type='network'>
<source network='red-network' mode='vepa'/>
<virtualport type='802.1Qbg'>
<parameters instanceid='09b11c53-8b5c-4eeb-8f00-d84eaa0aaa4f'/>
</virtualport>
<mac address='xx:xx:.....'/>
</interface>
(NB: if "mode" isn't specified, and the host network is actually a
bridge or virtual network, the contents of virtualport will be ignored.)
<network> will be expanded by giving it an optional "type" attribute
(which will default to 'virtual'), <source> subelement, and
<virtualport> subelement. When type='bridge', you can specify source
exactly as you would in a domain <interface> definition:
<network type='bridge'>
<name>red-network</name>
<source bridge='br0'/>
</network>
When type='direct', again you can specify source and virtualport pretty
much as you would in an interface definition:
<network type='direct'>
<name>red-network</name>
<source dev='eth0' mode='vepa'/>
<virtualport type='802.1Qbg'>
<parameters managerid="11" typeid="1193047"
typeidversion="2"
instanceid='09b11c53-8b5c-4eeb-8f00-d84eaa0aaa4f'/>
</virtualport>
</network>
However, dev would be optional - if not specified, we would expect a
pool of interfaces to be defined within source, eg:
<network type='direct'>
<name>red-network</name>
<source mode='vepa'>
<pool>
<interface name='eth10' maxConnect='1'/>
<interface name='eth11' maxConnect='1'/>
<interface name='eth12' maxConnect='1'/>
<interface name='eth13' maxConnect='1'/>
<interface name='eth14' maxConnect='1'/>
<interface name='eth25' maxConnect='5'/>
</pool>
</source>
<virtualport ...... />
</network>
at connect time, source dev would be allocated from the pool of
interfaces, round robin, with each interface having at most maxConnect
connections to guests at any given time.
Again, <virtualport> is optional, and if specified would have the same
purpose as <virtualport> in the interface definition.
Does this look like it covers everything?
BTW, for all the people asking about sectunnel, openvswitch, and vde -
can you see how those would fit in with this? In particular, do you see
any conflicts? (It's easy to add more stuff on later if something is
just missing, but much more problematic if I put something in that is
just plain wrong).
9:21 a.m.
New subject: [libvirt] migration of vnlink VMs
On Fri, Apr 29, 2011 at 04:12:55PM -0400, Laine Stump wrote:
Okay, here's a brief description of what I *think* will work.
I'll
build up the RNG based on this pseudo-xml:
For the <interface> definition in the guest XML, the main change
will be that <source .. mode='something'> will be valid (but
optional) when interface type='network' - in this case, it will just
be used to match against the source mode of the network on the host.
<virtualport> will also become valid for type='network', and will
serve two purposes:
1) if there is a mismatch with the virtualport on the host network,
the migrate/start will fail.
2) It will be ORed with <virtualport> on the host network to arrive
at the virtualport settings actually used.
For example:
<interface type='network'>
<source network='red-network' mode='vepa'/>
IMHO having a 'mode' here is throwing away the main reason for
using type=network in the first place - namely independance
from this host config element.
<virtualport type='802.1Qbg'>
<parameters instanceid='09b11c53-8b5c-4eeb-8f00-d84eaa0aaa4f'/>
</virtualport>
<mac address='xx:xx:.....'/>
</interface>
(NB: if "mode" isn't specified, and the host network is actually a
bridge or virtual network, the contents of virtualport will be
ignored.)
<network> will be expanded by giving it an optional "type" attribute
(which will default to 'virtual'), <source> subelement, and
<virtualport> subelement. When type='bridge', you can specify source
exactly as you would in a domain <interface> definition:
<network type='bridge'>
<name>red-network</name>
<source bridge='br0'/>
</network>
When type='direct', again you can specify source and virtualport
pretty much as you would in an interface definition:
<network type='direct'>
<name>red-network</name>
<source dev='eth0' mode='vepa'/>
<virtualport type='802.1Qbg'>
<parameters managerid="11" typeid="1193047"
typeidversion="2"
instanceid='09b11c53-8b5c-4eeb-8f00-d84eaa0aaa4f'/>
</virtualport>
</network>
None of this really feels right to me. With this proposed
schema, there is basically nothing in common between the
existing functionality for <network> and this new functionality
except for the <name> and <uuid> elements.
Apps which know how to deal with existing <network> schema
will have no ability to interpret this new data at all.
Quite probably they will mis-interpet it as providing an
isolated virtual network, with no IP addr set, since this
design isn't actually changing any attribute value that
they currently look for.
Either we need to make this align with the existing schema,
or we need to put this under a completely separate set of
APIs. I think we can likely do better with the schema design
and achieve the former.
However, dev would be optional - if not specified, we would expect a
pool of interfaces to be defined within source, eg:
<network type='direct'>
<name>red-network</name>
<source mode='vepa'>
<pool>
<interface name='eth10' maxConnect='1'/>
<interface name='eth11' maxConnect='1'/>
<interface name='eth12' maxConnect='1'/>
<interface name='eth13' maxConnect='1'/>
<interface name='eth14' maxConnect='1'/>
<interface name='eth25' maxConnect='5'/>
</pool>
</source>
<virtualport ...... />
</network>
I don't really like the fact that this design has special
cased the num(intefaces) == 1 case to have a completely
different XML schema. eg we have this:
<source dev='eth0' mode='vepa'/>
And this
<source mode='vepa'>
<pool>
<interface name='eth10' maxConnect='1'/>
</pool>
both meaning the same thing. There should only be one
representation in the schema for this kind of thing.
BTW, for all the people asking about sectunnel, openvswitch, and vde
- can you see how those would fit in with this? In particular, do
you see any conflicts? (It's easy to add more stuff on later if
something is just missing, but much more problematic if I put
something in that is just plain wrong).
As mentioned above, I think this design is wrong, because it is not
taking any account of the current schema for <network> which defines
the various routed modes.
Currently <network> supports 3 connectivity modes
- Non-routed network, separate subnet (no <forward> element present)
- Routed network, separate subnet with NAT (<forward mode='nat'/>)
- Routed network, separate subnet (<forward mode='route'/>)
Following on from this, I can see another couple of routed modes
- Routed network, IP subnetting
- Routed network, separate subnet with VPN
And the core goal here is to replae type=bridge and type=direct from the
domain XML, which means we're adding several bridging modes
- Bridged network, eth + bridge + tap (akin to type=bridge)
- Bridged network, eth + macvtap (akin to type=direct)
- Bridged network, sriov eth + bridge + tap (akin to type=bridge)
- Bridged network, sriov eth + macvtap (akin to type=direct)
The macvtap can be in 4 modes, so perhaps it is probably better to
consider them separately
- Bridged network, eth + bridge + tap
- Bridged network, eth + macvtap + vepa
- Bridged network, eth + macvtap + private
- Bridged network, eth + macvtap + passthrough
- Bridged network, eth + macvtap + bridge
- Bridged network, sriov eth + bridge + tap
- Bridged network, sriov eth + macvtap + vepa
- Bridged network, sriov eth + macvtap + private
- Bridged network, sriov eth + macvtap + passthrough
- Bridged network, sriov eth + macvtap + bridge
I can also perhaps imagine another VPN mode:
- Bridged network, with VPN
The current routed modes can route to anywhere, or be restricted to
a particular network interface eg with <forward dev='eth0'/>. It
only allows for a single interface, though even for routed modes it
could be desirable to list multiple devs.
The other big distinction is that the <network> modes which do routing,
include interface configuration data (ie the IP addrs & bridge name)
which is configured on the fly. It looks like with the bridged modes,
you're assuming the app has statically configured the interfaces via
the virInterface APIs already, and this just points to an existing
configured interface. This isn't neccessarily a bad thing, just an
observation of a significant difference.
So if we ignore the <ip> and <domain> elements from the current
<network>
schema, then there are a handful of others which we need to have a plan
for
<forward mode='nat|route'/> (omitted completely for isolated networks)
<bridge name="virbr0" /> (auto-generated/filled if omitted)
<mac address='....'/> (auto-generated/filled if omitted)
The <forward> element can have an optional dev= attribute.
I think the key attribute is the <forward> mode= attribute. I think we
should be adding further values to that attribute for the new network
modes we want to support. We should also make use of the dev= attribute
on <forward> where practical, and/or extend it.
We could expand the list of <foward> mode values in a flat list
- route
- nat
- bridge (brctl)
- vepa
- private
- passthru
- bridge (macvtap)
NB: really need to avoid using 'bridge' in terminology, since all
5 of the last options are really 'bridge'.
Or we could introduce a extra attribute, and have a 2 level list
- <forward layer='link'/> (for all ethernet layer bridging)
- <forward layer='network'/> (for all IP layer bridging aka routing)
So the current modes would be
<forward layer='network' mode='route|nat'/>
And new bridging modes would be
<forward layer='link'
mode='bridge-brctl|vepa|private|passthru|bridge-macvtap'/>
For the brctl/macvtap modes, the dev= attribute on <forward> could point to
the NIC being used, while with brctl modes, <bridge> would also be present.
In the SRIOV case, we potentiallly need a list of interfaces. For this we
probably want to use
<forward dev='eth0'>
<interface dev='eth0'/>
<interface dev='eth1'/>
<interface dev='eth2'/>
...
</forward>
NB, the first interface is always to be listed both as a dev= attribute
(for compat with existing apps) *and* as a child <interface> element (for
apps knowing the new schema).
The maxConnect= attribute from your examples above is an interesting
thing. I'm not sure whether that is neccessarily a good idea. It feels
similar to VMWare's "port group" idea, but I don't think having a
simple 'maxConnect=' attribute is sufficient to let us represent the
vmware port group idea. I think we might need an more explicit
element eg
<portgroup count='5'>
<interface dev='eth2'/>
</portgroup>
eg, so this associates a port group which allows 5 clients (VM NICs)
with the uplink provided by eth2 (which is assumed to be listed
under <forward>).
So a complete SRIOV example might be
<network>
<name>Foo</name>
<forward dev='eth0' layer='link' mode='vepa'>
<interface dev='eth0'/>
<interface dev='eth1'/>
<interface dev='eth2'/>
...
</forward>
<portgroup count='10'>
<interface dev='eth0'/>
</portgroup>
<portgroup count='5'>
<interface dev='eth1'/>
</portgroup>
<portgroup count='5'>
<interface dev='eth2'/>
</portgroup>
</network>
The <virtualport> parameters for VEPA/VNLink could either be stored at
the top level under <network>, or inside <portgroup> or both.
Regards,
Daniel
--
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org -o- http://virt-manager.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
1:40 a.m.
New subject: [libvirt] migration of vnlink VMs
On Tue, May 24, 2011 at 10:29:00PM +0200, Jérémie Tarot wrote:
Hi,
2011/5/24 Daniel P. Berrange <berrange(a)redhat.com>
> ...
>
> And new bridging modes would be
>
> <forward layer='link'
> mode='bridge-brctl|vepa|private|passthru|bridge-macvtap'/>
>
VDE ? Openvswitch ?
I would be very happy to push openvswitch and suggest to people
to rely on it ... once it is accepted in the upstream kernel.
Unfortunately that's still not the case, and history of hypervisors
like openvz and xen really shows why this should be their #1 focus,
instead of being put in the backburner TODO list as it seems [*]
Daniel
[*] I would love to be proved wrong on this !
--
Daniel Veillard | libxml Gnome XML XSLT toolkit http://xmlsoft.org/
daniel(a)veillard.com | Rpmfind RPM search engine http://rpmfind.net/
http://veillard.com/ | virtualization library http://libvirt.org/
1:30 p.m.
New subject: [libvirt] migration of vnlink VMs
I'll send a separate email (in a new thread, so it doesn't get lost! ;-)
with a new draft of what the network XML should look like, but wanted to
respond to Dan's comments inline...
On 05/24/2011 10:21 AM, Daniel P. Berrange wrote:
On Fri, Apr 29, 2011 at 04:12:55PM -0400, Laine Stump wrote:
> Okay, here's a brief description of what I *think* will work. I'll
> build up the RNG based on this pseudo-xml:
>
>
> For the<interface> definition in the guest XML, the main change
> will be that<source .. mode='something'> will be valid (but
> optional) when interface type='network' - in this case, it will just
> be used to match against the source mode of the network on the host.
> <virtualport> will also become valid for type='network', and will
> serve two purposes:
>
> 1) if there is a mismatch with the virtualport on the host network,
> the migrate/start will fail.
> 2) It will be ORed with<virtualport> on the host network to arrive
> at the virtualport settings actually used.
>
> For example:
>
> <interface type='network'>
> <source network='red-network' mode='vepa'/>
IMHO having a 'mode' here is throwing away the main reason for
using type=network in the first place - namely independance
from this host config element.
I agree, but was being accommodating :-) Since then, Dave has pointed
out that the same functionality can be achieved by having the management
application grab the XML for the network on the targetted host, and
check for matches of any important parameters before deciding to migrate
to that host. This has 2 advantages:
1) It is more flexible. The management application can check for more
than just mode='vepa', but also any number of other attributes of the
network on the target.
2) The result of a host's network not matching the desired mode will be
"management app looks elsewhere", rather than "migration fails".
The management application will need to do this anyway (even if just to
check that the given network is present at all) or, again, face the
prospect of the migration failing.
So I'll withdraw this piece from the next draft.
> <virtualport type='802.1Qbg'>
> <parameters instanceid='09b11c53-8b5c-4eeb-8f00-d84eaa0aaa4f'/>
> </virtualport>
> <mac address='xx:xx:.....'/>
> </interface>
>
> (NB: if "mode" isn't specified, and the host network is actually a
> bridge or virtual network, the contents of virtualport will be
> ignored.)
>
>
> <network> will be expanded by giving it an optional "type"
attribute
> (which will default to 'virtual'),<source> subelement, and
> <virtualport> subelement. When type='bridge', you can specify source
> exactly as you would in a domain<interface> definition:
>
> <network type='bridge'>
> <name>red-network</name>
> <source bridge='br0'/>
> </network>
>
> When type='direct', again you can specify source and virtualport
> pretty much as you would in an interface definition:
>
> <network type='direct'>
> <name>red-network</name>
> <source dev='eth0' mode='vepa'/>
> <virtualport type='802.1Qbg'>
> <parameters managerid="11" typeid="1193047"
typeidversion="2"
> instanceid='09b11c53-8b5c-4eeb-8f00-d84eaa0aaa4f'/>
> </virtualport>
> </network>
None of this really feels right to me. With this proposed
schema, there is basically nothing in common between the
existing functionality for<network> and this new functionality
except for the<name> and<uuid> elements.
Apps which know how to deal with existing<network> schema
will have no ability to interpret this new data at all.
Quite probably they will mis-interpet it as providing an
isolated virtual network, with no IP addr set, since this
design isn't actually changing any attribute value that
they currently look for.
Either we need to make this align with the existing schema,
or we need to put this under a completely separate set of
APIs. I think we can likely do better with the schema design
and achieve the former.
So the problem is that the new uses are so orthogonal to the current
usage that existing management apps encountering this new XML will
mistakenly believe that it's "old" XML with a bit of extra stuff that
can be ignored (thus leading to mayhem).
I think the most important thing is to make sure that a config for one
of these new types will have at least one change to an *existing*
element/attribute (mine just added a *new* attribute specifying type)
that causes existing apps to realize this isn't just an old school
network definition that happens to have a few kinks on the side. Your
suggestion of using new values for <forward mode="..."> seems like as
good an idea as any (actually I can't think of anything else that works
as well :-)
> However, dev would be optional - if not specified, we would
expect a
> pool of interfaces to be defined within source, eg:
>
> <network type='direct'>
> <name>red-network</name>
> <source mode='vepa'>
> <pool>
> <interface name='eth10' maxConnect='1'/>
> <interface name='eth11' maxConnect='1'/>
> <interface name='eth12' maxConnect='1'/>
> <interface name='eth13' maxConnect='1'/>
> <interface name='eth14' maxConnect='1'/>
> <interface name='eth25' maxConnect='5'/>
> </pool>
> </source>
> <virtualport ...... />
> </network>
I don't really like the fact that this design has special
cased the num(intefaces) == 1 case to have a completely
different XML schema. eg we have this:
<source dev='eth0' mode='vepa'/>
And this
<source mode='vepa'>
<pool>
<interface name='eth10' maxConnect='1'/>
</pool>
both meaning the same thing. There should only be one
representation in the schema for this kind of thing.
> BTW, for all the people asking about sectunnel, openvswitch, and vde
> - can you see how those would fit in with this? In particular, do
> you see any conflicts? (It's easy to add more stuff on later if
> something is just missing, but much more problematic if I put
> something in that is just plain wrong).
As mentioned above, I think this design is wrong, because it is not
taking any account of the current schema for<network> which defines
the various routed modes.
Currently<network> supports 3 connectivity modes
- Non-routed network, separate subnet (no<forward> element present)
- Routed network, separate subnet with NAT (<forward mode='nat'/>)
- Routed network, separate subnet (<forward mode='route'/>)
Following on from this, I can see another couple of routed modes
- Routed network, IP subnetting
- Routed network, separate subnet with VPN
And the core goal here is to replae type=bridge and type=direct from the
domain XML, which means we're adding several bridging modes
- Bridged network, eth + bridge + tap (akin to type=bridge)
- Bridged network, eth + macvtap (akin to type=direct)
- Bridged network, sriov eth + bridge + tap (akin to type=bridge)
- Bridged network, sriov eth + macvtap (akin to type=direct)
The macvtap can be in 4 modes, so perhaps it is probably better to
consider them separately
- Bridged network, eth + bridge + tap
- Bridged network, eth + macvtap + vepa
- Bridged network, eth + macvtap + private
- Bridged network, eth + macvtap + passthrough
- Bridged network, eth + macvtap + bridge
- Bridged network, sriov eth + bridge + tap
- Bridged network, sriov eth + macvtap + vepa
- Bridged network, sriov eth + macvtap + private
- Bridged network, sriov eth + macvtap + passthrough
- Bridged network, sriov eth + macvtap + bridge
I can also perhaps imagine another VPN mode:
- Bridged network, with VPN
The current routed modes can route to anywhere, or be restricted to
a particular network interface eg with<forward dev='eth0'/>. It
only allows for a single interface, though even for routed modes it
could be desirable to list multiple devs.
The other big distinction is that the<network> modes which do routing,
include interface configuration data (ie the IP addrs& bridge name)
which is configured on the fly. It looks like with the bridged modes,
you're assuming the app has statically configured the interfaces via
the virInterface APIs already, and this just points to an existing
configured interface. This isn't neccessarily a bad thing, just an
observation of a significant difference.
Right. Perhaps later it can be expanded
(at least in some of the modes)
to setup these devices when the network is started, but right now the
network definition is just used to point to something that already
exists and is functioning.
So if we ignore the<ip> and<domain> elements from the
current<network>
schema, then there are a handful of others which we need to have a plan
for
<forward mode='nat|route'/> (omitted completely for isolated
networks)
<bridge name="virbr0" /> (auto-generated/filled if omitted)
<mac address='....'/> (auto-generated/filled if omitted)
The<forward> element can have an optional dev= attribute.
I think the key attribute is the<forward> mode= attribute. I think we
should be adding further values to that attribute for the new network
modes we want to support. We should also make use of the dev= attribute
on<forward> where practical, and/or extend it.
We could expand the list of<foward> mode values in a flat list
- route
- nat
- bridge (brctl)
- vepa
- private
- passthru
- bridge (macvtap)
NB: really need to avoid using 'bridge' in terminology, since all
5 of the last options are really 'bridge'.
Or we could introduce a extra attribute, and have a 2 level list
-<forward layer='link'/> (for all ethernet layer bridging)
Does that gain us anything, though? While it's correct information, it
seems redundant (the layer can always be implied from the mode).
-<forward layer='network'/> (for all IP layer
bridging aka routing)
So the current modes would be
<forward layer='network' mode='route|nat'/>
And new bridging modes would be
<forward layer='link'
mode='bridge-brctl|vepa|private|passthru|bridge-macvtap'/>
For the brctl/macvtap modes, the dev= attribute on<forward> could point to
the NIC being used, while with brctl modes,<bridge> would also be present.
Are you saying that in the case of a brctl mode, it would be required to
fill in both of these?
<forward mode="bridge-brctl" dev="br0" .../>
<bridge name="br0" .../>
I think I would prefer to only use the one in <forward>. Are you
suggesting putting it there to help older management apps cope with the
new modes? I don't really think it would help; it's really just an
accident of implementation that the device in "bridge-brctl" mode
happens to be a bridge device.
In the SRIOV case, we potentiallly need a list of interfaces. For
this we
probably want to use
BTW, just to clarify, when you say "SRIOV", what you really mean is "any
situation where there are multiple network interface devices connected
to the same physical network, and identical connectivity to the guest
could be provided by any one of these devices". In other words, it
doesn't need to be an SRIOV ethernet card with multiple virtual
functions, it could also be an older style setup with multiple physical
cards, or multiple complete devices on a single card.
<forward dev='eth0'>
<interface dev='eth0'/>
<interface dev='eth1'/>
<interface dev='eth2'/>
...
</forward>
NB, the first interface is always to be listed both as a dev= attribute
(for compat with existing apps) *and* as a child<interface> element (for
apps knowing the new schema).
But since the pool of devices would only ever be used in one of the new
forward modes, which an existing app wouldn't understand anyway, would
that really buy us anything?
The maxConnect= attribute from your examples above is an interesting
thing. I'm not sure whether that is neccessarily a good idea. It feels
similar to VMWare's "port group" idea, but I don't think having a
simple 'maxConnect=' attribute is sufficient to let us represent the
vmware port group idea. I think we might need an more explicit
element eg
<portgroup count='5'>
<interface dev='eth2'/>
</portgroup>
eg, so this associates a port group which allows 5 clients (VM NICs)
with the uplink provided by eth2 (which is assumed to be listed
under<forward>).
I've thought about this a bit, and I think portgroup is a good idea, but
I don't think the name of the device being used fits there. portgroup is
a good place to put information about the characteristics of a set of
connections, but which device to use is a backend implementation detail,
and there isn't necessarily a 1:1 correspondence between the two.
portgroup would be used, for example, to configure bandwidth (that's
pretty much all VMWare uses it for, plus a blob of "vendor-specific"
data), and the guest interface XML would specify which portgroup a guest
was going to belong to - if you also set which physical device to use
based on portgroup, that would leave the guest XML specifying which
physical device to use, which is what we're trying to get away from.
(and also it would mean that each physical device would need its own
portgroup, which I don't think we want.
Thinking more about the maxCount thing, it seems like it might be
overkill for now. The case where there must be a limitation of 1 guest
per NIC is macvtap passthrough mode, but that's already implied by the
fact that it's passthrough. Other than that, libvirt can just attempt to
load-balance as best as possible by keeping track of how many
connections there are on each device, but not force any artificial
limit. We may need to provide some method of reporting the number of
connections to any particular network, to be used by a management
application for load balancing decisions (although the amount of traffic
is probably more important, and that can already be learned).
Conclusion on portgroup - a good idea, but not for this, probably for
configuration of bandwidth limiting.
So a complete SRIOV example might be
<network>
<name>Foo</name>
<forward dev='eth0' layer='link' mode='vepa'>
<interface dev='eth0'/>
<interface dev='eth1'/>
<interface dev='eth2'/>
...
</forward>
<portgroup count='10'>
<interface dev='eth0'/>
</portgroup>
<portgroup count='5'>
<interface dev='eth1'/>
</portgroup>
<portgroup count='5'>
<interface dev='eth2'/>
</portgroup>
</network>
The<virtualport> parameters for VEPA/VNLink could either be stored at
the top level under<network>, or inside<portgroup> or both.
Ah, now *there's* something that fits in portgroup (since that's likely
exactly what it's used for on the vepa/vnlink capable switch).
I think it's reasonable to put it in both places, at the top-level
(which would apply to all connections) and in portgroup (which would
override the global setting for connections using that portgroup). (I
think the bandwidth config could be done in the same way.
7:28 a.m.
New subject: [libvirt] migration of vnlink VMs
Sorry for hijacking the thread for something more general than vnlink:
It just occured to me today that the concept of virtual network should
be used also in the <graphics> section. Currently, if you specify
<graphics listen="some-ip-address"> you make the domain practically
unmigratable.
There should be a method to specify a simbolic network name, to be
evaluated to IP on the destination host, something like
<graphics listenNetwork="some-virtual-network">
Regards,
Dan.
7:37 a.m.
New subject: [libvirt] migration of vnlink VMs
On Wed, May 11, 2011 at 03:28:30PM +0300, Dan Kenigsberg wrote:
Sorry for hijacking the thread for something more general than
vnlink:
It just occured to me today that the concept of virtual network should
be used also in the <graphics> section. Currently, if you specify
<graphics listen="some-ip-address"> you make the domain practically
unmigratable.
There should be a method to specify a simbolic network name, to be
evaluated to IP on the destination host, something like
<graphics listenNetwork="some-virtual-network">
Yep, I pretty much agree, but we'll need more help from QEMU
todo that. A virtual network may have multiple IP addresses
and so we'll need to be able to specify multiple IPs to the
VNC server.
Regards,
Daniel
--
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org -o- http://virt-manager.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
7:55 a.m.
New subject: [libvirt] migration of vnlink VMs
On Wed, May 11, 2011 at 01:37:25PM +0100, Daniel P. Berrange wrote:
On Wed, May 11, 2011 at 03:28:30PM +0300, Dan Kenigsberg wrote:
> Sorry for hijacking the thread for something more general than vnlink:
>
>
> It just occured to me today that the concept of virtual network should
> be used also in the <graphics> section. Currently, if you specify
> <graphics listen="some-ip-address"> you make the domain practically
> unmigratable.
>
> There should be a method to specify a simbolic network name, to be
> evaluated to IP on the destination host, something like
>
> <graphics listenNetwork="some-virtual-network">
Yep, I pretty much agree, but we'll need more help from QEMU
todo that. A virtual network may have multiple IP addresses
and so we'll need to be able to specify multiple IPs to the
VNC server.
That would be nice, but I think that a semantics where libvirt is in
charge of telling which qemu process is to listen on which of the
network IP addresses is as reasonable (and may potentially better, as
libvirt has more knowledge of the host)
Anyway, if the virtual network is bridge-based, I could live with
passing the single IP of the bridge for qemus that do not support this
feature.
Regards,
Dan.
1:28 p.m.
New subject: [libvirt] migration of vnlink VMs
Inline...
-----Original Message-----
From: libvir-list-bounces(a)redhat.com [mailto:libvir-list-
bounces(a)redhat.com] On Behalf Of Oved Ourfalli
Sent: Thursday, April 28, 2011 1:15 AM
To: Laine Stump; libvir-list(a)redhat.com
Subject: Re: [libvirt] migration of vnlink VMs
----- Original Message -----
> From: "Laine Stump" <lstump(a)redhat.com>
> To: "Oved Ourfalli" <ovedo(a)redhat.com>
> Cc: "Ayal Baron" <abaron(a)redhat.com>, "Barak Azulay"
<bazulay(a)redhat.com>, "Shahar Havivi" <shaharh(a)redhat.com>,
> "Itamar Heim" <iheim(a)redhat.com>, "Dan Kenigsberg"
<danken(a)redhat.com>
> Sent: Thursday, April 28, 2011 10:20:35 AM
> Subject: Re: migration of vnlink VMs
> Oved,
>
> Would it be okay to repost this message to the thread on libvir-list
> so
> that other parties can add their thoughts?
>
Of course. I'm sending my answer to the libvirt list.
> On 04/27/2011 09:58 AM, Oved Ourfalli wrote:
> > Laine, hello.
> >
> > We read your proposal for abstraction of guest<--> host network
> > connection in libvirt.
> >
> > You has an open issue there regarding the vepa/vnlink attributes:
> > "3) What about the parameters in the<virtualport> element that are
> > currently used by vepa/vnlink. Do those belong with the host, or
> > with the guest?"
> >
> > The parameters for the virtualport element should be on the guest,
> > and not the host, because a specific interface can run multiple
> > profiles,
>
> Are you talking about host interface or guest interface? If you mean
> that multiple different profiles can be used when connecting to a
> particular switch - as long as there are only a few different
> profiles,
> rather than each guest having its own unique profile, then it still
> seems better to have the port profile live with the network
definition
> (and just define multiple networks, one for each port profile).
>
The profile names can be changed regularly, so it looks like it will
be
better to put them in the guest level, so that the network host file
won't have to be changed on all hosts once something has changed in
the
profiles.
Also, you will have a duplication of data, writing all the profile
name
on all the hosts that are connected to the vn-link/vepa switch.
>
> > so it will be a mistake to define a profile to be interface
> > specific on the host. Moreover, putting it in the guest level
will
> > enable us in the future (if supported by libvirt/qemu) to
migrate
> > a vm from a host with vepa/vnlink interfaces, to another
host
with
> > a bridge, for example.
>
> It seems to me like doing exactly the opposite would make it easier
to
> migrate to a host that used a different kind of switching (from vepa
> to
> vnlink, or from a bridged interface to vepa, etc), since the port
> profile required for a particular host's network would be at the
host
> waiting to be used.
You are right, but we would want to have the option to prevent that
from happening in case we wouldn't want to allow it.
We can make the ability to migrate between different network types
configurable, and we would like an easy way to tell libvirt - "please
allow/don't allow it".
>
> > So, in the networks at the host level you will have:
> > <network type='direct'>
> > <name>red-network</name>
> > <source mode='vepa'>
> > <pool>
> > <interface>
> > <name>eth0</name>
> > .....
> > </interface>
> > <interface>
> > <name>eth4</name>
> > .....
> > </interface>
> > <interface>
> > <name>eth18</name>
> > .....
> > </interface>
> > </pool>
> > </source>
> > </network>
> >
> > And in the guest you will have (for vepa):
> > <interface type='network'>
> > <source network='red-network'/>
> > <virtualport type="802.1Qbg">
> > <parameters managerid="11" typeid="1193047"
typeidversion="2"
> > instanceid="09b11c53-8b5c-4eeb-8f00-d84eaa0aaa4f"/>
> > </virtualport>
> > </interface>
> >
> > Or (for vnlink):
> > <interface type='network'>
> > <source network='red-network'/>
> > <virtualport type="802.1Qbh">
> > <parameters profile_name="profile1"/>
> > </virtualport>
> > </interface>
>
> This illustrates the problem I was wondering about - in your example
> it
> would not be possible for the guest to migrate from the host using a
> vepa switch to the host using a vnlink switch (and it would be
> possible
You are right. When trying to migrate between vepa and vnlink there
will be missing attributes in each in case we leave it on the host.
> to migrate to a host using a standard bridge only if the virtualport
> element was ignored). If the virtualport element lived with the
> network
> definition of red-network on each host, it could be migrated without
> problem.
>
> The only problematic thing would be if any of the attributes within
> <parameters> was unique for each guest (I don't know anything about
> the
> individual attributes, but "instanceid" sounds like it might be
> different for each guest).
Whether a given parameter is unique for each guest (or let's say,
whether
it can be shared by two or more guests) may also be a config/policy
detail
for certain parameters.
You should take into account that new parameters may be added later on
(not only for vepa/vn-link), and could be both unique or shared.
For this reason you design should be able to handle such a case.
BTW, "instanceid" identifies the virtual nic, therefore it is unique
(it is a UUID).
instanceid
The VSI Instance ID Identifier is generated when a VSI instance
(i.e. a virtual interface of a virtual machine) is created.
This is a globally unique identifier.
Please note that the IEEE specs (802.1Qbh/1Qbg) have undergone some
changes
in the last few months. The two protos are going to share the same proto
VDP for the port profile configuration. Also, new parameters may be
introduced and already existent ones may change type.
Later this year, when the IEEE drafts will be closer to final versions,
there
will be updates to the configuration parameters to reflect the changes
in
the drafts.
> > Then, when migrating from a vepa/vnlink host to another
vepa/vnlink
> > host containing red-network, the profile attributes will
be
> > available at the guest domain xml.
> > In case the target host has a red-network, which isn't
vepa/vnlink,
> > we want to be able to choose whether to make the use of the
profile
> > attributes optional (i.e., libvirt won't fail in case
of migrating
> > to a network of another type), or mandatory (i.e., libvirt will
fail
> > in case of migration to a non-vepa/vnlink network).
> >
> > We have something similar in CPU flags:
> > <cpu match="exact">
> > <model>qemu64</model>
> > <topology sockets="S" cores="C"
threads="T"/>
> > <feature policy="require/optional/disable......"
> > name="sse2"/>
> > </cpu>
>
> In this analogy, does "CPU flags" == "mode (vepa/vnlink/bridge)"
or
> does
> "CPU flags" == "virtualport parameters" ? It seems like what
you're
> wanting can be satisfied by simply not defining "red-network" on the
> hosts that don't have the proper networking setup available (maybe
> what
> you *really* want to call it is "red-vnlink-network").
What I meant to say in that is that we would like to have the ability
to say if an attribute must me used, or not.
"must" in the sense that it is "mandatory"?
I can see two cases:
1) src and dst networks are of different types
(NOTE: I consider vepa/vnlink different for now, but this will
probably change when BH and BG will both use VDP)
In this case I do not see why you need to worry about whether
parameter param-X used by src host should or should not be used
by dst host: it should only if it is a generic parameter and
(as such) it does not fall inside the config section that is
specific to the network type.
Trying to translate parameters between different network types
may not be always easy and clean. Even the property
"mandatory vs optional" may change with different network types.
2) src and dst networks are of the same type
In this case it _does_ make sense to have the possibility of
specifying whether a given param is needed or not. However, I
believe it would make sense mainly for those parameters that
represent optional/desirable features of the proto/net: such
config would then be used to decide whether migration will or
will not be possible, right?
I like the idea of the abstraction, especially the pool of interfaces.
However, I think you would have to lose a bit of abstraction in order
to make it possible to have migrations between network of different
types.
I guess your goal is not to make migration possible between each
possible combination of network types, is it?
There are parameters that are specific to a given network type.
How can you expect a migration from network type X to network type Y (Y
!= X)
if you only configure the parameters for type X (and assuming Y comes
with at least one mandatory parameter)?
What are the combinations of (src/net, dst/net) that you would like to
support?
Out of curiosity, have you taken into consideration the possibility of
defining an abstracted network config as a pool of network types?
For example something like this:
HOST: Pool of three network sub-types
<network type='network'>
<name>red-network</name>
<source type='direct' mode='vepa'>
...
</source>
<source type=XXX ...>
...
</source>
<source type=YYY ...>
...
</source>
</network>
GUEST WHICH ONLY ACCEPTS ONE SPECIFIC TYPE (direct/private):
<interface type='network'>
<source network='red-network'/>
<option prio=1 type='direct' mode='private'>
<virtualport type="802.1Qbh">
<parameters profile_name="profile_123"/>
</virtualport>
</option>
</interface>
GUEST WHICH ACCEPTS TWO TYPES (direct/private, direct/vepa):
<interface type='network'>
<source network='red-network'/>
<option prio=1 type='direct' mode='private'>
<virtualport type="802.1Qbh">
<parameters profile_name="profile_123"/>
</virtualport>
</option>
<option prio=2 type='direct' mode='vepa'>
<virtualport type="802.1Qbg">
<parameters managerid="11" typeid="1193047"
typeidversion="2"
instanceid="09b11c53-8b5c-4eeb-8f00-d84eaa0aaa4f"/>
</virtualport>
</option>
</interface>
During migration, the dst host would use the options/prios in the guest
config XML to select among the options available for the same 'network'
on the dst host, just like any other similar handshake.
I am not suggesting this approach, I just would like to know if you ever
thought about this option and, if you did, why you discarded it.
An example of scenario where the above pool of network types may make
sense would be the combination vepa+bridge or vnlink+bridge: if
something
goes wrong during migration and the vepa/OR/vnlink cannot associate the
port profile, the guest can at least have a backup net connection
through
the bridge. It could optionally also re-try the vepa/vnlink association
a number of times (libvirt would do it if configured to do so) ... while
maintaining temporary connectivity through the bridge.
/Chris
The issues you mention are indeed interesting. I'm cc-ing
libvirt-list
to see what other people think.
Putting it on the guest will indeed make it problematic to migrate
between networks that need different parameters (vnlink/vepa for
example).
Oved
--
libvir-list mailing list
libvir-list(a)redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list
4640
days inactive
4770
days old
11 comments
7 participants
participants (7)
-
Christian Benvenuti (benve) -
Dan Kenigsberg -
Daniel P. Berrange -
Daniel Veillard -
Jérémie Tarot -
Laine Stump -
Oved Ourfalli