6:34 p.m.
Thank you Laine, for your prompt reply. Adding the dev list to broaden the
audience.
On Mon, Sep 16, 2013 at 9:21 AM, Laine Stump <laine(a)laine.org> wrote:
On 09/16/2013 07:32 AM, Ajith Antony wrote:
> Hi, I'm not sure what the best way to pursue this is, but
since it
> appears you authored the relevant code, you could advise of a good
> next step.
Kind of. I think I wrote code for hostdev networks, but the original was
(if I remember correctly) written by Kyle Mestery with support only for
openvswitch.
> I'm trying to use libvirt to set up openvswitch vlans, but ran into a
> problem where the libvirt code is precluding using vlans if the
> network is not "Forwarded."[1] I don't think there is any technical
> reason not to support vlans for non-forwarded networks. I verified
> this by manually adding vlan tags to the ephemeral ports created by
> the libvirt network definition. The tags worked as expected.
> [1]
http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/network/bridge_driver....
You're misunderstanding the intent (at least) of that if condition. The
way that you configure a libvirt network to use openvswitch is with the
following:
<forward mode='bridge'/>
<virtualport type='openvswitch'>
<some other stuff I forget/>
</virtualport>
That condition is only checking if:
vlanAllowed = ((def->forward.type == VIR_NETWORK_FORWARD_BRIDGE &&
1) forward mode is 'bridge' (this doesn't necessarily mean that anything
will be forwarded beyond the bridge that the guest's network device is
connected to, only that it is connected to a bridge, i.e. as far as
libvirt is concerned any forwarding that *is* done will be L2
forwarding. The specifics of how that happens are left up to the bridge.)
Yes, I think this the test that interests me. The key fact in my scenario is
that I'm not preconfiguring a bridge. I wouldn't be able to set the forward
mode as 'bridge' unless I already prepared a bridge. I recongize that if I
prepared a bridge that I could add more portgroups to it with thier unique vlan
tags and not expect anything to actually cross the physical interface. I think
this is what is is reccomended among the blogs when one searches for
"openvswitch fake bridge."
You'll need to post a copy of your network config for me to
understand
for sure, but I'm guessing that you're missing one of the two items in
your config, possibly indicating that you've misunderstood how to
configure a libvirt network that uses an openvswitch bridge.
So can you send a copy of your network XML?
The network that I'm trying to define is described as the following. Note
there there is no <forward> element as I'm not expcting there to be a *real*
bridge and I don't expect that I need to preconfigre explicit bridges. I
expect an ephemeral virtual bridge to be created.
<network>
<name>ajith-net1</name>
<virtualport type='openvswitch'/>
<portgroup name='client_lan'>
<vlan>
<tag id='1'/>
</vlan>
</portgroup>
<portgroup name='client_wan'>
<vlan>
<tag id='2'/>
</vlan>
</portgroup>
</network>
When I include the VLAN tags I get the error:
error: unsupported configuration: <vlan> element specified for network
ajith-net1, whose type doesn't support vlan configuration
When I omit the VLAN tags as follows I am able to create the network:
<network>
<name>ajith-net1</name>
<virtualport type='openvswitch'/>
<portgroup name='client_lan'>
</portgroup>
<portgroup name='client_wan'>
</portgroup>
</network>
In my domain configs I specify the network like so:
<interface type='network'>
<source network='ajith-net1' portgroup='client_lan'/>
<virtualport type='openvswitch'/>
<model type='virtio'/>
</interface>
The resulting ephemeral bridge(virbr1) looks like the following when my
network(w/o vlans) and two domains are started. I don't know if the portgroup
was meaningful, but it was accepted in the definition:
$ sudo ovs-vsctl show
<...>
Bridge "virbr1"
Port "vnet23"
Interface "vnet23"
Port "vnet25"
Interface "vnet25"
Port "virbr1"
Interface "virbr1"
type: internal
Port "virbr1-nic"
Interface "virbr1-nic"
ovs_version: "1.9.3"
My goal is assign unique vlan tags to the two portgroups. While the network
and the domains are started, I can manually add tags to the ports:
$ sudo ovs-vsctl set port vnet25 tag=2
$ sudo ovs-vsctl set port vnet23 tag=1
$ sudo ovs-vsctl show
Bridge "virbr1"
Port "vnet23"
tag: 1
Interface "vnet23"
Port "vnet25"
tag: 2
Interface "vnet25"
Port "virbr1"
Interface "virbr1"
type: internal
Port "virbr1-nic"
Interface "virbr1-nic"
ovs_version: "1.9.3"
This configuration behaves exactly as I want. I'm interested to learn how to
prepare this without the manual step of tagging the ports outside of the
libvirt operations.
Ultimately my goal is to prepare isolated test environments that consist of
several VM's attached to a similar qty of vlans. I intend to create many of
these environments per host. I also recongnize that instead of portgroups, I
could use separate networks altogether. From an administrative standpoint, I'd
prefer to have one "network" per test environment, with several portgroups,
instead of *many* networks.
FWIW, I'm doing something similar on ESX, where I create a "vSwitch" and
then
create several "portgroups", each with its own vlan tag. These vSwitches are
not attached to any physical interfaces. (My esx work is directly against the
vsphere API, so I'm not sure if the same could be configured via libvirt) I'm
looking to port this work to KVM. It seems like a pretty natural thing to
support.
4:15 a.m.
New subject: [libvirt] openvswitch vlan support when not a forwarded network
On 09/16/2013 07:34 PM, Ajith Antony wrote:
Thank you Laine, for your prompt reply. Adding the dev list to
broaden the
audience.
On Mon, Sep 16, 2013 at 9:21 AM, Laine Stump <laine(a)laine.org> wrote:
On 09/16/2013 07:32 AM, Ajith Antony wrote:
>> Hi, I'm not sure what the best way to pursue this is, but since it
>> appears you authored the relevant code, you could advise of a good
>> next step.
> Kind of. I think I wrote code for hostdev networks, but the original was
> (if I remember correctly) written by Kyle Mestery with support only for
> openvswitch.
>
>> I'm trying to use libvirt to set up openvswitch vlans, but ran into a
>> problem where the libvirt code is precluding using vlans if the
>> network is not "Forwarded."[1] I don't think there is any
technical
>> reason not to support vlans for non-forwarded networks. I verified
>> this by manually adding vlan tags to the ephemeral ports created by
>> the libvirt network definition. The tags worked as expected.
>> [1]
http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/network/bridge_driver....
> You're misunderstanding the intent (at least) of that if condition. The
> way that you configure a libvirt network to use openvswitch is with the
> following:
>
> <forward mode='bridge'/>
> <virtualport type='openvswitch'>
> <some other stuff I forget/>
> </virtualport>
>
> That condition is only checking if:
>
> vlanAllowed = ((def->forward.type == VIR_NETWORK_FORWARD_BRIDGE &&
>
>
> 1) forward mode is 'bridge' (this doesn't necessarily mean that anything
> will be forwarded beyond the bridge that the guest's network device is
> connected to, only that it is connected to a bridge, i.e. as far as
> libvirt is concerned any forwarding that *is* done will be L2
> forwarding. The specifics of how that happens are left up to the bridge.)
Yes, I think this the test that interests me. The key fact in my scenario is
that I'm not preconfiguring a bridge. I wouldn't be able to set the forward
mode as 'bridge' unless I already prepared a bridge. I recongize that if I
prepared a bridge that I could add more portgroups to it with thier unique vlan
tags and not expect anything to actually cross the physical interface. I think
this is what is is reccomended among the blogs when one searches for
"openvswitch fake bridge."
1) libvirt will not create an openvswitch bridge for you. It assumes
that you have already created an openvswitch bridge outside of libvirt.
2) As far as I understand from what I've heard of the "openvswitch fake
bridge", you *do not* want one. My recollection is that people were
using that as some method of having multiple vlans on the same bridge
when the management software provided no method of setting vlan tags.
But libvirt *does* provide a method of setting the vlan tag.
> You'll need to post a copy of your network config for me to understand
> for sure, but I'm guessing that you're missing one of the two items in
> your config, possibly indicating that you've misunderstood how to
> configure a libvirt network that uses an openvswitch bridge.
>
> So can you send a copy of your network XML?
The network that I'm trying to define is described as the following. Note
there there is no <forward> element as I'm not expcting there to be a *real*
bridge and I don't expect that I need to preconfigre explicit bridges. I
expect an ephemeral virtual bridge to be created.
What you want to do makes sense, but it's not currently possible. If you
want to use an openvswitch bridge, you need to create the bridge outside
of libvirt, then create a network with <forward mode='bridge'/> and a
<virtualport type='openvswitch'>
<network>
<name>ajith-net1</name>
<virtualport type='openvswitch'/>
<portgroup name='client_lan'>
<vlan>
<tag id='1'/>
</vlan>
</portgroup>
<portgroup name='client_wan'>
<vlan>
<tag id='2'/>
</vlan>
</portgroup>
</network>
When I include the VLAN tags I get the error:
error: unsupported configuration: <vlan> element specified for network
ajith-net1, whose type doesn't support vlan configuration
for the reasons I give above. libvirt supports networks based on a
pre-existing openvswitch bridge, but isn't able to create such a bridge
itself.
That is a feature that I've contemplated though, and even tried
persuading a couple other people to write. Basically what you would need
to do is:
1) add new functions to src/util/virnetdevopenvswitch.c:
virNetDevOpenvswitchCreate()
virNetDevOpenvswitchSetSTPDelay()
virNetDevOpenvswitchSetSTP()
virNetDevOpenvswitchSetOnline()
[maybe some others]
2) modify src/network/bridge_driver.c:networkStartNetworkVirtual() to
check for a virtualport element with type openvswitch, and in that case
call the above virNetDevOpenvswitch*() functions in place of
virNetDevBridge*()
3) remove the restriction requiring forward mode == 'bridge' that you
mentioned in your original mail (since it will no longer be valid).
4) update src/docs/formatnetwork.html.in and
src/docs/schemas/network.rng to reflect that libvirt-managed virtual
networks can now use openvswitch for their bridges.
When I omit the VLAN tags as follows I am able to create the network:
<network>
<name>ajith-net1</name>
<virtualport type='openvswitch'/>
<portgroup name='client_lan'>
</portgroup>
<portgroup name='client_wan'>
</portgroup>
</network>
In my domain configs I specify the network like so:
<interface type='network'>
<source network='ajith-net1' portgroup='client_lan'/>
<virtualport type='openvswitch'/>
(If the network has <virtualport type='openvswitch'>, it's redundant to
put it in the domain's <interface> definition.)
<model type='virtio'/>
</interface>
The resulting ephemeral bridge(virbr1) looks like the following when my
network(w/o vlans) and two domains are started. I don't know if the portgroup
was meaningful, but it was accepted in the definition:
$ sudo ovs-vsctl show
<...>
Bridge "virbr1"
Port "vnet23"
Interface "vnet23"
Port "vnet25"
Interface "vnet25"
Port "virbr1"
Interface "virbr1"
type: internal
Port "virbr1-nic"
Interface "virbr1-nic"
ovs_version: "1.9.3"
You apparently have openvswitch's "Linux host bridge compatibility"
package installed on your machine. If you didn't, the network definition
you have would have created a Linux host bridge rather than an
openvswitch bridge. libvirt doesn't contain any code that can create an
openvswitch bridge directly, so that's the only possible way this could
be happening. The problem is that when you use compatibility mode,
you're limited to the Linux bridge-utils API, which has no method of
specifying a vlan tag for individual ports (because Linux host bridges
lack that capability).
My goal is assign unique vlan tags to the two portgroups. While the network
and the domains are started, I can manually add tags to the ports:
$ sudo ovs-vsctl set port vnet25 tag=2
$ sudo ovs-vsctl set port vnet23 tag=1
$ sudo ovs-vsctl show
Bridge "virbr1"
Port "vnet23"
tag: 1
Interface "vnet23"
Port "vnet25"
tag: 2
Interface "vnet25"
Port "virbr1"
Interface "virbr1"
type: internal
Port "virbr1-nic"
Interface "virbr1-nic"
ovs_version: "1.9.3"
This configuration behaves exactly as I want. I'm interested to learn how to
prepare this without the manual step of tagging the ports outside of the
libvirt operations.
Ultimately my goal is to prepare isolated test environments that consist of
several VM's attached to a similar qty of vlans. I intend to create many of
these environments per host. I also recongnize that instead of portgroups, I
could use separate networks altogether. From an administrative standpoint, I'd
prefer to have one "network" per test environment, with several portgroups,
instead of *many* networks.
Since this is all just numbers in memory (no real cables / switches),
there is little to no practical difference between having a single
bridge with lots of vlans, or having lots of bridges with no vlans. From
the point of view of client configuration, it would be the difference of:
[guest type 1]
<interface type='network'>
<source network='A' portgroup='1'/>
...
</interface>
[guest type 2]
<interface type='network'>
<source network='A' portgroup='2'/>
...
</interface>
versus
[guest type 1]
<interface type='network'>
<source network='A1'/>
...
</interface>
[guest type 2]
<interface type='network'>
<source network='A2'>
...
</interface>
and in the network config it would be:
<network>
<name>A</name>
....
<portgroup name='1'>
...
</portgroup>
<portgroup name='2'>
...
</portgroup>
</network>
versus:
<network>
<name>A1</name>
...
</network>
<network>
<name>A2</name>
...
</network>
One big difference is that you can do the latter today with existing
libvirt code (and you don't even need to have openvswitch installed on
your host). Unless you have > 255 guests on a single vlan, or need some
other openvswitch-specific feature not available with Linux host
bridges, I would just setup multiple networks and use the existing
libvirt networks.
FWIW, I'm doing something similar on ESX, where I create a
"vSwitch" and then
create several "portgroups", each with its own vlan tag. These vSwitches are
not attached to any physical interfaces. (My esx work is directly against the
vsphere API, so I'm not sure if the same could be configured via libvirt) I'm
looking to port this work to KVM. It seems like a pretty natural thing to
support.
11:48 a.m.
New subject: [libvirt] openvswitch vlan support when not a forwarded network
Thank you for your comprehensive reply!
On Tue, Sep 17, 2013 at 4:15 AM, Laine Stump <laine(a)laine.org> wrote:
On 09/16/2013 07:34 PM, Ajith Antony wrote:
> The resulting ephemeral bridge(virbr1) looks like the following when my
> network(w/o vlans) and two domains are started. I don't know if the portgroup
> was meaningful, but it was accepted in the definition:
>
> $ sudo ovs-vsctl show
> <...>
> Bridge "virbr1"
> Port "vnet23"
> Interface "vnet23"
> Port "vnet25"
> Interface "vnet25"
> Port "virbr1"
> Interface "virbr1"
> type: internal
> Port "virbr1-nic"
> Interface "virbr1-nic"
> ovs_version: "1.9.3"
You apparently have openvswitch's "Linux host bridge compatibility"
package installed on your machine. If you didn't, the network definition
you have would have created a Linux host bridge rather than an
openvswitch bridge. libvirt doesn't contain any code that can create an
openvswitch bridge directly, so that's the only possible way this could
be happening. The problem is that when you use compatibility mode,
you're limited to the Linux bridge-utils API, which has no method of
specifying a vlan tag for individual ports (because Linux host bridges
lack that capability).
Aha! I did not understand this. I was under the impression that libvirt was
managing this. I understand now.
> Ultimately my goal is to prepare isolated test environments that
consist of
> several VM's attached to a similar qty of vlans. I intend to create many of
> these environments per host. I also recongnize that instead of portgroups, I
> could use separate networks altogether. From an administrative standpoint, I'd
> prefer to have one "network" per test environment, with several
portgroups,
> instead of *many* networks.
Since this is all just numbers in memory (no real cables / switches),
there is little to no practical difference between having a single
bridge with lots of vlans, or having lots of bridges with no vlans.
One big difference is that you can do the latter today with existing
libvirt code (and you don't even need to have openvswitch installed on
your host). Unless you have > 255 guests on a single vlan, or need some
other openvswitch-specific feature not available with Linux host
bridges, I would just setup multiple networks and use the existing
libvirt networks.
Yes, I'll probably go with the regular bridge behavior for now. One very
attractive feature of using openvswitch is the ability to "re-wire" the whole
set-up by reassigning the vlan tags on-the-fly. My base usecase should be
consistent with the libvirt workflow, where things like changing domain
interface configs take effect when a domain is destroyed and started again, but
the opportunity to move interfaces around without a hard power-cycle could
prove valuable.
4086
days inactive
4087
days old
2 comments
2 participants
participants (2)
-
Ajith Antony -
Laine Stump