On 04/08/2013 02:44 PM, Will Dennis wrote:

Update: I have upgraded from libvirt 0.9.8 on the virtualization server to libvirt 1.0.4. Now the net-define command works with the "virtualport type='openvswitch'" directive in it. So I did a "net-edit" on the networks, added that line in, and saved them. So now I am assuming libvirt knows that the underlying bridge type is Open vSwitch. However, when I try to use virt-manager (which is 0.9.4 running on Fedora 18) to spin up a new VM and try to attach it to one of the networks I defined on the virtualization host (runs Ubuntu 12.04 Server with libvirt 1.0.4 as I had mentioned above), I am now getting the following different traceback:

Unable to complete install: 'Unable to add port vnet0 to OVS bridge vl20-ovsbr0: Operation not permitted'

Please send a copy of your network definition, as well as the <interface> element of your guest's definition so that we can see what you're working with. Does the OVS bridge named vl20-ovsbr0 exist?

Traceback (most recent call last): File "/usr/share/virt-manager/virtManager/asyncjob.py", line 96, in cb_wrapper callback(asyncjob, *args, **kwargs) File "/usr/share/virt-manager/virtManager/create.py", line 1943, in do_install guest.start_install(False, meter=meter) File "/usr/lib/python2.7/site-packages/virtinst/Guest.py", line 1249, in start_install noboot) File "/usr/lib/python2.7/site-packages/virtinst/Guest.py", line 1317, in _create_guest dom = self.conn.createLinux(start_xml or final_xml, 0) File "/usr/lib64/python2.7/site-packages/libvirt.py", line 2606, in createLinux if ret is None:raise libvirtError('virDomainCreateLinux() failed', conn=self) libvirtError: Unable to add port vnet0 to OVS bridge vl20-ovsbr0: Operation not permitted

So the error went from "Operation not supported" to "Operation not permitted"... Is this some sort of a permissions error now??

If someone could please help me figure this out, that would be great, as I'm up against a bit of a deadline with this to get it working...

Thanks, Will

-----Original Message----- From: Will Dennis Sent: Friday, April 05, 2013 12:12 PM To: libvirt-users@redhat.com Subject: Problem with net-define using Open vSwitch bridge

Hi all,

I used "net-define" to create a network to use to attach VMs to an Open vSwitch bridge. The net-define worked fine, I can see the network in "net-list" (I set it to started and to autostart) but when I try to attach a new VM's NIC to this network (via virt-manager), I'm getting the following traceback:

Unable to complete install: 'Unable to add bridge vl10-ovsbr0 port vnet0: Operation not supported'

Traceback (most recent call last): File "/usr/share/virt-manager/virtManager/asyncjob.py", line 96, in cb_wrapper callback(asyncjob, *args, **kwargs) File "/usr/share/virt-manager/virtManager/create.py", line 1943, in do_install guest.start_install(False, meter=meter) File "/usr/lib/python2.7/site-packages/virtinst/Guest.py", line 1249, in start_install noboot) File "/usr/lib/python2.7/site-packages/virtinst/Guest.py", line 1317, in _create_guest dom = self.conn.createLinux(start_xml or final_xml, 0) File "/usr/lib64/python2.7/site-packages/libvirt.py", line 2606, in createLinux if ret is None:raise libvirtError('virDomainCreateLinux() failed', conn=self) libvirtError: Unable to add bridge vl10-ovsbr0 port vnet0: Operation not supported

I'm going to assume the problem is that the bridge is not a linux bridge, but an Open vSwitch bridge. But I did have a line in the XML file that I used for net-define as follows:

<virtualport type='openvswitch'/>

But when I ran net-define, it failed to create the network with the error:

XML error: unexpected virtualport type -1

Removing that line and re-running the net-define worked, as I have stated above. How do I tell libvirt that this bridge is an Open vSwitch bridge and not a regular Linux bridge?

Thanks, Will

_______________________________________________ libvirt-users mailing list libvirt-users@redhat.com https://www.redhat.com/mailman/listinfo/libvirt-users

On 04/08/2013 05:20 PM, Will Dennis wrote:

Thanks, Laine, for your reply. The OVS bridge named 'vl20-ovsbr0' does exist, but it is an Open vSwitch "fake bridge" -- see http://blog.scottlowe.org/2012/10/19/vlans-with-open-vswitch-fake-bridge s/ for why I'm doing this. Here's some relevant output from my system:

# ifconfig | grep encap eth0 Link encap:Ethernet HWaddr 00:25:90:aa:bb:cc lo Link encap:Local Loopback ovsbr0 Link encap:Ethernet HWaddr a2:c6:37:cd:90:4a vl10-ovsbr0 Link encap:Ethernet HWaddr b6:c4:8c:69:2b:06 vl20-ovsbr0 Link encap:Ethernet HWaddr d6:d0:be:6e:b6:fb

# ovs-vsctl --real list-br ovsbr0

# ovs-vsctl --fake list-br vl10-ovsbr0 vl20-ovsbr0

# ovs-vsctl show 3e0d861b-efb7-46b1-af1b-4a76cd833558 Bridge "ovsbr0" Controller "tcp:127.0.0.1:6633" is_connected: true Port "ovsbr0" Interface "ovsbr0" type: internal Port "vl10-ovsbr0" tag: 10 Interface "vl10-ovsbr0" type: internal Port "vl20-ovsbr0" tag: 20 Interface "vl20-ovsbr0" type: internal

# ovs-vsctl br-to-parent vl20-ovsbr0 ovsbr0

virsh # net-dumpxml 20-net <network> <name>20-net</name> <uuid>9e2c2ce7-a193-17ec-d6c0-3f115e759e29</uuid> <forward mode='bridge'/> <bridge name='vl20-ovsbr0' /> <virtualport type='openvswitch'/> </network>

This looks fine.

(I have no XML from the guest, as virt-manager will not let me create it with the desired network...)

Is the problem that libvirt only can define networks with "real" OVS bridges? That would be a bummer, since I need the VM's port tagged with a specific VLAN... Or is there another workable way to do this?

Since this is the first time I've heard of an ovs "fake bridge", I can't really say for sure if it works correctly. However, I now see from your 2nd message that the reason for your initial failure was that you were using a version of libvirt (0.9.4) that had no OVS support at all, and that you've since upgraded to libvirt 1.0.4. Meanwhile, the blog post that you reference above as the reason you're using a "fake bridge" is saying that the reason to do this is to overcome some unnamed bug in libvirt's OVS vlan support. That blog entry was written in October 2012, and mentions that "Kyle" (probably Kyle Mestery) had submitted a patch for [whatever the problem was]. I'm unable to determine which bug or which patch they're talking about from that information, but I don't believe there are any open bugs in OVS vlan support, so it must have been pushed and so would definitely be fixed in libvirt 1.0.4. So, my advice is 1) stop using "fake bridges" and 2) use libvirt's ovs vlan support, as described here: http://www.siliconloons.com/?p=305 (It would be useful to make these "fake bridges" work properly though. You can probably find the error message output by ovs-vsctl by looking in /var/log/messages or /var/log/libvirt/libvirtd.log (it will be logged by the libvirtd process). If you can post that message to this thread, maybe someone will understand why it's failing, and we can push a fix for it.)

[Please do not top-post replies on a technical list. Put your responses inline to make following the conversation easier.] On 04/09/2013 11:56 AM, Will Dennis wrote:

So, looks like a permissions problem.. From /var/log/libvirt/libvirtd.log:

2013-04-08 16:58:53.897+0000: 6743: info : libvirt version: 1.0.4 2013-04-08 16:58:53.897+0000: 6743: error : virCommandWait:2315 : internal error Child process (/usr/local/bin/ovs-vsctl --timeout=5 -- --may-exist add-port vl20-ovsbr0 vnet0 -- set Interface vnet0 'external-ids:attached-mac="52:54:00:10:B0:EE"' -- set Interface vnet0 'external-ids:iface-id="6e4a326a-1595-e712-15ec-c4905725663e"' -- set Interface vnet0 'external-ids:vm-id="67e7842b-a47b-54df-bb7c-aca75368cd45"' -- set Interface vnet0 external-ids:iface-status=active) unexpected exit status 1: libvirt: error : cannot execute binary /usr/local/bin/ovs-vsctl: Permission denied 2013-04-08 16:58:53.897+0000: 6743: error : virNetDevOpenvswitchAddPort:136 : Unable to add port vnet0 to OVS bridge vl20-ovsbr0: Operation not permitted 2013-04-08 16:58:53.937+0000: 6743: error : virCommandWait:2315 : internal error Child process (/usr/local/bin/ovs-vsctl --timeout=5 -- --if-exists del-port) unexpected exit status 1: libvirt: error : cannot execute binary /usr/local/bin/ovs-vsctl: Permission denied

But ovs-vsctl does have read+exec set for 'others': # ls -la /usr/local/bin/ovs-vsctl -rwxr-xr-x 1 root root 3930719 Feb 26 17:12 /usr/local/bin/ovs-vsctl

What else is needed?

libvirtd is running as root, so that should be plenty of permissions. I assume you can run /usr/local/bin/ovs-vsctl from a root shell, correct? Does your system have AppArmor or SELinux installed? Anything in their logs that could indicate the reason for the denial? Anybody else have an idea?

The reason I want to use OVS "fake bridges" is to keep it simple for the team that will be using this virtualization server. If they just select the correct libvirt network (which is tied to the OVS "fake bridge") then the ports will be automatically tagged with the correct VLAN. (I think of OVS fake bridges as something akin to VMware "Port Groups" on an OVS level.) I have defined libvirt networks as detailed in Kyle's blogpost you referenced, but there does not seem to be a way in the virt-manager UI to select the desired portgroup as defined in the network XML. (Yes, you can edit the VM's XML config and hack the portgroup into there, but that's not what I want these users to have to do...)

Sigh. Yes, setting up a portgroup in the network definition for each vlan, then telling the users to select the proper portgroup would be the right way to do what you want, but unfortunately there has been little active development to support new libvirt features in virt-manager for "awhile" now, and the addition of portgroups to libvirt has happened in the meantime. If you're familiar with python and would like to add support for that, I'm sure virt-manager would *gladly* accept a patch. Otherwise, hopefully there will be some new blood in virt-manager, and they'll think this is a good request to take on. At any rate, the generic nature of your error leads me to believe that the problem isn't the "fake bridge" after all.

Thanks again for your assistance! Will

-----Original Message----- From: sendmail [mailto:justsendmailnothingelse@gmail.com] On Behalf Of Laine Stump Sent: Tuesday, April 09, 2013 10:47 AM To: libvirt-users@redhat.com Cc: Will Dennis Subject: Re: [libvirt-users] Problem with net-define using Open vSwitch bridge

On 04/08/2013 05:20 PM, Will Dennis wrote:

...

Thanks, Laine, for your reply. The OVS bridge named 'vl20-ovsbr0' does exist, but it is an Open vSwitch "fake bridge" -- see http://blog.scottlowe.org/2012/10/19/vlans-with-open-vswitch-fake-brid ge s/ for why I'm doing this. Here's some relevant output from my system:

# ifconfig | grep encap eth0 Link encap:Ethernet HWaddr 00:25:90:aa:bb:cc lo Link encap:Local Loopback ovsbr0 Link encap:Ethernet HWaddr a2:c6:37:cd:90:4a vl10-ovsbr0 Link encap:Ethernet HWaddr b6:c4:8c:69:2b:06 vl20-ovsbr0 Link encap:Ethernet HWaddr d6:d0:be:6e:b6:fb

# ovs-vsctl --real list-br ovsbr0

# ovs-vsctl --fake list-br vl10-ovsbr0 vl20-ovsbr0

# ovs-vsctl show 3e0d861b-efb7-46b1-af1b-4a76cd833558 Bridge "ovsbr0" Controller "tcp:127.0.0.1:6633" is_connected: true Port "ovsbr0" Interface "ovsbr0" type: internal Port "vl10-ovsbr0" tag: 10 Interface "vl10-ovsbr0" type: internal Port "vl20-ovsbr0" tag: 20 Interface "vl20-ovsbr0" type: internal

# ovs-vsctl br-to-parent vl20-ovsbr0 ovsbr0

virsh # net-dumpxml 20-net <network> <name>20-net</name> <uuid>9e2c2ce7-a193-17ec-d6c0-3f115e759e29</uuid> <forward mode='bridge'/> <bridge name='vl20-ovsbr0' /> <virtualport type='openvswitch'/> </network> This looks fine.

...

(I have no XML from the guest, as virt-manager will not let me create it with the desired network...)

Is the problem that libvirt only can define networks with "real" OVS bridges? That would be a bummer, since I need the VM's port tagged with a specific VLAN... Or is there another workable way to do this?

Since this is the first time I've heard of an ovs "fake bridge", I can't really say for sure if it works correctly. However, I now see from your 2nd message that the reason for your initial failure was that you were using a version of libvirt (0.9.4) that had no OVS support at all, and that you've since upgraded to libvirt 1.0.4.

Meanwhile, the blog post that you reference above as the reason you're using a "fake bridge" is saying that the reason to do this is to overcome some unnamed bug in libvirt's OVS vlan support. That blog entry was written in October 2012, and mentions that "Kyle" (probably Kyle Mestery) had submitted a patch for [whatever the problem was]. I'm unable to determine which bug or which patch they're talking about from that information, but I don't believe there are any open bugs in OVS vlan support, so it must have been pushed and so would definitely be fixed in libvirt 1.0.4.

So, my advice is 1) stop using "fake bridges" and 2) use libvirt's ovs vlan support, as described here:

http://www.siliconloons.com/?p=305

(It would be useful to make these "fake bridges" work properly though. You can probably find the error message output by ovs-vsctl by looking in /var/log/messages or /var/log/libvirt/libvirtd.log (it will be logged by the libvirtd process). If you can post that message to this thread, maybe someone will understand why it's failing, and we can push a fix for it.)

_______________________________________________ libvirt-users mailing list libvirt-users@redhat.com https://www.redhat.com/mailman/listinfo/libvirt-users