[libvirt-users] libvirt + openvswitch, <parameters interfaceid='x'/> seems less-than-useful?
Hey folks, I'm working on setting up a Cuckoo sandbox malware lab at home, and can't seem to get openvswitch and libvirt to work together as-documented. I'm not subscribed to the list, so please reply-all. Thanks in advance! I've debugged this, and I think the interfaceid stuff in network XML isn't doing much besides setting a custom interfaceid in an OpenVSwitch table that really has no effect on the operation of OpenVSwitch. I think the -- set Interface vnet0 "external-ids:iface-id="{uuid here}" command run by libvirt doesn't actually set the _uuid of a new interface. :( It's just for an external system (e.g. libvirt) to track an interface *it* creates in OVS. I've got an OVS switch, "malware0" which is my malware lab network. I made a "monitor0" interface, and told OVS to make that a span/mirror of all the traffic on the malware0 switch. I attach monitor0 to my cuckoo VM, and boot the VM. Well, actually libvirt REALLY wants to create the interface itself, so I have to delete the monitor0 interface from OVS. I boot the VM, libvirt creates the monitor0 interface (with a different UUID) and it's not a span/mirror port any more. :( I had all the ports in the cuckoo VM <parameters interfaceid='{uuid here}'/>, but libvirt keeps deleting/creating ports and the OVS internal UUID changes, which means any setting applied in OVS (e.g. setting a port as a mirror) is lost during the add/delete cycle. With my cuckoo sandbox vm up: (no bfc10d05 UUID in ports, the mirror port UUID) # ovs-vsctl list bridge _uuid : 16847994-eb75-4e71-a913-50edd8a89252 mirrors : [bfc10d05-846e-4653-8417-27e1f648da93] name : "malware0" ports : [3931cb83-1f85-4fc8-880d-ffe299a85857, 6c6e3d97-d55b-4d55-8179-302412242664, 8eb4e13b-b5ea-4994-8e95-87d61db87ba6] With the cuckoo vm down: (393, 83b go away) # ovs-vsctl list bridge _uuid : 16847994-eb75-4e71-a913-50edd8a89252 mirrors : [bfc10d05-846e-4653-8417-27e1f648da93] name : "malware0" ports : [6c6e3d97-d55b-4d55-8179-302412242664] ... and lets bring up the cuckoo vm again: (only 6c6 remains) # ovs-vsctl list bridge _uuid : 16847994-eb75-4e71-a913-50edd8a89252 mirrors : [bfc10d05-846e-4653-8417-27e1f648da93] name : "malware0" ports : [1c09dd43-52d0-449b-81a2-537ddafb4966, 6c6e3d97-d55b-4d55-8179-302412242664, f90820f9-056f-47a3-bd51-c5190ad1df46]
On 05/13/2016 09:57 AM, Richard Harman wrote:
Hey folks, I'm working on setting up a Cuckoo sandbox malware lab at home, and can't seem to get openvswitch and libvirt to work together as-documented. I'm not subscribed to the list, so please reply-all. Thanks in advance!
I've debugged this, and I think the interfaceid stuff in network XML isn't doing much besides setting a custom interfaceid in an OpenVSwitch table that really has no effect on the operation of OpenVSwitch. I think the -- set Interface vnet0 "external-ids:iface-id="{uuid here}" command run by libvirt doesn't actually set the _uuid of a new interface. :(
I don't use Open vSwitch, but I was around when the support was added, and reviewed the patches (which were authored by Kyle Mestery, whose email has changed since then, so I'm not sure how to Cc him). My understanding at the time was that the interfaceid was intended for exactly the purpose that you say doesn't work (i.e. so that OVS itself could recognize that interface as it appeared and disappeared at different locations on the network). I would classify that as a bug, but that's just my understanding as an OVS outsider. Anyone who is following OVS more closely?
It's just for an external system (e.g. libvirt) to track an interface *it* creates in OVS.
I've got an OVS switch, "malware0" which is my malware lab network. I made a "monitor0" interface, and told OVS to make that a span/mirror of all the traffic on the malware0 switch.
I attach monitor0 to my cuckoo VM, and boot the VM. Well, actually libvirt REALLY wants to create the interface itself, so I have to delete the monitor0 interface from OVS. I boot the VM, libvirt creates the monitor0 interface (with a different UUID) and it's not a span/mirror port any more. :(
I had all the ports in the cuckoo VM <parameters interfaceid='{uuid here}'/>, but libvirt keeps deleting/creating ports and the OVS internal UUID changes, which means any setting applied in OVS (e.g. setting a port as a mirror) is lost during the add/delete cycle.
With my cuckoo sandbox vm up: (no bfc10d05 UUID in ports, the mirror port UUID)
# ovs-vsctl list bridge _uuid : 16847994-eb75-4e71-a913-50edd8a89252 mirrors : [bfc10d05-846e-4653-8417-27e1f648da93] name : "malware0" ports : [3931cb83-1f85-4fc8-880d-ffe299a85857, 6c6e3d97-d55b-4d55-8179-302412242664, 8eb4e13b-b5ea-4994-8e95-87d61db87ba6]
With the cuckoo vm down: (393, 83b go away)
# ovs-vsctl list bridge _uuid : 16847994-eb75-4e71-a913-50edd8a89252 mirrors : [bfc10d05-846e-4653-8417-27e1f648da93] name : "malware0" ports : [6c6e3d97-d55b-4d55-8179-302412242664]
... and lets bring up the cuckoo vm again: (only 6c6 remains)
# ovs-vsctl list bridge _uuid : 16847994-eb75-4e71-a913-50edd8a89252 mirrors : [bfc10d05-846e-4653-8417-27e1f648da93] name : "malware0" ports : [1c09dd43-52d0-449b-81a2-537ddafb4966, 6c6e3d97-d55b-4d55-8179-302412242664, f90820f9-056f-47a3-bd51-c5190ad1df46]
_______________________________________________ libvirt-users mailing list libvirt-users@redhat.com https://www.redhat.com/mailman/listinfo/libvirt-users
On 05/13/16 20:04, Laine Stump wrote:
On 05/13/2016 09:57 AM, Richard Harman wrote:
I've debugged this, and I think the interfaceid stuff in network XML isn't doing much besides setting a custom interfaceid in an OpenVSwitch table that really has no effect on the operation of OpenVSwitch. I think the -- set Interface vnet0 "external-ids:iface-id="{uuid here}" command run by libvirt doesn't actually set the _uuid of a new interface. :(
I don't use Open vSwitch, but I was around when the support was added, and reviewed the patches (which were authored by Kyle Mestery, whose email has changed since then, so I'm not sure how to Cc him). My understanding at the time was that the interfaceid was intended for exactly the purpose that you say doesn't work (i.e. so that OVS itself could recognize that interface as it appeared and disappeared at different locations on the network). I would classify that as a bug, but that's just my understanding as an OVS outsider.
Further evidence, now that I figured out how to dump the external-ids stuff in OVS: Here's the relevant ports of the domain XML: <domain type='kvm' id='5'> <name>cuckoo</name> <uuid>cb4903aa-eaf4-432d-aa57-e9f74f1c8b79</uuid> .... <interface type='bridge'> <mac address='52:54:00:f5:29:72'/> <source network='malware' bridge='malware0'/> <virtualport type='openvswitch'> <parameters interfaceid='23e982f3-f344-48a2-8bed-b584acb95616'/> </virtualport> <target dev='monitor0'/> <model type='virtio'/> <alias name='net2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> </interface> .... cuckoo vm up: monitor0 _uuid is 969ef76c... # ovs-vsctl --columns=name,_uuid,external-ids list Interface | grep -A3 monitor0 name : "monitor0" _uuid : 969ef76c-c802-4e9f-97e0-9a06ce133252 external_ids : {attached-mac="52:54:00:f5:29:72", iface-id="23e982f3-f344-48a2-8bed-b584acb95616", iface-status=active, vm-id="cb4903aa-eaf4-432d-aa57-e9f74f1c8b79"} # cuckoo vm down: (no output, the interface has been deleted by libvirt) # ovs-vsctl --columns=name,_uuid,external-ids list Interface | grep -A3 monitor0 # I found this in my log, showing libvirt deleted the interface: ovs-vsctl[4590]: ovs|00001|vsctl|INFO|Called as ovs-vsctl --timeout=5 -- --if-exists del-port monitor0 cuckoo vm up again: uuid is now fb025df0... # ovs-vsctl --columns=name,_uuid,external-ids list Interface | grep -A3 monitor0 name : "monitor0" _uuid : fb025df0-d934-46d6-b9ed-314bc1de5ef3 external_ids : {attached-mac="52:54:00:f5:29:72", iface-id="23e982f3-f344-48a2-8bed-b584acb95616", iface-status=active, vm-id="cb4903aa-eaf4-432d-aa57-e9f74f1c8b79"} # Anyway, thanks for reading! Hopefully we'll get to the bottom of this, where I can keep my span port. :) In the mean time I'm going to go digging through the OVS source, specifically the ovs-vsctl command to see if an argument can be used to forcibly set the internal _uuid in OVS. Richard
On 05/13/2016 08:24 PM, Richard Harman wrote:
On 05/13/16 20:04, Laine Stump wrote:
On 05/13/2016 09:57 AM, Richard Harman wrote:
I've debugged this, and I think the interfaceid stuff in network XML isn't doing much besides setting a custom interfaceid in an OpenVSwitch table that really has no effect on the operation of OpenVSwitch. I think the -- set Interface vnet0 "external-ids:iface-id="{uuid here}" command run by libvirt doesn't actually set the _uuid of a new interface. :( I don't use Open vSwitch, but I was around when the support was added, and reviewed the patches (which were authored by Kyle Mestery, whose email has changed since then, so I'm not sure how to Cc him). My understanding at the time was that the interfaceid was intended for exactly the purpose that you say doesn't work (i.e. so that OVS itself could recognize that interface as it appeared and disappeared at different locations on the network). I would classify that as a bug, but that's just my understanding as an OVS outsider.
Further evidence, now that I figured out how to dump the external-ids stuff in OVS:
Here's the relevant ports of the domain XML:
<domain type='kvm' id='5'> <name>cuckoo</name> <uuid>cb4903aa-eaf4-432d-aa57-e9f74f1c8b79</uuid> .... <interface type='bridge'> <mac address='52:54:00:f5:29:72'/> <source network='malware' bridge='malware0'/> <virtualport type='openvswitch'> <parameters interfaceid='23e982f3-f344-48a2-8bed-b584acb95616'/> </virtualport> <target dev='monitor0'/> <model type='virtio'/> <alias name='net2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> </interface> ....
cuckoo vm up: monitor0 _uuid is 969ef76c...
# ovs-vsctl --columns=name,_uuid,external-ids list Interface | grep -A3 monitor0 name : "monitor0" _uuid : 969ef76c-c802-4e9f-97e0-9a06ce133252 external_ids : {attached-mac="52:54:00:f5:29:72", iface-id="23e982f3-f344-48a2-8bed-b584acb95616", iface-status=active, vm-id="cb4903aa-eaf4-432d-aa57-e9f74f1c8b79"} #
cuckoo vm down: (no output, the interface has been deleted by libvirt)
# ovs-vsctl --columns=name,_uuid,external-ids list Interface | grep -A3 monitor0 #
I found this in my log, showing libvirt deleted the interface: ovs-vsctl[4590]: ovs|00001|vsctl|INFO|Called as ovs-vsctl --timeout=5 -- --if-exists del-port monitor0
cuckoo vm up again: uuid is now fb025df0...
# ovs-vsctl --columns=name,_uuid,external-ids list Interface | grep -A3 monitor0 name : "monitor0" _uuid : fb025df0-d934-46d6-b9ed-314bc1de5ef3 external_ids : {attached-mac="52:54:00:f5:29:72", iface-id="23e982f3-f344-48a2-8bed-b584acb95616", iface-status=active, vm-id="cb4903aa-eaf4-432d-aa57-e9f74f1c8b79"} #
Anyway, thanks for reading! Hopefully we'll get to the bottom of this, where I can keep my span port. :) In the mean time I'm going to go digging through the OVS source, specifically the ovs-vsctl command to see if an argument can be used to forcibly set the internal _uuid in OVS.
Right. Either the ovs-vsctl command used by libvirt needs to change, or (if it's actually a bug in OVS) ovs-ctl's use of external-ids:iface-id needs to change. I'm Cc'ing libvir-list, just in case there's an OVS person who monitors that list but not this one.
participants (2)
-
Laine Stump -
Richard Harman