[libvirt] [PATCH] qemu: change the name of tap device for a tap and bridge network
Creating tap device and adding the device to bridge are not atomic operation. Similarly deleting tap device and removing it from bridge are not atomic operation. The Problem occurs when two vms start and shutdown. When one vm with the nic named "vnet0" stopping, it deleted tap device but not removing port from bridge. At this time, another vm created the tap device named "vnet0" and added port to the same bridge. Then, the first vm deleted the tap device from the same bridge. Finally, the tap device of the second vm don't attached to the bridge. So, we can add domid to vm's nic name. For example, the vm's domid is 1 and vnet0 is renamed to vnet1.0. Signed-off-by: ZhiPeng Lu <lu.zhipeng@zte.com.cn> --- src/qemu/qemu_interface.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/src/qemu/qemu_interface.c b/src/qemu/qemu_interface.c index d0850c0..17d40a7 100644 --- a/src/qemu/qemu_interface.c +++ b/src/qemu/qemu_interface.c @@ -512,6 +512,7 @@ qemuInterfaceBridgeConnect(virDomainDefPtr def, bool template_ifname = false; virQEMUDriverConfigPtr cfg = virQEMUDriverGetConfig(driver); const char *tunpath = "/dev/net/tun"; + char *domIdPlusIndex = NULL; if (net->backend.tap) { tunpath = net->backend.tap; @@ -531,8 +532,13 @@ qemuInterfaceBridgeConnect(virDomainDefPtr def, STRPREFIX(net->ifname, VIR_NET_GENERATED_PREFIX) || strchr(net->ifname, '%')) { VIR_FREE(net->ifname); - if (VIR_STRDUP(net->ifname, VIR_NET_GENERATED_PREFIX "%d") < 0) + if (virAsprintf(&domIdPlusIndex, "%s%d.%s", + VIR_NET_GENERATED_PREFIX, def->id, "%d") < 0) { + goto cleanup; + } + if (VIR_STRDUP(net->ifname, domIdPlusIndex) < 0) { goto cleanup; + } /* avoid exposing vnet%d in getXMLDesc or error outputs */ template_ifname = true; } @@ -594,6 +600,7 @@ qemuInterfaceBridgeConnect(virDomainDefPtr def, ret = 0; cleanup: + VIR_FREE(domIdPlusIndex); if (ret < 0) { size_t i; for (i = 0; i < *tapfdSize && tapfd[i] >= 0; i++) -- 1.8.3.1
On Fri, Apr 28, 2017 at 05:23:19PM +0800, ZhiPeng Lu wrote:
Creating tap device and adding the device to bridge are not atomic operation. Similarly deleting tap device and removing it from bridge are not atomic operation. The Problem occurs when two vms start and shutdown. When one vm with the nic named "vnet0" stopping, it deleted tap device but not removing port from bridge. At this time, another vm created the tap device named "vnet0" and added port to the same bridge. Then, the first vm deleted the tap device from the same bridge. Finally, the tap device of the second vm don't attached to the bridge. So, we can add domid to vm's nic name. For example, the vm's domid is 1 and vnet0 is renamed to vnet1.0.
Surely deleting the NIC automatically removes it from the bridge so we can just remove the code that delets the bridge port. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On 04/28/2017 05:33 AM, Daniel P. Berrange wrote:
On Fri, Apr 28, 2017 at 05:23:19PM +0800, ZhiPeng Lu wrote:
Creating tap device and adding the device to bridge are not atomic operation. Similarly deleting tap device and removing it from bridge are not atomic operation. The Problem occurs when two vms start and shutdown. When one vm with the nic named "vnet0" stopping, it deleted tap device but not removing port from bridge. At this time, another vm created the tap device named "vnet0" and added port to the same bridge. Then, the first vm deleted the tap device from the same bridge. Finally, the tap device of the second vm don't attached to the bridge. So, we can add domid to vm's nic name. For example, the vm's domid is 1 and vnet0 is renamed to vnet1.0.
Surely deleting the NIC automatically removes it from the bridge so we can just remove the code that delets the bridge port.
That is true when using a Linux host bridge, but I recall that for openvswitch (which I think is what ZhiPeng is using, based on an earlier patch), you must explicitly remove the port from the bridge - apparently the port is still there in openvswitch's table as some sort of "zombie" connection even after the tap device itself no longer exists. But instead of changing the naming scheme, maybe we should just delete the bridge port *before* deleting the tap device instead of after. (Am I recalling correctly that the tap device is deleted automatically when the qemu process is killed? If so, then what's needed is to move the loop in qemuProcessStop() that cleans up network interfaces so that it happens before qemuProcessKill() is called.
On Fri, Apr 28, 2017 at 01:08:45PM -0400, Laine Stump wrote:
On 04/28/2017 05:33 AM, Daniel P. Berrange wrote:
On Fri, Apr 28, 2017 at 05:23:19PM +0800, ZhiPeng Lu wrote:
Creating tap device and adding the device to bridge are not atomic operation. Similarly deleting tap device and removing it from bridge are not atomic operation. The Problem occurs when two vms start and shutdown. When one vm with the nic named "vnet0" stopping, it deleted tap device but not removing port from bridge. At this time, another vm created the tap device named "vnet0" and added port to the same bridge. Then, the first vm deleted the tap device from the same bridge. Finally, the tap device of the second vm don't attached to the bridge. So, we can add domid to vm's nic name. For example, the vm's domid is 1 and vnet0 is renamed to vnet1.0.
Surely deleting the NIC automatically removes it from the bridge so we can just remove the code that delets the bridge port.
That is true when using a Linux host bridge, but I recall that for openvswitch (which I think is what ZhiPeng is using, based on an earlier patch), you must explicitly remove the port from the bridge - apparently the port is still there in openvswitch's table as some sort of "zombie" connection even after the tap device itself no longer exists.
But instead of changing the naming scheme, maybe we should just delete the bridge port *before* deleting the tap device instead of after. (Am I recalling correctly that the tap device is deleted automatically when the qemu process is killed? If so, then what's needed is to move the loop in qemuProcessStop() that cleans up network interfaces so that it happens before qemuProcessKill() is called.
Agreed, we should just reverse the order. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
SSdtIHdvcnJpZWQgdGhhdCB0aGUgbmV0d29yayBwYWNrZXQgbG9zcyByYXRlIHdpbGwgaW5jcmVh c2UgYWZ0ZXIgIHJldmVyc2luZyB0aGUgb3JkZXIuDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoN Cg0KDQoNCuS4uuS6huiuqeaCqOeahFZQbGF06Jma5ouf5YyW5pWF6Zqc5b6X5Yiw6auY5pWI55qE 5aSE55CG77yM6K+35LiK5oql5pWF6Zqc5YiwOiAkVlBsYXTmioDmnK/mlK/mjIHjgIINCg0KDQro iqblv5fmnIsgbHV6aGlwZW5nDQoNCg0KDQoNCg0KDQpJVOW8gOWPkeW3peeoi+W4iCBJVCBEZXZl bG9wbWVudApFbmdpbmVlcg0K5pON5L2c57O757uf5Lqn5ZOB6YOoL+S4reW/g+eglOeptumZoi/n s7vnu5/kuqflk4EgT1MgUHJvZHVjdCBEZXB0Li9DZW50cmFsIFLvvIZEIEluc3RpdHV0ZS9TeXN0 ZW0gUHJvZHVjdA0KDQoNCg0KDQoNCg0KDQoNCg0K5rex5Zyz5biC5Y2X5bGx5Yy656eR5oqA5Y2X 6LevNTXlj7fkuK3lhbTpgJrorq/noJTlj5HlpKfmpbwzM+alvCANCjMzL0YsIFImRCBCdWlsZGlu ZywgWlRFCkNvcnBvcmF0aW9uIEhpLXRlY2ggUm9hZCBTb3V0aCwgDQpIaS10ZWNoCkluZHVzdHJp YWwgUGFyayBOYW5zaGFuIERpc3RyaWN0LCBTaGVuemhlbiwgUC5SLkNoaW5hLCA1MTgwNTcgDQpU OiArODYgNzU1IHh4eHh4eHh4IEY6Kzg2IDc1NSB4eHh4eHh4eCANCk06ICs4NiB4eHh4eHh4eHh4 eCANCkU6IGx1LnpoaXBlbmdAenRlLmNvbS5jbiANCnd3dy56dGUuY29tLmNuDQoNCg0KDQoNCg0K DQrljp/lp4vpgq7ku7YNCg0KDQoNCuWPkeS7tuS6uu+8miDvvJxiZXJyYW5nZUByZWRoYXQuY29t 77yeDQrmlLbku7bkurrvvJog77ycbGFpbmVAbGFpbmUub3Jn77yeDQrmioTpgIHkurrvvJog77yc bGlidmlyLWxpc3RAcmVkaGF0LmNvbe+8nuiKpuW/l+acizEwMTA4MjcyDQrml6Ug5pyfIO+8mjIw MTflubQwNeaciDAy5pelIDE2OjMwDQrkuLsg6aKYIO+8mlJlOiBbbGlidmlydF0gW1BBVENIXSBx ZW11OiBjaGFuZ2UgdGhlIG5hbWUgb2YgdGFwIGRldmljZSBmb3IgYSB0YXBhbmQgYnJpZGdlIG5l dHdvcmsNCg0KDQoNCg0KDQpPbiBGcmksIEFwciAyOCwgMjAxNyBhdCAwMTowODo0NVBNIC0wNDAw LCBMYWluZSBTdHVtcCB3cm90ZToNCu+8niBPbiAwNC8yOC8yMDE3IDA1OjMzIEFNLCBEYW5pZWwg UC4gQmVycmFuZ2Ugd3JvdGU6DQrvvJ4g77yeIE9uIEZyaSwgQXByIDI4LCAyMDE3IGF0IDA1OjIz OjE5UE0gKzA4MDAsIFpoaVBlbmcgTHUgd3JvdGU6DQrvvJ4g77ye77yeIENyZWF0aW5nIHRhcCBk ZXZpY2UgYW5kIGFkZGluZyB0aGUgZGV2aWNlIHRvIGJyaWRnZSBhcmUgbm90IGF0b21pYyBvcGVy YXRpb24uDQrvvJ4g77ye77yeIFNpbWlsYXJseSBkZWxldGluZyB0YXAgZGV2aWNlIGFuZCByZW1v dmluZyBpdCBmcm9tIGJyaWRnZSBhcmUgbm90IGF0b21pYyBvcGVyYXRpb24uDQrvvJ4g77ye77ye IFRoZSBQcm9ibGVtIG9jY3VycyB3aGVuIHR3byB2bXMgc3RhcnQgYW5kIHNodXRkb3duLiBXaGVu IG9uZSB2bSB3aXRoIHRoZSBuaWMNCu+8niDvvJ7vvJ4gbmFtZWQgInZuZXQwIiBzdG9wcGluZywg aXQgZGVsZXRlZCB0YXAgZGV2aWNlIGJ1dCBub3QgcmVtb3ZpbmcgcG9ydCBmcm9tIGJyaWRnZS4N Cu+8niDvvJ7vvJ4gQXQgdGhpcyB0aW1lLCBhbm90aGVyIHZtIGNyZWF0ZWQgdGhlIHRhcCBkZXZp Y2UgbmFtZWQgInZuZXQwIiBhbmQgYWRkZWQgcG9ydCB0byB0aGUNCu+8niDvvJ7vvJ4gc2FtZSBi cmlkZ2UuIFRoZW4sIHRoZSBmaXJzdCB2bSBkZWxldGVkIHRoZSB0YXAgZGV2aWNlIGZyb20gdGhl IHNhbWUgYnJpZGdlLg0K77yeIO+8nu+8niBGaW5hbGx5LCB0aGUgdGFwIGRldmljZSBvZiB0aGUg c2Vjb25kIHZtIGRvbid0IGF0dGFjaGVkIHRvIHRoZSBicmlkZ2UuDQrvvJ4g77ye77yeIFNvLCB3 ZSBjYW4gYWRkIGRvbWlkIHRvIHZtJ3MgbmljIG5hbWUuIEZvciBleGFtcGxlLCB0aGUgdm0ncyBk b21pZCBpcyAxIGFuZCB2bmV0MA0K77yeIO+8nu+8niBpcyByZW5hbWVkIHRvIHZuZXQxLjAuDQrv vJ4g77yeIA0K77yeIO+8niBTdXJlbHkgZGVsZXRpbmcgdGhlIE5JQyBhdXRvbWF0aWNhbGx5IHJl bW92ZXMgaXQgZnJvbSB0aGUgYnJpZGdlIHNvIHdlDQrvvJ4g77yeIGNhbiBqdXN0IHJlbW92ZSB0 aGUgY29kZSB0aGF0IGRlbGV0cyB0aGUgYnJpZGdlIHBvcnQuDQrvvJ4gDQrvvJ4gVGhhdCBpcyB0 cnVlIHdoZW4gdXNpbmcgYSBMaW51eCBob3N0IGJyaWRnZSwgYnV0IEkgcmVjYWxsIHRoYXQgZm9y DQrvvJ4gb3BlbnZzd2l0Y2ggKHdoaWNoIEkgdGhpbmsgaXMgd2hhdCBaaGlQZW5nIGlzIHVzaW5n LCBiYXNlZCBvbiBhbiBlYXJsaWVyDQrvvJ4gcGF0Y2gpLCB5b3UgbXVzdCBleHBsaWNpdGx5IHJl bW92ZSB0aGUgcG9ydCBmcm9tIHRoZSBicmlkZ2UgLSBhcHBhcmVudGx5DQrvvJ4gdGhlIHBvcnQg aXMgc3RpbGwgdGhlcmUgaW4gb3BlbnZzd2l0Y2gncyB0YWJsZSBhcyBzb21lIHNvcnQgb2YgInpv bWJpZSINCu+8niBjb25uZWN0aW9uIGV2ZW4gYWZ0ZXIgdGhlIHRhcCBkZXZpY2UgaXRzZWxmIG5v IGxvbmdlciBleGlzdHMuDQrvvJ4gDQrvvJ4gDQrvvJ4gQnV0IGluc3RlYWQgb2YgY2hhbmdpbmcg dGhlIG5hbWluZyBzY2hlbWUsIG1heWJlIHdlIHNob3VsZCBqdXN0IGRlbGV0ZQ0K77yeIHRoZSBi cmlkZ2UgcG9ydCAqYmVmb3JlKiBkZWxldGluZyB0aGUgdGFwIGRldmljZSBpbnN0ZWFkIG9mIGFm dGVyLiAoQW0gSQ0K77yeIHJlY2FsbGluZyBjb3JyZWN0bHkgdGhhdCB0aGUgdGFwIGRldmljZSBp cyBkZWxldGVkIGF1dG9tYXRpY2FsbHkgd2hlbg0K77yeIHRoZSBxZW11IHByb2Nlc3MgaXMga2ls bGVkPyBJZiBzbywgdGhlbiB3aGF0J3MgbmVlZGVkIGlzIHRvIG1vdmUgdGhlDQrvvJ4gbG9vcCBp biBxZW11UHJvY2Vzc1N0b3AoKSB0aGF0IGNsZWFucyB1cCBuZXR3b3JrIGludGVyZmFjZXMgc28g dGhhdCBpdA0K77yeIGhhcHBlbnMgYmVmb3JlIHFlbXVQcm9jZXNzS2lsbCgpIGlzIGNhbGxlZC4N Cg0KQWdyZWVkLCB3ZSBzaG91bGQganVzdCByZXZlcnNlIHRoZSBvcmRlci4NCg0KUmVnYXJkcywN CkRhbmllbA0KLS0gDQp8OiBodHRwczovL2JlcnJhbmdlLmNvbSAgICAgIC1vLSAgICBodHRwczov L3d3dy5mbGlja3IuY29tL3Bob3Rvcy9kYmVycmFuZ2UgOnwNCnw6IGh0dHBzOi8vbGlidmlydC5v cmcgICAgICAgICAtby0gICAgICAgICAgICBodHRwczovL2ZzdG9wMTM4LmJlcnJhbmdlLmNvbSA6 fA0KfDogaHR0cHM6Ly9lbnRhbmdsZS1waG90by5vcmcgICAgLW8tICAgIGh0dHBzOi8vd3d3Lmlu c3RhZ3JhbS5jb20vZGJlcnJhbmdlIDp8DQoNCi0tDQpsaWJ2aXItbGlzdCBtYWlsaW5nIGxpc3QN CmxpYnZpci1saXN0QHJlZGhhdC5jb20NCmh0dHBzOi8vd3d3LnJlZGhhdC5jb20vbWFpbG1hbi9s aXN0aW5mby9saWJ2aXItbGlzdA==
On Thu, May 04, 2017 at 08:28:03AM +0800, lu.zhipeng@zte.com.cn wrote:
I'm worried that the network packet loss rate will increase after reversing the order.
I don't see why that is relevant - we're killing the guest or unplugging the NIC, so there's no traffic being sent on this tap device. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
T24gTW9uLCBNYXkgMDgsIDIwMTcgYXQgMDM6MDM6MzBQTSArMDgwMCwgWmhpUGVuZyBMdSB3cm90 ZTrvvJ7vvJ4gSW4gcWVtdVByb2Nlc3NTdG9wIHdlIGV4cGxpY2l0bHkgcmVtb3ZlIHRoZSBwb3J0 IGZyb20gdGhlIG9wZW52c3dpdGNoIGJyaWRnZSBhZnRlcu+8nu+8niBxZW11UHJvY2Vzc0tpbGwg aXMgY2FsbGVkLiBCdXQgdGhlcmUgaXMgYSBjZXJ0YWluIGludGVydmFsIG9mIHRpbWUgYmV0d2Vl bu+8nu+8niBkZWxldGluZyB0YXAgZGV2aWNlIGFuZCByZW1vdmluZyBpdCBmcm9tIGJyaWRnZS4g VGhlIHByb2JsZW0gb2NjdXJzIHdoZW4gdHdvIHZtc++8nu+8niBzdGFydCBhbmQgc2h1dGRvd24g d2l0aCB0aGUgc2FtZSBuYW1lJ3MgbmV0d29yayBpbnRlcmZhY2UgYXR0YWNoZWQgdG8gdGhlIHNh bWXvvJ7vvJ4gb3BlbnZzd2l0Y2ggYnJpZGdlLiBXaGVuIG9uZSB2bSB3aXRoIHRoZSBuaWMgbmFt ZWQgInZuZXQwIiBzdG9wcGluZywgaXQgZGVsZXRlZO+8nu+8nnRhcCBkZXZpY2Ugd2l0aG91dCB0 aW1lbHkgcmVtb3ZpbmcgdGhlIHBvcnQgZnJvbSBicmlkZ2Uu77ye77yeLkF0IHRoaXMgdGltZSwg YW5vdGhlciB2bSBjcmVhdGVkIHRoZSB0YXAgZGV2aWNlIG5hbWVkICJ2bmV0MCIgYW5kIGFkZGVk IHBvcnQgdG8gdGhl77ye77yeIHNhbWUgYnJpZGdlLiBUaGVuLCB0aGUgZmlyc3Qgdm0gcmVtb3Zl ZCB0aGUgcG9ydCBmcm9tIHRoZSBzYW1lIGJyaWRnZS7vvJ7vvJ4gRmluYWxseSwgdGhlIHRhcCBk ZXZpY2Ugb2YgdGhlIHNlY29uZCB2bSBkaWQgbm90IGF0dGFjaGVkIHRvIHRoZSBicmlkZ2Uu77ye 77yeIFdlIG5lZWQgdG8gZGVsZXRlIHRoZSBicmlkZ2UgcG9ydCBiZWZvcmUgZGVsZXRpbmcgdGhl IHRhcCBkZXZpY2UgaW5zdGVhZCBvZiBhZnRlci7vvJ7vvJ4gU28gd2hhdCdzIG5lZWRlZCBpcyB0 byBtb3ZlIHRoZSBsb29wIGluIHFlbXVQcm9jZXNzU3RvcCB0aGF0IGNsZWFucyB1cO+8nu+8niBu ZXR3b3JrIGludGVyZmFjZXMgc28gdGhhdCBpdCBoYXBwZW5zIGJlZm9yZSBxZW11UHJvY2Vzc0tp bGwgaXMgY2FsbGVkLu+8nlRoaXMgZml4IHdvbid0IHdvcmsgY29ycmVjdGx5IGVpdGhlci4gWW91 IGNhbm5vdCBhc3N1bWUgdGhhdCBsaWJ2aXJ0IGhhc++8nmNvbnRyb2wgb3ZlciB3aGVuIHRoZSBR RU1VIHByb2Nlc3MgZXhpdHMuIEl0IG1heSBleGl0IGl0c2VsZiAqYmVmb3JlKu+8nmxpYnZpcnQg cnVucyBhbnkgb2YgaXRzIGNsZWFudXAgY29kZS4NCg0KDQoNCg0KT24gTW9uLCBNYXkgMDgsIDIw MTcgYXQgMDY6MTg6MjVQTSArMDgwMCwgbHUuemhpcGVuZ0B6dGUuY29tLmNuIHdyb3RlOu+8niBJ IG1heSBub3QgaGF2ZSBkZXNjcmliZWQgaXQgY2xlYXJseS4gIO+8niDvvJ4gaSBuZWVkIHRvIGVu c3VyZSAgdGhlIG9yZGVyIG9mICBhZGRpbmcgcG9ydCB0byBicmlkZ2UgYW5kIGRlbGV0aW5nIGZy b20gYnJpZGdlLu+8niDvvJ4gaSAgcmVuYW1lIHRhcCBkZXZpY2UgdG8gYXZvaWQgdGhlIHByb2Js ZW0gIGluIG15IGZpcnN0IHBhdGNoLmkgdGhpbmsgaXQgY2FuIHNvbHZlIHRoZSBwcm9ibGVtLu+8 niDvvJ4gbXkgc2Vjb25kIHBhdGNoIG1heSBjYW4ndCByZXNvbHZlIHRoZSBwcm9ibGVtIC7vvJ4g 77yeIERvIHlvdSAgaGF2ZSBhbnkgYmV0dGVyIGlkZWFzPw0KDQoNCg0KDQoNCg0KDQoNCg0KDQoN Cg0KDQoNCg0KDQoNCg0KDQrkuLrkuoborqnmgqjnmoRWUGxhdOiZmuaLn+WMluaVhemanOW+l+WI sOmrmOaViOeahOWkhOeQhu+8jOivt+S4iuaKpeaVhemanOWIsDogJFZQbGF05oqA5pyv5pSv5oyB 44CCDQoNCg0K6Iqm5b+X5pyLIGx1emhpcGVuZw0KDQoNCg0KDQoNCg0KSVTlvIDlj5Hlt6XnqIvl uIggSVQgRGV2ZWxvcG1lbnQKRW5naW5lZXINCuaTjeS9nOezu+e7n+S6p+WTgemDqC/kuK3lv4Pn oJTnqbbpmaIv57O757uf5Lqn5ZOBIE9TIFByb2R1Y3QgRGVwdC4vQ2VudHJhbCBS77yGRCBJbnN0 aXR1dGUvU3lzdGVtIFByb2R1Y3QNCg0KDQoNCg0KDQoNCg0KDQoNCua3seWcs+W4guWNl+WxseWM uuenkeaKgOWNl+i3rzU15Y+35Lit5YW06YCa6K6v56CU5Y+R5aSn5qW8MzPmpbwgDQozMy9GLCBS JkQgQnVpbGRpbmcsIFpURQpDb3Jwb3JhdGlvbiBIaS10ZWNoIFJvYWQgU291dGgsIA0KSGktdGVj aApJbmR1c3RyaWFsIFBhcmsgTmFuc2hhbiBEaXN0cmljdCwgU2hlbnpoZW4sIFAuUi5DaGluYSwg NTE4MDU3IA0KVDogKzg2IDc1NSB4eHh4eHh4eCBGOis4NiA3NTUgeHh4eHh4eHggDQpNOiArODYg eHh4eHh4eHh4eHggDQpFOiBsdS56aGlwZW5nQHp0ZS5jb20uY24gDQp3d3cuenRlLmNvbS5jbg0K DQoNCg0KDQoNCg0K5Y6f5aeL6YKu5Lu2DQoNCg0KDQrlj5Hku7bkurrvvJog77ycYmVycmFuZ2VA cmVkaGF0LmNvbe+8ng0K5pS25Lu25Lq677yaIO+8nGxhaW5lQGxhaW5lLm9yZ++8ng0K5oqE6YCB 5Lq677yaIO+8nGxpYnZpci1saXN0QHJlZGhhdC5jb23vvJ7oiqblv5fmnIsxMDEwODI3Mg0K5pel IOacnyDvvJoyMDE35bm0MDXmnIgwMuaXpSAxNjo0MQ0K5Li7IOmimCDvvJpSZTogW2xpYnZpcnRd IFtQQVRDSF0gcWVtdTogY2hhbmdlIHRoZSBuYW1lIG9mIHRhcCBkZXZpY2UgZm9yIGEgdGFwYW5k IGJyaWRnZSBuZXR3b3JrDQoNCg0KDQoNCg0KT24gRnJpLCBBcHIgMjgsIDIwMTcgYXQgMDE6MDg6 NDVQTSAtMDQwMCwgTGFpbmUgU3R1bXAgd3JvdGU6DQrvvJ4gT24gMDQvMjgvMjAxNyAwNTozMyBB TSwgRGFuaWVsIFAuIEJlcnJhbmdlIHdyb3RlOg0K77yeIO+8niBPbiBGcmksIEFwciAyOCwgMjAx NyBhdCAwNToyMzoxOVBNICswODAwLCBaaGlQZW5nIEx1IHdyb3RlOg0K77yeIO+8nu+8niBDcmVh dGluZyB0YXAgZGV2aWNlIGFuZCBhZGRpbmcgdGhlIGRldmljZSB0byBicmlkZ2UgYXJlIG5vdCBh dG9taWMgb3BlcmF0aW9uLg0K77yeIO+8nu+8niBTaW1pbGFybHkgZGVsZXRpbmcgdGFwIGRldmlj ZSBhbmQgcmVtb3ZpbmcgaXQgZnJvbSBicmlkZ2UgYXJlIG5vdCBhdG9taWMgb3BlcmF0aW9uLg0K 77yeIO+8nu+8niBUaGUgUHJvYmxlbSBvY2N1cnMgd2hlbiB0d28gdm1zIHN0YXJ0IGFuZCBzaHV0 ZG93bi4gV2hlbiBvbmUgdm0gd2l0aCB0aGUgbmljDQrvvJ4g77ye77yeIG5hbWVkICJ2bmV0MCIg c3RvcHBpbmcsIGl0IGRlbGV0ZWQgdGFwIGRldmljZSBidXQgbm90IHJlbW92aW5nIHBvcnQgZnJv bSBicmlkZ2UuDQrvvJ4g77ye77yeIEF0IHRoaXMgdGltZSwgYW5vdGhlciB2bSBjcmVhdGVkIHRo ZSB0YXAgZGV2aWNlIG5hbWVkICJ2bmV0MCIgYW5kIGFkZGVkIHBvcnQgdG8gdGhlDQrvvJ4g77ye 77yeIHNhbWUgYnJpZGdlLiBUaGVuLCB0aGUgZmlyc3Qgdm0gZGVsZXRlZCB0aGUgdGFwIGRldmlj ZSBmcm9tIHRoZSBzYW1lIGJyaWRnZS4NCu+8niDvvJ7vvJ4gRmluYWxseSwgdGhlIHRhcCBkZXZp Y2Ugb2YgdGhlIHNlY29uZCB2bSBkb24ndCBhdHRhY2hlZCB0byB0aGUgYnJpZGdlLg0K77yeIO+8 nu+8niBTbywgd2UgY2FuIGFkZCBkb21pZCB0byB2bSdzIG5pYyBuYW1lLiBGb3IgZXhhbXBsZSwg dGhlIHZtJ3MgZG9taWQgaXMgMSBhbmQgdm5ldDANCu+8niDvvJ7vvJ4gaXMgcmVuYW1lZCB0byB2 bmV0MS4wLg0K77yeIO+8niANCu+8niDvvJ4gU3VyZWx5IGRlbGV0aW5nIHRoZSBOSUMgYXV0b21h dGljYWxseSByZW1vdmVzIGl0IGZyb20gdGhlIGJyaWRnZSBzbyB3ZQ0K77yeIO+8niBjYW4ganVz dCByZW1vdmUgdGhlIGNvZGUgdGhhdCBkZWxldHMgdGhlIGJyaWRnZSBwb3J0Lg0K77yeIA0K77ye IFRoYXQgaXMgdHJ1ZSB3aGVuIHVzaW5nIGEgTGludXggaG9zdCBicmlkZ2UsIGJ1dCBJIHJlY2Fs bCB0aGF0IGZvcg0K77yeIG9wZW52c3dpdGNoICh3aGljaCBJIHRoaW5rIGlzIHdoYXQgWmhpUGVu ZyBpcyB1c2luZywgYmFzZWQgb24gYW4gZWFybGllcg0K77yeIHBhdGNoKSwgeW91IG11c3QgZXhw bGljaXRseSByZW1vdmUgdGhlIHBvcnQgZnJvbSB0aGUgYnJpZGdlIC0gYXBwYXJlbnRseQ0K77ye IHRoZSBwb3J0IGlzIHN0aWxsIHRoZXJlIGluIG9wZW52c3dpdGNoJ3MgdGFibGUgYXMgc29tZSBz b3J0IG9mICJ6b21iaWUiDQrvvJ4gY29ubmVjdGlvbiBldmVuIGFmdGVyIHRoZSB0YXAgZGV2aWNl IGl0c2VsZiBubyBsb25nZXIgZXhpc3RzLg0K77yeIA0K77yeIA0K77yeIEJ1dCBpbnN0ZWFkIG9m IGNoYW5naW5nIHRoZSBuYW1pbmcgc2NoZW1lLCBtYXliZSB3ZSBzaG91bGQganVzdCBkZWxldGUN Cu+8niB0aGUgYnJpZGdlIHBvcnQgKmJlZm9yZSogZGVsZXRpbmcgdGhlIHRhcCBkZXZpY2UgaW5z dGVhZCBvZiBhZnRlci4gKEFtIEkNCu+8niByZWNhbGxpbmcgY29ycmVjdGx5IHRoYXQgdGhlIHRh cCBkZXZpY2UgaXMgZGVsZXRlZCBhdXRvbWF0aWNhbGx5IHdoZW4NCu+8niB0aGUgcWVtdSBwcm9j ZXNzIGlzIGtpbGxlZD8gSWYgc28sIHRoZW4gd2hhdCdzIG5lZWRlZCBpcyB0byBtb3ZlIHRoZQ0K 77yeIGxvb3AgaW4gcWVtdVByb2Nlc3NTdG9wKCkgdGhhdCBjbGVhbnMgdXAgbmV0d29yayBpbnRl cmZhY2VzIHNvIHRoYXQgaXQNCu+8niBoYXBwZW5zIGJlZm9yZSBxZW11UHJvY2Vzc0tpbGwoKSBp cyBjYWxsZWQuDQoNCkFncmVlZCwgd2Ugc2hvdWxkIGp1c3QgcmV2ZXJzZSB0aGUgb3JkZXIuDQoN ClJlZ2FyZHMsDQpEYW5pZWwNCi0tIA0KfDogaHR0cHM6Ly9iZXJyYW5nZS5jb20gICAgICAtby0g ICAgaHR0cHM6Ly93d3cuZmxpY2tyLmNvbS9waG90b3MvZGJlcnJhbmdlIDp8DQp8OiBodHRwczov L2xpYnZpcnQub3JnICAgICAgICAgLW8tICAgICAgICAgICAgaHR0cHM6Ly9mc3RvcDEzOC5iZXJy YW5nZS5jb20gOnwNCnw6IGh0dHBzOi8vZW50YW5nbGUtcGhvdG8ub3JnICAgIC1vLSAgICBodHRw czovL3d3dy5pbnN0YWdyYW0uY29tL2RiZXJyYW5nZSA6fA==
FYI your replies to messages on the list are almost impossible to read & understand. Your email client is sending HTML and badly mangled plain text. Please make it send plain text only to public mailing lists, with line wrapping and sensibly trim & quote text you are replying to. I can't see any comment you made in this latest message so I can't reply to you. On Tue, May 09, 2017 at 08:44:55AM +0800, lu.zhipeng@zte.com.cn wrote:
On Mon, May 08, 2017 at 03:03:30PM +0800, ZhiPeng Lu wrote:＞＞ In qemuProcessStop we explicitly remove the port from the openvswitch bridge after＞＞ qemuProcessKill is called. But there is a certain interval of time between＞＞ deleting tap device and removing it from bridge. The problem occurs when two vms＞＞ start and shutdown with the same name's network interface attached to the same＞＞ openvswitch bridge. When one vm with the nic named "vnet0" stopping, it deleted＞＞tap device without timely removing the port from bridge.＞＞.At this time, another vm created the tap device named "vnet0" and added port to the＞＞ same bridge. Then, the first vm removed the port from the same bridge.＞＞ Finally, the tap device of the second vm did not attached to the bridge.＞＞ We need to delete the bridge port before deleting the tap device instead of after.＞＞ So what's needed is to move the loop in qemuProcessStop that cleans up＞＞ network interfaces so that it happens before qemuProcessKill is called.＞This fix won't work correctly either. You cannot assume that libvirt has＞control over when the QEMU process exits. It may exit itself *before*＞libvirt runs any of its cleanup code.
On Mon, May 08, 2017 at 06:18:25PM +0800, lu.zhipeng@zte.com.cn wrote:＞ I may not have described it clearly. ＞ ＞ i need to ensure the order of adding port to bridge and deleting from bridge.＞ ＞ i rename tap device to avoid the problem in my first patch.i think it can solve the problem.＞ ＞ my second patch may can't resolve the problem .＞ ＞ Do you have any better ideas?
g>
为了让您的VPlat虚拟化故障得到高效的处理，请上报故障到: $VPlat技术支持。
芦志朋 luzhipeng
IT开发工程师 IT Development Engineer 操作系统产品部/中心研究院/系统产品 OS Product Dept./Central R＆D Institute/System Product
深圳市南山区科技南路55号中兴通讯研发大楼33楼 33/F, R&D Building, ZTE Corporation Hi-tech Road South, Hi-tech Industrial Park Nanshan District, Shenzhen, P.R.China, 518057 T: +86 755 xxxxxxxx F:+86 755 xxxxxxxx M: +86 xxxxxxxxxxx E: lu.zhipeng@zte.com.cn www.zte.com.cn
原始邮件
发件人： ＜berrange@redhat.com＞ 收件人： ＜laine@laine.org＞ 抄送人： ＜libvir-list@redhat.com＞芦志朋10108272 日 期 ：2017年05月02日 16:41 主 题 ：Re: [libvirt] [PATCH] qemu: change the name of tap device for a tapand bridge network
On Fri, Apr 28, 2017 at 01:08:45PM -0400, Laine Stump wrote: ＞ On 04/28/2017 05:33 AM, Daniel P. Berrange wrote: ＞ ＞ On Fri, Apr 28, 2017 at 05:23:19PM +0800, ZhiPeng Lu wrote: ＞ ＞＞ Creating tap device and adding the device to bridge are not atomic operation. ＞ ＞＞ Similarly deleting tap device and removing it from bridge are not atomic operation. ＞ ＞＞ The Problem occurs when two vms start and shutdown. When one vm with the nic ＞ ＞＞ named "vnet0" stopping, it deleted tap device but not removing port from bridge. ＞ ＞＞ At this time, another vm created the tap device named "vnet0" and added port to the ＞ ＞＞ same bridge. Then, the first vm deleted the tap device from the same bridge. ＞ ＞＞ Finally, the tap device of the second vm don't attached to the bridge. ＞ ＞＞ So, we can add domid to vm's nic name. For example, the vm's domid is 1 and vnet0 ＞ ＞＞ is renamed to vnet1.0. ＞ ＞ ＞ ＞ Surely deleting the NIC automatically removes it from the bridge so we ＞ ＞ can just remove the code that delets the bridge port. ＞ ＞ That is true when using a Linux host bridge, but I recall that for ＞ openvswitch (which I think is what ZhiPeng is using, based on an earlier ＞ patch), you must explicitly remove the port from the bridge - apparently ＞ the port is still there in openvswitch's table as some sort of "zombie" ＞ connection even after the tap device itself no longer exists. ＞ ＞ ＞ But instead of changing the naming scheme, maybe we should just delete ＞ the bridge port *before* deleting the tap device instead of after. (Am I ＞ recalling correctly that the tap device is deleted automatically when ＞ the qemu process is killed? If so, then what's needed is to move the ＞ loop in qemuProcessStop() that cleans up network interfaces so that it ＞ happens before qemuProcessKill() is called.
Agreed, we should just reverse the order.
Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
77yeT24gRnJpLCBBcHIgMjgsIDIwMTcgYXQgMDU6MjM6MTlQTSArMDgwMCwgWmhpUGVuZyBMdSB3 cm90ZTrvvJ7vvJ4gQ3JlYXRpbmcgdGFwIGRldmljZSBhbmQgYWRkaW5nIHRoZSBkZXZpY2UgdG8g YnJpZGdlIGFyZSBub3QgYXRvbWljIG9wZXJhdGlvbi7vvJ7vvJ4gU2ltaWxhcmx5IGRlbGV0aW5n IHRhcCBkZXZpY2UgYW5kIHJlbW92aW5nIGl0IGZyb20gYnJpZGdlIGFyZSBub3QgYXRvbWljIG9w ZXJhdGlvbi7vvJ7vvJ5UaGUgUHJvYmxlbSBvY2N1cnMgd2hlbiB0d28gdm1zIHN0YXJ0IGFuZCBz aHV0ZG93bi4gV2hlbiBvbmUgdm0gd2l0aCB0aGUgbmlj77ye77yeIG5hbWVkICJ2bmV0MCIgc3Rv cHBpbmcsIGl0IGRlbGV0ZWQgdGFwIGRldmljZSBidXQgbm90IHJlbW92aW5nIHBvcnQgZnJvbSBi cmlkZ2Uu77ye77yeIEF0IHRoaXMgdGltZSwgYW5vdGhlciB2bSBjcmVhdGVkIHRoZSB0YXAgZGV2 aWNlIG5hbWVkICJ2bmV0MCIgYW5kIGFkZGVkIHBvcnQgdG8gdGhl77ye77yeIHNhbWUgYnJpZGdl LiBUaGVuLCB0aGUgZmlyc3Qgdm0gZGVsZXRlZCB0aGUgdGFwIGRldmljZSBmcm9tIHRoZSBzYW1l IGJyaWRnZS7vvJ7vvJ4gRmluYWxseSwgdGhlIHRhcCBkZXZpY2Ugb2YgdGhlIHNlY29uZCB2bSBk b24ndCBhdHRhY2hlZCB0byB0aGUgYnJpZGdlLu+8nu+8niBTbywgd2UgY2FuIGFkZCBkb21pZCB0 byB2bSdzIG5pYyBuYW1lLiBGb3IgZXhhbXBsZSwgdGhlIHZtJ3MgZG9taWQgaXMgMSBhbmQgdm5l dDDvvJ7vvJ4gaXMgcmVuYW1lZCB0byB2bmV0MS4wLu+8nlN1cmVseSBkZWxldGluZyB0aGUgTklD IGF1dG9tYXRpY2FsbHkgcmVtb3ZlcyBpdCBmcm9tIHRoZSBicmlkZ2Ugc28gd2XvvJ5jYW4ganVz dCByZW1vdmUgdGhlIGNvZGUgdGhhdCBkZWxldHMgdGhlIGJyaWRnZSBwb3J0Lg0KDQppIGhhdmUg ZG9uZSBzb21lIHRlc3RzIGZvciBhIHRhcCArIG9wZW52c3dpdGNoIGJyaWRnZSBuZXR3b3JrLiAg aSBmaW5kICB0aGUgbmljIG5hbWVkICJ2bmV0MCIgZG9uJ3QgZXhzaXQgYmVmb3JlIGNhbGxpbmcg dGhlIHZpck5ldERldk9wZW52c3dpdGNoUmVtb3ZlUG9ydC4NCg0KaSB0aGluayAgdGhlIHRhcCBp cyBkZWxldGVkIG5vdCBieSByZW1vdmluZ3BvcnQgIGZyb20gYnJpZGdlLiBpIHRoaW5rIGhvdHBs dWdpbmcgbmV0IGhhcyB0aGUgc2FtZSBwcm9ibGVtLg0KDQoNCg0KDQpieSB0aGUgd2F5Og0KDQog ICAgICBteSBjb21wYW55J3MgZS1tYWlsICBkb2VzIG5vdCBzdXBwb3J0IHRocmVhZCBwb3N0aW5n ICBhbmQgaGFzIHNvbWUgb3RoZXIgcHJvYmxlbXMuIGkgYmVsaWV2ZSB0aGF0IG15IGNvbGxlYWd1 ZXMgY2FuIHNvb24gcmVzbG92ZWQgdGhlbS4NCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoN Cg0KDQoNCg0K5Li65LqG6K6p5oKo55qEVlBsYXTomZrmi5/ljJbmlYXpmpzlvpfliLDpq5jmlYjn moTlpITnkIbvvIzor7fkuIrmiqXmlYXpmpzliLA6ICRWUGxhdOaKgOacr+aUr+aMgeOAgg0KDQoN CuiKpuW/l+aciyBsdXpoaXBlbmcNCg0KDQoNCg0KDQoNCklU5byA5Y+R5bel56iL5biIIElUIERl dmVsb3BtZW50CkVuZ2luZWVyDQrmk43kvZzns7vnu5/kuqflk4Hpg6gv5Lit5b+D56CU56m26Zmi L+ezu+e7n+S6p+WTgSBPUyBQcm9kdWN0IERlcHQuL0NlbnRyYWwgUu+8hkQgSW5zdGl0dXRlL1N5 c3RlbSBQcm9kdWN0DQoNCg0KDQoNCg0KDQoNCg0KDQrmt7HlnLPluILljZflsbHljLrnp5HmioDl jZfot681NeWPt+S4reWFtOmAmuiur+eglOWPkeWkp+alvDMz5qW8IA0KMzMvRiwgUiZEIEJ1aWxk aW5nLCBaVEUKQ29ycG9yYXRpb24gSGktdGVjaCBSb2FkIFNvdXRoLCANCkhpLXRlY2gKSW5kdXN0 cmlhbCBQYXJrIE5hbnNoYW4gRGlzdHJpY3QsIFNoZW56aGVuLCBQLlIuQ2hpbmEsIDUxODA1NyAN ClQ6ICs4NiA3NTUgeHh4eHh4eHggRjorODYgNzU1IHh4eHh4eHh4IA0KTTogKzg2IHh4eHh4eHh4 eHh4IA0KRTogbHUuemhpcGVuZ0B6dGUuY29tLmNuIA0Kd3d3Lnp0ZS5jb20uY24NCg0KDQoNCg0K DQoNCuWOn+Wni+mCruS7tg0KDQoNCg0K5Y+R5Lu25Lq677yaIO+8nGJlcnJhbmdlQHJlZGhhdC5j b23vvJ4NCuaUtuS7tuS6uu+8muiKpuW/l+acizEwMTA4MjcyDQrmioTpgIHkurrvvJog77ycbGli dmlyLWxpc3RAcmVkaGF0LmNvbe+8ng0K5pelIOacnyDvvJoyMDE35bm0MDTmnIgyOOaXpSAxOToy Nw0K5Li7IOmimCDvvJpSZTogW2xpYnZpcnRdIFtQQVRDSF0gcWVtdTogY2hhbmdlIHRoZSBuYW1l IG9mIHRhcCBkZXZpY2UgZm9yIGEgdGFwYW5kIGJyaWRnZSBuZXR3b3JrDQoNCg0KDQoNCg0KT24g RnJpLCBBcHIgMjgsIDIwMTcgYXQgMDU6MjM6MTlQTSArMDgwMCwgWmhpUGVuZyBMdSB3cm90ZToN Cu+8niBDcmVhdGluZyB0YXAgZGV2aWNlIGFuZCBhZGRpbmcgdGhlIGRldmljZSB0byBicmlkZ2Ug YXJlIG5vdCBhdG9taWMgb3BlcmF0aW9uLg0K77yeIFNpbWlsYXJseSBkZWxldGluZyB0YXAgZGV2 aWNlIGFuZCByZW1vdmluZyBpdCBmcm9tIGJyaWRnZSBhcmUgbm90IGF0b21pYyBvcGVyYXRpb24u DQrvvJ4gVGhlIFByb2JsZW0gb2NjdXJzIHdoZW4gdHdvIHZtcyBzdGFydCBhbmQgc2h1dGRvd24u IFdoZW4gb25lIHZtIHdpdGggdGhlIG5pYw0K77yeIG5hbWVkICJ2bmV0MCIgc3RvcHBpbmcsIGl0 IGRlbGV0ZWQgdGFwIGRldmljZSBidXQgbm90IHJlbW92aW5nIHBvcnQgZnJvbSBicmlkZ2UuDQrv vJ4gQXQgdGhpcyB0aW1lLCBhbm90aGVyIHZtIGNyZWF0ZWQgdGhlIHRhcCBkZXZpY2UgbmFtZWQg InZuZXQwIiBhbmQgYWRkZWQgcG9ydCB0byB0aGUNCu+8niBzYW1lIGJyaWRnZS4gVGhlbiwgdGhl IGZpcnN0IHZtIGRlbGV0ZWQgdGhlIHRhcCBkZXZpY2UgZnJvbSB0aGUgc2FtZSBicmlkZ2UuDQrv vJ4gRmluYWxseSwgdGhlIHRhcCBkZXZpY2Ugb2YgdGhlIHNlY29uZCB2bSBkb24ndCBhdHRhY2hl ZCB0byB0aGUgYnJpZGdlLg0K77yeIFNvLCB3ZSBjYW4gYWRkIGRvbWlkIHRvIHZtJ3MgbmljIG5h bWUuIEZvciBleGFtcGxlLCB0aGUgdm0ncyBkb21pZCBpcyAxIGFuZCB2bmV0MA0K77yeIGlzIHJl bmFtZWQgdG8gdm5ldDEuMC4NCg0KU3VyZWx5IGRlbGV0aW5nIHRoZSBOSUMgYXV0b21hdGljYWxs eSByZW1vdmVzIGl0IGZyb20gdGhlIGJyaWRnZSBzbyB3ZQ0KY2FuIGp1c3QgcmVtb3ZlIHRoZSBj b2RlIHRoYXQgZGVsZXRzIHRoZSBicmlkZ2UgcG9ydC4NCg0KDQpSZWdhcmRzLA0KRGFuaWVsDQot LSANCnw6IGh0dHBzOi8vYmVycmFuZ2UuY29tICAgICAgLW8tICAgIGh0dHBzOi8vd3d3LmZsaWNr ci5jb20vcGhvdG9zL2RiZXJyYW5nZSA6fA0KfDogaHR0cHM6Ly9saWJ2aXJ0Lm9yZyAgICAgICAg IC1vLSAgICAgICAgICAgIGh0dHBzOi8vZnN0b3AxMzguYmVycmFuZ2UuY29tIDp8DQp8OiBodHRw czovL2VudGFuZ2xlLXBob3RvLm9yZyAgICAtby0gICAgaHR0cHM6Ly93d3cuaW5zdGFncmFtLmNv bS9kYmVycmFuZ2UgOnwNCg0KLS0NCmxpYnZpci1saXN0IG1haWxpbmcgbGlzdA0KbGlidmlyLWxp c3RAcmVkaGF0LmNvbQ0KaHR0cHM6Ly93d3cucmVkaGF0LmNvbS9tYWlsbWFuL2xpc3RpbmZvL2xp YnZpci1saXN0
Possibly related I notice race conditions caused by vnic never getting loaded if an existing bridge is already up (by OS init scripts etc) and stopping VM's from getting started. Often this is behavior you want ; i.e having Host Hypervisor NIC's added and up before libvirtd sets up it's nics/bridges. On 29 April 2017 at 11:15, <lu.zhipeng@zte.com.cn> wrote:
＞On Fri, Apr 28, 2017 at 05:23:19PM +0800, ZhiPeng Lu wrote: ＞＞ Creating tap device and adding the device to bridge are not atomic operation. ＞＞ Similarly deleting tap device and removing it from bridge are not atomic operation. ＞＞The Problem occurs when two vms start and shutdown. When one vm with the nic ＞＞ named "vnet0" stopping, it deleted tap device but not removing port from bridge. ＞＞ At this time, another vm created the tap device named " vnet0" and added port to the ＞＞ same bridge. Then, the first vm deleted the tap device from the same bridge. ＞＞ Finally, the tap device of the second vm don't attached to the bridge. ＞＞ So, we can add domid to vm's nic name. For example, the vm' s domid is 1 and vnet0 ＞＞ is renamed to vnet1.0.
＞Surely deleting the NIC automatically removes it from the bridge so we ＞can just remove the code that delets the bridge port.
i have done some tests for a tap + openvswitch bridge network. i find the nic named "vnet0" don't exsit before calling the virNetDevOpenvswitchRemovePort.
i think the tap is deleted not by removingport from bridge. i think hotpluging net has the same problem.
by the way:
my company's e-mail does not support thread posting and has some other problems. i believe that my colleagues can soon resloved them.
*为了让您的VPlat虚拟化故障得到高效的处理，请上报故障到: $VPlat技术支持。*
芦志朋 luzhipeng
IT开发工程师 IT Development Engineer 操作系统产品部/中心研究院/系统产品 OS Product Dept./Central R＆D Institute/System Product
深圳市南山区科技南路55号中兴通讯研发大楼33楼 33/F, R&D Building, ZTE Corporation Hi-tech Road South, Hi-tech Industrial Park Nanshan District, Shenzhen, P.R.China, 518057 T: +86 755 xxxxxxxx F:+86 755 xxxxxxxx M: +86 xxxxxxxxxxx E: lu.zhipeng@zte.com.cn www.zte.com.cn 原始邮件 *发件人：* ＜berrange@redhat.com＞; *收件人：*芦志朋10108272; *抄送人：* ＜libvir-list@redhat.com＞; *日 期 ：*2017年04月28日 19:27 *主 题 ：**Re: [libvirt] [PATCH] qemu: change the name of tap device for a tapand bridge network*
On Fri, Apr 28, 2017 at 05:23:19PM +0800, ZhiPeng Lu wrote: ＞ Creating tap device and adding the device to bridge are not atomic operation. ＞ Similarly deleting tap device and removing it from bridge are not atomic operation. ＞ The Problem occurs when two vms start and shutdown. When one vm with the nic ＞ named "vnet0" stopping, it deleted tap device but not removing port from bridge. ＞ At this time, another vm created the tap device named " vnet0" and added port to the ＞ same bridge. Then, the first vm deleted the tap device from the same bridge. ＞ Finally, the tap device of the second vm don't attached to the bridge. ＞ So, we can add domid to vm's nic name. For example, the vm's domid is 1 and vnet0 ＞ is renamed to vnet1.0.
Surely deleting the NIC automatically removes it from the bridge so we can just remove the code that delets the bridge port.
Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/ photos/dberrange :| |: https://libvirt.org -o- https:// fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www. instagram.com/dberrange :|
-- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
-- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
On 04/28/2017 07:23 PM, Joel Wirāmu Pauling wrote:
Possibly related I notice race conditions caused by vnic never getting loaded if an existing bridge is already up (by OS init scripts etc) and stopping VM's from getting started.
Can you explain this more exactly? In particular, by "vnic" do you mean the network device as seen by the guest? And what do you mean by the term "getting loaded"? (It doesn't make sense to me that you should mean the network device in the guest, unless "stopping VM's from getting started" just means that the guest doesn't become fully functional, rather than that the qemu process doesn't start). Beyond that, of course the bridge that the tap device will be connected to needs to exist before you can connect something to it - I don't understand how its existence could cause a failure; rather its *non*-existence would cause a failure. (As you can see by my wild suppositions that make no sense, the terms you've used are a bit too vague/open to interpretation for me to understand exactly the problem you're referring to) (Hmm - perhaps you're referring to the situation where libvirt attempts to create a bridge on the host for one of its virtual networks, but either a bridge by that name has already been created by "someone else" or another netdev already exists on the host that is on the same subnet (and thus has the same route)? Your description doesn't fit that very well, but that is a known problem and unrelated to the patch/problem we're discussing here.
Often this is behavior you want ; i.e having Host Hypervisor NIC's added and up before libvirtd sets up it's nics/bridges.
On 29 April 2017 at 11:15, <lu.zhipeng@zte.com.cn <mailto:lu.zhipeng@zte.com.cn>> wrote:
＞On Fri, Apr 28, 2017 at 05:23:19PM +0800, ZhiPeng Lu wrote: ＞＞ Creating tap device and adding the device to bridge are not atomic operation. ＞＞ Similarly deleting tap device and removing it from bridge are not atomic operation. ＞＞The Problem occurs when two vms start and shutdown. When one vm with the nic ＞＞ named "vnet0" stopping, it deleted tap device but not removing port from bridge. ＞＞ At this time, another vm created the tap device named "vnet0" and added port to the ＞＞ same bridge. Then, the first vm deleted the tap device from the same bridge. ＞＞ Finally, the tap device of the second vm don't attached to the bridge. ＞＞ So, we can add domid to vm's nic name. For example, the vm's domid is 1 and vnet0 ＞＞ is renamed to vnet1.0.
＞Surely deleting the NIC automatically removes it from the bridge so we ＞can just remove the code that delets the bridge port.
i have done some tests for a tap + openvswitch bridge network. i find the nic named "vnet0" don't exsit before calling the virNetDevOpenvswitchRemovePort.
i think the tap is deleted not by removingport from bridge. i think hotpluging net has the same problem.
by the way:
my company's e-mail does not support thread posting and has some other problems. i believe that my colleagues can soon resloved them.
*为了让您的VPlat虚拟化故障得到高效的处理，请上报故障到: $VPlat技术支 持。*
芦志朋 luzhipeng
IT开发工程师 IT Development Engineer 操作系统产品部/中心研究院/系统产品 OS Product Dept./Central R＆D Institute/System Product
深圳市南山区科技南路55号中兴通讯研发大楼33楼 33/F, R&D Building, ZTE Corporation Hi-tech Road South, Hi-tech Industrial Park Nanshan District, Shenzhen, P.R.China, 518057 T: +86 755 xxxxxxxx F:+86 755 xxxxxxxx M: +86 xxxxxxxxxxx E: lu.zhipeng@zte.com.cn <mailto:lu.zhipeng@zte.com.cn> www.zte.com.cn <http://www.zte.com.cn/>
原始邮件 *发件人：*＜berrange@redhat.com <mailto:berrange@redhat.com>＞; *收件人：*芦志朋10108272; *抄送人：*＜libvir-list@redhat.com <mailto:libvir-list@redhat.com>＞; *日 期 ：*2017年04月28日 19:27 *主 题 ：**Re: [libvirt] [PATCH] qemu: change the name of tap device for a tapand bridge network*
On Fri, Apr 28, 2017 at 05:23:19PM +0800, ZhiPeng Lu wrote: ＞ Creating tap device and adding the device to bridge are not atomic operation. ＞ Similarly deleting tap device and removing it from bridge are not atomic operation. ＞ The Problem occurs when two vms start and shutdown. When one vm with the nic ＞ named "vnet0" stopping, it deleted tap device but not removing port from bridge. ＞ At this time, another vm created the tap device named "vnet0" and added port to the ＞ same bridge. Then, the first vm deleted the tap device from the same bridge. ＞ Finally, the tap device of the second vm don't attached to the bridge. ＞ So, we can add domid to vm's nic name. For example, the vm's domid is 1 and vnet0 ＞ is renamed to vnet1.0.
Surely deleting the NIC automatically removes it from the bridge so we can just remove the code that delets the bridge port.
Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange <https://www.flickr.com/photos/dberrange> :| |: https://libvirt.org -o- https://fstop138.berrange.com <https://fstop138.berrange.com> :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange <https://www.instagram.com/dberrange> :|
-- libvir-list mailing list libvir-list@redhat.com <mailto:libvir-list@redhat.com> https://www.redhat.com/mailman/listinfo/libvir-list <https://www.redhat.com/mailman/listinfo/libvir-list>
-- libvir-list mailing list libvir-list@redhat.com <mailto:libvir-list@redhat.com> https://www.redhat.com/mailman/listinfo/libvir-list <https://www.redhat.com/mailman/listinfo/libvir-list>
-- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
I'll try and explain the steps/situation where this is an issue. Say for example you have your router/dhcpd done inside a VM guest - you use macvtap devices and vlans off the hypervisor to get the traffic in and out (WAN and LAN). And you want the LAN side to provide connectivity to the host as well as to the network beyond the host. this is on a Ubuntu 16.04 host. 1) Create and isolated virtual Network via libvirt xml - say 'LAN' this is the second network (assume default exists) creates virbr1 and a vnic1 This is addition to the WAN and LAN macvtap devices. Unbeknown-st to libvirt the LAN macvtap is going to end up bridge to the 'isolated' network by the VM. 2) the virbr1 bridge needs numbering vnic1 is attached inside the VM to a dhcpd server but you still need the virbr1 interface to be numbered on the hypervisor (but you don't want the libvirt networking to take care of the numbering etc becuase that's what your fancy NFV VM is going to handle). Manually it's easy at this point you just run dhclient et.al on the bridge and you are good to go. BUT you want this to happen at boot; adding any sort of etc/network or /etc/sysconfig/networking scripts refering to that virbr1 will create a race condition which will stop the vm from getting started as you mention. On 29 April 2017 at 14:45, Laine Stump <laine@laine.org> wrote:
On 04/28/2017 07:23 PM, Joel Wirāmu Pauling wrote:
Possibly related I notice race conditions caused by vnic never getting loaded if an existing bridge is already up (by OS init scripts etc) and stopping VM's from getting started.
Can you explain this more exactly? In particular, by "vnic" do you mean the network device as seen by the guest? And what do you mean by the term "getting loaded"? (It doesn't make sense to me that you should mean the network device in the guest, unless "stopping VM's from getting started" just means that the guest doesn't become fully functional, rather than that the qemu process doesn't start). Beyond that, of course the bridge that the tap device will be connected to needs to exist before you can connect something to it - I don't understand how its existence could cause a failure; rather its *non*-existence would cause a failure. (As you can see by my wild suppositions that make no sense, the terms you've used are a bit too vague/open to interpretation for me to understand exactly the problem you're referring to)
(Hmm - perhaps you're referring to the situation where libvirt attempts to create a bridge on the host for one of its virtual networks, but either a bridge by that name has already been created by "someone else" or another netdev already exists on the host that is on the same subnet (and thus has the same route)? Your description doesn't fit that very well, but that is a known problem and unrelated to the patch/problem we're discussing here.
Often this is behavior you want ; i.e having Host Hypervisor NIC's added and up before libvirtd sets up it's nics/bridges.
On 29 April 2017 at 11:15, <lu.zhipeng@zte.com.cn <mailto:lu.zhipeng@zte.com.cn>> wrote:
＞On Fri, Apr 28, 2017 at 05:23:19PM +0800, ZhiPeng Lu wrote: ＞＞ Creating tap device and adding the device to bridge are not atomic
operation.
＞＞ Similarly deleting tap device and removing it from bridge are not
atomic operation.
＞＞The Problem occurs when two vms start and shutdown. When one vm
with the nic
＞＞ named "vnet0" stopping, it deleted tap device but not removing port
from bridge.
＞＞ At this time, another vm created the tap device named "vnet0" and
added port to the
＞＞ same bridge. Then, the first vm deleted the tap device from the same
bridge.
＞＞ Finally, the tap device of the second vm don't attached to the
bridge.
＞＞ So, we can add domid to vm's nic name. For example, the vm's domid
is 1 and vnet0
＞＞ is renamed to vnet1.0.
＞Surely deleting the NIC automatically removes it from the bridge so
we
＞can just remove the code that delets the bridge port.
i have done some tests for a tap + openvswitch bridge network. i find the nic named "vnet0" don't exsit before calling the virNetDevOpenvswitchRemovePort.
i think the tap is deleted not by removingport from bridge. i think hotpluging net has the same problem.
by the way:
my company's e-mail does not support thread posting and has some other problems. i believe that my colleagues can soon resloved them.
*为了让您的VPlat虚拟化故障得到高效的处理，请上报故障到: $VPlat技术支 持。*
芦志朋 luzhipeng
IT开发工程师 IT Development Engineer 操作系统产品部/中心研究院/系统产品 OS Product Dept./Central R＆D Institute/System Product
深圳市南山区科技南路55号中兴通讯研发大楼33楼 33/F, R&D Building, ZTE Corporation Hi-tech Road South, Hi-tech Industrial Park Nanshan District, Shenzhen, P.R.China, 518057 T: +86 755 xxxxxxxx F:+86 755 xxxxxxxx M: +86 xxxxxxxxxxx E: lu.zhipeng@zte.com.cn <mailto:lu.zhipeng@zte.com.cn> www.zte.com.cn <http://www.zte.com.cn/>
原始邮件 *发件人：*＜berrange@redhat.com <mailto:berrange@redhat.com>＞; *收件人：*芦志朋10108272; *抄送人：*＜libvir-list@redhat.com <mailto:libvir-list@redhat.com>＞; *日 期 ：*2017年04月28日 19:27 *主 题 ：**Re: [libvirt] [PATCH] qemu: change the name of tap device for a tapand bridge network*
On Fri, Apr 28, 2017 at 05:23:19PM +0800, ZhiPeng Lu wrote: ＞ Creating tap device and adding the device to bridge are not atomic
operation.
＞ Similarly deleting tap device and removing it from bridge are not
atomic operation.
＞ The Problem occurs when two vms start and shutdown. When one vm
with the nic
＞ named "vnet0" stopping, it deleted tap device but not removing
port from bridge.
＞ At this time, another vm created the tap device named "vnet0" and
added port to the
＞ same bridge. Then, the first vm deleted the tap device from the
same bridge.
＞ Finally, the tap device of the second vm don't attached to the
bridge.
＞ So, we can add domid to vm's nic name. For example, the vm's domid
is 1 and vnet0
＞ is renamed to vnet1.0.
Surely deleting the NIC automatically removes it from the bridge so
we
can just remove the code that delets the bridge port.
Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/
dberrange
<https://www.flickr.com/photos/dberrange> :| |: https://libvirt.org -o-
<https://fstop138.berrange.com> :| |: https://entangle-photo.org -o- https://www.instagram.com/
dberrange
<https://www.instagram.com/dberrange> :|
-- libvir-list mailing list libvir-list@redhat.com <mailto:libvir-list@redhat.com> https://www.redhat.com/mailman/listinfo/libvir-list <https://www.redhat.com/mailman/listinfo/libvir-list>
-- libvir-list mailing list libvir-list@redhat.com <mailto:libvir-list@redhat.com> https://www.redhat.com/mailman/listinfo/libvir-list <https://www.redhat.com/mailman/listinfo/libvir-list>
-- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
aSBleHBsYWluIHRoZSBjYXNlIGFnYWluLiAgdGhlIG5ldCBjb25maWcgeG1sIG9mIHR3byB2bXMg aXMgOg0KDQog77ycaW50ZXJmYWNlIHR5cGU9J2JyaWRnZSfvvJ4NCg0KICAgICAg77ycbWFjIGFk ZHJlc3M9J2ZhOjE2OjNlOmUxOmIyOjAxJy/vvJ4NCg0KICAgICAg77ycc291cmNlIGJyaWRnZT0n YnIxMDAnL++8ng0KDQogICAgICDvvJxtb2RlbCB0eXBlPSd2aXJ0aW8nL++8ng0KDQogICAgICDv vJxkcml2ZXIgbmFtZT0ndmhvc3QnL++8ng0KDQogICAgICDvvJx2aXJ0dWFscG9ydCB0eXBlPSdv cGVudnN3aXRjaCcv77yeDQoNCiAgICDvvJwvaW50ZXJmYWNl77yeDQoNCm9mIGNhdXNlIHRoZSBt YWMgb2YgdHdvIHZtcyBpcyBkaWZmZXJlbnQuIHRoZSBicmlkZ2UgbmFtZWQgYnIxMDAgaXMgY3Jl YXRlZCBpbiBhZHZhbmNlLg0KDQp3aGVuIHRoZSBmaXJzdCB2bSBzdGFydGVkICx3ZSBjYW4gZmlu ZCB0aGUgdGFvIGRldmljZSBuYW1lZCAidm5ldDAiIGJ5IHZpcnNoIGRvbWlmbGlzdCBkb21pZCBj b21tYW5kLg0KDQp3ZSBjYW4gc2h1dGRvdyB0aGUgZmlyc3Qgdm0gYW5kIHN0YXJ0IHRoZSBzZWNv bmQgdm0gYXQgb3IgYXJvdW5kIHRoZSBzYW1lIHRpbWUuIHdlICBjYW4gZmluZCAgdGhlIHRhcCBk ZXZpY2Ugb2YgIHRoZSBzZWNvbmQgdm0gIGlzIGFsc28gbmFtZWQgInZuZXQwIi4NCg0Kc3RhcnRp bmcgdXAgYW5kIHNodXRkb3duaW5nIHZtIGFyZSBub3QgbXV0dWFsbHkgZXhjbHVzaXZlLiBhZGRp bmcgcG9ydCBuYW1lZCAidm5ldDAiIHRvIGJyaWRnZSBhbmQgIGRlbGV0aW5nIHBvcnQgZnJvbSBi cmlkZ2UgYXJlIG5vdCBtdXR1YWxseSBleGNsdXNpdmUuDQoNCnNvIHRoZSBwcm9ibGVtIG9jY3Vy cmVkLiB0aGUgcG9ydCBvZiB0aGUgc2Vjb25kIHZtIGFkZGVkIHRvIHRoZSBicmlkZ2UgaXMgZGV0 ZWxlZCBieSBjYWxsaW5nIHZpck5ldERldk9wZW52c3dpdGNoUmVtb3ZlUG9ydCBpbiBzaHV0ZG93 bmluZyAgdGhlIGZpcnN0IHZtIC4NCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoN Cg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQrkuLrkuoborqnmgqjnmoRWUGxhdOiZmuaLn+WM luaVhemanOW+l+WIsOmrmOaViOeahOWkhOeQhu+8jOivt+S4iuaKpeaVhemanOWIsDogJFZQbGF0 5oqA5pyv5pSv5oyB44CCDQoNCg0K6Iqm5b+X5pyLIGx1emhpcGVuZw0KDQoNCg0KDQoNCg0KSVTl vIDlj5Hlt6XnqIvluIggSVQgRGV2ZWxvcG1lbnQKRW5naW5lZXINCuaTjeS9nOezu+e7n+S6p+WT gemDqC/kuK3lv4PnoJTnqbbpmaIv57O757uf5Lqn5ZOBIE9TIFByb2R1Y3QgRGVwdC4vQ2VudHJh bCBS77yGRCBJbnN0aXR1dGUvU3lzdGVtIFByb2R1Y3QNCg0KDQoNCg0KDQoNCg0KDQoNCua3seWc s+W4guWNl+WxseWMuuenkeaKgOWNl+i3rzU15Y+35Lit5YW06YCa6K6v56CU5Y+R5aSn5qW8MzPm pbwgDQozMy9GLCBSJkQgQnVpbGRpbmcsIFpURQpDb3Jwb3JhdGlvbiBIaS10ZWNoIFJvYWQgU291 dGgsIA0KSGktdGVjaApJbmR1c3RyaWFsIFBhcmsgTmFuc2hhbiBEaXN0cmljdCwgU2hlbnpoZW4s IFAuUi5DaGluYSwgNTE4MDU3IA0KVDogKzg2IDc1NSB4eHh4eHh4eCBGOis4NiA3NTUgeHh4eHh4 eHggDQpNOiArODYgeHh4eHh4eHh4eHggDQpFOiBsdS56aGlwZW5nQHp0ZS5jb20uY24gDQp3d3cu enRlLmNvbS5jbg0KDQoNCg0KDQoNCg0K5Y6f5aeL6YKu5Lu2DQoNCg0KDQrlj5Hku7bkurrvvJog 77ycam9lbEBhZW5lcnRpYS5uZXTvvJ4NCuaUtuS7tuS6uu+8miDvvJxsYWluZUBsYWluZS5vcmfv vJ4NCuaKhOmAgeS6uu+8miDvvJxsaWJ2aXItbGlzdEByZWRoYXQuY29t77ye6Iqm5b+X5pyLMTAx MDgyNzINCuaXpSDmnJ8g77yaMjAxN+W5tDA05pyIMjnml6UgMTE6MjENCuS4uyDpopgg77yaUmU6 IFtsaWJ2aXJ0XeetlOWkjTogUmU6IFtQQVRDSF0gcWVtdTogY2hhbmdlIHRoZSBuYW1lIG9mIHRh cCBkZXZpY2UgZm9yIGEgdGFwYW5kIGJyaWRnZSBuZXR3b3JrDQoNCg0KDQoNCg0KDQoNCkknbGwg dHJ5IGFuZCBleHBsYWluIHRoZSBzdGVwcy9zaXR1YXRpb24gd2hlcmUgdGhpcyBpcyBhbiBpc3N1 ZS4NCg0KDQoNClNheSBmb3IgZXhhbXBsZSB5b3UgaGF2ZSB5b3VyIHJvdXRlci9kaGNwZCBkb25l IGluc2lkZSBhIFZNIGd1ZXN0IC0geW91IHVzZSBtYWN2dGFwIGRldmljZXMgYW5kIHZsYW5zIG9m ZiB0aGUgaHlwZXJ2aXNvciB0byBnZXQgdGhlIHRyYWZmaWMgaW4gYW5kIG91dCAoV0FOIGFuZCBM QU4pLiBBbmQgeW91IHdhbnQgdGhlIExBTiBzaWRlIHRvIHByb3ZpZGUgY29ubmVjdGl2aXR5IHRv IHRoZSBob3N0IGFzIHdlbGwgYXMgdG8gdGhlIG5ldHdvcmsgYmV5b25kIHRoZSBob3N0Lg0KDQp0 aGlzIGlzIG9uIGEgVWJ1bnR1IDE2LjA0IGhvc3QuDQoNCg0KDQoxKSBDcmVhdGUgYW5kIGlzb2xh dGVkIHZpcnR1YWwgTmV0d29yayB2aWEgbGlidmlydCB4bWwgLSBzYXkgJ0xBTicgdGhpcyBpcyB0 aGUgc2Vjb25kIG5ldHdvcmsgKGFzc3VtZSBkZWZhdWx0IGV4aXN0cykgY3JlYXRlcyB2aXJicjEg YW5kIGEgdm5pYzEgVGhpcyBpcyBhZGRpdGlvbiB0byB0aGUgV0FOIGFuZCBMQU4gbWFjdnRhcCBk ZXZpY2VzLiBVbmJla25vd24tc3QgdG8gbGlidmlydCB0aGUgTEFOIG1hY3Z0YXAgaXMgZ29pbmcg dG8gZW5kIHVwIGJyaWRnZSB0byB0aGUgJ2lzb2xhdGVkJyBuZXR3b3JrIGJ5IHRoZSBWTS4NCg0K DQoyKSB0aGUgdmlyYnIxIGJyaWRnZSBuZWVkcyBudW1iZXJpbmcNCg0KDQoNCnZuaWMxIGlzIGF0 dGFjaGVkIGluc2lkZSB0aGUgVk0gdG8gYSBkaGNwZCBzZXJ2ZXIgYnV0IHlvdSBzdGlsbCBuZWVk IHRoZSB2aXJicjEgaW50ZXJmYWNlIHRvIGJlIG51bWJlcmVkIG9uIHRoZSBoeXBlcnZpc29yIChi dXQgeW91IGRvbid0IHdhbnQgdGhlIGxpYnZpcnQgbmV0d29ya2luZyB0byB0YWtlIGNhcmUgb2Yg dGhlIG51bWJlcmluZyBldGMgYmVjdWFzZSB0aGF0J3Mgd2hhdCB5b3VyIGZhbmN5IE5GViBWTSBp cyBnb2luZyB0byBoYW5kbGUpLiBNYW51YWxseSBpdCdzIGVhc3kgYXQgdGhpcyBwb2ludCB5b3Ug anVzdCBydW4gZGhjbGllbnQgZXQuYWwgb24gdGhlIGJyaWRnZSBhbmQgeW91IGFyZSBnb29kIHRv IGdvLg0KDQoNCg0KQlVUIHlvdSB3YW50IHRoaXMgdG8gaGFwcGVuIGF0IGJvb3QgYWRkaW5nIGFu eSBzb3J0IG9mIGV0Yy9uZXR3b3JrIG9yIC9ldGMvc3lzY29uZmlnL25ldHdvcmtpbmcgc2NyaXB0 cyByZWZlcmluZyB0byB0aGF0IHZpcmJyMSB3aWxsIGNyZWF0ZSBhIHJhY2UgY29uZGl0aW9uIHdo aWNoIHdpbGwgc3RvcCB0aGUgdm0gZnJvbSBnZXR0aW5nIHN0YXJ0ZWQgYXMgeW91IG1lbnRpb24u DQoNCg0KDQoNCg0KDQoNCg0KT24gMjkgQXByaWwgMjAxNyBhdCAxNDo0NSwgTGFpbmUgU3R1bXAg 77ycbGFpbmVAbGFpbmUub3Jn77yeIHdyb3RlOg0KT24gMDQvMjgvMjAxNyAwNzoyMyBQTSwgSm9l bCBXaXLEgW11IFBhdWxpbmcgd3JvdGU6DQog77yeIFBvc3NpYmx5IHJlbGF0ZWQgSSBub3RpY2Ug cmFjZSBjb25kaXRpb25zIGNhdXNlZCBieSB2bmljIG5ldmVyIGdldHRpbmcNCiDvvJ4gbG9hZGVk IGlmIGFuIGV4aXN0aW5nIGJyaWRnZSBpcyBhbHJlYWR5IHVwIChieSBPUyBpbml0IHNjcmlwdHMg ZXRjKSBhbmQNCiDvvJ4gc3RvcHBpbmcgVk0ncyBmcm9tIGdldHRpbmcgc3RhcnRlZC4NCiANCiBD YW4geW91IGV4cGxhaW4gdGhpcyBtb3JlIGV4YWN0bHk/IEluIHBhcnRpY3VsYXIsIGJ5ICJ2bmlj IiBkbyB5b3UgbWVhbg0KIHRoZSBuZXR3b3JrIGRldmljZSBhcyBzZWVuIGJ5IHRoZSBndWVzdD8g QW5kIHdoYXQgZG8geW91IG1lYW4gYnkgdGhlDQogdGVybSAiZ2V0dGluZyBsb2FkZWQiPyAoSXQg ZG9lc24ndCBtYWtlIHNlbnNlIHRvIG1lIHRoYXQgeW91IHNob3VsZCBtZWFuDQogdGhlIG5ldHdv cmsgZGV2aWNlIGluIHRoZSBndWVzdCwgdW5sZXNzICJzdG9wcGluZyBWTSdzIGZyb20gZ2V0dGlu Zw0KIHN0YXJ0ZWQiIGp1c3QgbWVhbnMgdGhhdCB0aGUgZ3Vlc3QgZG9lc24ndCBiZWNvbWUgZnVs bHkgZnVuY3Rpb25hbCwNCiByYXRoZXIgdGhhbiB0aGF0IHRoZSBxZW11IHByb2Nlc3MgZG9lc24n dCBzdGFydCkuIEJleW9uZCB0aGF0LCBvZiBjb3Vyc2UNCiB0aGUgYnJpZGdlIHRoYXQgdGhlIHRh cCBkZXZpY2Ugd2lsbCBiZSBjb25uZWN0ZWQgdG8gbmVlZHMgdG8gZXhpc3QNCiBiZWZvcmUgeW91 IGNhbiBjb25uZWN0IHNvbWV0aGluZyB0byBpdCAtIEkgZG9uJ3QgdW5kZXJzdGFuZCBob3cgaXRz DQogZXhpc3RlbmNlIGNvdWxkIGNhdXNlIGEgZmFpbHVyZSByYXRoZXIgaXRzICpub24qLWV4aXN0 ZW5jZSB3b3VsZCBjYXVzZQ0KIGEgZmFpbHVyZS4gKEFzIHlvdSBjYW4gc2VlIGJ5IG15IHdpbGQg c3VwcG9zaXRpb25zIHRoYXQgbWFrZSBubyBzZW5zZSwNCiB0aGUgdGVybXMgeW91J3ZlIHVzZWQg YXJlIGEgYml0IHRvbyB2YWd1ZS9vcGVuIHRvIGludGVycHJldGF0aW9uIGZvciBtZQ0KIHRvIHVu ZGVyc3RhbmQgZXhhY3RseSB0aGUgcHJvYmxlbSB5b3UncmUgcmVmZXJyaW5nIHRvKQ0KIA0KIChI bW0gLSBwZXJoYXBzIHlvdSdyZSByZWZlcnJpbmcgdG8gdGhlIHNpdHVhdGlvbiB3aGVyZSBsaWJ2 aXJ0IGF0dGVtcHRzDQogdG8gY3JlYXRlIGEgYnJpZGdlIG9uIHRoZSBob3N0IGZvciBvbmUgb2Yg aXRzIHZpcnR1YWwgbmV0d29ya3MsIGJ1dA0KIGVpdGhlciBhIGJyaWRnZSBieSB0aGF0IG5hbWUg aGFzIGFscmVhZHkgYmVlbiBjcmVhdGVkIGJ5ICJzb21lb25lIGVsc2UiDQogb3IgYW5vdGhlciBu ZXRkZXYgYWxyZWFkeSBleGlzdHMgb24gdGhlIGhvc3QgdGhhdCBpcyBvbiB0aGUgc2FtZSBzdWJu ZXQNCiAoYW5kIHRodXMgaGFzIHRoZSBzYW1lIHJvdXRlKT8gWW91ciBkZXNjcmlwdGlvbiBkb2Vz bid0IGZpdCB0aGF0IHZlcnkNCiB3ZWxsLCBidXQgdGhhdCBpcyBhIGtub3duIHByb2JsZW0gYW5k IHVucmVsYXRlZCB0byB0aGUgcGF0Y2gvcHJvYmxlbQ0KIHdlJ3JlIGRpc2N1c3NpbmcgaGVyZS4N CiANCiDvvJ4NCiDvvJ4gT2Z0ZW4gdGhpcyBpcyBiZWhhdmlvciB5b3Ugd2FudCAgaS5lIGhhdmlu ZyBIb3N0IEh5cGVydmlzb3IgTklDJ3MgYWRkZWQNCiDvvJ4gYW5kIHVwIGJlZm9yZSBsaWJ2aXJ0 ZCBzZXRzIHVwIGl0J3Mgbmljcy9icmlkZ2VzLg0KIA0KIA0KIA0KIO+8ng0KIO+8ng0KIO+8niBP biAyOSBBcHJpbCAyMDE3IGF0IDExOjE1LCDvvJxsdS56aGlwZW5nQHp0ZS5jb20uY24NCiDvvJ4g 77ycbWFpbHRvOmx1LnpoaXBlbmdAenRlLmNvbS5jbu+8nu+8niB3cm90ZToNCiDvvJ4NCiDvvJ4N CiDvvJ4gICAgIO+8nk9uIEZyaSwgQXByIDI4LCAyMDE3IGF0IDA1OjIzOjE5UE0gKzA4MDAsIFpo aVBlbmcgTHUgd3JvdGU6DQog77yeICAgICDvvJ7vvJ4NCiDvvJ4gICAgIENyZWF0aW5nIHRhcCBk ZXZpY2UgYW5kIGFkZGluZyB0aGUgZGV2aWNlIHRvIGJyaWRnZSBhcmUgbm90IGF0b21pYyBvcGVy YXRpb24uDQog77yeICAgICDvvJ7vvJ4NCiDvvJ4gICAgIFNpbWlsYXJseSBkZWxldGluZyB0YXAg ZGV2aWNlIGFuZCByZW1vdmluZyBpdCBmcm9tIGJyaWRnZSBhcmUgbm90IGF0b21pYyBvcGVyYXRp b24uDQog77yeICAgICDvvJ7vvJ5UaGUgUHJvYmxlbSBvY2N1cnMgd2hlbiB0d28gdm1zIHN0YXJ0 IGFuZCBzaHV0ZG93bi4gV2hlbiBvbmUgdm0gd2l0aCB0aGUgbmljDQog77yeICAgICDvvJ7vvJ4N CiDvvJ4gICAgIG5hbWVkICJ2bmV0MCIgc3RvcHBpbmcsIGl0IGRlbGV0ZWQgdGFwIGRldmljZSBi dXQgbm90IHJlbW92aW5nIHBvcnQgZnJvbSBicmlkZ2UuDQog77yeICAgICDvvJ7vvJ4NCiDvvJ4g ICAgIEF0IHRoaXMgdGltZSwgYW5vdGhlciB2bSBjcmVhdGVkIHRoZSB0YXAgZGV2aWNlIG5hbWVk ICJ2bmV0MCIgYW5kIGFkZGVkIHBvcnQgdG8gdGhlDQog77yeICAgICDvvJ7vvJ4NCiDvvJ4gICAg IHNhbWUgYnJpZGdlLiBUaGVuLCB0aGUgZmlyc3Qgdm0gZGVsZXRlZCB0aGUgdGFwIGRldmljZSBm cm9tIHRoZSBzYW1lIGJyaWRnZS4NCiDvvJ4gICAgIO+8nu+8ng0KIO+8niAgICAgRmluYWxseSwg dGhlIHRhcCBkZXZpY2Ugb2YgdGhlIHNlY29uZCB2bSBkb24ndCBhdHRhY2hlZCB0byB0aGUgYnJp ZGdlLg0KIO+8niAgICAg77ye77yeDQog77yeICAgICBTbywgd2UgY2FuIGFkZCBkb21pZCB0byB2 bSdzIG5pYyBuYW1lLiBGb3IgZXhhbXBsZSwgdGhlIHZtJ3MgZG9taWQgaXMgMSBhbmQgdm5ldDAN CiDvvJ4gICAgIO+8nu+8niBpcyByZW5hbWVkIHRvIHZuZXQxLjAuDQog77yeDQog77yeICAgICDv vJ5TdXJlbHkgZGVsZXRpbmcgdGhlIE5JQyBhdXRvbWF0aWNhbGx5IHJlbW92ZXMgaXQgZnJvbSB0 aGUgYnJpZGdlIHNvIHdlDQog77yeICAgICDvvJ5jYW4ganVzdCByZW1vdmUgdGhlIGNvZGUgdGhh dCBkZWxldHMgdGhlIGJyaWRnZSBwb3J0Lg0KIO+8ng0KIO+8niAgICAgaSBoYXZlIGRvbmUgc29t ZSB0ZXN0cyBmb3IgYSB0YXAgKyBvcGVudnN3aXRjaCBicmlkZ2UgbmV0d29yay4gIGkNCiDvvJ4g ICAgIGZpbmQgIHRoZSBuaWMgbmFtZWQgInZuZXQwIiBkb24ndCBleHNpdCBiZWZvcmUgY2FsbGlu Zw0KIO+8niAgICAgdGhlIHZpck5ldERldk9wZW52c3dpdGNoUmVtb3ZlUG9ydC4NCiDvvJ4NCiDv vJ4gICAgIGkgdGhpbmsgIHRoZSB0YXAgaXMgZGVsZXRlZCBub3QgYnkgcmVtb3Zpbmdwb3J0ICBm cm9tIGJyaWRnZS4gaQ0KIO+8niAgICAgdGhpbmsgaG90cGx1Z2luZyBuZXQgaGFzIHRoZSBzYW1l IHByb2JsZW0uDQog77yeDQog77yeDQog77yeICAgICBieSB0aGUgd2F5Og0KIO+8ng0KIO+8niAg ICAgICAgICAgbXkgY29tcGFueSdzIGUtbWFpbCAgZG9lcyBub3Qgc3VwcG9ydCB0aHJlYWQgcG9z dGluZyAgYW5kIGhhcw0KIO+8niAgICAgc29tZSBvdGhlciBwcm9ibGVtcy4gaSBiZWxpZXZlIHRo YXQgbXkgY29sbGVhZ3VlcyBjYW4gc29vbiByZXNsb3ZlZA0KIO+8niAgICAgdGhlbS4NCiDvvJ4N CiDvvJ4NCiDvvJ4NCiDvvJ4NCiDvvJ4NCiDvvJ4NCiDvvJ4gICAgICrkuLrkuoborqnmgqjnmoRW UGxhdOiZmuaLn+WMluaVhemanOW+l+WIsOmrmOaViOeahOWkhOeQhu+8jOivt+S4iuaKpeaVhema nOWIsDogJFZQbGF05oqA5pyv5pSvDQog77yeICAgICDmjIHjgIIqDQog77yeDQog77yeICAgICDo iqblv5fmnIsgbHV6aGlwZW5nDQog77yeDQog77yeDQog77yeICAgICBJVOW8gOWPkeW3peeoi+W4 iCBJVCBEZXZlbG9wbWVudCBFbmdpbmVlcg0KIO+8niAgICAg5pON5L2c57O757uf5Lqn5ZOB6YOo L+S4reW/g+eglOeptumZoi/ns7vnu5/kuqflk4EgT1MgUHJvZHVjdCBEZXB0Li9DZW50cmFsIFLv vIZEDQog77yeICAgICBJbnN0aXR1dGUvU3lzdGVtIFByb2R1Y3QNCiDvvJ4NCiDvvJ4NCiDvvJ4N CiDvvJ4gICAgIOa3seWcs+W4guWNl+WxseWMuuenkeaKgOWNl+i3rzU15Y+35Lit5YW06YCa6K6v 56CU5Y+R5aSn5qW8MzPmpbwNCiDvvJ4gICAgIDMzL0YsIFImRCBCdWlsZGluZywgWlRFIENvcnBv cmF0aW9uIEhpLXRlY2ggUm9hZCBTb3V0aCwNCiDvvJ4gICAgIEhpLXRlY2ggSW5kdXN0cmlhbCBQ YXJrIE5hbnNoYW4gRGlzdHJpY3QsIFNoZW56aGVuLCBQLlIuQ2hpbmEsIDUxODA1Nw0KIO+8niAg ICAgVDogKzg2IDc1NSB4eHh4eHh4eCBGOis4NiA3NTUgeHh4eHh4eHgNCiDvvJ4gICAgIE06ICs4 NiB4eHh4eHh4eHh4eA0KIO+8niAgICAgRTogbHUuemhpcGVuZ0B6dGUuY29tLmNuIO+8nG1haWx0 bzpsdS56aGlwZW5nQHp0ZS5jb20uY27vvJ4NCiDvvJ4gICAgIHd3dy56dGUuY29tLmNuIO+8nGh0 dHA6Ly93d3cuenRlLmNvbS5jbi/vvJ4NCiDvvJ4NCiDvvJ4gICAgIOWOn+Wni+mCruS7tg0KIO+8 niAgICAgKuWPkeS7tuS6uu+8mirvvJxiZXJyYW5nZUByZWRoYXQuY29tIO+8nG1haWx0bzpiZXJy YW5nZUByZWRoYXQuY29t77ye77yeDQog77yeICAgICAq5pS25Lu25Lq677yaKuiKpuW/l+acizEw MTA4MjcyDQog77yeICAgICAq5oqE6YCB5Lq677yaKu+8nGxpYnZpci1saXN0QHJlZGhhdC5jb20g 77ycbWFpbHRvOmxpYnZpci1saXN0QHJlZGhhdC5jb23vvJ7vvJ4NCiDvvJ4gICAgICrml6Ug5pyf IO+8mioyMDE35bm0MDTmnIgyOOaXpSAxOToyNw0KIO+8niAgICAgKuS4uyDpopgg77yaKipSZTog W2xpYnZpcnRdIFtQQVRDSF0gcWVtdTogY2hhbmdlIHRoZSBuYW1lIG9mIHRhcCBkZXZpY2UNCiDv vJ4gICAgIGZvciBhIHRhcGFuZCBicmlkZ2UgbmV0d29yayoNCiDvvJ4NCiDvvJ4NCiDvvJ4gICAg IE9uIEZyaSwgQXByIDI4LCAyMDE3IGF0IDA1OjIzOjE5UE0gKzA4MDAsIFpoaVBlbmcgTHUgd3Jv dGU6DQog77yeICAgICDvvJ4gQ3JlYXRpbmcgdGFwIGRldmljZSBhbmQgYWRkaW5nIHRoZSBkZXZp Y2UgdG8gYnJpZGdlIGFyZSBub3QgYXRvbWljIG9wZXJhdGlvbi4NCiDvvJ4gICAgIO+8niBTaW1p bGFybHkgZGVsZXRpbmcgdGFwIGRldmljZSBhbmQgcmVtb3ZpbmcgaXQgZnJvbSBicmlkZ2UgYXJl IG5vdCBhdG9taWMgb3BlcmF0aW9uLg0KIO+8niAgICAg77yeIFRoZSBQcm9ibGVtIG9jY3VycyB3 aGVuIHR3byB2bXMgc3RhcnQgYW5kIHNodXRkb3duLiBXaGVuIG9uZSB2bSB3aXRoIHRoZSBuaWMN CiDvvJ4gICAgIO+8niBuYW1lZCAidm5ldDAiIHN0b3BwaW5nLCBpdCBkZWxldGVkIHRhcCBkZXZp Y2UgYnV0IG5vdCByZW1vdmluZyBwb3J0IGZyb20gYnJpZGdlLg0KIO+8niAgICAg77yeIEF0IHRo aXMgdGltZSwgYW5vdGhlciB2bSBjcmVhdGVkIHRoZSB0YXAgZGV2aWNlIG5hbWVkICJ2bmV0MCIg YW5kIGFkZGVkIHBvcnQgdG8gdGhlDQog77yeICAgICDvvJ4gc2FtZSBicmlkZ2UuIFRoZW4sIHRo ZSBmaXJzdCB2bSBkZWxldGVkIHRoZSB0YXAgZGV2aWNlIGZyb20gdGhlIHNhbWUgYnJpZGdlLg0K IO+8niAgICAg77yeIEZpbmFsbHksIHRoZSB0YXAgZGV2aWNlIG9mIHRoZSBzZWNvbmQgdm0gZG9u J3QgYXR0YWNoZWQgdG8gdGhlIGJyaWRnZS4NCiDvvJ4gICAgIO+8niBTbywgd2UgY2FuIGFkZCBk b21pZCB0byB2bSdzIG5pYyBuYW1lLi4gRm9yIGV4YW1wbGUsIHRoZSB2bSdzIGRvbWlkIGlzIDEg YW5kIHZuZXQwDQog77yeICAgICDvvJ4gaXMgcmVuYW1lZCB0byB2bmV0MS4wLg0KIO+8ng0KIO+8 niAgICAgU3VyZWx5IGRlbGV0aW5nIHRoZSBOSUMgYXV0b21hdGljYWxseSByZW1vdmVzIGl0IGZy b20gdGhlIGJyaWRnZSBzbyB3ZQ0KIO+8niAgICAgY2FuIGp1c3QgcmVtb3ZlIHRoZSBjb2RlIHRo YXQgZGVsZXRzIHRoZSBicmlkZ2UgcG9ydC4NCiDvvJ4NCiDvvJ4NCiDvvJ4gICAgIFJlZ2FyZHMs DQog77yeICAgICBEYW5pZWwNCiDvvJ4gICAgIC0tDQog77yeICAgICB8OiBodHRwczovL2JlcnJh bmdlLmNvbSAgICAgIC1vLSAgICBodHRwczovL3d3dy5mbGlja3IuY29tL3Bob3Rvcy9kYmVycmFu Z2UNCiDvvJ4gICAgIO+8nGh0dHBzOi8vd3d3LmZsaWNrci5jb20vcGhvdG9zL2RiZXJyYW5nZe+8 niA6fA0KIO+8niAgICAgfDogaHR0cHM6Ly9saWJ2aXJ0Lm9yZyAgICAgICAgIC1vLSAgICAgICAg ICAgIGh0dHBzOi8vZnN0b3AxMzguYmVycmFuZ2UuY29tDQog77yeICAgICDvvJxodHRwczovL2Zz dG9wMTM4LmJlcnJhbmdlLmNvbe+8niA6fA0KIO+8niAgICAgfDogaHR0cHM6Ly9lbnRhbmdsZS1w aG90by5vcmcgICAgLW8tICAgIGh0dHBzOi8vd3d3Lmluc3RhZ3JhbS5jb20vZGJlcnJhbmdlDQog 77yeICAgICDvvJxodHRwczovL3d3dy5pbnN0YWdyYW0uY29tL2RiZXJyYW5nZe+8niA6fA0KIO+8 ng0KIO+8niAgICAgLS0NCiDvvJ4gICAgIGxpYnZpci1saXN0IG1haWxpbmcgbGlzdA0KIO+8niAg ICAgbGlidmlyLWxpc3RAcmVkaGF0LmNvbSDvvJxtYWlsdG86bGlidmlyLWxpc3RAcmVkaGF0LmNv be+8ng0KIO+8niAgICAgaHR0cHM6Ly93d3cucmVkaGF0LmNvbS9tYWlsbWFuL2xpc3RpbmZvL2xp YnZpci1saXN0DQog77yeICAgICDvvJxodHRwczovL3d3dy5yZWRoYXQuY29tL21haWxtYW4vbGlz dGluZm8vbGlidmlyLWxpc3TvvJ4NCiDvvJ4NCiDvvJ4NCiDvvJ4NCiDvvJ4NCiDvvJ4gICAgIC0t DQog77yeICAgICBsaWJ2aXItbGlzdCBtYWlsaW5nIGxpc3QNCiDvvJ4gICAgIGxpYnZpci1saXN0 QHJlZGhhdC5jb20g77ycbWFpbHRvOmxpYnZpci1saXN0QHJlZGhhdC5jb23vvJ4NCiDvvJ4gICAg IGh0dHBzOi8vd3d3LnJlZGhhdC5jb20vbWFpbG1hbi9saXN0aW5mby9saWJ2aXItbGlzdA0KIA0K DQrvvJ4gICAgIO+8nGh0dHBzOi8vd3d3LnJlZGhhdC5jb20vbWFpbG1hbi9saXN0aW5mby9saWJ2 aXItbGlzdO+8ng0KIO+8ng0KIO+8ng0KIO+8ng0KIO+8ng0KIO+8niAtLQ0KIO+8niBsaWJ2aXIt bGlzdCBtYWlsaW5nIGxpc3QNCiDvvJ4gbGlidmlyLWxpc3RAcmVkaGF0LmNvbQ0KIO+8niBodHRw czovL3d3dy5yZWRoYXQuY29tL21haWxtYW4vbGlzdGluZm8vbGlidmlyLWxpc3QNCiDvvJ4=
participants (5)
-
Daniel P. Berrange -
Joel Wirāmu Pauling -
Laine Stump -
lu.zhipeng＠zte.com.cn -
ZhiPeng Lu