4:23 a.m.
Creating tap device and adding the device to bridge are not atomic operation.
Similarly deleting tap device and removing it from bridge are not atomic operation.
The Problem occurs when two vms start and shutdown. When one vm with the nic
named "vnet0" stopping, it deleted tap device but not removing port from
bridge.
At this time, another vm created the tap device named "vnet0" and added port to
the
same bridge. Then, the first vm deleted the tap device from the same bridge.
Finally, the tap device of the second vm don't attached to the bridge.
So, we can add domid to vm's nic name. For example, the vm's domid is 1 and vnet0
is renamed to vnet1.0.
Signed-off-by: ZhiPeng Lu <lu.zhipeng(a)zte.com.cn>
---
src/qemu/qemu_interface.c | 9 ++++++++-
1 file changed, 8 insertions(+), 1 deletion(-)
diff --git a/src/qemu/qemu_interface.c b/src/qemu/qemu_interface.c
index d0850c0..17d40a7 100644
--- a/src/qemu/qemu_interface.c
+++ b/src/qemu/qemu_interface.c
@@ -512,6 +512,7 @@ qemuInterfaceBridgeConnect(virDomainDefPtr def,
bool template_ifname = false;
virQEMUDriverConfigPtr cfg = virQEMUDriverGetConfig(driver);
const char *tunpath = "/dev/net/tun";
+ char *domIdPlusIndex = NULL;
if (net->backend.tap) {
tunpath = net->backend.tap;
@@ -531,8 +532,13 @@ qemuInterfaceBridgeConnect(virDomainDefPtr def,
STRPREFIX(net->ifname, VIR_NET_GENERATED_PREFIX) ||
strchr(net->ifname, '%')) {
VIR_FREE(net->ifname);
- if (VIR_STRDUP(net->ifname, VIR_NET_GENERATED_PREFIX "%d") < 0)
+ if (virAsprintf(&domIdPlusIndex, "%s%d.%s",
+ VIR_NET_GENERATED_PREFIX, def->id, "%d") < 0) {
+ goto cleanup;
+ }
+ if (VIR_STRDUP(net->ifname, domIdPlusIndex) < 0) {
goto cleanup;
+ }
/* avoid exposing vnet%d in getXMLDesc or error outputs */
template_ifname = true;
}
@@ -594,6 +600,7 @@ qemuInterfaceBridgeConnect(virDomainDefPtr def,
ret = 0;
cleanup:
+ VIR_FREE(domIdPlusIndex);
if (ret < 0) {
size_t i;
for (i = 0; i < *tapfdSize && tapfd[i] >= 0; i++)
--
1.8.3.1
4:33 a.m.
On Fri, Apr 28, 2017 at 05:23:19PM +0800, ZhiPeng Lu wrote:
Creating tap device and adding the device to bridge are not atomic
operation.
Similarly deleting tap device and removing it from bridge are not atomic operation.
The Problem occurs when two vms start and shutdown. When one vm with the nic
named "vnet0" stopping, it deleted tap device but not removing port from
bridge.
At this time, another vm created the tap device named "vnet0" and added port to
the
same bridge. Then, the first vm deleted the tap device from the same bridge.
Finally, the tap device of the second vm don't attached to the bridge.
So, we can add domid to vm's nic name. For example, the vm's domid is 1 and
vnet0
is renamed to vnet1.0.
Surely deleting the NIC automatically removes it from the bridge so we
can just remove the code that delets the bridge port.
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
12:08 p.m.
On 04/28/2017 05:33 AM, Daniel P. Berrange wrote:
On Fri, Apr 28, 2017 at 05:23:19PM +0800, ZhiPeng Lu wrote:
> Creating tap device and adding the device to bridge are not atomic operation.
> Similarly deleting tap device and removing it from bridge are not atomic operation.
> The Problem occurs when two vms start and shutdown. When one vm with the nic
> named "vnet0" stopping, it deleted tap device but not removing port from
bridge.
> At this time, another vm created the tap device named "vnet0" and added
port to the
> same bridge. Then, the first vm deleted the tap device from the same bridge.
> Finally, the tap device of the second vm don't attached to the bridge.
> So, we can add domid to vm's nic name. For example, the vm's domid is 1 and
vnet0
> is renamed to vnet1.0.
Surely deleting the NIC automatically removes it from the bridge so we
can just remove the code that delets the bridge port.
That is true when using a Linux host bridge, but I recall that for
openvswitch (which I think is what ZhiPeng is using, based on an earlier
patch), you must explicitly remove the port from the bridge - apparently
the port is still there in openvswitch's table as some sort of "zombie"
connection even after the tap device itself no longer exists.
But instead of changing the naming scheme, maybe we should just delete
the bridge port *before* deleting the tap device instead of after. (Am I
recalling correctly that the tap device is deleted automatically when
the qemu process is killed? If so, then what's needed is to move the
loop in qemuProcessStop() that cleans up network interfaces so that it
happens before qemuProcessKill() is called.
3:05 a.m.
On Fri, Apr 28, 2017 at 01:08:45PM -0400, Laine Stump wrote:
On 04/28/2017 05:33 AM, Daniel P. Berrange wrote:
> On Fri, Apr 28, 2017 at 05:23:19PM +0800, ZhiPeng Lu wrote:
>> Creating tap device and adding the device to bridge are not atomic operation.
>> Similarly deleting tap device and removing it from bridge are not atomic
operation.
>> The Problem occurs when two vms start and shutdown. When one vm with the nic
>> named "vnet0" stopping, it deleted tap device but not removing port
from bridge.
>> At this time, another vm created the tap device named "vnet0" and
added port to the
>> same bridge. Then, the first vm deleted the tap device from the same bridge.
>> Finally, the tap device of the second vm don't attached to the bridge.
>> So, we can add domid to vm's nic name. For example, the vm's domid is 1
and vnet0
>> is renamed to vnet1.0.
>
> Surely deleting the NIC automatically removes it from the bridge so we
> can just remove the code that delets the bridge port.
That is true when using a Linux host bridge, but I recall that for
openvswitch (which I think is what ZhiPeng is using, based on an earlier
patch), you must explicitly remove the port from the bridge - apparently
the port is still there in openvswitch's table as some sort of "zombie"
connection even after the tap device itself no longer exists.
But instead of changing the naming scheme, maybe we should just delete
the bridge port *before* deleting the tap device instead of after. (Am I
recalling correctly that the tap device is deleted automatically when
the qemu process is killed? If so, then what's needed is to move the
loop in qemuProcessStop() that cleans up network interfaces so that it
happens before qemuProcessKill() is called.
Agreed, we should just reverse the order.
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
8:28 a.m.
New subject: [libvirt] 答复: Re: [PATCH] qemu: change the name of tap device for a tapand bridge network
SSdtIHdvcnJpZWQgdGhhdCB0aGUgbmV0d29yayBwYWNrZXQgbG9zcyByYXRlIHdpbGwgaW5jcmVh
c2UgYWZ0ZXIgIHJldmVyc2luZyB0aGUgb3JkZXIuDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoN
Cg0KDQoNCuS4uuS6huiuqeaCqOeahFZQbGF06Jma5ouf5YyW5pWF6Zqc5b6X5Yiw6auY5pWI55qE
5aSE55CG77yM6K+35LiK5oql5pWF6Zqc5YiwOiAkVlBsYXTmioDmnK/mlK/mjIHjgIINCg0KDQro
iqblv5fmnIsgbHV6aGlwZW5nDQoNCg0KDQoNCg0KDQpJVOW8gOWPkeW3peeoi+W4iCBJVCBEZXZl
bG9wbWVudApFbmdpbmVlcg0K5pON5L2c57O757uf5Lqn5ZOB6YOoL+S4reW/g+eglOeptumZoi/n
s7vnu5/kuqflk4EgT1MgUHJvZHVjdCBEZXB0Li9DZW50cmFsIFLvvIZEIEluc3RpdHV0ZS9TeXN0
ZW0gUHJvZHVjdA0KDQoNCg0KDQoNCg0KDQoNCg0K5rex5Zyz5biC5Y2X5bGx5Yy656eR5oqA5Y2X
6LevNTXlj7fkuK3lhbTpgJrorq/noJTlj5HlpKfmpbwzM+alvCANCjMzL0YsIFImRCBCdWlsZGlu
ZywgWlRFCkNvcnBvcmF0aW9uIEhpLXRlY2ggUm9hZCBTb3V0aCwgDQpIaS10ZWNoCkluZHVzdHJp
YWwgUGFyayBOYW5zaGFuIERpc3RyaWN0LCBTaGVuemhlbiwgUC5SLkNoaW5hLCA1MTgwNTcgDQpU
OiArODYgNzU1IHh4eHh4eHh4IEY6Kzg2IDc1NSB4eHh4eHh4eCANCk06ICs4NiB4eHh4eHh4eHh4
eCANCkU6IGx1LnpoaXBlbmdAenRlLmNvbS5jbiANCnd3dy56dGUuY29tLmNuDQoNCg0KDQoNCg0K
DQrljp/lp4vpgq7ku7YNCg0KDQoNCuWPkeS7tuS6uu+8miDvvJxiZXJyYW5nZUByZWRoYXQuY29t
77yeDQrmlLbku7bkurrvvJog77ycbGFpbmVAbGFpbmUub3Jn77yeDQrmioTpgIHkurrvvJog77yc
bGlidmlyLWxpc3RAcmVkaGF0LmNvbe+8nuiKpuW/l+acizEwMTA4MjcyDQrml6Ug5pyfIO+8mjIw
MTflubQwNeaciDAy5pelIDE2OjMwDQrkuLsg6aKYIO+8mlJlOiBbbGlidmlydF0gW1BBVENIXSBx
ZW11OiBjaGFuZ2UgdGhlIG5hbWUgb2YgdGFwIGRldmljZSBmb3IgYSB0YXBhbmQgYnJpZGdlIG5l
dHdvcmsNCg0KDQoNCg0KDQpPbiBGcmksIEFwciAyOCwgMjAxNyBhdCAwMTowODo0NVBNIC0wNDAw
LCBMYWluZSBTdHVtcCB3cm90ZToNCu+8niBPbiAwNC8yOC8yMDE3IDA1OjMzIEFNLCBEYW5pZWwg
UC4gQmVycmFuZ2Ugd3JvdGU6DQrvvJ4g77yeIE9uIEZyaSwgQXByIDI4LCAyMDE3IGF0IDA1OjIz
OjE5UE0gKzA4MDAsIFpoaVBlbmcgTHUgd3JvdGU6DQrvvJ4g77ye77yeIENyZWF0aW5nIHRhcCBk
ZXZpY2UgYW5kIGFkZGluZyB0aGUgZGV2aWNlIHRvIGJyaWRnZSBhcmUgbm90IGF0b21pYyBvcGVy
YXRpb24uDQrvvJ4g77ye77yeIFNpbWlsYXJseSBkZWxldGluZyB0YXAgZGV2aWNlIGFuZCByZW1v
dmluZyBpdCBmcm9tIGJyaWRnZSBhcmUgbm90IGF0b21pYyBvcGVyYXRpb24uDQrvvJ4g77ye77ye
IFRoZSBQcm9ibGVtIG9jY3VycyB3aGVuIHR3byB2bXMgc3RhcnQgYW5kIHNodXRkb3duLiBXaGVu
IG9uZSB2bSB3aXRoIHRoZSBuaWMNCu+8niDvvJ7vvJ4gbmFtZWQgInZuZXQwIiBzdG9wcGluZywg
aXQgZGVsZXRlZCB0YXAgZGV2aWNlIGJ1dCBub3QgcmVtb3ZpbmcgcG9ydCBmcm9tIGJyaWRnZS4N
Cu+8niDvvJ7vvJ4gQXQgdGhpcyB0aW1lLCBhbm90aGVyIHZtIGNyZWF0ZWQgdGhlIHRhcCBkZXZp
Y2UgbmFtZWQgInZuZXQwIiBhbmQgYWRkZWQgcG9ydCB0byB0aGUNCu+8niDvvJ7vvJ4gc2FtZSBi
cmlkZ2UuIFRoZW4sIHRoZSBmaXJzdCB2bSBkZWxldGVkIHRoZSB0YXAgZGV2aWNlIGZyb20gdGhl
IHNhbWUgYnJpZGdlLg0K77yeIO+8nu+8niBGaW5hbGx5LCB0aGUgdGFwIGRldmljZSBvZiB0aGUg
c2Vjb25kIHZtIGRvbid0IGF0dGFjaGVkIHRvIHRoZSBicmlkZ2UuDQrvvJ4g77ye77yeIFNvLCB3
ZSBjYW4gYWRkIGRvbWlkIHRvIHZtJ3MgbmljIG5hbWUuIEZvciBleGFtcGxlLCB0aGUgdm0ncyBk
b21pZCBpcyAxIGFuZCB2bmV0MA0K77yeIO+8nu+8niBpcyByZW5hbWVkIHRvIHZuZXQxLjAuDQrv
vJ4g77yeIA0K77yeIO+8niBTdXJlbHkgZGVsZXRpbmcgdGhlIE5JQyBhdXRvbWF0aWNhbGx5IHJl
bW92ZXMgaXQgZnJvbSB0aGUgYnJpZGdlIHNvIHdlDQrvvJ4g77yeIGNhbiBqdXN0IHJlbW92ZSB0
aGUgY29kZSB0aGF0IGRlbGV0cyB0aGUgYnJpZGdlIHBvcnQuDQrvvJ4gDQrvvJ4gVGhhdCBpcyB0
cnVlIHdoZW4gdXNpbmcgYSBMaW51eCBob3N0IGJyaWRnZSwgYnV0IEkgcmVjYWxsIHRoYXQgZm9y
DQrvvJ4gb3BlbnZzd2l0Y2ggKHdoaWNoIEkgdGhpbmsgaXMgd2hhdCBaaGlQZW5nIGlzIHVzaW5n
LCBiYXNlZCBvbiBhbiBlYXJsaWVyDQrvvJ4gcGF0Y2gpLCB5b3UgbXVzdCBleHBsaWNpdGx5IHJl
bW92ZSB0aGUgcG9ydCBmcm9tIHRoZSBicmlkZ2UgLSBhcHBhcmVudGx5DQrvvJ4gdGhlIHBvcnQg
aXMgc3RpbGwgdGhlcmUgaW4gb3BlbnZzd2l0Y2gncyB0YWJsZSBhcyBzb21lIHNvcnQgb2YgInpv
bWJpZSINCu+8niBjb25uZWN0aW9uIGV2ZW4gYWZ0ZXIgdGhlIHRhcCBkZXZpY2UgaXRzZWxmIG5v
IGxvbmdlciBleGlzdHMuDQrvvJ4gDQrvvJ4gDQrvvJ4gQnV0IGluc3RlYWQgb2YgY2hhbmdpbmcg
dGhlIG5hbWluZyBzY2hlbWUsIG1heWJlIHdlIHNob3VsZCBqdXN0IGRlbGV0ZQ0K77yeIHRoZSBi
cmlkZ2UgcG9ydCAqYmVmb3JlKiBkZWxldGluZyB0aGUgdGFwIGRldmljZSBpbnN0ZWFkIG9mIGFm
dGVyLiAoQW0gSQ0K77yeIHJlY2FsbGluZyBjb3JyZWN0bHkgdGhhdCB0aGUgdGFwIGRldmljZSBp
cyBkZWxldGVkIGF1dG9tYXRpY2FsbHkgd2hlbg0K77yeIHRoZSBxZW11IHByb2Nlc3MgaXMga2ls
bGVkPyBJZiBzbywgdGhlbiB3aGF0J3MgbmVlZGVkIGlzIHRvIG1vdmUgdGhlDQrvvJ4gbG9vcCBp
biBxZW11UHJvY2Vzc1N0b3AoKSB0aGF0IGNsZWFucyB1cCBuZXR3b3JrIGludGVyZmFjZXMgc28g
dGhhdCBpdA0K77yeIGhhcHBlbnMgYmVmb3JlIHFlbXVQcm9jZXNzS2lsbCgpIGlzIGNhbGxlZC4N
Cg0KQWdyZWVkLCB3ZSBzaG91bGQganVzdCByZXZlcnNlIHRoZSBvcmRlci4NCg0KUmVnYXJkcywN
CkRhbmllbA0KLS0gDQp8OiBodHRwczovL2JlcnJhbmdlLmNvbSAgICAgIC1vLSAgICBodHRwczov
L3d3dy5mbGlja3IuY29tL3Bob3Rvcy9kYmVycmFuZ2UgOnwNCnw6IGh0dHBzOi8vbGlidmlydC5v
cmcgICAgICAgICAtby0gICAgICAgICAgICBodHRwczovL2ZzdG9wMTM4LmJlcnJhbmdlLmNvbSA6
fA0KfDogaHR0cHM6Ly9lbnRhbmdsZS1waG90by5vcmcgICAgLW8tICAgIGh0dHBzOi8vd3d3Lmlu
c3RhZ3JhbS5jb20vZGJlcnJhbmdlIDp8DQoNCi0tDQpsaWJ2aXItbGlzdCBtYWlsaW5nIGxpc3QN
CmxpYnZpci1saXN0QHJlZGhhdC5jb20NCmh0dHBzOi8vd3d3LnJlZGhhdC5jb20vbWFpbG1hbi9s
aXN0aW5mby9saWJ2aXItbGlzdA==
4:06 a.m.
New subject: [libvirt] 答复: Re: [PATCH] qemu: change the name of tap device for a tapand bridge network
On Thu, May 04, 2017 at 08:28:03AM +0800, lu.zhipeng(a)zte.com.cn wrote:
I'm worried that the network packet loss rate will increase
after
reversing the order.
I don't see why that is relevant - we're killing the guest or unplugging
the NIC, so there's no traffic being sent on this tap device.
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
8:44 a.m.
New subject: [libvirt] 答复: Re: [PATCH] qemu: change the name of tap device for a tapand bridge network
T24gTW9uLCBNYXkgMDgsIDIwMTcgYXQgMDM6MDM6MzBQTSArMDgwMCwgWmhpUGVuZyBMdSB3cm90
ZTrvvJ7vvJ4gSW4gcWVtdVByb2Nlc3NTdG9wIHdlIGV4cGxpY2l0bHkgcmVtb3ZlIHRoZSBwb3J0
IGZyb20gdGhlIG9wZW52c3dpdGNoIGJyaWRnZSBhZnRlcu+8nu+8niBxZW11UHJvY2Vzc0tpbGwg
aXMgY2FsbGVkLiBCdXQgdGhlcmUgaXMgYSBjZXJ0YWluIGludGVydmFsIG9mIHRpbWUgYmV0d2Vl
bu+8nu+8niBkZWxldGluZyB0YXAgZGV2aWNlIGFuZCByZW1vdmluZyBpdCBmcm9tIGJyaWRnZS4g
VGhlIHByb2JsZW0gb2NjdXJzIHdoZW4gdHdvIHZtc++8nu+8niBzdGFydCBhbmQgc2h1dGRvd24g
d2l0aCB0aGUgc2FtZSBuYW1lJ3MgbmV0d29yayBpbnRlcmZhY2UgYXR0YWNoZWQgdG8gdGhlIHNh
bWXvvJ7vvJ4gb3BlbnZzd2l0Y2ggYnJpZGdlLiBXaGVuIG9uZSB2bSB3aXRoIHRoZSBuaWMgbmFt
ZWQgInZuZXQwIiBzdG9wcGluZywgaXQgZGVsZXRlZO+8nu+8nnRhcCBkZXZpY2Ugd2l0aG91dCB0
aW1lbHkgcmVtb3ZpbmcgdGhlIHBvcnQgZnJvbSBicmlkZ2Uu77ye77yeLkF0IHRoaXMgdGltZSwg
YW5vdGhlciB2bSBjcmVhdGVkIHRoZSB0YXAgZGV2aWNlIG5hbWVkICJ2bmV0MCIgYW5kIGFkZGVk
IHBvcnQgdG8gdGhl77ye77yeIHNhbWUgYnJpZGdlLiBUaGVuLCB0aGUgZmlyc3Qgdm0gcmVtb3Zl
ZCB0aGUgcG9ydCBmcm9tIHRoZSBzYW1lIGJyaWRnZS7vvJ7vvJ4gRmluYWxseSwgdGhlIHRhcCBk
ZXZpY2Ugb2YgdGhlIHNlY29uZCB2bSBkaWQgbm90IGF0dGFjaGVkIHRvIHRoZSBicmlkZ2Uu77ye
77yeIFdlIG5lZWQgdG8gZGVsZXRlIHRoZSBicmlkZ2UgcG9ydCBiZWZvcmUgZGVsZXRpbmcgdGhl
IHRhcCBkZXZpY2UgaW5zdGVhZCBvZiBhZnRlci7vvJ7vvJ4gU28gd2hhdCdzIG5lZWRlZCBpcyB0
byBtb3ZlIHRoZSBsb29wIGluIHFlbXVQcm9jZXNzU3RvcCB0aGF0IGNsZWFucyB1cO+8nu+8niBu
ZXR3b3JrIGludGVyZmFjZXMgc28gdGhhdCBpdCBoYXBwZW5zIGJlZm9yZSBxZW11UHJvY2Vzc0tp
bGwgaXMgY2FsbGVkLu+8nlRoaXMgZml4IHdvbid0IHdvcmsgY29ycmVjdGx5IGVpdGhlci4gWW91
IGNhbm5vdCBhc3N1bWUgdGhhdCBsaWJ2aXJ0IGhhc++8nmNvbnRyb2wgb3ZlciB3aGVuIHRoZSBR
RU1VIHByb2Nlc3MgZXhpdHMuIEl0IG1heSBleGl0IGl0c2VsZiAqYmVmb3JlKu+8nmxpYnZpcnQg
cnVucyBhbnkgb2YgaXRzIGNsZWFudXAgY29kZS4NCg0KDQoNCg0KT24gTW9uLCBNYXkgMDgsIDIw
MTcgYXQgMDY6MTg6MjVQTSArMDgwMCwgbHUuemhpcGVuZ0B6dGUuY29tLmNuIHdyb3RlOu+8niBJ
IG1heSBub3QgaGF2ZSBkZXNjcmliZWQgaXQgY2xlYXJseS4gIO+8niDvvJ4gaSBuZWVkIHRvIGVu
c3VyZSAgdGhlIG9yZGVyIG9mICBhZGRpbmcgcG9ydCB0byBicmlkZ2UgYW5kIGRlbGV0aW5nIGZy
b20gYnJpZGdlLu+8niDvvJ4gaSAgcmVuYW1lIHRhcCBkZXZpY2UgdG8gYXZvaWQgdGhlIHByb2Js
ZW0gIGluIG15IGZpcnN0IHBhdGNoLmkgdGhpbmsgaXQgY2FuIHNvbHZlIHRoZSBwcm9ibGVtLu+8
niDvvJ4gbXkgc2Vjb25kIHBhdGNoIG1heSBjYW4ndCByZXNvbHZlIHRoZSBwcm9ibGVtIC7vvJ4g
77yeIERvIHlvdSAgaGF2ZSBhbnkgYmV0dGVyIGlkZWFzPw0KDQoNCg0KDQoNCg0KDQoNCg0KDQoN
Cg0KDQoNCg0KDQoNCg0KDQrkuLrkuoborqnmgqjnmoRWUGxhdOiZmuaLn+WMluaVhemanOW+l+WI
sOmrmOaViOeahOWkhOeQhu+8jOivt+S4iuaKpeaVhemanOWIsDogJFZQbGF05oqA5pyv5pSv5oyB
44CCDQoNCg0K6Iqm5b+X5pyLIGx1emhpcGVuZw0KDQoNCg0KDQoNCg0KSVTlvIDlj5Hlt6XnqIvl
uIggSVQgRGV2ZWxvcG1lbnQKRW5naW5lZXINCuaTjeS9nOezu+e7n+S6p+WTgemDqC/kuK3lv4Pn
oJTnqbbpmaIv57O757uf5Lqn5ZOBIE9TIFByb2R1Y3QgRGVwdC4vQ2VudHJhbCBS77yGRCBJbnN0
aXR1dGUvU3lzdGVtIFByb2R1Y3QNCg0KDQoNCg0KDQoNCg0KDQoNCua3seWcs+W4guWNl+WxseWM
uuenkeaKgOWNl+i3rzU15Y+35Lit5YW06YCa6K6v56CU5Y+R5aSn5qW8MzPmpbwgDQozMy9GLCBS
JkQgQnVpbGRpbmcsIFpURQpDb3Jwb3JhdGlvbiBIaS10ZWNoIFJvYWQgU291dGgsIA0KSGktdGVj
aApJbmR1c3RyaWFsIFBhcmsgTmFuc2hhbiBEaXN0cmljdCwgU2hlbnpoZW4sIFAuUi5DaGluYSwg
NTE4MDU3IA0KVDogKzg2IDc1NSB4eHh4eHh4eCBGOis4NiA3NTUgeHh4eHh4eHggDQpNOiArODYg
eHh4eHh4eHh4eHggDQpFOiBsdS56aGlwZW5nQHp0ZS5jb20uY24gDQp3d3cuenRlLmNvbS5jbg0K
DQoNCg0KDQoNCg0K5Y6f5aeL6YKu5Lu2DQoNCg0KDQrlj5Hku7bkurrvvJog77ycYmVycmFuZ2VA
cmVkaGF0LmNvbe+8ng0K5pS25Lu25Lq677yaIO+8nGxhaW5lQGxhaW5lLm9yZ++8ng0K5oqE6YCB
5Lq677yaIO+8nGxpYnZpci1saXN0QHJlZGhhdC5jb23vvJ7oiqblv5fmnIsxMDEwODI3Mg0K5pel
IOacnyDvvJoyMDE35bm0MDXmnIgwMuaXpSAxNjo0MQ0K5Li7IOmimCDvvJpSZTogW2xpYnZpcnRd
IFtQQVRDSF0gcWVtdTogY2hhbmdlIHRoZSBuYW1lIG9mIHRhcCBkZXZpY2UgZm9yIGEgdGFwYW5k
IGJyaWRnZSBuZXR3b3JrDQoNCg0KDQoNCg0KT24gRnJpLCBBcHIgMjgsIDIwMTcgYXQgMDE6MDg6
NDVQTSAtMDQwMCwgTGFpbmUgU3R1bXAgd3JvdGU6DQrvvJ4gT24gMDQvMjgvMjAxNyAwNTozMyBB
TSwgRGFuaWVsIFAuIEJlcnJhbmdlIHdyb3RlOg0K77yeIO+8niBPbiBGcmksIEFwciAyOCwgMjAx
NyBhdCAwNToyMzoxOVBNICswODAwLCBaaGlQZW5nIEx1IHdyb3RlOg0K77yeIO+8nu+8niBDcmVh
dGluZyB0YXAgZGV2aWNlIGFuZCBhZGRpbmcgdGhlIGRldmljZSB0byBicmlkZ2UgYXJlIG5vdCBh
dG9taWMgb3BlcmF0aW9uLg0K77yeIO+8nu+8niBTaW1pbGFybHkgZGVsZXRpbmcgdGFwIGRldmlj
ZSBhbmQgcmVtb3ZpbmcgaXQgZnJvbSBicmlkZ2UgYXJlIG5vdCBhdG9taWMgb3BlcmF0aW9uLg0K
77yeIO+8nu+8niBUaGUgUHJvYmxlbSBvY2N1cnMgd2hlbiB0d28gdm1zIHN0YXJ0IGFuZCBzaHV0
ZG93bi4gV2hlbiBvbmUgdm0gd2l0aCB0aGUgbmljDQrvvJ4g77ye77yeIG5hbWVkICJ2bmV0MCIg
c3RvcHBpbmcsIGl0IGRlbGV0ZWQgdGFwIGRldmljZSBidXQgbm90IHJlbW92aW5nIHBvcnQgZnJv
bSBicmlkZ2UuDQrvvJ4g77ye77yeIEF0IHRoaXMgdGltZSwgYW5vdGhlciB2bSBjcmVhdGVkIHRo
ZSB0YXAgZGV2aWNlIG5hbWVkICJ2bmV0MCIgYW5kIGFkZGVkIHBvcnQgdG8gdGhlDQrvvJ4g77ye
77yeIHNhbWUgYnJpZGdlLiBUaGVuLCB0aGUgZmlyc3Qgdm0gZGVsZXRlZCB0aGUgdGFwIGRldmlj
ZSBmcm9tIHRoZSBzYW1lIGJyaWRnZS4NCu+8niDvvJ7vvJ4gRmluYWxseSwgdGhlIHRhcCBkZXZp
Y2Ugb2YgdGhlIHNlY29uZCB2bSBkb24ndCBhdHRhY2hlZCB0byB0aGUgYnJpZGdlLg0K77yeIO+8
nu+8niBTbywgd2UgY2FuIGFkZCBkb21pZCB0byB2bSdzIG5pYyBuYW1lLiBGb3IgZXhhbXBsZSwg
dGhlIHZtJ3MgZG9taWQgaXMgMSBhbmQgdm5ldDANCu+8niDvvJ7vvJ4gaXMgcmVuYW1lZCB0byB2
bmV0MS4wLg0K77yeIO+8niANCu+8niDvvJ4gU3VyZWx5IGRlbGV0aW5nIHRoZSBOSUMgYXV0b21h
dGljYWxseSByZW1vdmVzIGl0IGZyb20gdGhlIGJyaWRnZSBzbyB3ZQ0K77yeIO+8niBjYW4ganVz
dCByZW1vdmUgdGhlIGNvZGUgdGhhdCBkZWxldHMgdGhlIGJyaWRnZSBwb3J0Lg0K77yeIA0K77ye
IFRoYXQgaXMgdHJ1ZSB3aGVuIHVzaW5nIGEgTGludXggaG9zdCBicmlkZ2UsIGJ1dCBJIHJlY2Fs
bCB0aGF0IGZvcg0K77yeIG9wZW52c3dpdGNoICh3aGljaCBJIHRoaW5rIGlzIHdoYXQgWmhpUGVu
ZyBpcyB1c2luZywgYmFzZWQgb24gYW4gZWFybGllcg0K77yeIHBhdGNoKSwgeW91IG11c3QgZXhw
bGljaXRseSByZW1vdmUgdGhlIHBvcnQgZnJvbSB0aGUgYnJpZGdlIC0gYXBwYXJlbnRseQ0K77ye
IHRoZSBwb3J0IGlzIHN0aWxsIHRoZXJlIGluIG9wZW52c3dpdGNoJ3MgdGFibGUgYXMgc29tZSBz
b3J0IG9mICJ6b21iaWUiDQrvvJ4gY29ubmVjdGlvbiBldmVuIGFmdGVyIHRoZSB0YXAgZGV2aWNl
IGl0c2VsZiBubyBsb25nZXIgZXhpc3RzLg0K77yeIA0K77yeIA0K77yeIEJ1dCBpbnN0ZWFkIG9m
IGNoYW5naW5nIHRoZSBuYW1pbmcgc2NoZW1lLCBtYXliZSB3ZSBzaG91bGQganVzdCBkZWxldGUN
Cu+8niB0aGUgYnJpZGdlIHBvcnQgKmJlZm9yZSogZGVsZXRpbmcgdGhlIHRhcCBkZXZpY2UgaW5z
dGVhZCBvZiBhZnRlci4gKEFtIEkNCu+8niByZWNhbGxpbmcgY29ycmVjdGx5IHRoYXQgdGhlIHRh
cCBkZXZpY2UgaXMgZGVsZXRlZCBhdXRvbWF0aWNhbGx5IHdoZW4NCu+8niB0aGUgcWVtdSBwcm9j
ZXNzIGlzIGtpbGxlZD8gSWYgc28sIHRoZW4gd2hhdCdzIG5lZWRlZCBpcyB0byBtb3ZlIHRoZQ0K
77yeIGxvb3AgaW4gcWVtdVByb2Nlc3NTdG9wKCkgdGhhdCBjbGVhbnMgdXAgbmV0d29yayBpbnRl
cmZhY2VzIHNvIHRoYXQgaXQNCu+8niBoYXBwZW5zIGJlZm9yZSBxZW11UHJvY2Vzc0tpbGwoKSBp
cyBjYWxsZWQuDQoNCkFncmVlZCwgd2Ugc2hvdWxkIGp1c3QgcmV2ZXJzZSB0aGUgb3JkZXIuDQoN
ClJlZ2FyZHMsDQpEYW5pZWwNCi0tIA0KfDogaHR0cHM6Ly9iZXJyYW5nZS5jb20gICAgICAtby0g
ICAgaHR0cHM6Ly93d3cuZmxpY2tyLmNvbS9waG90b3MvZGJlcnJhbmdlIDp8DQp8OiBodHRwczov
L2xpYnZpcnQub3JnICAgICAgICAgLW8tICAgICAgICAgICAgaHR0cHM6Ly9mc3RvcDEzOC5iZXJy
YW5nZS5jb20gOnwNCnw6IGh0dHBzOi8vZW50YW5nbGUtcGhvdG8ub3JnICAgIC1vLSAgICBodHRw
czovL3d3dy5pbnN0YWdyYW0uY29tL2RiZXJyYW5nZSA6fA==
3:24 a.m.
New subject: [libvirt] 答复: Re: [PATCH] qemu: change the name of tap device for a tapand bridge network
FYI your replies to messages on the list are almost impossible
to read & understand.
Your email client is sending HTML and badly mangled plain text.
Please make it send plain text only to public mailing lists,
with line wrapping and sensibly trim & quote text you are
replying to.
I can't see any comment you made in this latest message so I can't
reply to you.
On Tue, May 09, 2017 at 08:44:55AM +0800, lu.zhipeng(a)zte.com.cn wrote:
On Mon, May 08, 2017 at 03:03:30PM +0800, ZhiPeng Lu wrote:＞＞ In
qemuProcessStop we explicitly remove the port from the openvswitch bridge after＞＞
qemuProcessKill is called. But there is a certain interval of time between＞＞ deleting tap
device and removing it from bridge. The problem occurs when two vms＞＞ start and shutdown
with the same name's network interface attached to the same＞＞ openvswitch bridge. When
one vm with the nic named "vnet0" stopping, it deleted＞＞tap device without
timely removing the port from bridge.＞＞.At this time, another vm created the tap device
named "vnet0" and added port to the＞＞ same bridge. Then, the first vm removed
the port from the same bridge.＞＞ Finally, the tap device of the second vm did not attached
to the bridge.＞＞ We need to delete the bridge port before deleting the tap device instead
of after.＞＞ So what's needed is to move the loop in qemuProcessStop that cleans up＞＞
network interfaces so that it happens before qemuProcessKill is called.＞This fix won't
work correctly either. You cannot assume that libvirt has＞control over when the QEMU
process exits. It may exit itself *before*＞libvirt runs any of its cleanup code.
On Mon, May 08, 2017 at 06:18:25PM +0800, lu.zhipeng(a)zte.com.cn wrote:＞ I may not have
described it clearly. ＞ ＞ i need to ensure the order of adding port to bridge and
deleting from bridge.＞ ＞ i rename tap device to avoid the problem in my first patch.i
think it can solve the problem.＞ ＞ my second patch may can't resolve the problem .＞ ＞
Do you have any better ideas?
g>
为了让您的VPlat虚拟化故障得到高效的处理，请上报故障到: $VPlat技术支持。
芦志朋 luzhipeng
IT开发工程师 IT Development
Engineer
操作系统产品部/中心研究院/系统产品 OS Product Dept./Central R＆D Institute/System Product
深圳市南山区科技南路55号中兴通讯研发大楼33楼
33/F, R&D Building, ZTE
Corporation Hi-tech Road South,
Hi-tech
Industrial Park Nanshan District, Shenzhen, P.R.China, 518057
T: +86 755 xxxxxxxx F:+86 755 xxxxxxxx
M: +86 xxxxxxxxxxx
E: lu.zhipeng(a)zte.com.cn
www.zte.com.cn
原始邮件
发件人： ＜berrange(a)redhat.com>
收件人： ＜laine(a)laine.org>
抄送人： ＜libvir-list(a)redhat.xn--com>10108272-u92xv15exu8g
日 期 ：2017年05月02日 16:41
主 题 ：Re: [libvirt] [PATCH] qemu: change the name of tap device for a tapand bridge
network
On Fri, Apr 28, 2017 at 01:08:45PM -0400, Laine Stump wrote:
＞ On 04/28/2017 05:33 AM, Daniel P. Berrange wrote:
＞ ＞ On Fri, Apr 28, 2017 at 05:23:19PM +0800, ZhiPeng Lu wrote:
＞ ＞＞ Creating tap device and adding the device to bridge are not atomic operation.
＞ ＞＞ Similarly deleting tap device and removing it from bridge are not atomic operation.
＞ ＞＞ The Problem occurs when two vms start and shutdown. When one vm with the nic
＞ ＞＞ named "vnet0" stopping, it deleted tap device but not removing port from
bridge.
＞ ＞＞ At this time, another vm created the tap device named "vnet0" and added
port to the
＞ ＞＞ same bridge. Then, the first vm deleted the tap device from the same bridge.
＞ ＞＞ Finally, the tap device of the second vm don't attached to the bridge.
＞ ＞＞ So, we can add domid to vm's nic name. For example, the vm's domid is 1 and
vnet0
＞ ＞＞ is renamed to vnet1.0.
＞ ＞
＞ ＞ Surely deleting the NIC automatically removes it from the bridge so we
＞ ＞ can just remove the code that delets the bridge port.
＞
＞ That is true when using a Linux host bridge, but I recall that for
＞ openvswitch (which I think is what ZhiPeng is using, based on an earlier
＞ patch), you must explicitly remove the port from the bridge - apparently
＞ the port is still there in openvswitch's table as some sort of "zombie"
＞ connection even after the tap device itself no longer exists.
＞
＞
＞ But instead of changing the naming scheme, maybe we should just delete
＞ the bridge port *before* deleting the tap device instead of after. (Am I
＞ recalling correctly that the tap device is deleted automatically when
＞ the qemu process is killed? If so, then what's needed is to move the
＞ loop in qemuProcessStop() that cleans up network interfaces so that it
＞ happens before qemuProcessKill() is called.
Agreed, we should just reverse the order.
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
7:15 a.m.
New subject: [libvirt] 答复: Re: [PATCH] qemu: change the name of tap device for a tapand bridge network
77yeT24gRnJpLCBBcHIgMjgsIDIwMTcgYXQgMDU6MjM6MTlQTSArMDgwMCwgWmhpUGVuZyBMdSB3
cm90ZTrvvJ7vvJ4gQ3JlYXRpbmcgdGFwIGRldmljZSBhbmQgYWRkaW5nIHRoZSBkZXZpY2UgdG8g
YnJpZGdlIGFyZSBub3QgYXRvbWljIG9wZXJhdGlvbi7vvJ7vvJ4gU2ltaWxhcmx5IGRlbGV0aW5n
IHRhcCBkZXZpY2UgYW5kIHJlbW92aW5nIGl0IGZyb20gYnJpZGdlIGFyZSBub3QgYXRvbWljIG9w
ZXJhdGlvbi7vvJ7vvJ5UaGUgUHJvYmxlbSBvY2N1cnMgd2hlbiB0d28gdm1zIHN0YXJ0IGFuZCBz
aHV0ZG93bi4gV2hlbiBvbmUgdm0gd2l0aCB0aGUgbmlj77ye77yeIG5hbWVkICJ2bmV0MCIgc3Rv
cHBpbmcsIGl0IGRlbGV0ZWQgdGFwIGRldmljZSBidXQgbm90IHJlbW92aW5nIHBvcnQgZnJvbSBi
cmlkZ2Uu77ye77yeIEF0IHRoaXMgdGltZSwgYW5vdGhlciB2bSBjcmVhdGVkIHRoZSB0YXAgZGV2
aWNlIG5hbWVkICJ2bmV0MCIgYW5kIGFkZGVkIHBvcnQgdG8gdGhl77ye77yeIHNhbWUgYnJpZGdl
LiBUaGVuLCB0aGUgZmlyc3Qgdm0gZGVsZXRlZCB0aGUgdGFwIGRldmljZSBmcm9tIHRoZSBzYW1l
IGJyaWRnZS7vvJ7vvJ4gRmluYWxseSwgdGhlIHRhcCBkZXZpY2Ugb2YgdGhlIHNlY29uZCB2bSBk
b24ndCBhdHRhY2hlZCB0byB0aGUgYnJpZGdlLu+8nu+8niBTbywgd2UgY2FuIGFkZCBkb21pZCB0
byB2bSdzIG5pYyBuYW1lLiBGb3IgZXhhbXBsZSwgdGhlIHZtJ3MgZG9taWQgaXMgMSBhbmQgdm5l
dDDvvJ7vvJ4gaXMgcmVuYW1lZCB0byB2bmV0MS4wLu+8nlN1cmVseSBkZWxldGluZyB0aGUgTklD
IGF1dG9tYXRpY2FsbHkgcmVtb3ZlcyBpdCBmcm9tIHRoZSBicmlkZ2Ugc28gd2XvvJ5jYW4ganVz
dCByZW1vdmUgdGhlIGNvZGUgdGhhdCBkZWxldHMgdGhlIGJyaWRnZSBwb3J0Lg0KDQppIGhhdmUg
ZG9uZSBzb21lIHRlc3RzIGZvciBhIHRhcCArIG9wZW52c3dpdGNoIGJyaWRnZSBuZXR3b3JrLiAg
aSBmaW5kICB0aGUgbmljIG5hbWVkICJ2bmV0MCIgZG9uJ3QgZXhzaXQgYmVmb3JlIGNhbGxpbmcg
dGhlIHZpck5ldERldk9wZW52c3dpdGNoUmVtb3ZlUG9ydC4NCg0KaSB0aGluayAgdGhlIHRhcCBp
cyBkZWxldGVkIG5vdCBieSByZW1vdmluZ3BvcnQgIGZyb20gYnJpZGdlLiBpIHRoaW5rIGhvdHBs
dWdpbmcgbmV0IGhhcyB0aGUgc2FtZSBwcm9ibGVtLg0KDQoNCg0KDQpieSB0aGUgd2F5Og0KDQog
ICAgICBteSBjb21wYW55J3MgZS1tYWlsICBkb2VzIG5vdCBzdXBwb3J0IHRocmVhZCBwb3N0aW5n
ICBhbmQgaGFzIHNvbWUgb3RoZXIgcHJvYmxlbXMuIGkgYmVsaWV2ZSB0aGF0IG15IGNvbGxlYWd1
ZXMgY2FuIHNvb24gcmVzbG92ZWQgdGhlbS4NCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoN
Cg0KDQoNCg0K5Li65LqG6K6p5oKo55qEVlBsYXTomZrmi5/ljJbmlYXpmpzlvpfliLDpq5jmlYjn
moTlpITnkIbvvIzor7fkuIrmiqXmlYXpmpzliLA6ICRWUGxhdOaKgOacr+aUr+aMgeOAgg0KDQoN
CuiKpuW/l+aciyBsdXpoaXBlbmcNCg0KDQoNCg0KDQoNCklU5byA5Y+R5bel56iL5biIIElUIERl
dmVsb3BtZW50CkVuZ2luZWVyDQrmk43kvZzns7vnu5/kuqflk4Hpg6gv5Lit5b+D56CU56m26Zmi
L+ezu+e7n+S6p+WTgSBPUyBQcm9kdWN0IERlcHQuL0NlbnRyYWwgUu+8hkQgSW5zdGl0dXRlL1N5
c3RlbSBQcm9kdWN0DQoNCg0KDQoNCg0KDQoNCg0KDQrmt7HlnLPluILljZflsbHljLrnp5HmioDl
jZfot681NeWPt+S4reWFtOmAmuiur+eglOWPkeWkp+alvDMz5qW8IA0KMzMvRiwgUiZEIEJ1aWxk
aW5nLCBaVEUKQ29ycG9yYXRpb24gSGktdGVjaCBSb2FkIFNvdXRoLCANCkhpLXRlY2gKSW5kdXN0
cmlhbCBQYXJrIE5hbnNoYW4gRGlzdHJpY3QsIFNoZW56aGVuLCBQLlIuQ2hpbmEsIDUxODA1NyAN
ClQ6ICs4NiA3NTUgeHh4eHh4eHggRjorODYgNzU1IHh4eHh4eHh4IA0KTTogKzg2IHh4eHh4eHh4
eHh4IA0KRTogbHUuemhpcGVuZ0B6dGUuY29tLmNuIA0Kd3d3Lnp0ZS5jb20uY24NCg0KDQoNCg0K
DQoNCuWOn+Wni+mCruS7tg0KDQoNCg0K5Y+R5Lu25Lq677yaIO+8nGJlcnJhbmdlQHJlZGhhdC5j
b23vvJ4NCuaUtuS7tuS6uu+8muiKpuW/l+acizEwMTA4MjcyDQrmioTpgIHkurrvvJog77ycbGli
dmlyLWxpc3RAcmVkaGF0LmNvbe+8ng0K5pelIOacnyDvvJoyMDE35bm0MDTmnIgyOOaXpSAxOToy
Nw0K5Li7IOmimCDvvJpSZTogW2xpYnZpcnRdIFtQQVRDSF0gcWVtdTogY2hhbmdlIHRoZSBuYW1l
IG9mIHRhcCBkZXZpY2UgZm9yIGEgdGFwYW5kIGJyaWRnZSBuZXR3b3JrDQoNCg0KDQoNCg0KT24g
RnJpLCBBcHIgMjgsIDIwMTcgYXQgMDU6MjM6MTlQTSArMDgwMCwgWmhpUGVuZyBMdSB3cm90ZToN
Cu+8niBDcmVhdGluZyB0YXAgZGV2aWNlIGFuZCBhZGRpbmcgdGhlIGRldmljZSB0byBicmlkZ2Ug
YXJlIG5vdCBhdG9taWMgb3BlcmF0aW9uLg0K77yeIFNpbWlsYXJseSBkZWxldGluZyB0YXAgZGV2
aWNlIGFuZCByZW1vdmluZyBpdCBmcm9tIGJyaWRnZSBhcmUgbm90IGF0b21pYyBvcGVyYXRpb24u
DQrvvJ4gVGhlIFByb2JsZW0gb2NjdXJzIHdoZW4gdHdvIHZtcyBzdGFydCBhbmQgc2h1dGRvd24u
IFdoZW4gb25lIHZtIHdpdGggdGhlIG5pYw0K77yeIG5hbWVkICJ2bmV0MCIgc3RvcHBpbmcsIGl0
IGRlbGV0ZWQgdGFwIGRldmljZSBidXQgbm90IHJlbW92aW5nIHBvcnQgZnJvbSBicmlkZ2UuDQrv
vJ4gQXQgdGhpcyB0aW1lLCBhbm90aGVyIHZtIGNyZWF0ZWQgdGhlIHRhcCBkZXZpY2UgbmFtZWQg
InZuZXQwIiBhbmQgYWRkZWQgcG9ydCB0byB0aGUNCu+8niBzYW1lIGJyaWRnZS4gVGhlbiwgdGhl
IGZpcnN0IHZtIGRlbGV0ZWQgdGhlIHRhcCBkZXZpY2UgZnJvbSB0aGUgc2FtZSBicmlkZ2UuDQrv
vJ4gRmluYWxseSwgdGhlIHRhcCBkZXZpY2Ugb2YgdGhlIHNlY29uZCB2bSBkb24ndCBhdHRhY2hl
ZCB0byB0aGUgYnJpZGdlLg0K77yeIFNvLCB3ZSBjYW4gYWRkIGRvbWlkIHRvIHZtJ3MgbmljIG5h
bWUuIEZvciBleGFtcGxlLCB0aGUgdm0ncyBkb21pZCBpcyAxIGFuZCB2bmV0MA0K77yeIGlzIHJl
bmFtZWQgdG8gdm5ldDEuMC4NCg0KU3VyZWx5IGRlbGV0aW5nIHRoZSBOSUMgYXV0b21hdGljYWxs
eSByZW1vdmVzIGl0IGZyb20gdGhlIGJyaWRnZSBzbyB3ZQ0KY2FuIGp1c3QgcmVtb3ZlIHRoZSBj
b2RlIHRoYXQgZGVsZXRzIHRoZSBicmlkZ2UgcG9ydC4NCg0KDQpSZWdhcmRzLA0KRGFuaWVsDQot
LSANCnw6IGh0dHBzOi8vYmVycmFuZ2UuY29tICAgICAgLW8tICAgIGh0dHBzOi8vd3d3LmZsaWNr
ci5jb20vcGhvdG9zL2RiZXJyYW5nZSA6fA0KfDogaHR0cHM6Ly9saWJ2aXJ0Lm9yZyAgICAgICAg
IC1vLSAgICAgICAgICAgIGh0dHBzOi8vZnN0b3AxMzguYmVycmFuZ2UuY29tIDp8DQp8OiBodHRw
czovL2VudGFuZ2xlLXBob3RvLm9yZyAgICAtby0gICAgaHR0cHM6Ly93d3cuaW5zdGFncmFtLmNv
bS9kYmVycmFuZ2UgOnwNCg0KLS0NCmxpYnZpci1saXN0IG1haWxpbmcgbGlzdA0KbGlidmlyLWxp
c3RAcmVkaGF0LmNvbQ0KaHR0cHM6Ly93d3cucmVkaGF0LmNvbS9tYWlsbWFuL2xpc3RpbmZvL2xp
YnZpci1saXN0
6:23 p.m.
New subject: [libvirt] 答复: Re: [PATCH] qemu: change the name of tap device for a tapand bridge network
Possibly related I notice race conditions caused by vnic never getting
loaded if an existing bridge is already up (by OS init scripts etc) and
stopping VM's from getting started.
Often this is behavior you want ; i.e having Host Hypervisor NIC's added
and up before libvirtd sets up it's nics/bridges.
On 29 April 2017 at 11:15, <lu.zhipeng(a)zte.com.cn> wrote:
＞On Fri, Apr 28, 2017 at 05:23:19PM +0800, ZhiPeng Lu wrote:
＞＞ Creating tap device and adding the device to bridge
are not atomic operation.
＞＞ Similarly deleting tap device and removing it from bridge
are not atomic operation.
＞＞The Problem occurs when two vms start and shutdown. When
one vm with the nic
＞＞ named "vnet0" stopping, it deleted tap device but not
removing port from bridge.
＞＞ At this time, another vm created the tap device named "
vnet0" and added port to the
＞＞ same bridge. Then, the first vm deleted the tap device
from the same bridge.
＞＞ Finally, the tap device of the second vm don't attached to the bridge.
＞＞ So, we can add domid to vm's nic name. For example, the vm'
s domid is 1 and vnet0
＞＞ is renamed to vnet1.0.
＞Surely deleting the NIC automatically removes it from the bridge so we
＞can just remove the code that delets the bridge port.
i have done some tests for a tap + openvswitch bridge network. i find
the nic named "vnet0" don't exsit before calling the
virNetDevOpenvswitchRemovePort.
i think the tap is deleted not by removingport from bridge. i think
hotpluging net has the same problem.
by the way:
my company's e-mail does not support thread posting and has some
other problems. i believe that my colleagues can soon resloved them.
*为了让您的VPlat虚拟化故障得到高效的处理，请上报故障到: $VPlat技术支持。*
芦志朋 luzhipeng
IT开发工程师 IT Development Engineer
操作系统产品部/中心研究院/系统产品 OS Product Dept./Central R＆D Institute/System Product
深圳市南山区科技南路55号中兴通讯研发大楼33楼
33/F, R&D Building, ZTE Corporation Hi-tech Road South,
Hi-tech Industrial Park Nanshan District, Shenzhen, P.R.China, 518057
T: +86 755 xxxxxxxx F:+86 755 xxxxxxxx
M: +86 xxxxxxxxxxx
E: lu.zhipeng(a)zte.com.cn
www.zte.com.cn
原始邮件
*发件人：* ＜berrange(a)redhat.com>;
*收件人：*芦志朋10108272;
*抄送人：* ＜libvir-list(a)redhat.com>;
*日 期 ：*2017年04月28日 19:27
*主 题 ：**Re: [libvirt] [PATCH] qemu: change the name of tap device for a
tapand bridge network*
On Fri, Apr 28, 2017 at 05:23:19PM +0800, ZhiPeng Lu wrote:
＞ Creating tap device and adding the device to bridge
are not atomic operation.
＞ Similarly deleting tap device and removing it from
bridge are not atomic operation.
＞ The Problem occurs when two vms start and shutdown. When
one vm with the nic
＞ named "vnet0" stopping, it deleted tap device but not
removing port from bridge.
＞ At this time, another vm created the tap device named "
vnet0" and added port to the
＞ same bridge. Then, the first vm deleted the tap
device from the same bridge.
＞ Finally, the tap device of the second vm don't attached to the bridge.
＞ So, we can add domid to vm's nic name. For example, the
vm's domid is 1 and vnet0
＞ is renamed to vnet1.0.
Surely deleting the NIC automatically removes it from the bridge so we
can just remove the code that delets the bridge port.
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/
photos/dberrange :|
|: https://libvirt.org -o- https://
fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.
instagram.com/dberrange :|
--
libvir-list mailing list
libvir-list(a)redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list
--
libvir-list mailing list
libvir-list(a)redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list
9:45 p.m.
New subject: [libvirt] 答复: Re: [PATCH] qemu: change the name of tap device for a tapand bridge network
On 04/28/2017 07:23 PM, Joel Wirāmu Pauling wrote:
Possibly related I notice race conditions caused by vnic never
getting
loaded if an existing bridge is already up (by OS init scripts etc) and
stopping VM's from getting started.
Can you explain this more exactly? In particular, by "vnic" do you mean
the network device as seen by the guest? And what do you mean by the
term "getting loaded"? (It doesn't make sense to me that you should mean
the network device in the guest, unless "stopping VM's from getting
started" just means that the guest doesn't become fully functional,
rather than that the qemu process doesn't start). Beyond that, of course
the bridge that the tap device will be connected to needs to exist
before you can connect something to it - I don't understand how its
existence could cause a failure; rather its *non*-existence would cause
a failure. (As you can see by my wild suppositions that make no sense,
the terms you've used are a bit too vague/open to interpretation for me
to understand exactly the problem you're referring to)
(Hmm - perhaps you're referring to the situation where libvirt attempts
to create a bridge on the host for one of its virtual networks, but
either a bridge by that name has already been created by "someone else"
or another netdev already exists on the host that is on the same subnet
(and thus has the same route)? Your description doesn't fit that very
well, but that is a known problem and unrelated to the patch/problem
we're discussing here.
Often this is behavior you want ; i.e having Host Hypervisor NIC's added
and up before libvirtd sets up it's nics/bridges.
On 29 April 2017 at 11:15, <lu.zhipeng(a)zte.com.cn
<mailto:lu.zhipeng@zte.com.cn>> wrote:
＞On Fri, Apr 28, 2017 at 05:23:19PM +0800, ZhiPeng Lu wrote:
＞＞
Creating tap device and adding the device to bridge are not atomic operation.
＞＞
Similarly deleting tap device and removing it from bridge are not atomic operation.
＞＞The Problem occurs when two vms start and shutdown. When one vm with the nic
＞＞
named "vnet0" stopping, it deleted tap device but not removing port from
bridge.
＞＞
At this time, another vm created the tap device named "vnet0" and added
port to the
＞＞
same bridge. Then, the first vm deleted the tap device from the same bridge.
＞＞
Finally, the tap device of the second vm don't attached to the bridge.
＞＞
So, we can add domid to vm's nic name. For example, the vm's domid is 1 and
vnet0
＞＞ is renamed to vnet1.0.
＞Surely deleting the NIC automatically removes it from the bridge so we
＞can just remove the code that delets the bridge port.
i have done some tests for a tap + openvswitch bridge network. i
find the nic named "vnet0" don't exsit before calling
the virNetDevOpenvswitchRemovePort.
i think the tap is deleted not by removingport from bridge. i
think hotpluging net has the same problem.
by the way:
my company's e-mail does not support thread posting and has
some other problems. i believe that my colleagues can soon resloved
them.
*为了让您的VPlat虚拟化故障得到高效的处理，请上报故障到: $VPlat技术支
持。*
芦志朋 luzhipeng
IT开发工程师 IT Development Engineer
操作系统产品部/中心研究院/系统产品 OS Product Dept./Central R＆D
Institute/System Product
深圳市南山区科技南路55号中兴通讯研发大楼33楼
33/F, R&D Building, ZTE Corporation Hi-tech Road South,
Hi-tech Industrial Park Nanshan District, Shenzhen, P.R.China, 518057
T: +86 755 xxxxxxxx F:+86 755 xxxxxxxx
M: +86 xxxxxxxxxxx
E: lu.zhipeng(a)zte.com.cn <mailto:lu.zhipeng@zte.com.cn>
www.zte.com.cn <http://www.zte.com.cn/>
原始邮件
*发件人：*＜berrange(a)redhat.com <mailto:berrange@redhat.com>＞;
*收件人：*芦志朋10108272;
*抄送人：*＜libvir-list(a)redhat.com <mailto:libvir-list@redhat.com>＞;
*日 期 ：*2017年04月28日 19:27
*主 题 ：**Re: [libvirt] [PATCH] qemu: change the name of tap device
for a tapand bridge network*
On Fri, Apr 28, 2017 at 05:23:19PM +0800, ZhiPeng Lu wrote:
＞ Creating tap device and adding the device to bridge are not atomic operation.
＞ Similarly deleting tap device and removing it from bridge are not atomic
operation.
＞ The Problem occurs when two vms start and shutdown. When one vm with the nic
＞ named "vnet0" stopping, it deleted tap device but not removing port from
bridge.
＞ At this time, another vm created the tap device named "vnet0" and added
port to the
＞ same bridge. Then, the first vm deleted the tap device from the same bridge.
＞ Finally, the tap device of the second vm don't attached to the bridge.
＞ So, we can add domid to vm's nic name. For example, the vm's domid is 1 and
vnet0
＞ is renamed to vnet1.0.
Surely deleting the NIC automatically removes it from the bridge so we
can just remove the code that delets the bridge port.
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange
<https://www.flickr.com/photos/dberrange> :|
|: https://libvirt.org -o- https://fstop138.berrange.com
<https://fstop138.berrange.com> :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange
<https://www.instagram.com/dberrange> :|
--
libvir-list mailing list
libvir-list(a)redhat.com <mailto:libvir-list@redhat.com>
https://www.redhat.com/mailman/listinfo/libvir-list
<https://www.redhat.com/mailman/listinfo/libvir-list>
--
libvir-list mailing list
libvir-list(a)redhat.com <mailto:libvir-list@redhat.com>
https://www.redhat.com/mailman/listinfo/libvir-list
<https://www.redhat.com/mailman/listinfo/libvir-list>
--
libvir-list mailing list
libvir-list(a)redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list
10:20 p.m.
New subject: [libvirt] 答复: Re: [PATCH] qemu: change the name of tap device for a tapand bridge network
I'll try and explain the steps/situation where this is an issue.
Say for example you have your router/dhcpd done inside a VM guest - you use
macvtap devices and vlans off the hypervisor to get the traffic in and out
(WAN and LAN). And you want the LAN side to provide connectivity to the
host as well as to the network beyond the host.
this is on a Ubuntu 16.04 host.
1) Create and isolated virtual Network via libvirt xml - say 'LAN' this is
the second network (assume default exists) creates virbr1 and a vnic1 This
is addition to the WAN and LAN macvtap devices. Unbeknown-st to libvirt the
LAN macvtap is going to end up bridge to the 'isolated' network by the VM.
2) the virbr1 bridge needs numbering
vnic1 is attached inside the VM to a dhcpd server but you still need the
virbr1 interface to be numbered on the hypervisor (but you don't want the
libvirt networking to take care of the numbering etc becuase that's what
your fancy NFV VM is going to handle). Manually it's easy at this point you
just run dhclient et.al on the bridge and you are good to go.
BUT you want this to happen at boot; adding any sort of etc/network or
/etc/sysconfig/networking scripts refering to that virbr1 will create a
race condition which will stop the vm from getting started as you mention.
On 29 April 2017 at 14:45, Laine Stump <laine(a)laine.org> wrote:
On 04/28/2017 07:23 PM, Joel Wirāmu Pauling wrote:
> Possibly related I notice race conditions caused by vnic never getting
> loaded if an existing bridge is already up (by OS init scripts etc) and
> stopping VM's from getting started.
Can you explain this more exactly? In particular, by "vnic" do you mean
the network device as seen by the guest? And what do you mean by the
term "getting loaded"? (It doesn't make sense to me that you should mean
the network device in the guest, unless "stopping VM's from getting
started" just means that the guest doesn't become fully functional,
rather than that the qemu process doesn't start). Beyond that, of course
the bridge that the tap device will be connected to needs to exist
before you can connect something to it - I don't understand how its
existence could cause a failure; rather its *non*-existence would cause
a failure. (As you can see by my wild suppositions that make no sense,
the terms you've used are a bit too vague/open to interpretation for me
to understand exactly the problem you're referring to)
(Hmm - perhaps you're referring to the situation where libvirt attempts
to create a bridge on the host for one of its virtual networks, but
either a bridge by that name has already been created by "someone else"
or another netdev already exists on the host that is on the same subnet
(and thus has the same route)? Your description doesn't fit that very
well, but that is a known problem and unrelated to the patch/problem
we're discussing here.
>
> Often this is behavior you want ; i.e having Host Hypervisor NIC's added
> and up before libvirtd sets up it's nics/bridges.
>
>
> On 29 April 2017 at 11:15, <lu.zhipeng(a)zte.com.cn
> <mailto:lu.zhipeng@zte.com.cn>> wrote:
>
>
> ＞On Fri, Apr 28, 2017 at 05:23:19PM +0800, ZhiPeng Lu wrote:
> ＞＞
> Creating tap device and adding the device to bridge are not atomic
operation.
> ＞＞
> Similarly deleting tap device and removing it from bridge are not
atomic operation.
> ＞＞The Problem occurs when two vms start and shutdown. When one vm
with the nic
> ＞＞
> named "vnet0" stopping, it deleted tap device but not removing port
from bridge.
> ＞＞
> At this time, another vm created the tap device named "vnet0" and
added port to the
> ＞＞
> same bridge. Then, the first vm deleted the tap device from the same
bridge.
> ＞＞
> Finally, the tap device of the second vm don't attached to the
bridge.
> ＞＞
> So, we can add domid to vm's nic name. For example, the vm's domid
is 1 and vnet0
> ＞＞ is renamed to vnet1.0.
>
> ＞Surely deleting the NIC automatically removes it from the bridge so
we
> ＞can just remove the code that delets the bridge port.
>
> i have done some tests for a tap + openvswitch bridge network. i
> find the nic named "vnet0" don't exsit before calling
> the virNetDevOpenvswitchRemovePort.
>
> i think the tap is deleted not by removingport from bridge. i
> think hotpluging net has the same problem.
>
>
> by the way:
>
> my company's e-mail does not support thread posting and has
> some other problems. i believe that my colleagues can soon resloved
> them.
>
>
>
>
>
>
> *为了让您的VPlat虚拟化故障得到高效的处理，请上报故障到: $VPlat技术支
> 持。*
>
> 芦志朋 luzhipeng
>
>
> IT开发工程师 IT Development Engineer
> 操作系统产品部/中心研究院/系统产品 OS Product Dept./Central R＆D
> Institute/System Product
>
>
>
> 深圳市南山区科技南路55号中兴通讯研发大楼33楼
> 33/F, R&D Building, ZTE Corporation Hi-tech Road South,
> Hi-tech Industrial Park Nanshan District, Shenzhen, P.R.China, 518057
> T: +86 755 xxxxxxxx F:+86 755 xxxxxxxx
> M: +86 xxxxxxxxxxx
> E: lu.zhipeng(a)zte.com.cn <mailto:lu.zhipeng@zte.com.cn>
> www.zte.com.cn <http://www.zte.com.cn/>
>
> 原始邮件
> *发件人：*＜berrange(a)redhat.com <mailto:berrange@redhat.com>＞;
> *收件人：*芦志朋10108272;
> *抄送人：*＜libvir-list(a)redhat.com <mailto:libvir-list@redhat.com>＞;
> *日 期 ：*2017年04月28日 19:27
> *主 题 ：**Re: [libvirt] [PATCH] qemu: change the name of tap device
> for a tapand bridge network*
>
>
> On Fri, Apr 28, 2017 at 05:23:19PM +0800, ZhiPeng Lu wrote:
> ＞ Creating tap device and adding the device to bridge are not atomic
operation.
> ＞ Similarly deleting tap device and removing it from bridge are not
atomic operation.
> ＞ The Problem occurs when two vms start and shutdown. When one vm
with the nic
> ＞ named "vnet0" stopping, it deleted tap device but not removing
port from bridge.
> ＞ At this time, another vm created the tap device named "vnet0" and
added port to the
> ＞ same bridge. Then, the first vm deleted the tap device from the
same bridge.
> ＞ Finally, the tap device of the second vm don't attached to the
bridge.
> ＞ So, we can add domid to vm's nic name. For example, the vm's domid
is 1 and vnet0
> ＞ is renamed to vnet1.0.
>
> Surely deleting the NIC automatically removes it from the bridge so
we
> can just remove the code that delets the bridge port.
>
>
> Regards,
> Daniel
> --
> |: https://berrange.com -o- https://www.flickr.com/photos/
dberrange
> <https://www.flickr.com/photos/dberrange> :|
> |: https://libvirt.org -o-
https://fstop138.berrange.com
> <https://fstop138.berrange.com> :|
> |: https://entangle-photo.org -o- https://www.instagram.com/
dberrange
> <https://www.instagram.com/dberrange> :|
>
> --
> libvir-list mailing list
> libvir-list(a)redhat.com <mailto:libvir-list@redhat.com>
> https://www.redhat.com/mailman/listinfo/libvir-list
> <https://www.redhat.com/mailman/listinfo/libvir-list>
>
>
>
>
> --
> libvir-list mailing list
> libvir-list(a)redhat.com <mailto:libvir-list@redhat.com>
> https://www.redhat.com/mailman/listinfo/libvir-list
> <https://www.redhat.com/mailman/listinfo/libvir-list>
>
>
>
>
> --
> libvir-list mailing list
> libvir-list(a)redhat.com
> https://www.redhat.com/mailman/listinfo/libvir-list
>
12:51 p.m.
New subject: [libvirt] 答复: Re: 答复: Re: [PATCH] qemu: change the name of tap device for a tapand bridge network
aSBleHBsYWluIHRoZSBjYXNlIGFnYWluLiAgdGhlIG5ldCBjb25maWcgeG1sIG9mIHR3byB2bXMg
aXMgOg0KDQog77ycaW50ZXJmYWNlIHR5cGU9J2JyaWRnZSfvvJ4NCg0KICAgICAg77ycbWFjIGFk
ZHJlc3M9J2ZhOjE2OjNlOmUxOmIyOjAxJy/vvJ4NCg0KICAgICAg77ycc291cmNlIGJyaWRnZT0n
YnIxMDAnL++8ng0KDQogICAgICDvvJxtb2RlbCB0eXBlPSd2aXJ0aW8nL++8ng0KDQogICAgICDv
vJxkcml2ZXIgbmFtZT0ndmhvc3QnL++8ng0KDQogICAgICDvvJx2aXJ0dWFscG9ydCB0eXBlPSdv
cGVudnN3aXRjaCcv77yeDQoNCiAgICDvvJwvaW50ZXJmYWNl77yeDQoNCm9mIGNhdXNlIHRoZSBt
YWMgb2YgdHdvIHZtcyBpcyBkaWZmZXJlbnQuIHRoZSBicmlkZ2UgbmFtZWQgYnIxMDAgaXMgY3Jl
YXRlZCBpbiBhZHZhbmNlLg0KDQp3aGVuIHRoZSBmaXJzdCB2bSBzdGFydGVkICx3ZSBjYW4gZmlu
ZCB0aGUgdGFvIGRldmljZSBuYW1lZCAidm5ldDAiIGJ5IHZpcnNoIGRvbWlmbGlzdCBkb21pZCBj
b21tYW5kLg0KDQp3ZSBjYW4gc2h1dGRvdyB0aGUgZmlyc3Qgdm0gYW5kIHN0YXJ0IHRoZSBzZWNv
bmQgdm0gYXQgb3IgYXJvdW5kIHRoZSBzYW1lIHRpbWUuIHdlICBjYW4gZmluZCAgdGhlIHRhcCBk
ZXZpY2Ugb2YgIHRoZSBzZWNvbmQgdm0gIGlzIGFsc28gbmFtZWQgInZuZXQwIi4NCg0Kc3RhcnRp
bmcgdXAgYW5kIHNodXRkb3duaW5nIHZtIGFyZSBub3QgbXV0dWFsbHkgZXhjbHVzaXZlLiBhZGRp
bmcgcG9ydCBuYW1lZCAidm5ldDAiIHRvIGJyaWRnZSBhbmQgIGRlbGV0aW5nIHBvcnQgZnJvbSBi
cmlkZ2UgYXJlIG5vdCBtdXR1YWxseSBleGNsdXNpdmUuDQoNCnNvIHRoZSBwcm9ibGVtIG9jY3Vy
cmVkLiB0aGUgcG9ydCBvZiB0aGUgc2Vjb25kIHZtIGFkZGVkIHRvIHRoZSBicmlkZ2UgaXMgZGV0
ZWxlZCBieSBjYWxsaW5nIHZpck5ldERldk9wZW52c3dpdGNoUmVtb3ZlUG9ydCBpbiBzaHV0ZG93
bmluZyAgdGhlIGZpcnN0IHZtIC4NCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoN
Cg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQrkuLrkuoborqnmgqjnmoRWUGxhdOiZmuaLn+WM
luaVhemanOW+l+WIsOmrmOaViOeahOWkhOeQhu+8jOivt+S4iuaKpeaVhemanOWIsDogJFZQbGF0
5oqA5pyv5pSv5oyB44CCDQoNCg0K6Iqm5b+X5pyLIGx1emhpcGVuZw0KDQoNCg0KDQoNCg0KSVTl
vIDlj5Hlt6XnqIvluIggSVQgRGV2ZWxvcG1lbnQKRW5naW5lZXINCuaTjeS9nOezu+e7n+S6p+WT
gemDqC/kuK3lv4PnoJTnqbbpmaIv57O757uf5Lqn5ZOBIE9TIFByb2R1Y3QgRGVwdC4vQ2VudHJh
bCBS77yGRCBJbnN0aXR1dGUvU3lzdGVtIFByb2R1Y3QNCg0KDQoNCg0KDQoNCg0KDQoNCua3seWc
s+W4guWNl+WxseWMuuenkeaKgOWNl+i3rzU15Y+35Lit5YW06YCa6K6v56CU5Y+R5aSn5qW8MzPm
pbwgDQozMy9GLCBSJkQgQnVpbGRpbmcsIFpURQpDb3Jwb3JhdGlvbiBIaS10ZWNoIFJvYWQgU291
dGgsIA0KSGktdGVjaApJbmR1c3RyaWFsIFBhcmsgTmFuc2hhbiBEaXN0cmljdCwgU2hlbnpoZW4s
IFAuUi5DaGluYSwgNTE4MDU3IA0KVDogKzg2IDc1NSB4eHh4eHh4eCBGOis4NiA3NTUgeHh4eHh4
eHggDQpNOiArODYgeHh4eHh4eHh4eHggDQpFOiBsdS56aGlwZW5nQHp0ZS5jb20uY24gDQp3d3cu
enRlLmNvbS5jbg0KDQoNCg0KDQoNCg0K5Y6f5aeL6YKu5Lu2DQoNCg0KDQrlj5Hku7bkurrvvJog
77ycam9lbEBhZW5lcnRpYS5uZXTvvJ4NCuaUtuS7tuS6uu+8miDvvJxsYWluZUBsYWluZS5vcmfv
vJ4NCuaKhOmAgeS6uu+8miDvvJxsaWJ2aXItbGlzdEByZWRoYXQuY29t77ye6Iqm5b+X5pyLMTAx
MDgyNzINCuaXpSDmnJ8g77yaMjAxN+W5tDA05pyIMjnml6UgMTE6MjENCuS4uyDpopgg77yaUmU6
IFtsaWJ2aXJ0XeetlOWkjTogUmU6IFtQQVRDSF0gcWVtdTogY2hhbmdlIHRoZSBuYW1lIG9mIHRh
cCBkZXZpY2UgZm9yIGEgdGFwYW5kIGJyaWRnZSBuZXR3b3JrDQoNCg0KDQoNCg0KDQoNCkknbGwg
dHJ5IGFuZCBleHBsYWluIHRoZSBzdGVwcy9zaXR1YXRpb24gd2hlcmUgdGhpcyBpcyBhbiBpc3N1
ZS4NCg0KDQoNClNheSBmb3IgZXhhbXBsZSB5b3UgaGF2ZSB5b3VyIHJvdXRlci9kaGNwZCBkb25l
IGluc2lkZSBhIFZNIGd1ZXN0IC0geW91IHVzZSBtYWN2dGFwIGRldmljZXMgYW5kIHZsYW5zIG9m
ZiB0aGUgaHlwZXJ2aXNvciB0byBnZXQgdGhlIHRyYWZmaWMgaW4gYW5kIG91dCAoV0FOIGFuZCBM
QU4pLiBBbmQgeW91IHdhbnQgdGhlIExBTiBzaWRlIHRvIHByb3ZpZGUgY29ubmVjdGl2aXR5IHRv
IHRoZSBob3N0IGFzIHdlbGwgYXMgdG8gdGhlIG5ldHdvcmsgYmV5b25kIHRoZSBob3N0Lg0KDQp0
aGlzIGlzIG9uIGEgVWJ1bnR1IDE2LjA0IGhvc3QuDQoNCg0KDQoxKSBDcmVhdGUgYW5kIGlzb2xh
dGVkIHZpcnR1YWwgTmV0d29yayB2aWEgbGlidmlydCB4bWwgLSBzYXkgJ0xBTicgdGhpcyBpcyB0
aGUgc2Vjb25kIG5ldHdvcmsgKGFzc3VtZSBkZWZhdWx0IGV4aXN0cykgY3JlYXRlcyB2aXJicjEg
YW5kIGEgdm5pYzEgVGhpcyBpcyBhZGRpdGlvbiB0byB0aGUgV0FOIGFuZCBMQU4gbWFjdnRhcCBk
ZXZpY2VzLiBVbmJla25vd24tc3QgdG8gbGlidmlydCB0aGUgTEFOIG1hY3Z0YXAgaXMgZ29pbmcg
dG8gZW5kIHVwIGJyaWRnZSB0byB0aGUgJ2lzb2xhdGVkJyBuZXR3b3JrIGJ5IHRoZSBWTS4NCg0K
DQoyKSB0aGUgdmlyYnIxIGJyaWRnZSBuZWVkcyBudW1iZXJpbmcNCg0KDQoNCnZuaWMxIGlzIGF0
dGFjaGVkIGluc2lkZSB0aGUgVk0gdG8gYSBkaGNwZCBzZXJ2ZXIgYnV0IHlvdSBzdGlsbCBuZWVk
IHRoZSB2aXJicjEgaW50ZXJmYWNlIHRvIGJlIG51bWJlcmVkIG9uIHRoZSBoeXBlcnZpc29yIChi
dXQgeW91IGRvbid0IHdhbnQgdGhlIGxpYnZpcnQgbmV0d29ya2luZyB0byB0YWtlIGNhcmUgb2Yg
dGhlIG51bWJlcmluZyBldGMgYmVjdWFzZSB0aGF0J3Mgd2hhdCB5b3VyIGZhbmN5IE5GViBWTSBp
cyBnb2luZyB0byBoYW5kbGUpLiBNYW51YWxseSBpdCdzIGVhc3kgYXQgdGhpcyBwb2ludCB5b3Ug
anVzdCBydW4gZGhjbGllbnQgZXQuYWwgb24gdGhlIGJyaWRnZSBhbmQgeW91IGFyZSBnb29kIHRv
IGdvLg0KDQoNCg0KQlVUIHlvdSB3YW50IHRoaXMgdG8gaGFwcGVuIGF0IGJvb3QgYWRkaW5nIGFu
eSBzb3J0IG9mIGV0Yy9uZXR3b3JrIG9yIC9ldGMvc3lzY29uZmlnL25ldHdvcmtpbmcgc2NyaXB0
cyByZWZlcmluZyB0byB0aGF0IHZpcmJyMSB3aWxsIGNyZWF0ZSBhIHJhY2UgY29uZGl0aW9uIHdo
aWNoIHdpbGwgc3RvcCB0aGUgdm0gZnJvbSBnZXR0aW5nIHN0YXJ0ZWQgYXMgeW91IG1lbnRpb24u
DQoNCg0KDQoNCg0KDQoNCg0KT24gMjkgQXByaWwgMjAxNyBhdCAxNDo0NSwgTGFpbmUgU3R1bXAg
77ycbGFpbmVAbGFpbmUub3Jn77yeIHdyb3RlOg0KT24gMDQvMjgvMjAxNyAwNzoyMyBQTSwgSm9l
bCBXaXLEgW11IFBhdWxpbmcgd3JvdGU6DQog77yeIFBvc3NpYmx5IHJlbGF0ZWQgSSBub3RpY2Ug
cmFjZSBjb25kaXRpb25zIGNhdXNlZCBieSB2bmljIG5ldmVyIGdldHRpbmcNCiDvvJ4gbG9hZGVk
IGlmIGFuIGV4aXN0aW5nIGJyaWRnZSBpcyBhbHJlYWR5IHVwIChieSBPUyBpbml0IHNjcmlwdHMg
ZXRjKSBhbmQNCiDvvJ4gc3RvcHBpbmcgVk0ncyBmcm9tIGdldHRpbmcgc3RhcnRlZC4NCiANCiBD
YW4geW91IGV4cGxhaW4gdGhpcyBtb3JlIGV4YWN0bHk/IEluIHBhcnRpY3VsYXIsIGJ5ICJ2bmlj
IiBkbyB5b3UgbWVhbg0KIHRoZSBuZXR3b3JrIGRldmljZSBhcyBzZWVuIGJ5IHRoZSBndWVzdD8g
QW5kIHdoYXQgZG8geW91IG1lYW4gYnkgdGhlDQogdGVybSAiZ2V0dGluZyBsb2FkZWQiPyAoSXQg
ZG9lc24ndCBtYWtlIHNlbnNlIHRvIG1lIHRoYXQgeW91IHNob3VsZCBtZWFuDQogdGhlIG5ldHdv
cmsgZGV2aWNlIGluIHRoZSBndWVzdCwgdW5sZXNzICJzdG9wcGluZyBWTSdzIGZyb20gZ2V0dGlu
Zw0KIHN0YXJ0ZWQiIGp1c3QgbWVhbnMgdGhhdCB0aGUgZ3Vlc3QgZG9lc24ndCBiZWNvbWUgZnVs
bHkgZnVuY3Rpb25hbCwNCiByYXRoZXIgdGhhbiB0aGF0IHRoZSBxZW11IHByb2Nlc3MgZG9lc24n
dCBzdGFydCkuIEJleW9uZCB0aGF0LCBvZiBjb3Vyc2UNCiB0aGUgYnJpZGdlIHRoYXQgdGhlIHRh
cCBkZXZpY2Ugd2lsbCBiZSBjb25uZWN0ZWQgdG8gbmVlZHMgdG8gZXhpc3QNCiBiZWZvcmUgeW91
IGNhbiBjb25uZWN0IHNvbWV0aGluZyB0byBpdCAtIEkgZG9uJ3QgdW5kZXJzdGFuZCBob3cgaXRz
DQogZXhpc3RlbmNlIGNvdWxkIGNhdXNlIGEgZmFpbHVyZSByYXRoZXIgaXRzICpub24qLWV4aXN0
ZW5jZSB3b3VsZCBjYXVzZQ0KIGEgZmFpbHVyZS4gKEFzIHlvdSBjYW4gc2VlIGJ5IG15IHdpbGQg
c3VwcG9zaXRpb25zIHRoYXQgbWFrZSBubyBzZW5zZSwNCiB0aGUgdGVybXMgeW91J3ZlIHVzZWQg
YXJlIGEgYml0IHRvbyB2YWd1ZS9vcGVuIHRvIGludGVycHJldGF0aW9uIGZvciBtZQ0KIHRvIHVu
ZGVyc3RhbmQgZXhhY3RseSB0aGUgcHJvYmxlbSB5b3UncmUgcmVmZXJyaW5nIHRvKQ0KIA0KIChI
bW0gLSBwZXJoYXBzIHlvdSdyZSByZWZlcnJpbmcgdG8gdGhlIHNpdHVhdGlvbiB3aGVyZSBsaWJ2
aXJ0IGF0dGVtcHRzDQogdG8gY3JlYXRlIGEgYnJpZGdlIG9uIHRoZSBob3N0IGZvciBvbmUgb2Yg
aXRzIHZpcnR1YWwgbmV0d29ya3MsIGJ1dA0KIGVpdGhlciBhIGJyaWRnZSBieSB0aGF0IG5hbWUg
aGFzIGFscmVhZHkgYmVlbiBjcmVhdGVkIGJ5ICJzb21lb25lIGVsc2UiDQogb3IgYW5vdGhlciBu
ZXRkZXYgYWxyZWFkeSBleGlzdHMgb24gdGhlIGhvc3QgdGhhdCBpcyBvbiB0aGUgc2FtZSBzdWJu
ZXQNCiAoYW5kIHRodXMgaGFzIHRoZSBzYW1lIHJvdXRlKT8gWW91ciBkZXNjcmlwdGlvbiBkb2Vz
bid0IGZpdCB0aGF0IHZlcnkNCiB3ZWxsLCBidXQgdGhhdCBpcyBhIGtub3duIHByb2JsZW0gYW5k
IHVucmVsYXRlZCB0byB0aGUgcGF0Y2gvcHJvYmxlbQ0KIHdlJ3JlIGRpc2N1c3NpbmcgaGVyZS4N
CiANCiDvvJ4NCiDvvJ4gT2Z0ZW4gdGhpcyBpcyBiZWhhdmlvciB5b3Ugd2FudCAgaS5lIGhhdmlu
ZyBIb3N0IEh5cGVydmlzb3IgTklDJ3MgYWRkZWQNCiDvvJ4gYW5kIHVwIGJlZm9yZSBsaWJ2aXJ0
ZCBzZXRzIHVwIGl0J3Mgbmljcy9icmlkZ2VzLg0KIA0KIA0KIA0KIO+8ng0KIO+8ng0KIO+8niBP
biAyOSBBcHJpbCAyMDE3IGF0IDExOjE1LCDvvJxsdS56aGlwZW5nQHp0ZS5jb20uY24NCiDvvJ4g
77ycbWFpbHRvOmx1LnpoaXBlbmdAenRlLmNvbS5jbu+8nu+8niB3cm90ZToNCiDvvJ4NCiDvvJ4N
CiDvvJ4gICAgIO+8nk9uIEZyaSwgQXByIDI4LCAyMDE3IGF0IDA1OjIzOjE5UE0gKzA4MDAsIFpo
aVBlbmcgTHUgd3JvdGU6DQog77yeICAgICDvvJ7vvJ4NCiDvvJ4gICAgIENyZWF0aW5nIHRhcCBk
ZXZpY2UgYW5kIGFkZGluZyB0aGUgZGV2aWNlIHRvIGJyaWRnZSBhcmUgbm90IGF0b21pYyBvcGVy
YXRpb24uDQog77yeICAgICDvvJ7vvJ4NCiDvvJ4gICAgIFNpbWlsYXJseSBkZWxldGluZyB0YXAg
ZGV2aWNlIGFuZCByZW1vdmluZyBpdCBmcm9tIGJyaWRnZSBhcmUgbm90IGF0b21pYyBvcGVyYXRp
b24uDQog77yeICAgICDvvJ7vvJ5UaGUgUHJvYmxlbSBvY2N1cnMgd2hlbiB0d28gdm1zIHN0YXJ0
IGFuZCBzaHV0ZG93bi4gV2hlbiBvbmUgdm0gd2l0aCB0aGUgbmljDQog77yeICAgICDvvJ7vvJ4N
CiDvvJ4gICAgIG5hbWVkICJ2bmV0MCIgc3RvcHBpbmcsIGl0IGRlbGV0ZWQgdGFwIGRldmljZSBi
dXQgbm90IHJlbW92aW5nIHBvcnQgZnJvbSBicmlkZ2UuDQog77yeICAgICDvvJ7vvJ4NCiDvvJ4g
ICAgIEF0IHRoaXMgdGltZSwgYW5vdGhlciB2bSBjcmVhdGVkIHRoZSB0YXAgZGV2aWNlIG5hbWVk
ICJ2bmV0MCIgYW5kIGFkZGVkIHBvcnQgdG8gdGhlDQog77yeICAgICDvvJ7vvJ4NCiDvvJ4gICAg
IHNhbWUgYnJpZGdlLiBUaGVuLCB0aGUgZmlyc3Qgdm0gZGVsZXRlZCB0aGUgdGFwIGRldmljZSBm
cm9tIHRoZSBzYW1lIGJyaWRnZS4NCiDvvJ4gICAgIO+8nu+8ng0KIO+8niAgICAgRmluYWxseSwg
dGhlIHRhcCBkZXZpY2Ugb2YgdGhlIHNlY29uZCB2bSBkb24ndCBhdHRhY2hlZCB0byB0aGUgYnJp
ZGdlLg0KIO+8niAgICAg77ye77yeDQog77yeICAgICBTbywgd2UgY2FuIGFkZCBkb21pZCB0byB2
bSdzIG5pYyBuYW1lLiBGb3IgZXhhbXBsZSwgdGhlIHZtJ3MgZG9taWQgaXMgMSBhbmQgdm5ldDAN
CiDvvJ4gICAgIO+8nu+8niBpcyByZW5hbWVkIHRvIHZuZXQxLjAuDQog77yeDQog77yeICAgICDv
vJ5TdXJlbHkgZGVsZXRpbmcgdGhlIE5JQyBhdXRvbWF0aWNhbGx5IHJlbW92ZXMgaXQgZnJvbSB0
aGUgYnJpZGdlIHNvIHdlDQog77yeICAgICDvvJ5jYW4ganVzdCByZW1vdmUgdGhlIGNvZGUgdGhh
dCBkZWxldHMgdGhlIGJyaWRnZSBwb3J0Lg0KIO+8ng0KIO+8niAgICAgaSBoYXZlIGRvbmUgc29t
ZSB0ZXN0cyBmb3IgYSB0YXAgKyBvcGVudnN3aXRjaCBicmlkZ2UgbmV0d29yay4gIGkNCiDvvJ4g
ICAgIGZpbmQgIHRoZSBuaWMgbmFtZWQgInZuZXQwIiBkb24ndCBleHNpdCBiZWZvcmUgY2FsbGlu
Zw0KIO+8niAgICAgdGhlIHZpck5ldERldk9wZW52c3dpdGNoUmVtb3ZlUG9ydC4NCiDvvJ4NCiDv
vJ4gICAgIGkgdGhpbmsgIHRoZSB0YXAgaXMgZGVsZXRlZCBub3QgYnkgcmVtb3Zpbmdwb3J0ICBm
cm9tIGJyaWRnZS4gaQ0KIO+8niAgICAgdGhpbmsgaG90cGx1Z2luZyBuZXQgaGFzIHRoZSBzYW1l
IHByb2JsZW0uDQog77yeDQog77yeDQog77yeICAgICBieSB0aGUgd2F5Og0KIO+8ng0KIO+8niAg
ICAgICAgICAgbXkgY29tcGFueSdzIGUtbWFpbCAgZG9lcyBub3Qgc3VwcG9ydCB0aHJlYWQgcG9z
dGluZyAgYW5kIGhhcw0KIO+8niAgICAgc29tZSBvdGhlciBwcm9ibGVtcy4gaSBiZWxpZXZlIHRo
YXQgbXkgY29sbGVhZ3VlcyBjYW4gc29vbiByZXNsb3ZlZA0KIO+8niAgICAgdGhlbS4NCiDvvJ4N
CiDvvJ4NCiDvvJ4NCiDvvJ4NCiDvvJ4NCiDvvJ4NCiDvvJ4gICAgICrkuLrkuoborqnmgqjnmoRW
UGxhdOiZmuaLn+WMluaVhemanOW+l+WIsOmrmOaViOeahOWkhOeQhu+8jOivt+S4iuaKpeaVhema
nOWIsDogJFZQbGF05oqA5pyv5pSvDQog77yeICAgICDmjIHjgIIqDQog77yeDQog77yeICAgICDo
iqblv5fmnIsgbHV6aGlwZW5nDQog77yeDQog77yeDQog77yeICAgICBJVOW8gOWPkeW3peeoi+W4
iCBJVCBEZXZlbG9wbWVudCBFbmdpbmVlcg0KIO+8niAgICAg5pON5L2c57O757uf5Lqn5ZOB6YOo
L+S4reW/g+eglOeptumZoi/ns7vnu5/kuqflk4EgT1MgUHJvZHVjdCBEZXB0Li9DZW50cmFsIFLv
vIZEDQog77yeICAgICBJbnN0aXR1dGUvU3lzdGVtIFByb2R1Y3QNCiDvvJ4NCiDvvJ4NCiDvvJ4N
CiDvvJ4gICAgIOa3seWcs+W4guWNl+WxseWMuuenkeaKgOWNl+i3rzU15Y+35Lit5YW06YCa6K6v
56CU5Y+R5aSn5qW8MzPmpbwNCiDvvJ4gICAgIDMzL0YsIFImRCBCdWlsZGluZywgWlRFIENvcnBv
cmF0aW9uIEhpLXRlY2ggUm9hZCBTb3V0aCwNCiDvvJ4gICAgIEhpLXRlY2ggSW5kdXN0cmlhbCBQ
YXJrIE5hbnNoYW4gRGlzdHJpY3QsIFNoZW56aGVuLCBQLlIuQ2hpbmEsIDUxODA1Nw0KIO+8niAg
ICAgVDogKzg2IDc1NSB4eHh4eHh4eCBGOis4NiA3NTUgeHh4eHh4eHgNCiDvvJ4gICAgIE06ICs4
NiB4eHh4eHh4eHh4eA0KIO+8niAgICAgRTogbHUuemhpcGVuZ0B6dGUuY29tLmNuIO+8nG1haWx0
bzpsdS56aGlwZW5nQHp0ZS5jb20uY27vvJ4NCiDvvJ4gICAgIHd3dy56dGUuY29tLmNuIO+8nGh0
dHA6Ly93d3cuenRlLmNvbS5jbi/vvJ4NCiDvvJ4NCiDvvJ4gICAgIOWOn+Wni+mCruS7tg0KIO+8
niAgICAgKuWPkeS7tuS6uu+8mirvvJxiZXJyYW5nZUByZWRoYXQuY29tIO+8nG1haWx0bzpiZXJy
YW5nZUByZWRoYXQuY29t77ye77yeDQog77yeICAgICAq5pS25Lu25Lq677yaKuiKpuW/l+acizEw
MTA4MjcyDQog77yeICAgICAq5oqE6YCB5Lq677yaKu+8nGxpYnZpci1saXN0QHJlZGhhdC5jb20g
77ycbWFpbHRvOmxpYnZpci1saXN0QHJlZGhhdC5jb23vvJ7vvJ4NCiDvvJ4gICAgICrml6Ug5pyf
IO+8mioyMDE35bm0MDTmnIgyOOaXpSAxOToyNw0KIO+8niAgICAgKuS4uyDpopgg77yaKipSZTog
W2xpYnZpcnRdIFtQQVRDSF0gcWVtdTogY2hhbmdlIHRoZSBuYW1lIG9mIHRhcCBkZXZpY2UNCiDv
vJ4gICAgIGZvciBhIHRhcGFuZCBicmlkZ2UgbmV0d29yayoNCiDvvJ4NCiDvvJ4NCiDvvJ4gICAg
IE9uIEZyaSwgQXByIDI4LCAyMDE3IGF0IDA1OjIzOjE5UE0gKzA4MDAsIFpoaVBlbmcgTHUgd3Jv
dGU6DQog77yeICAgICDvvJ4gQ3JlYXRpbmcgdGFwIGRldmljZSBhbmQgYWRkaW5nIHRoZSBkZXZp
Y2UgdG8gYnJpZGdlIGFyZSBub3QgYXRvbWljIG9wZXJhdGlvbi4NCiDvvJ4gICAgIO+8niBTaW1p
bGFybHkgZGVsZXRpbmcgdGFwIGRldmljZSBhbmQgcmVtb3ZpbmcgaXQgZnJvbSBicmlkZ2UgYXJl
IG5vdCBhdG9taWMgb3BlcmF0aW9uLg0KIO+8niAgICAg77yeIFRoZSBQcm9ibGVtIG9jY3VycyB3
aGVuIHR3byB2bXMgc3RhcnQgYW5kIHNodXRkb3duLiBXaGVuIG9uZSB2bSB3aXRoIHRoZSBuaWMN
CiDvvJ4gICAgIO+8niBuYW1lZCAidm5ldDAiIHN0b3BwaW5nLCBpdCBkZWxldGVkIHRhcCBkZXZp
Y2UgYnV0IG5vdCByZW1vdmluZyBwb3J0IGZyb20gYnJpZGdlLg0KIO+8niAgICAg77yeIEF0IHRo
aXMgdGltZSwgYW5vdGhlciB2bSBjcmVhdGVkIHRoZSB0YXAgZGV2aWNlIG5hbWVkICJ2bmV0MCIg
YW5kIGFkZGVkIHBvcnQgdG8gdGhlDQog77yeICAgICDvvJ4gc2FtZSBicmlkZ2UuIFRoZW4sIHRo
ZSBmaXJzdCB2bSBkZWxldGVkIHRoZSB0YXAgZGV2aWNlIGZyb20gdGhlIHNhbWUgYnJpZGdlLg0K
IO+8niAgICAg77yeIEZpbmFsbHksIHRoZSB0YXAgZGV2aWNlIG9mIHRoZSBzZWNvbmQgdm0gZG9u
J3QgYXR0YWNoZWQgdG8gdGhlIGJyaWRnZS4NCiDvvJ4gICAgIO+8niBTbywgd2UgY2FuIGFkZCBk
b21pZCB0byB2bSdzIG5pYyBuYW1lLi4gRm9yIGV4YW1wbGUsIHRoZSB2bSdzIGRvbWlkIGlzIDEg
YW5kIHZuZXQwDQog77yeICAgICDvvJ4gaXMgcmVuYW1lZCB0byB2bmV0MS4wLg0KIO+8ng0KIO+8
niAgICAgU3VyZWx5IGRlbGV0aW5nIHRoZSBOSUMgYXV0b21hdGljYWxseSByZW1vdmVzIGl0IGZy
b20gdGhlIGJyaWRnZSBzbyB3ZQ0KIO+8niAgICAgY2FuIGp1c3QgcmVtb3ZlIHRoZSBjb2RlIHRo
YXQgZGVsZXRzIHRoZSBicmlkZ2UgcG9ydC4NCiDvvJ4NCiDvvJ4NCiDvvJ4gICAgIFJlZ2FyZHMs
DQog77yeICAgICBEYW5pZWwNCiDvvJ4gICAgIC0tDQog77yeICAgICB8OiBodHRwczovL2JlcnJh
bmdlLmNvbSAgICAgIC1vLSAgICBodHRwczovL3d3dy5mbGlja3IuY29tL3Bob3Rvcy9kYmVycmFu
Z2UNCiDvvJ4gICAgIO+8nGh0dHBzOi8vd3d3LmZsaWNrci5jb20vcGhvdG9zL2RiZXJyYW5nZe+8
niA6fA0KIO+8niAgICAgfDogaHR0cHM6Ly9saWJ2aXJ0Lm9yZyAgICAgICAgIC1vLSAgICAgICAg
ICAgIGh0dHBzOi8vZnN0b3AxMzguYmVycmFuZ2UuY29tDQog77yeICAgICDvvJxodHRwczovL2Zz
dG9wMTM4LmJlcnJhbmdlLmNvbe+8niA6fA0KIO+8niAgICAgfDogaHR0cHM6Ly9lbnRhbmdsZS1w
aG90by5vcmcgICAgLW8tICAgIGh0dHBzOi8vd3d3Lmluc3RhZ3JhbS5jb20vZGJlcnJhbmdlDQog
77yeICAgICDvvJxodHRwczovL3d3dy5pbnN0YWdyYW0uY29tL2RiZXJyYW5nZe+8niA6fA0KIO+8
ng0KIO+8niAgICAgLS0NCiDvvJ4gICAgIGxpYnZpci1saXN0IG1haWxpbmcgbGlzdA0KIO+8niAg
ICAgbGlidmlyLWxpc3RAcmVkaGF0LmNvbSDvvJxtYWlsdG86bGlidmlyLWxpc3RAcmVkaGF0LmNv
be+8ng0KIO+8niAgICAgaHR0cHM6Ly93d3cucmVkaGF0LmNvbS9tYWlsbWFuL2xpc3RpbmZvL2xp
YnZpci1saXN0DQog77yeICAgICDvvJxodHRwczovL3d3dy5yZWRoYXQuY29tL21haWxtYW4vbGlz
dGluZm8vbGlidmlyLWxpc3TvvJ4NCiDvvJ4NCiDvvJ4NCiDvvJ4NCiDvvJ4NCiDvvJ4gICAgIC0t
DQog77yeICAgICBsaWJ2aXItbGlzdCBtYWlsaW5nIGxpc3QNCiDvvJ4gICAgIGxpYnZpci1saXN0
QHJlZGhhdC5jb20g77ycbWFpbHRvOmxpYnZpci1saXN0QHJlZGhhdC5jb23vvJ4NCiDvvJ4gICAg
IGh0dHBzOi8vd3d3LnJlZGhhdC5jb20vbWFpbG1hbi9saXN0aW5mby9saWJ2aXItbGlzdA0KIA0K
DQrvvJ4gICAgIO+8nGh0dHBzOi8vd3d3LnJlZGhhdC5jb20vbWFpbG1hbi9saXN0aW5mby9saWJ2
aXItbGlzdO+8ng0KIO+8ng0KIO+8ng0KIO+8ng0KIO+8ng0KIO+8niAtLQ0KIO+8niBsaWJ2aXIt
bGlzdCBtYWlsaW5nIGxpc3QNCiDvvJ4gbGlidmlyLWxpc3RAcmVkaGF0LmNvbQ0KIO+8niBodHRw
czovL3d3dy5yZWRoYXQuY29tL21haWxtYW4vbGlzdGluZm8vbGlidmlyLWxpc3QNCiDvvJ4=
2757
days inactive
2768
days old
12 comments
5 participants
participants (5)
-
Daniel P. Berrange -
Joel Wirāmu Pauling -
Laine Stump -
lu.zhipeng＠zte.com.cn -
ZhiPeng Lu