[libvirt] [PATCH RESEND] qemu: Remove inactive vm when failed to start it
Libvirt forgets to remove inactive vm when failed to start a defined vm. That may result in residual domain in driver->domains on such condition: during the process of starting a vm, undefine it, and qemu exit because of some exception. As we undefined the vm successfully, the vm->persistent was set to 0, we will always fail to undefine it until restart libvirtd. Signed-off-by: Yi Wang <wang.yi59@zte.com.cn> --- src/qemu/qemu_driver.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/qemu/qemu_driver.c b/src/qemu/qemu_driver.c index cdb727b..af8afab 100644 --- a/src/qemu/qemu_driver.c +++ b/src/qemu/qemu_driver.c @@ -7185,6 +7185,8 @@ qemuDomainCreateWithFlags(virDomainPtr dom, unsigned int flags) endjob: qemuProcessEndJob(driver, vm); + if (ret < 0) + qemuDomainRemoveInactive(driver, vm); cleanup: virDomainObjEndAPI(&vm); -- 1.8.3.1
SGkgYWxsLA0KDQpDb3VsZCBzb21lb25lIHJldmlldyBteSBwYXRjaCBwbGVhc2U/DQoNCg0KDQoN Cj4gTGlidmlydCBmb3JnZXRzIHRvIHJlbW92ZSBpbmFjdGl2ZSB2bSB3aGVuIGZhaWxlZCB0byBz dGFydCBhIGRlZmluZWQgdm0uDQoNCj4gVGhhdCBtYXkgcmVzdWx0IGluIHJlc2lkdWFsIGRvbWFp biBpbiBkcml2ZXItPmRvbWFpbnMgb24gc3VjaCBjb25kaXRpb246DQoNCj4gZHVyaW5nIHRoZSBw cm9jZXNzIG9mIHN0YXJ0aW5nIGEgdm0sIHVuZGVmaW5lIGl0LCBhbmQgcWVtdSBleGl0IGJlY2F1 c2UNCg0KPiBvZiBzb21lIGV4Y2VwdGlvbi4gQXMgd2UgdW5kZWZpbmVkIHRoZSB2bSBzdWNjZXNz ZnVsbHksIHRoZSB2bS0+cGVyc2lzdGVudA0KDQo+IHdhcyBzZXQgdG8gMCwgd2Ugd2lsbCBhbHdh eXMgZmFpbCB0byB1bmRlZmluZSBpdCB1bnRpbCByZXN0YXJ0IGxpYnZpcnRkLg0KDQo+IA0KDQo+ IFNpZ25lZC1vZmYtYnk6IFlpIFdhbmcgPHdhbmcueWk1OUB6dGUuY29tLmNuPg0KDQo+IC0tLQ0K DQo+ICBzcmMvcWVtdS9xZW11X2RyaXZlci5jIHwgMiArKw0KDQo+ICAxIGZpbGUgY2hhbmdlZCwg MiBpbnNlcnRpb25zKCspDQoNCj4gDQoNCj4gZGlmZiAtLWdpdCBhL3NyYy9xZW11L3FlbXVfZHJp dmVyLmMgYi9zcmMvcWVtdS9xZW11X2RyaXZlci5jDQoNCj4gaW5kZXggY2RiNzI3Yi4uYWY4YWZh YiAxMDA2NDQNCg0KPiAtLS0gYS9zcmMvcWVtdS9xZW11X2RyaXZlci5jDQoNCj4gKysrIGIvc3Jj L3FlbXUvcWVtdV9kcml2ZXIuYw0KDQo+IEBAIC03MTg1LDYgKzcxODUsOCBAQCBxZW11RG9tYWlu Q3JlYXRlV2l0aEZsYWdzKHZpckRvbWFpblB0ciBkb20sIHVuc2lnbmVkIGludCBmbGFncykNCg0K PiAgDQoNCj4gICBlbmRqb2I6DQoNCj4gICAgICBxZW11UHJvY2Vzc0VuZEpvYihkcml2ZXIsIHZt KQ0KDQo+ICsgICAgaWYgKHJldCA8IDApDQoNCj4gKyAgICAgICAgcWVtdURvbWFpblJlbW92ZUlu YWN0aXZlKGRyaXZlciwgdm0pDQoNCj4gIA0KDQo+ICAgY2xlYW51cDoNCg0KPiAgICAgIHZpckRv bWFpbk9iakVuZEFQSSgmdm0pDQoNCj4gLS0gDQoNCj4gMS44LjMuMQ0KDQoNCg0KDQoNCg0KDQoN Ci0tLQ0KDQpCZXN0IHdpc2hlcw0KDQpZaSBXYW5n
On 06/25/2017 10:56 PM, Yi Wang wrote:
Libvirt forgets to remove inactive vm when failed to start a defined vm. That may result in residual domain in driver->domains on such condition: during the process of starting a vm, undefine it, and qemu exit because of some exception. As we undefined the vm successfully, the vm->persistent was set to 0, we will always fail to undefine it until restart libvirtd.
This seems to be a strange sequence of operations, but the claim is that by adding this logic to CreateWithFlags, then the problem you're facing is resolved. However, is adding this to the Create logic the right thing to do? IIUC: CreateXMLWithFlags is successful Undefine domain is successful -> Logic at top says, "if !vm->persistent", then fail -> Logic at bottom has: -> vm->persistent = 0 -> if !virDomainObjIsActive then qemuDomainRemoveInactive ... Thus if domain is active, expectation is that RemoveInactive is done when the domain is "normally" shutdown... ... time goes by... QEMU exits w/ some exception -> qemuDomainDestroyFlags isn't run -> ? Thus avoiding some libvirt logic to handle the exception (would that be qemuProcessAutoDestroyAdd assuming VIR_DOMAIN_START_AUTODESTROY was used) ? So your fix is, the next time a CreateWithFlags happens, if anything fails, then domain removal can be done with the assumption that something within ObjStart will fail eventually because Undefine deleted most traces of the guest. Of course if libvirtd restart happens, then because the config no longer exists the domain object isn't created. I wonder if perhaps (yuck) another flag is required to indicate that a domain went from persistent to !persistent in Undefine, but because it was still running the RemoveInactive wasn't called. The interesting part of that is how would the reaping happen. Seems this would be part of some qemuProcessHandle* function. Something that would be resetting vm->def->id and/or running qemuProcessStop (e.g. processing via AutoDestroy) Some more details and understanding why/what qemuProcessHandle* may (or may not be) called I think is necessary. It doesn't seem to me that domain removal for this condition should require going through Start processing, but something in that monitor event processing logic would seem to need to be taught about this possibility.
Signed-off-by: Yi Wang <wang.yi59@zte.com.cn> --- src/qemu/qemu_driver.c | 2 ++ 1 file changed, 2 insertions(+)
There's two reasons to get to the endjob: label... #1 virDomainObjIsActive (domain already active) #2 qemuDomainObjStart fails So this certainly wouldn't be right for the IsActive condition, under "normal" circumstances. And again, I don't think requiring a subsequent call to Start would be "correct" either. I am curious though if when in this condition, is the vm->def->id set? IOW: What does 'virsh list' indicate or even virsh dominfo $domain? John
diff --git a/src/qemu/qemu_driver.c b/src/qemu/qemu_driver.c index cdb727b..af8afab 100644 --- a/src/qemu/qemu_driver.c +++ b/src/qemu/qemu_driver.c @@ -7185,6 +7185,8 @@ qemuDomainCreateWithFlags(virDomainPtr dom, unsigned int flags)
endjob: qemuProcessEndJob(driver, vm); + if (ret < 0) + qemuDomainRemoveInactive(driver, vm);
cleanup: virDomainObjEndAPI(&vm);
SGkgSm9obiwNCg0KVGhhbmtzIGZvciB5b3VyIHJldmlldyENCg0KPiBUaGlzIHNlZW1zIHRvIGJl IGEgc3RyYW5nZSBzZXF1ZW5jZSBvZiBvcGVyYXRpb25zLCBidXQgdGhlIGNsYWltIGlzIHRoYXQN Cg0KPiBieSBhZGRpbmcgdGhpcyBsb2dpYyB0byBDcmVhdGVXaXRoRmxhZ3MsIHRoZW4gdGhlIHBy b2JsZW0geW91J3JlIGZhY2luZw0KDQo+IGlzIHJlc29sdmVkLiBIb3dldmVyLCBpcyBhZGRpbmcg dGhpcyB0byB0aGUgQ3JlYXRlIGxvZ2ljIHRoZSByaWdodCB0aGluZw0KDQo+IHRvIGRvPw0KDQo+ IElJVUM6DQoNClRoaXMgY29uZGl0aW9uIGlzIHJhcmUsIHNvIEkgZGVzaWduZWQgc29tZSBvcGVy YXRpb25zIHdoaWNoIGNhbiBoZWxwIHVzIHRvIHVuZGVyc3RhbmQgd2hhdCBoYXBwZW5lZDoNCg0K IyB2aXJzaCBkZWZpbmUgd2luNyAtPiBzdWNjZXNzZnVsDQoNCiMgdmlyc2ggc3RhcnQgd2luNyAm IHNsZWVwIDAuMiB2aXJzaCB1bmRlZmluZSB3aW43IC0+IHN0YXJ0IGZhaWxlZCBhbmQgdW5kZWZp bmUgc3VjY2Vzc2Z1bA0KDQogDQoNCmFuZCB3ZSBtYXkgc2VlIGxpYnZpcnRkIG91dHB1dCBzdWNo IGxvZ3MgbGlrZSB0aGlzOg0KDQpxZW11RG9tYWluRGVmaW5lWE1MRmxhZ3M6NzM4NiA6IENyZWF0 aW5nIGRvbWFpbiB3aW43DQoNCnFlbXVQcm9jZXNzU3RhcnQ6NjA4NiA6IGNvbm49Li4uLg0KDQpx ZW11RG9tYWluVW5kZWZpbmVGbGFnczo3NTAxIDogVW5kZWZpbmluZyBkb21haW4gd2luNw0KDQpl cnJvciA6IHFlbXVNb25pdG9yT3BlblVuaXg6Mzc5IDogZmFpbGVkIHRvIGNvbm5lY3QgdG8gbW9u aXRvciBzb2NrZXQNCg0KIA0KDQpMaWJ2aXJ0ZCB1bmxvY2tlZCB2bSBkdXJpbmcgcWVtdVByb2Nl c3NTdGFydCwgc28gcWVtdURvbWFpblVuZGVmaW5lRmxhZ3MgY2FuIGdldCB0aGUgbG9jayBhbmQg dW5kZWZpbmUgdm0gc3VjY2Vzc2Z1bGx5Lg0KDQpBZnRlciBmaW5pc2hpbmcgdGhlIGFib3ZlIHN0 ZXBzLCB3ZSBjYW4gZ2V0IHNvbWV0aGluZyB3cm9uZzoNCg0KIyB2aXJzaCBsaXN0IC0tYWxsDQoN CiBJZCAgICBOYW1lICAgICAgICAgICAgICAgICAgICAgICAgICAgU3RhdGUNCg0KLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQ0KDQogLSAgICAgd2lu NyAgICAgICAgICAgICAgICAgICAgICAgICAgIHNodXQgb2ZmDQoNCiANCg0KIyB2aXJzaCB1bmRl ZmluZSB3aW43DQoNCmVycm9yOiBGYWlsZWQgdG8gdW5kZWZpbmUgZG9tYWluIHdpbjcNCg0KZXJy b3I6IFJlcXVlc3RlZCBvcGVyYXRpb24gaXMgbm90IHZhbGlkOiBjYW5ub3QgdW5kZWZpbmUgdHJh bnNpZW50IGRvbWFpbg0KDQogDQoNCg0KDQoNCj4gVGhlcmUncyB0d28gcmVhc29ucyB0byBnZXQg dG8gdGhlIGVuZGpvYjogbGFiZWwuLi4NCg0KPiAjMSB2aXJEb21haW5PYmpJc0FjdGl2ZSAoZG9t YWluIGFscmVhZHkgYWN0aXZlKQ0KDQo+ICMyIHFlbXVEb21haW5PYmpTdGFydCBmYWlscw0KDQo+ IFNvIHRoaXMgY2VydGFpbmx5IHdvdWxkbid0IGJlIHJpZ2h0IGZvciB0aGUgSXNBY3RpdmUgY29u ZGl0aW9uLCB1bmRlcg0KDQo+ICJub3JtYWwiIGNpcmN1bXN0YW5jZXMuIEFuZCBhZ2FpbiwgSSBk b24ndCB0aGluayByZXF1aXJpbmcgYSBzdWJzZXF1ZW50DQoNCj4gY2FsbCB0byBTdGFydCB3b3Vs ZCBiZSAiY29ycmVjdCIgZWl0aGVyLiAgSSBhbSBjdXJpb3VzIHRob3VnaCBpZiB3aGVuIGluDQoN Cj4gdGhpcyBjb25kaXRpb24sIGlzIHRoZSB2bS0+ZGVmLT5pZCBzZXQ/ICBJT1c6IFdoYXQgZG9l cyAndmlyc2ggbGlzdCcNCg0KPiBpbmRpY2F0ZSBvciBldmVuIHZpcnNoIGRvbWluZm8gJGRvbWFp bj8NCg0KV2hhdCB5b3Ugc2FpZCBzb3VuZHMgcmVhc29uYWJsZSwgSSBjYW4gY2FsbCBxZW11RG9t YWluT2JqU3RhcnQgYXQgb25jZSBhZnRlciBxZW11RG9tYWluT2JqU3RhcnQgZmFpbHMgbGlrZSBs aWJ2aXJ0IGRvZXMgaW4gcWVtdURvbWFpbkNyZWF0ZVhNTCwgYnV0IGVuZCBqb2Igc2hvdWxkIGJl IGNhbGxlZCBiZWZvcmUgcWVtdURvbWFpblJlbW92ZUluYWN0aXZlIDop
On 07/12/2017 07:34 AM, wang.yi59@zte.com.cn wrote:
Hi John,
Thanks for your review!
Somehow your response is out of synch with the rest of the series - things like this get lost very quickly.
This seems to be a strange sequence of operations, but the claim is that
by adding this logic to CreateWithFlags, then the problem you're facing
is resolved. However, is adding this to the Create logic the right thing
to do?
IIUC:
This condition is rare, so I designed some operations which can help us to understand what happened:
# virsh define win7 -> successful
# virsh start win7 &; sleep 0.2; virsh undefine win7 -> start failed and undefine successful
and we may see libvirtd output such logs like this:
qemuDomainDefineXMLFlags:7386 : Creating domain win7
qemuProcessStart:6086 : conn=....
qemuDomainUndefineFlags:7501 : Undefining domain win7
error : qemuMonitorOpenUnix:379 : failed to connect to monitor socket
Libvirtd unlocked vm during qemuProcessStart, so qemuDomainUndefineFlags can get the lock and undefine vm successfully.
Can you be a bit more specific. Do you mean during qemuConnectMonitor processing during ProcessLaunch? Or perhaps Agent processing? In any case, perhaps then the better fix is to prohibit undefine whilst unlocked for Monitor or Agent start. The perhaps interesting question is where to set flag. Setting/clearing just around Monitor/Agent startup would be a seemingly short window and perhaps where you're seeing the virObjectUnlock; however, I wonder if perhaps all of qemuProcessLaunch should be "protected" as there's some really critical things happening in there. Thus at the top one sets a boolean "obj->starting = true" and at cleanup "obj->starting = false". In qemuDomainUndefineFlags, there'd need to be a similar check to the if (!persistent) along the lines of "if (starting)" then an error message indicating that the domain is starting up and cannot be undefined. Perhaps similarly for Destroy just to be safe. I didn't think all that long about it, but hopefully it's enough to perhaps have to generate more patches... I don't believe with that it'd be possible to have the need to qemuDomainRemoveInactive in the CreateWithFlags, but I could be wrong. John
After finishing the above steps, we can get something wrong:
# virsh list --all
Id Name State
----------------------------------------------------
- win7 shut off
# virsh undefine win7
error: Failed to undefine domain win7
error: Requested operation is not valid: cannot undefine transient domain
There's two reasons to get to the endjob: label...
#1 virDomainObjIsActive (domain already active)
#2 qemuDomainObjStart fails
So this certainly wouldn't be right for the IsActive condition, under
"normal" circumstances. And again, I don't think requiring a subsequent
call to Start would be "correct" either. I am curious though if when in
this condition, is the vm->def->id set? IOW: What does 'virsh list'
indicate or even virsh dominfo $domain?
What you said sounds reasonable, I can call qemuDomainObjStart at once after qemuDomainObjStart fails like libvirt does in qemuDomainCreateXML, but end job should be called before qemuDomainRemoveInactive :)
Pk9uIDA3LzEyLzIwMTcgMDc6MzQgQU0sIHdhbmcgeWk1OSB6dGUgY29tIGNuIHdyb3RlOg0KDQo+ PiBIaSBKb2huLA0KDQo+Pg0KDQo+PiBUaGFua3MgZm9yIHlvdXIgcmV2aWV3IQ0KDQo+Pg0KDQo+ DQoNCj5Tb21laG93IHlvdXIgcmVzcG9uc2UgaXMgb3V0IG9mIHN5bmNoIHdpdGggdGhlIHJlc3Qg b2YgdGhlIHNlcmllcyAtDQoNCj50aGluZ3MgbGlrZSB0aGlzIGdldCBsb3N0IHZlcnkgcXVpY2ts eS4NCg0KPg0KDQo+Pj4gVGhpcyBzZWVtcyB0byBiZSBhIHN0cmFuZ2Ugc2VxdWVuY2Ugb2Ygb3Bl cmF0aW9ucywgYnV0IHRoZSBjbGFpbSBpcyB0aGF0DQoNCj4+DQoNCj4+PiBieSBhZGRpbmcgdGhp cyBsb2dpYyB0byBDcmVhdGVXaXRoRmxhZ3MsIHRoZW4gdGhlIHByb2JsZW0geW91J3JlIGZhY2lu Zw0KDQo+Pg0KDQo+Pj4gaXMgcmVzb2x2ZWQuIEhvd2V2ZXIsIGlzIGFkZGluZyB0aGlzIHRvIHRo ZSBDcmVhdGUgbG9naWMgdGhlIHJpZ2h0IHRoaW5nDQoNCj4+DQoNCj4+PiB0byBkbz8NCg0KPj4N Cg0KPj4+IElJVUM6DQoNCj4+DQoNCj4+IFRoaXMgY29uZGl0aW9uIGlzIHJhcmUsIHNvIEkgZGVz aWduZWQgc29tZSBvcGVyYXRpb25zIHdoaWNoIGNhbiBoZWxwIHVzDQoNCj4+IHRvIHVuZGVyc3Rh bmQgd2hhdCBoYXBwZW5lZDoNCg0KPj4NCg0KPj4gIyB2aXJzaCBkZWZpbmUgd2luNyAtPiBzdWNj ZXNzZnVsDQoNCj4+DQoNCj4+ICMgdmlyc2ggc3RhcnQgd2luNyAmIHNsZWVwIDAuMiB2aXJzaCB1 bmRlZmluZSB3aW43IC0+IHN0YXJ0IGZhaWxlZCBhbmQNCg0KPj4gdW5kZWZpbmUgc3VjY2Vzc2Z1 bA0KDQo+Pg0KDQo+Pg0KDQo+Pg0KDQo+PiBhbmQgd2UgbWF5IHNlZSBsaWJ2aXJ0ZCBvdXRwdXQg c3VjaCBsb2dzIGxpa2UgdGhpczoNCg0KPj4NCg0KPj4gcWVtdURvbWFpbkRlZmluZVhNTEZsYWdz OjczODYgOiBDcmVhdGluZyBkb21haW4gd2luNw0KDQo+Pg0KDQo+PiBxZW11UHJvY2Vzc1N0YXJ0 OjYwODYgOiBjb25uPS4uLi4NCg0KPj4NCg0KPj4gcWVtdURvbWFpblVuZGVmaW5lRmxhZ3M6NzUw MSA6IFVuZGVmaW5pbmcgZG9tYWluIHdpbjcNCg0KPj4NCg0KPj4gZXJyb3IgOiBxZW11TW9uaXRv ck9wZW5Vbml4OjM3OSA6IGZhaWxlZCB0byBjb25uZWN0IHRvIG1vbml0b3Igc29ja2V0DQoNCj4+ DQoNCj4+DQoNCj4+DQoNCj4+IExpYnZpcnRkIHVubG9ja2VkIHZtIGR1cmluZyBxZW11UHJvY2Vz c1N0YXJ0LCBzbyBxZW11RG9tYWluVW5kZWZpbmVGbGFncw0KDQo+PiBjYW4gZ2V0IHRoZSBsb2Nr IGFuZCB1bmRlZmluZSB2bSBzdWNjZXNzZnVsbHkuDQoNCj4NCg0KPkNhbiB5b3UgYmUgYSBiaXQg bW9yZSBzcGVjaWZpYy4gRG8geW91IG1lYW4gZHVyaW5nIHFlbXVDb25uZWN0TW9uaXRvcg0KDQo+ cHJvY2Vzc2luZyBkdXJpbmcgUHJvY2Vzc0xhdW5jaD8gT3IgcGVyaGFwcyBBZ2VudCBwcm9jZXNz aW5nPw0KDQoNCg0KDQpJdCBmYWlsZWQgaW4gcHJvY2Vzc2luZyBkdXJpbmcgUHJvY2Vzc0xhdW5j aCwgb24gd2hpY2ggcGVyaW9kIHFlbXUgZXhpdGVkDQoNCnVuZXhwZWN0ZWRseS4NCg0KDQoNCg0K Pg0KDQo+SW4gYW55IGNhc2UsIHBlcmhhcHMgdGhlbiB0aGUgYmV0dGVyIGZpeCBpcyB0byBwcm9o aWJpdCB1bmRlZmluZSB3aGlsc3QNCg0KPnVubG9ja2VkIGZvciBNb25pdG9yIG9yIEFnZW50IHN0 YXJ0LiBUaGUgcGVyaGFwcyBpbnRlcmVzdGluZyBxdWVzdGlvbiBpcw0KDQo+d2hlcmUgdG8gc2V0 IGZsYWcuIFNldHRpbmcvY2xlYXJpbmcganVzdCBhcm91bmQgTW9uaXRvci9BZ2VudCBzdGFydHVw DQoNCj53b3VsZCBiZSBhIHNlZW1pbmdseSBzaG9ydCB3aW5kb3cgYW5kIHBlcmhhcHMgd2hlcmUg eW91J3JlIHNlZWluZyB0aGUNCg0KPnZpck9iamVjdFVubG9jayBob3dldmVyLCBJIHdvbmRlciBp ZiBwZXJoYXBzIGFsbCBvZiBxZW11UHJvY2Vzc0xhdW5jaA0KDQo+c2hvdWxkIGJlICJwcm90ZWN0 ZWQiIGFzIHRoZXJlJ3Mgc29tZSByZWFsbHkgY3JpdGljYWwgdGhpbmdzIGhhcHBlbmluZw0KDQo+ aW4gdGhlcmUuIFRodXMgYXQgdGhlIHRvcCBvbmUgc2V0cyBhIGJvb2xlYW4gIm9iai0+c3RhcnRp bmcgPSB0cnVlIiBhbmQNCg0KPmF0IGNsZWFudXAgIm9iai0+c3RhcnRpbmcgPSBmYWxzZSIuDQoN Cj4NCg0KPkluIHFlbXVEb21haW5VbmRlZmluZUZsYWdzLCB0aGVyZSdkIG5lZWQgdG8gYmUgYSBz aW1pbGFyIGNoZWNrIHRvIHRoZSBpZg0KDQo+KCFwZXJzaXN0ZW50KSBhbG9uZyB0aGUgbGluZXMg b2YgImlmIChzdGFydGluZykiIHRoZW4gYW4gZXJyb3IgbWVzc2FnZQ0KDQo+aW5kaWNhdGluZyB0 aGF0IHRoZSBkb21haW4gaXMgc3RhcnRpbmcgdXAgYW5kIGNhbm5vdCBiZSB1bmRlZmluZWQuDQoN Cj5QZXJoYXBzIHNpbWlsYXJseSBmb3IgRGVzdHJveSBqdXN0IHRvIGJlIHNhZmUuDQoNCg0KDQoN ClllcywgeW91ciBhZHZpY2UgbG9va3MgYmV0dGVyLiBTbyBpdCBzZWVtcyBsaWtlIHRoYXQgd2Ug c2hvdWxkIHByb3RlY3QgYWxsDQoNCm9mIHFlbXVQcm9jZXNzU3RhcnQsICdjYXVzZSBzb21lIGZ1 bmN0aW9ucyBzdWNoIGxpa2UgcWVtdVByb2Nlc3NGaW5pc2hTdGFydHVwDQoNCm1heSBhbHNvIGlu dm9rZSBNb25pdG9yLg0KDQpJZiB0aGlzIHdvcmtzLCBJIHdpbGwgc2VuZCB0aGUgVjIgcGF0Y2gu DQoNCg0KDQoNCj4NCg0KPkkgZGlkbid0IHRoaW5rIGFsbCB0aGF0IGxvbmcgYWJvdXQgaXQsIGJ1 dCBob3BlZnVsbHkgaXQncyBlbm91Z2ggdG8NCg0KPnBlcmhhcHMgaGF2ZSB0byBnZW5lcmF0ZSBt b3JlIHBhdGNoZXMuLi4gIEkgZG9uJ3QgYmVsaWV2ZSB3aXRoIHRoYXQgaXQnZA0KDQo+YmUgcG9z c2libGUgdG8gaGF2ZSB0aGUgbmVlZCB0byBxZW11RG9tYWluUmVtb3ZlSW5hY3RpdmUgaW4gdGhl DQoNCj5DcmVhdGVXaXRoRmxhZ3MsIGJ1dCBJIGNvdWxkIGJlIHdyb25nLg0KDQoNCg0KDQotLS0N Cg0KQmVzdCB3aXNoZXMNCg0KWWkgV2FuZw==
SGkgSm9obiwNCg0KSSBoYXZlIHJld29ya2VkIHRoaXMgcGF0Y2ggb24gdGhlIG90aGVyIG5ldyB0 aHJlYWRbMV0sIHBsZWFzZSByZXZpZXcgdGhhdA0KDQpwYXRjaC4NCg0KVGhhbmsgeW91IDotKQ0K DQoNCg0KDQpbMV0gaHR0cHM6Ly93d3cucmVkaGF0LmNvbS9hcmNoaXZlcy9saWJ2aXItbGlzdC8y MDE3LUp1bHkvbXNnMDA5MjEuaHRtbA0KDQoNCg0KDQo+T24gMDcvMTIvMjAxNyAwNzozNCBBTSwg d2FuZy55aTU5QHp0ZS5jb20uY24gd3JvdGU6DQoNCj4+IEhpIEpvaG4sDQoNCj4+IA0KDQo+PiBU aGFua3MgZm9yIHlvdXIgcmV2aWV3IQ0KDQo+PiANCg0KPg0KDQo+U29tZWhvdyB5b3VyIHJlc3Bv bnNlIGlzIG91dCBvZiBzeW5jaCB3aXRoIHRoZSByZXN0IG9mIHRoZSBzZXJpZXMgLQ0KDQo+dGhp bmdzIGxpa2UgdGhpcyBnZXQgbG9zdCB2ZXJ5IHF1aWNrbHkuDQoNCj4NCg0KPj4+IFRoaXMgc2Vl bXMgdG8gYmUgYSBzdHJhbmdlIHNlcXVlbmNlIG9mIG9wZXJhdGlvbnMsIGJ1dCB0aGUgY2xhaW0g aXMgdGhhdA0KDQo+PiANCg0KPj4+IGJ5IGFkZGluZyB0aGlzIGxvZ2ljIHRvIENyZWF0ZVdpdGhG bGFncywgdGhlbiB0aGUgcHJvYmxlbSB5b3UncmUgZmFjaW5nDQoNCj4+IA0KDQo+Pj4gaXMgcmVz b2x2ZWQuIEhvd2V2ZXIsIGlzIGFkZGluZyB0aGlzIHRvIHRoZSBDcmVhdGUgbG9naWMgdGhlIHJp Z2h0IHRoaW5nDQoNCj4+IA0KDQo+Pj4gdG8gZG8/DQoNCj4+IA0KDQo+Pj4gSUlVQzoNCg0KPj4g DQoNCj4+IFRoaXMgY29uZGl0aW9uIGlzIHJhcmUsIHNvIEkgZGVzaWduZWQgc29tZSBvcGVyYXRp b25zIHdoaWNoIGNhbiBoZWxwIHVzDQoNCj4+IHRvIHVuZGVyc3RhbmQgd2hhdCBoYXBwZW5lZDoN Cg0KPj4gDQoNCj4+ICMgdmlyc2ggZGVmaW5lIHdpbjcgLT4gc3VjY2Vzc2Z1bA0KDQo+PiANCg0K Pj4gIyB2aXJzaCBzdGFydCB3aW43ICYgc2xlZXAgMC4yIHZpcnNoIHVuZGVmaW5lIHdpbjcgLT4g c3RhcnQgZmFpbGVkIGFuZA0KDQo+PiB1bmRlZmluZSBzdWNjZXNzZnVsDQoNCj4+IA0KDQo+PiAg DQoNCj4+IA0KDQo+PiBhbmQgd2UgbWF5IHNlZSBsaWJ2aXJ0ZCBvdXRwdXQgc3VjaCBsb2dzIGxp a2UgdGhpczoNCg0KPj4gDQoNCj4+IHFlbXVEb21haW5EZWZpbmVYTUxGbGFnczo3Mzg2IDogQ3Jl YXRpbmcgZG9tYWluIHdpbjcNCg0KPj4gDQoNCj4+IHFlbXVQcm9jZXNzU3RhcnQ6NjA4NiA6IGNv bm49Li4uLg0KDQo+PiANCg0KPj4gcWVtdURvbWFpblVuZGVmaW5lRmxhZ3M6NzUwMSA6IFVuZGVm aW5pbmcgZG9tYWluIHdpbjcNCg0KPj4gDQoNCj4+IGVycm9yIDogcWVtdU1vbml0b3JPcGVuVW5p eDozNzkgOiBmYWlsZWQgdG8gY29ubmVjdCB0byBtb25pdG9yIHNvY2tldA0KDQo+PiANCg0KPj4g IA0KDQo+PiANCg0KPj4gTGlidmlydGQgdW5sb2NrZWQgdm0gZHVyaW5nIHFlbXVQcm9jZXNzU3Rh cnQsIHNvIHFlbXVEb21haW5VbmRlZmluZUZsYWdzDQoNCj4+IGNhbiBnZXQgdGhlIGxvY2sgYW5k IHVuZGVmaW5lIHZtIHN1Y2Nlc3NmdWxseS4NCg0KPg0KDQo+Q2FuIHlvdSBiZSBhIGJpdCBtb3Jl IHNwZWNpZmljLiBEbyB5b3UgbWVhbiBkdXJpbmcgcWVtdUNvbm5lY3RNb25pdG9yDQoNCj5wcm9j ZXNzaW5nIGR1cmluZyBQcm9jZXNzTGF1bmNoPyBPciBwZXJoYXBzIEFnZW50IHByb2Nlc3Npbmc/ DQoNCj4NCg0KPkluIGFueSBjYXNlLCBwZXJoYXBzIHRoZW4gdGhlIGJldHRlciBmaXggaXMgdG8g cHJvaGliaXQgdW5kZWZpbmUgd2hpbHN0DQoNCj51bmxvY2tlZCBmb3IgTW9uaXRvciBvciBBZ2Vu dCBzdGFydC4gVGhlIHBlcmhhcHMgaW50ZXJlc3RpbmcgcXVlc3Rpb24gaXMNCg0KPndoZXJlIHRv IHNldCBmbGFnLiBTZXR0aW5nL2NsZWFyaW5nIGp1c3QgYXJvdW5kIE1vbml0b3IvQWdlbnQgc3Rh cnR1cA0KDQo+d291bGQgYmUgYSBzZWVtaW5nbHkgc2hvcnQgd2luZG93IGFuZCBwZXJoYXBzIHdo ZXJlIHlvdSdyZSBzZWVpbmcgdGhlDQoNCj52aXJPYmplY3RVbmxvY2sgaG93ZXZlciwgSSB3b25k ZXIgaWYgcGVyaGFwcyBhbGwgb2YgcWVtdVByb2Nlc3NMYXVuY2gNCg0KPnNob3VsZCBiZSAicHJv dGVjdGVkIiBhcyB0aGVyZSdzIHNvbWUgcmVhbGx5IGNyaXRpY2FsIHRoaW5ncyBoYXBwZW5pbmcN Cg0KPmluIHRoZXJlLiBUaHVzIGF0IHRoZSB0b3Agb25lIHNldHMgYSBib29sZWFuICJvYmotPnN0 YXJ0aW5nID0gdHJ1ZSIgYW5kDQoNCj5hdCBjbGVhbnVwICJvYmotPnN0YXJ0aW5nID0gZmFsc2Ui Lg0KDQo+DQoNCj5JbiBxZW11RG9tYWluVW5kZWZpbmVGbGFncywgdGhlcmUnZCBuZWVkIHRvIGJl IGEgc2ltaWxhciBjaGVjayB0byB0aGUgaWYNCg0KPighcGVyc2lzdGVudCkgYWxvbmcgdGhlIGxp bmVzIG9mICJpZiAoc3RhcnRpbmcpIiB0aGVuIGFuIGVycm9yIG1lc3NhZ2UNCg0KPmluZGljYXRp bmcgdGhhdCB0aGUgZG9tYWluIGlzIHN0YXJ0aW5nIHVwIGFuZCBjYW5ub3QgYmUgdW5kZWZpbmVk Lg0KDQo+UGVyaGFwcyBzaW1pbGFybHkgZm9yIERlc3Ryb3kganVzdCB0byBiZSBzYWZlLg0KDQo+ DQoNCj5JIGRpZG4ndCB0aGluayBhbGwgdGhhdCBsb25nIGFib3V0IGl0LCBidXQgaG9wZWZ1bGx5 IGl0J3MgZW5vdWdoIHRvDQoNCj5wZXJoYXBzIGhhdmUgdG8gZ2VuZXJhdGUgbW9yZSBwYXRjaGVz Li4uICBJIGRvbid0IGJlbGlldmUgd2l0aCB0aGF0IGl0J2QNCg0KPmJlIHBvc3NpYmxlIHRvIGhh dmUgdGhlIG5lZWQgdG8gcWVtdURvbWFpblJlbW92ZUluYWN0aXZlIGluIHRoZQ0KDQo+Q3JlYXRl V2l0aEZsYWdzLCBidXQgSSBjb3VsZCBiZSB3cm9uZy4NCg0KDQoNCg0KDQoNCg0KLS0tDQoNCkJl c3Qgd2lzaGVzDQoNCllpIFdhbmc=
participants (3)
-
John Ferlan -
wang.yi59＠zte.com.cn -
Yi Wang