[libvirt] [PATCH] rpc : fix a access for null pointer
virNetSocketRemoveIOCallback get sock's ObjectLock and will call virNetSocketEventFree. virNetSocketEventFree may be free sock object and virNetSocketRemoveIOCallback will access a null pointer in release sock's ObjectLock. Signed-off-by: Liu Yun <liu.yunh@zte.com.cn> Signed-off-by: Peng Hao <peng.hao2@zte.com.cn> --- src/rpc/virnetsocket.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/src/rpc/virnetsocket.c b/src/rpc/virnetsocket.c index d228c8a..8b550e8 100644 --- a/src/rpc/virnetsocket.c +++ b/src/rpc/virnetsocket.c @@ -2140,14 +2140,12 @@ static void virNetSocketEventFree(void *opaque) virFreeCallback ff; void *eopaque; - virObjectLock(sock); ff = sock->ff; eopaque = sock->opaque; sock->func = NULL; sock->ff = NULL; sock->opaque = NULL; - virObjectUnlock(sock); - + if (ff) ff(eopaque); @@ -2207,6 +2205,7 @@ void virNetSocketUpdateIOCallback(virNetSocketPtr sock, void virNetSocketRemoveIOCallback(virNetSocketPtr sock) { + virObjectRef(sock); virObjectLock(sock); if (sock->watch < 0) { @@ -2220,6 +2219,7 @@ void virNetSocketRemoveIOCallback(virNetSocketPtr sock) sock->watch = -1; virObjectUnlock(sock); + virObjectRef(sock); } void virNetSocketClose(virNetSocketPtr sock) -- 1.8.3.1
On 07/15/2017 05:00 PM, Peng Hao wrote:
virNetSocketRemoveIOCallback get sock's ObjectLock and will call virNetSocketEventFree. virNetSocketEventFree may be free sock object and virNetSocketRemoveIOCallback will access a null pointer in release sock's ObjectLock.
Signed-off-by: Liu Yun <liu.yunh@zte.com.cn> Signed-off-by: Peng Hao <peng.hao2@zte.com.cn> --- src/rpc/virnetsocket.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
I don't think this can work.
diff --git a/src/rpc/virnetsocket.c b/src/rpc/virnetsocket.c index d228c8a..8b550e8 100644 --- a/src/rpc/virnetsocket.c +++ b/src/rpc/virnetsocket.c @@ -2140,14 +2140,12 @@ static void virNetSocketEventFree(void *opaque) virFreeCallback ff; void *eopaque;
- virObjectLock(sock); ff = sock->ff; eopaque = sock->opaque; sock->func = NULL; sock->ff = NULL; sock->opaque = NULL; - virObjectUnlock(sock);
I think we need the lock here. This function is called from the event loop thread. So even if virNetSocketUpdateIOCallback() locks the @socket this code can see it unlocked. Or locked. But the crucial part is it's modifying the object and thus should have lock held.
- + if (ff) ff(eopaque);
@@ -2207,6 +2205,7 @@ void virNetSocketUpdateIOCallback(virNetSocketPtr sock,
void virNetSocketRemoveIOCallback(virNetSocketPtr sock) { + virObjectRef(sock); virObjectLock(sock);
I think this is what actually fixes your problem. However, I also think it introduces uneven ratio of ref:unref calls.
if (sock->watch < 0) { @@ -2220,6 +2219,7 @@ void virNetSocketRemoveIOCallback(virNetSocketPtr sock) sock->watch = -1;
virObjectUnlock(sock); + virObjectRef(sock);
It definitely does so because you ref twice. Anyway, do you perhaps have a backtrace to share? Michal
SGkgTWljaGFsLA0KDQogICAgVGhpcyBwcm9ibGVtIGlzIHRyaWdnZXJyZWQgYnkgbGlidmlydCBw eXRob24ncyBleGFtcGxlIGV2ZW50LXRlc3QucHkuIHRoZSBvcmlnaW5hbCBleGFtcGxlcyBoYXMg cmVzb3VjZSBsZWFrIGlzc3VlDQoNCmF0IHRoZSByZW1vdmVfaGFuZGxlIGFuZCByZW1vdmVfdGlt ZXIuIA0KDQogICAgd2l0aCAicHl0aG9uIC11IGV2ZW50LXRlc3QucHkiIHJ1biB0aGlzIGV4YW1w bGUgYW5kICJzeXN0ZW1jdGwgcmVzdGFydCBsaWJ2aXJ0ZC5zZXJ2aWNlIiB3aWxsIHRyaWdnZXIg cmVzb3VyY2UgbGVhayBwcm9ibGVtLg0KDQp3aXRoIGxzb2YgLXAgPGV2ZW50LXRlc3QucGlkPiBj YW4gc2VlIHNvY2tldCBoYW5kbGVyJ3MgbnVtYmVyIGluY3JlYXNlZCAsIGFmdGVyIHJlc3RhcnQg bGlidmlydGQuc2VyaXZjZSBlYWNoIHRpbWUuDQoNCiAgICB0aGUgcmVhc29uIGlzIHJlbW92ZV9o YW5kbGUgYW5kIHJlbW92ZV90aW1lciBkbyBub3QgcmV0dXJuIHRoZSByZW1vdmUgaGFuZGxlIGlu Zm9ybWF0aW9uIHRvIGxpYnZpcnQtcHl0aG9uJ3MgZnJhbWV3b3JrLiANCg0KbGl0dGxlIHBhdGNo IHdhcyBhcHBseSB0byB0aGlzIGV4YW1wbGUsIHRvIGZpeCB0aGlzIHByb2JsZW0uDQoNCiAgIE5v dywgcnVuIHRoaXMgZXhhbXBsZSBhZ2FpbiBhbmQgcmVzdGFydCBsaWJ2aXJ0ZC5zZXJ2aWNlICwg Y2FsbCBzZXF1ZW5jZSB2aXJOZXRTb2NrZXRSZW1vdmVJT0NhbGxiYWNrLT52aXJOZXRTb2NrZXRF dmVudEZyZWUgDQoNCmNhbiBiZSBvYnNlcnZlZCAsIHRoZSBuby1yZWN1cnNpdmUgbXV0ZXgsIGxv Y2sgd2l0aCByZWN1cnNpdmUgaXNzdWUgY2FuIGJlIHNlZW4uIA0KDQogICAgeW91IGNhbiBjaGVj ayB0aGUgZGV0YWlsIHN0YWNrIHRyYWNlIGFuZCBvdXIgY29tbWVudHMgYWJvdXQgdGhlIGxvY2sn cyBpc3N1ZSBpbiBmdW5jdGlvbiB2aXJOZXRTb2NrZXRFdmVudEZyZWUgIGluIHRoZSBmb2xsb3dp bmcuDQoNCg0KDQoNCiAgPT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT0gIA0KDQogZGVmIGFkZF90aW1lcihzZWxmLCBpbnRl cnZhbCwgY2IsIG9wYXF1ZSk6DQoNCiAgICAgICAgdGltZXJJRCA9IHNlbGYubmV4dFRpbWVySUQg KyAxDQoNCiAgICAgICAgc2VsZi5uZXh0VGltZXJJRCA9IHNlbGYubmV4dFRpbWVySUQgKyAxDQoN Cg0KDQoNCiAgICAgICAgaCA9IHNlbGYudmlyRXZlbnRMb29wUHVyZVRpbWVyKHRpbWVySUQsIGlu dGVydmFsLCBjYiwgb3BhcXVlKQ0KDQogICAgICAgIHNlbGYudGltZXJzLmFwcGVuZChoKQ0KDQot ICAgICAgIHNlbGYudGltZXJzX29wYXF1ZVt0aW1lcklEXSA9IG9wYXF1ZQ0KDQogICAgICAgIHNl bGYuaW50ZXJydXB0KCkNCg0KDQoNCg0KICAgICAgICBkZWJ1ZygiQWRkIHRpbWVyICVkIGludGVy dmFsICVkIiAlICh0aW1lcklELCBpbnRlcnZhbCkpDQoNCg0KDQoNCiAgICAgICAgcmV0dXJuIHRp bWVySUQNCg0KDQoNCg0KDQoNCg0KICAgICBkZWYgcmVtb3ZlX2hhbmRsZShzZWxmLCBoYW5kbGVJ RCk6DQoNCiAgICAgICAgIGhhbmRsZXMgPSBbXQ0KDQotICAgICAgICBvcGFxdWUgPSBOb25lDQoN CiAgICAgICAgIGZvciBoIGluIHNlbGYuaGFuZGxlczoNCg0KICAgICAgICAgICAgIGlmIGguZ2V0 X2lkKCkgPT0gaGFuZGxlSUQ6DQoNCiAgICAgICAgICAgICAgICAgc2VsZi5wb2xsLnVucmVnaXN0 ZXIoaC5nZXRfZmQoKSkNCg0KLSAgICAgICAgICAgICAgICBvcGFxdWUgPSBzZWxmLm9wYXF1ZXNb aGFuZGxlSURdDQoNCi0gICAgICAgICAgICAgICAgZGVsIHNlbGYub3BhcXVlc1toYW5kbGVJRF0N Cg0KICAgICAgICAgICAgICAgICBkZWJ1ZygiUmVtb3ZlIGhhbmRsZSAlZCBmZCAlZCIgJSAoaGFu ZGxlSUQsIGguZ2V0X2ZkKCkpKQ0KDQogICAgICAgICAgICAgZWxzZToNCg0KICAgICAgICAgICAg ICAgICBoYW5kbGVzLmFwcGVuZChoKQ0KDQogICAgICAgICBzZWxmLmhhbmRsZXMgPSBoYW5kbGVz DQoNCiAgICAgICAgIHNlbGYuaW50ZXJydXB0KCkNCg0KLSAgICAgICAgcmV0dXJuIG9wYXF1ZQ0K DQoNCg0KDQogICAgICMgU3RvcCBmaXJpbmcgdGhlIHBlcmlvZGljIHRpbWVyDQoNCiAgICAgZGVm IHJlbW92ZV90aW1lcihzZWxmLCB0aW1lcklEKToNCg0KICAgICAgICAgdGltZXJzID0gW10NCg0K LSAgICAgICAgb3BhcXVlID0gTm9uZQ0KDQogICAgICAgICBmb3IgaCBpbiBzZWxmLnRpbWVyczoN Cg0KICAgICAgICAgICAgIGlmIGguZ2V0X2lkKCkgIT0gdGltZXJJRDoNCg0KICAgICAgICAgICAg ICAgICB0aW1lcnMuYXBwZW5kKGgpDQoNCi0gICAgICAgICAgICBlbHNlOg0KDQotICAgICAgICAg ICAgICAgIG9wYXF1ZSA9IHNlbGYudGltZXJzX29wYXF1ZVt0aW1lcklEXQ0KDQogICAgICAgICAg ICAgICAgIGRlYnVnKCJSZW1vdmUgdGltZXIgJWQiICUgdGltZXJJRCkNCg0KICAgICAgICAgc2Vs Zi50aW1lcnMgPSB0aW1lcnMNCg0KICAgICAgICAgc2VsZi5pbnRlcnJ1cHQoKQ0KDQotICAgICAg ICByZXR1cm4gb3BhcXVlDQoNCj09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PQ0KDQoNCg0KDQoN Cj4+T24gMDcvMTUvMjAxNyAwNTowMCBQTSwgUGVuZyBIYW8gd3JvdGU6DQoNCj4+IHZpck5ldFNv Y2tldFJlbW92ZUlPQ2FsbGJhY2sgZ2V0IHNvY2sncyBPYmplY3RMb2NrIGFuZCB3aWxsIGNhbGwN Cg0KPj4gdmlyTmV0U29ja2V0RXZlbnRGcmVlLiB2aXJOZXRTb2NrZXRFdmVudEZyZWUgbWF5IGJl IGZyZWUgc29jaw0KDQo+PiBvYmplY3QgYW5kIHZpck5ldFNvY2tldFJlbW92ZUlPQ2FsbGJhY2sg d2lsbCBhY2Nlc3MgYSBudWxsIHBvaW50ZXINCg0KPj4gaW4gcmVsZWFzZSBzb2NrJ3MgT2JqZWN0 TG9jay4NCg0KPj4gDQoNCj4+IFNpZ25lZC1vZmYtYnk6IExpdSBZdW4gPGxpdS55dW5oQHp0ZS5j b20uY24+DQoNCj4+IFNpZ25lZC1vZmYtYnk6IFBlbmcgSGFvIDxwZW5nLmhhbzJAenRlLmNvbS5j bj4NCg0KPj4gLS0tDQoNCj4+ICBzcmMvcnBjL3Zpcm5ldHNvY2tldC5jIHwgNiArKystLS0NCg0K Pj4gIDEgZmlsZSBjaGFuZ2VkLCAzIGluc2VydGlvbnMoKyksIDMgZGVsZXRpb25zKC0pDQoNCj4+ IA0KDQo+DQoNCj5JIGRvbid0IHRoaW5rIHRoaXMgY2FuIHdvcmsuDQoNCj4NCg0KPj4gZGlmZiAt LWdpdCBhL3NyYy9ycGMvdmlybmV0c29ja2V0LmMgYi9zcmMvcnBjL3Zpcm5ldHNvY2tldC5jDQoN Cj4+IGluZGV4IGQyMjhjOGEuLjhiNTUwZTggMTAwNjQ0DQoNCj4+IC0tLSBhL3NyYy9ycGMvdmly bmV0c29ja2V0LmMNCg0KPj4gKysrIGIvc3JjL3JwYy92aXJuZXRzb2NrZXQuYw0KDQo+PiBAQCAt MjE0MCwxNCArMjE0MCwxMiBAQCBzdGF0aWMgdm9pZCB2aXJOZXRTb2NrZXRFdmVudEZyZWUodm9p ZCAqb3BhcXVlKQ0KDQo+PiAgICAgIHZpckZyZWVDYWxsYmFjayBmZg0KDQo+PiAgICAgIHZvaWQg KmVvcGFxdWUNCg0KPj4gIA0KDQo+PiAtICAgIHZpck9iamVjdExvY2soc29jaykNCg0KPj4gICAg ICBmZiA9IHNvY2stPmZmDQoNCj4+ICAgICAgZW9wYXF1ZSA9IHNvY2stPm9wYXF1ZQ0KDQo+PiAg ICAgIHNvY2stPmZ1bmMgPSBOVUxMDQoNCj4+ICAgICAgc29jay0+ZmYgPSBOVUxMDQoNCj4+ICAg ICAgc29jay0+b3BhcXVlID0gTlVMTA0KDQo+PiAtICAgIHZpck9iamVjdFVubG9jayhzb2NrKQ0K DQo+DQoNCj5JIHRoaW5rIHdlIG5lZWQgdGhlIGxvY2sgaGVyZS4gVGhpcyBmdW5jdGlvbiBpcyBj YWxsZWQgZnJvbSB0aGUgZXZlbnQNCg0KPmxvb3AgdGhyZWFkLiBTbyBldmVuIGlmIHZpck5ldFNv Y2tldFVwZGF0ZUlPQ2FsbGJhY2soKSBsb2NrcyB0aGUgQHNvY2tldA0KDQo+dGhpcyBjb2RlIGNh biBzZWUgaXQgdW5sb2NrZWQuIE9yIGxvY2tlZC4gQnV0IHRoZSBjcnVjaWFsIHBhcnQgaXMgaXQn cw0KDQo+bW9kaWZ5aW5nIHRoZSBvYmplY3QgYW5kIHRodXMgc2hvdWxkIGhhdmUgbG9jayBoZWxk Lg0KDQo+DQoNCiAgIEkgaGF2ZSBjaGVjayB0aGUgY29kZSAsIGluIGRlZmF1bHQgaW1wbGVtZW50 YXRpb24gb2YgZXZlbnRQb2xsLCB2aXJFdmVudFBvbGxSdW5PbmNlIGFsd2F5cyBkaXNwYXRjaCBh bmQgY2xlYXIgaW4gb25lIHRocmVhZCBsb29wLA0KDQpzbywgdGhlIGxvY2sgaW4gdGhlIHZpck5l dFNvY2tldEV2ZW50RnJlZSBtYXkgYmUgdW5uZXNzYXJ5Lg0KDQoNCg0KDQo+PiAtDQoNCj4+ICsg IA0KDQo+PiAgICAgIGlmIChmZikNCg0KPj4gICAgICAgICAgZmYoZW9wYXF1ZSkNCg0KPj4gIA0K DQo+PiBAQCAtMjIwNyw2ICsyMjA1LDcgQEAgdm9pZCB2aXJOZXRTb2NrZXRVcGRhdGVJT0NhbGxi YWNrKHZpck5ldFNvY2tldFB0ciBzb2NrLA0KDQo+PiAgDQoNCj4+ICB2b2lkIHZpck5ldFNvY2tl dFJlbW92ZUlPQ2FsbGJhY2sodmlyTmV0U29ja2V0UHRyIHNvY2spDQoNCj4+ICB7DQoNCj4+ICsg ICAgdmlyT2JqZWN0UmVmKHNvY2spDQoNCg0KDQoNClRoaXMgc2hvdWxkIGJlIG1pc3Rha2Ugd2hl biBnZW5lcmF0ZSB0aGUgcGF0Y2guIFRoZSBjb3JyZWN0IG9uZSBpcyANCg0KICAgICArICAgIHZp ck9iamVjdFVucmVmKHNvY2spDQoNCj4+ICAgICAgdmlyT2JqZWN0TG9jayhzb2NrKQ0KDQo+DQoN Cj5JIHRoaW5rIHRoaXMgaXMgd2hhdCBhY3R1YWxseSBmaXhlcyB5b3VyIHByb2JsZW0uIEhvd2V2 ZXIsIEkgYWxzbyB0aGluaw0KDQo+aXQgaW50cm9kdWNlcyB1bmV2ZW4gcmF0aW8gb2YgcmVmOnVu cmVmIGNhbGxzLg0KDQo+DQoNCj4+ICANCg0KPj4gICAgICBpZiAoc29jay0+d2F0Y2ggPCAwKSB7 DQoNCj4+IEBAIC0yMjIwLDYgKzIyMTksNyBAQCB2b2lkIHZpck5ldFNvY2tldFJlbW92ZUlPQ2Fs bGJhY2sodmlyTmV0U29ja2V0UHRyIHNvY2spDQoNCj4+ICAgICAgc29jay0+d2F0Y2ggPSAtMQ0K DQo+PiAgDQoNCj4+ICAgICAgdmlyT2JqZWN0VW5sb2NrKHNvY2spDQoNCj4+ICsgICAgdmlyT2Jq ZWN0UmVmKHNvY2spDQoNCg0KDQoNCg0KVGhpcyBzaG91bGQgYmUgbWlzdGFrZSB3aGVuIGdlbmVy YXRlIHRoZSBwYXRjaC4gVGhlIGNvcnJlY3Qgb25lIGlzIA0KDQoNCiAgICAgKyAgICB2aXJPYmpl Y3RVbnJlZihzb2NrKQ0KDQo+DQoNCj5JdCBkZWZpbml0ZWx5IGRvZXMgc28gYmVjYXVzZSB5b3Ug cmVmIHR3aWNlLiBBbnl3YXksIGRvIHlvdSBwZXJoYXBzIGhhdmUNCg0KPmEgYmFja3RyYWNlIHRv IHNoYXJlPw0KDQogICAgDQoNCiMwICBfX2xsbF9sb2NrX3dhaXQgKCkNCg0KICAgIGF0IC4uL25w dGwvc3lzZGVwcy91bml4L3N5c3YvbGludXgveDg2XzY0L2xvd2xldmVsbG9jay5TOjEzNQ0KDQoj MSAgMHgwMDAwN2ZkZTYyMDdjZDAyIGluIF9MX2xvY2tfNzkxICgpIGZyb20gL2xpYjY0L2xpYnB0 aHJlYWQuc28uMA0KDQojMiAgMHgwMDAwN2ZkZTYyMDdjYzA4IGluIF9fR0lfX19wdGhyZWFkX211 dGV4X2xvY2sgKA0KDQogICAgbXV0ZXg9bXV0ZXhAZW50cnk9MHgxMTljM2UwKSBhdCBwdGhyZWFk X211dGV4X2xvY2suYzo2NA0KDQojMyAgMHgwMDAwN2ZkZTVhOTdlZTE1IGluIHZpck11dGV4TG9j ayAobT1tQGVudHJ5PTB4MTE5YzNlMCkNCg0KICAgIGF0IHV0aWwvdmlydGhyZWFkLmM6ODkNCg0K IzQgIDB4MDAwMDdmZGU1YTk2MDhhZSBpbiB2aXJPYmplY3RMb2NrIChhbnlvYmo9YW55b2JqQGVu dHJ5PTB4MTE5YzNkMCkNCg0KICAgIGF0IHV0aWwvdmlyb2JqZWN0LmM6MzIzDQoNCiM1ICAweDAw MDA3ZmRlNWFhYTc1MmMgaW4gdmlyTmV0U29ja2V0RXZlbnRGcmVlIChvcGFxdWU9MHgxMTljM2Qw KQ0KDQogICAgYXQgcnBjL3Zpcm5ldHNvY2tldC5jOjIxMzQNCg0KIzYgIDB4MDAwMDdmZGU1YWU1 N2Y4NyBpbiBsaWJ2aXJ0X3ZpckV2ZW50UmVtb3ZlSGFuZGxlRnVuYyAoDQoNCiAgICB3YXRjaD08 b3B0aW1pemVkIG91dD4pIGF0IGxpYnZpcnQtb3ZlcnJpZGUuYzo1NDk2DQoNCiM3ICAweDAwMDA3 ZmRlNWFhYWFjNjkgaW4gdmlyTmV0U29ja2V0UmVtb3ZlSU9DYWxsYmFjayAoc29jaz0weDExOWMz ZDApDQoNCiAgICBhdCBycGMvdmlybmV0c29ja2V0LmM6MjIxMg0KDQojOCAgMHgwMDAwN2ZkZTVh YTk2ZDc2IGluIHZpck5ldENsaWVudE1hcmtDbG9zZSAoY2xpZW50PTB4MTE5YzY1MCwgcmVhc29u PTApDQoNCiAgICBhdCBycGMvdmlybmV0Y2xpZW50LmM6Nzc5DQoNCiM5ICAweDAwMDA3ZmRlNWFh OTcwZWIgaW4gdmlyTmV0Q2xpZW50SW5jb21pbmdFdmVudCAoc29jaz0weDExOWMzZDAsIGV2ZW50 cz05LA0KDQogICAgb3BhcXVlPTB4MTE5YzY1MCkgYXQgcnBjL3Zpcm5ldGNsaWVudC5jOjE5ODUN Cg0KIzEwIDB4MDAwMDdmZGU1YWU0YjM0NyBpbiBsaWJ2aXJ0X3ZpckV2ZW50SW52b2tlSGFuZGxl Q2FsbGJhY2sgKA0KDQogICAgc2VsZj08b3B0aW1pemVkIG91dD4sIGFyZ3M9PG9wdGltaXplZCBv dXQ+KSBhdCBsaWJ2aXJ0LW92ZXJyaWRlLmM6NTcxOA0KDQojMTEgMHgwMDAwN2ZkZTYyMzZmYWE0 IGluIFB5RXZhbF9FdmFsRnJhbWVFeCAoKSBmcm9tIC9saWI2NC9saWJweXRob24yLjcuc28uMS4w DQoNCiMxMiAweDAwMDA3ZmRlNjIzNzEwYmQgaW4gUHlFdmFsX0V2YWxDb2RlRXggKCkgZnJvbSAv bGliNjQvbGlicHl0aG9uMi43LnNvLjEuMA0KDQotLS1UeXBlIDxyZXR1cm4+IHRvIGNvbnRpbnVl LCBvciBxIDxyZXR1cm4+IHRvIHF1aXQtLS0NCg0KIzEzIDB4MDAwMDdmZGU2MjM2Zjc2ZiBpbiBQ eUV2YWxfRXZhbEZyYW1lRXggKCkgZnJvbSAvbGliNjQvbGlicHl0aG9uMi43LnNvLjEuMA0KDQoj MTQgMHgwMDAwN2ZkZTYyMzZmODYwIGluIFB5RXZhbF9FdmFsRnJhbWVFeCAoKSBmcm9tIC9saWI2 NC9saWJweXRob24yLjcuc28uMS4wDQoNCiMxNSAweDAwMDA3ZmRlNjIzNmY4NjAgaW4gUHlFdmFs X0V2YWxGcmFtZUV4ICgpIGZyb20gL2xpYjY0L2xpYnB5dGhvbjIuNy5zby4xLjANCg0KIzE2IDB4 MDAwMDdmZGU2MjM2Zjg2MCBpbiBQeUV2YWxfRXZhbEZyYW1lRXggKCkgZnJvbSAvbGliNjQvbGli cHl0aG9uMi43LnNvLjEuMA0KDQojMTcgMHgwMDAwN2ZkZTYyMzcxMGJkIGluIFB5RXZhbF9FdmFs Q29kZUV4ICgpIGZyb20gL2xpYjY0L2xpYnB5dGhvbjIuNy5zby4xLjANCg0KIzE4IDB4MDAwMDdm ZGU2MjJmZTA1ZCBpbiBmdW5jdGlvbl9jYWxsICgpIGZyb20gL2xpYjY0L2xpYnB5dGhvbjIuNy5z by4xLjANCg0KIzE5IDB4MDAwMDdmZGU2MjJkOTBiMyBpbiBQeU9iamVjdF9DYWxsICgpIGZyb20g L2xpYjY0L2xpYnB5dGhvbjIuNy5zby4xLjANCg0KIzIwIDB4MDAwMDdmZGU2MjM2YzJmNyBpbiBQ eUV2YWxfRXZhbEZyYW1lRXggKCkgZnJvbSAvbGliNjQvbGlicHl0aG9uMi43LnNvLjEuMA0KDQoj MjEgMHgwMDAwN2ZkZTYyMzZmODYwIGluIFB5RXZhbF9FdmFsRnJhbWVFeCAoKSBmcm9tIC9saWI2 NC9saWJweXRob24yLjcuc28uMS4wDQoNCiMyMiAweDAwMDA3ZmRlNjIzNmY4NjAgaW4gUHlFdmFs X0V2YWxGcmFtZUV4ICgpIGZyb20gL2xpYjY0L2xpYnB5dGhvbjIuNy5zby4xLjANCg0KIzIzIDB4 MDAwMDdmZGU2MjM3MTBiZCBpbiBQeUV2YWxfRXZhbENvZGVFeCAoKSBmcm9tIC9saWI2NC9saWJw eXRob24yLjcuc28uMS4wDQoNCiMyNCAweDAwMDA3ZmRlNjIyZmRmNjggaW4gZnVuY3Rpb25fY2Fs bCAoKSBmcm9tIC9saWI2NC9saWJweXRob24yLjcuc28uMS4wDQoNCiMyNSAweDAwMDA3ZmRlNjIy ZDkwYjMgaW4gUHlPYmplY3RfQ2FsbCAoKSBmcm9tIC9saWI2NC9saWJweXRob24yLjcuc28uMS4w DQoNCiMyNiAweDAwMDA3ZmRlNjIyZTgwYTUgaW4gaW5zdGFuY2VtZXRob2RfY2FsbCAoKQ0KDQoN Cg0KDQo+DQoNCj5NaWNoYWwNCg0KDQoNCg0KDQoNCg0KDQpCZXN0UmVnYXJkcw0KDQogIExpdVl1 bg0KDQoNCg0KDQoNCg0K5Y6f5aeL6YKu5Lu2
On 07/21/2017 08:20 AM, liu.yunh@zte.com.cn wrote:
Hi Michal,
This problem is triggerred by libvirt python's example event-test.py. the original examples has resouce leak issue
at the remove_handle and remove_timer.
with "python -u event-test.py" run this example and "systemctl restart libvirtd.service" will trigger resource leak problem.
with lsof -p <event-test.pid> can see socket handler's number increased , after restart libvirtd.serivce each time.
This is interesting. When I try this out, the python script just gets disconnected and never connects back. So I don't see any number (FD) getting increased.
the reason is remove_handle and remove_timer do not return the remove handle information to libvirt-python's framework.
little patch was apply to this example, to fix this problem.
Now, run this example again and restart libvirtd.service , call sequence virNetSocketRemoveIOCallback->virNetSocketEventFree
can be observed , the no-recursive mutex, lock with recursive issue can be seen.
Recursive mutexes are usually sign of bad design. Anyway in this case, one lock should be held by thread doing virNetSocketRemoveIOCallback(). The other (event loop) should be trying to lock the socket lock from virNetSocketEventFree(). Since these are two different threads, each one of them is modifying the state of the socket we have to have them use the lock.
you can check the detail stack trace and our comments about the lock's issue in function virNetSocketEventFree in the following.
====================================================================
def add_timer(self, interval, cb, opaque):
timerID = self.nextTimerID + 1
self.nextTimerID = self.nextTimerID + 1
h = self.virEventLoopPureTimer(timerID, interval, cb, opaque)
self.timers.append(h)
- self.timers_opaque[timerID] = opaque
self.interrupt()
debug("Add timer %d interval %d" % (timerID, interval))
return timerID
def remove_handle(self, handleID):
handles = []
- opaque = None
for h in self.handles:
if h.get_id() == handleID:
self.poll.unregister(h.get_fd())
- opaque = self.opaques[handleID]
- del self.opaques[handleID]
debug("Remove handle %d fd %d" % (handleID, h.get_fd()))
else:
handles.append(h)
self.handles = handles
self.interrupt()
- return opaque
# Stop firing the periodic timer
def remove_timer(self, timerID):
timers = []
- opaque = None
for h in self.timers:
if h.get_id() != timerID:
timers.append(h)
- else:
- opaque = self.timers_opaque[timerID]
debug("Remove timer %d" % timerID)
self.timers = timers
self.interrupt()
- return opaque
I don't see this code anywhere and thus cannot perform the changes you've suggested. Sorry. Is this current git HEAD?
====================================================================================
On 07/15/2017 05:00 PM, Peng Hao wrote:
virNetSocketRemoveIOCallback get sock's ObjectLock and will call
virNetSocketEventFree. virNetSocketEventFree may be free sock
object and virNetSocketRemoveIOCallback will access a null pointer
in release sock's ObjectLock.
Signed-off-by: Liu Yun <liu.yunh@zte.com.cn>
Signed-off-by: Peng Hao <peng.hao2@zte.com.cn>
---
src/rpc/virnetsocket.c | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
I don't think this can work.
diff --git a/src/rpc/virnetsocket.c b/src/rpc/virnetsocket.c
index d228c8a..8b550e8 100644
--- a/src/rpc/virnetsocket.c
+++ b/src/rpc/virnetsocket.c
@@ -2140,14 +2140,12 @@ static void virNetSocketEventFree(void *opaque)
virFreeCallback ff
void *eopaque
- virObjectLock(sock)
ff = sock->ff
eopaque = sock->opaque
sock->func = NULL
sock->ff = NULL
sock->opaque = NULL
- virObjectUnlock(sock)
I think we need the lock here. This function is called from the event
loop thread. So even if virNetSocketUpdateIOCallback() locks the @socket
this code can see it unlocked. Or locked. But the crucial part is it's
modifying the object and thus should have lock held.
I have check the code , in default implementation of eventPoll, virEventPollRunOnce always dispatch and clear in one thread loop,
so, the lock in the virNetSocketEventFree may be unnessary.
-
+
if (ff)
ff(eopaque)
@@ -2207,6 +2205,7 @@ void virNetSocketUpdateIOCallback(virNetSocketPtr sock,
void virNetSocketRemoveIOCallback(virNetSocketPtr sock)
{
+ virObjectRef(sock)
This should be mistake when generate the patch. The correct one is
+ virObjectUnref(sock)
virObjectLock(sock)
I think this is what actually fixes your problem. However, I also think
it introduces uneven ratio of ref:unref calls.
if (sock->watch < 0) {
@@ -2220,6 +2219,7 @@ void virNetSocketRemoveIOCallback(virNetSocketPtr sock)
sock->watch = -1
virObjectUnlock(sock)
+ virObjectRef(sock)
This should be mistake when generate the patch. The correct one is
+ virObjectUnref(sock)
It definitely does so because you ref twice. Anyway, do you perhaps have
a backtrace to share?
Can you provide the output of 't a a bt'? I wonder if this is the only thread (and thus something left socket locked) or we have some deadlock here.
#0 __lll_lock_wait ()
at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1 0x00007fde6207cd02 in _L_lock_791 () from /lib64/libpthread.so.0
#2 0x00007fde6207cc08 in __GI___pthread_mutex_lock (
mutex=mutex@entry=0x119c3e0) at pthread_mutex_lock.c:64
#3 0x00007fde5a97ee15 in virMutexLock (m=m@entry=0x119c3e0)
at util/virthread.c:89
#4 0x00007fde5a9608ae in virObjectLock (anyobj=anyobj@entry=0x119c3d0)
at util/virobject.c:323
#5 0x00007fde5aaa752c in virNetSocketEventFree (opaque=0x119c3d0)
at rpc/virnetsocket.c:2134
#6 0x00007fde5ae57f87 in libvirt_virEventRemoveHandleFunc (
watch=<optimized out>) at libvirt-override.c:5496
#7 0x00007fde5aaaac69 in virNetSocketRemoveIOCallback (sock=0x119c3d0)
at rpc/virnetsocket.c:2212
#8 0x00007fde5aa96d76 in virNetClientMarkClose (client=0x119c650, reason=0)
at rpc/virnetclient.c:779
#9 0x00007fde5aa970eb in virNetClientIncomingEvent (sock=0x119c3d0, events=9,
opaque=0x119c650) at rpc/virnetclient.c:1985
#10 0x00007fde5ae4b347 in libvirt_virEventInvokeHandleCallback (
self=<optimized out>, args=<optimized out>) at libvirt-override.c:5718
#11 0x00007fde6236faa4 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#12 0x00007fde623710bd in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
---Type <return> to continue, or q <return> to quit---
#13 0x00007fde6236f76f in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#14 0x00007fde6236f860 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#15 0x00007fde6236f860 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#16 0x00007fde6236f860 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#17 0x00007fde623710bd in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
#18 0x00007fde622fe05d in function_call () from /lib64/libpython2.7.so.1.0
#19 0x00007fde622d90b3 in PyObject_Call () from /lib64/libpython2.7.so.1.0
#20 0x00007fde6236c2f7 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#21 0x00007fde6236f860 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#22 0x00007fde6236f860 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#23 0x00007fde623710bd in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
#24 0x00007fde622fdf68 in function_call () from /lib64/libpython2.7.so.1.0
#25 0x00007fde622d90b3 in PyObject_Call () from /lib64/libpython2.7.so.1.0
#26 0x00007fde622e80a5 in instancemethod_call ()
Michal
TWljaGFsLA0KDQogICBwbGVhc2UgY2hlY2sgbXkgY29tbWVudHMgaW4gdGhlIGZvbGxvd2luZy4N Cg0KDQoNCg0KDQo+T24gMDcvMjEvMjAxNyAwODoyMCBBTSwgbGl1Lnl1bmhAenRlLmNvbS5jbiB3 cm90ZToNCg0KPj4gSGkgTWljaGFsLA0KDQo+PiANCg0KPj4gICAgIFRoaXMgcHJvYmxlbSBpcyB0 cmlnZ2VycmVkIGJ5IGxpYnZpcnQgcHl0aG9uJ3MgZXhhbXBsZSBldmVudC10ZXN0LnB5LiB0aGUg b3JpZ2luYWwgZXhhbXBsZXMgaGFzIHJlc291Y2UgbGVhayBpc3N1ZQ0KDQo+PiANCg0KPj4gYXQg dGhlIHJlbW92ZV9oYW5kbGUgYW5kIHJlbW92ZV90aW1lci4gDQoNCj4+IA0KDQo+PiAgICAgd2l0 aCAicHl0aG9uIC11IGV2ZW50LXRlc3QucHkiIHJ1biB0aGlzIGV4YW1wbGUgYW5kICJzeXN0ZW1j dGwgcmVzdGFydCBsaWJ2aXJ0ZC5zZXJ2aWNlIiB3aWxsIHRyaWdnZXIgcmVzb3VyY2UgbGVhayBw cm9ibGVtLg0KDQo+PiANCg0KPj4gd2l0aCBsc29mIC1wIDxldmVudC10ZXN0LnBpZD4gY2FuIHNl ZSBzb2NrZXQgaGFuZGxlcidzIG51bWJlciBpbmNyZWFzZWQgLCBhZnRlciByZXN0YXJ0IGxpYnZp cnRkLnNlcml2Y2UgZWFjaCB0aW1lLg0KDQo+DQoNCj5UaGlzIGlzIGludGVyZXN0aW5nLiBXaGVu IEkgdHJ5IHRoaXMgb3V0LCB0aGUgcHl0aG9uIHNjcmlwdCBqdXN0IGdldHMNCg0KPmRpc2Nvbm5l Y3RlZCBhbmQgbmV2ZXIgY29ubmVjdHMgYmFjay4gU28gSSBkb24ndCBzZWUgYW55IG51bWJlciAo RkQpDQoNCj5nZXR0aW5nIGluY3JlYXNlZC4NCg0KPg0KDQogICB3ZSBhcmUgZXZhbHVhdGluZyB0 aGUgZXZlbnQgZHJpdmVuIGZyYW1ld29yayBpbiB0aGUgZXZlbnQtdGVzdC5weSBleGFtcGxlLiBi ZWNhdXNlIGl0J3Mgb25seSBpbGx1cnN0cmF0ZSBvbmUgc2hvdCBjb25uZWN0aW9uLCB3ZSBtb2Rp ZnkgdmlyRXZlbnRMb29wUHVyZVN0YXJ0IHdpdGggYSBsb29wDQoNCnRvIGFsbG93IHRoZSB0aHJl YWQgcmVjb25uZWN0IHRvIGxpYnZpcnRkLnNlcnZpY2UuIHRoZSBvcmlnaW5hbCBleGFtcGxlIGFz IHlvdSBzZWVuLCBvbmNlIHRoZSBsaWJ2aXJ0ZC5zZXJ2aWNlIHdhcyByZXN0YXJ0LCB0aGUgZXhh bXBsZSBzdG9wIHJ1bm5pbmcuIHNvIHlvdSBhbHNvIG5lZWQgZG8gbGl0dGxlIG1vZGlmaWNhdGlv biANCg0KdG8gYWxsb3cgdGhlIHRocmVhZCByZWNvbm5lY3QgdG8gbGlidmlydGQuc2VydmljZS4g dGhlIHJlc291cmNlIGxlYWsgcHJvYmxlbSB3aWxsIGJlIHNlZW4gYWZ0ZXIgdGhlIG1vZGlmaWNh dGlvbi4NCg0KDQoNCg0KPj4gDQoNCg0KPj4gICAgIHRoZSByZWFzb24gaXMgcmVtb3ZlX2hhbmRs ZSBhbmQgcmVtb3ZlX3RpbWVyIGRvIG5vdCByZXR1cm4gdGhlIHJlbW92ZSBoYW5kbGUgaW5mb3Jt YXRpb24gdG8gbGlidmlydC1weXRob24ncyBmcmFtZXdvcmsuIA0KDQo+PiANCg0KPj4gbGl0dGxl IHBhdGNoIHdhcyBhcHBseSB0byB0aGlzIGV4YW1wbGUsIHRvIGZpeCB0aGlzIHByb2JsZW0uDQoN Cj4+IA0KDQo+PiAgICBOb3csIHJ1biB0aGlzIGV4YW1wbGUgYWdhaW4gYW5kIHJlc3RhcnQgbGli dmlydGQuc2VydmljZSAsIGNhbGwgc2VxdWVuY2UgdmlyTmV0U29ja2V0UmVtb3ZlSU9DYWxsYmFj ay0+dmlyTmV0U29ja2V0RXZlbnRGcmVlIA0KDQo+PiANCg0KPj4gY2FuIGJlIG9ic2VydmVkICwg dGhlIG5vLXJlY3Vyc2l2ZSBtdXRleCwgbG9jayB3aXRoIHJlY3Vyc2l2ZSBpc3N1ZSBjYW4gYmUg c2Vlbi4gDQoNCj4NCg0KPlJlY3Vyc2l2ZSBtdXRleGVzIGFyZSB1c3VhbGx5IHNpZ24gb2YgYmFk IGRlc2lnbi4gQW55d2F5IGluIHRoaXMgY2FzZSwNCg0KPm9uZSBsb2NrIHNob3VsZCBiZSBoZWxk IGJ5IHRocmVhZCBkb2luZyB2aXJOZXRTb2NrZXRSZW1vdmVJT0NhbGxiYWNrKCkuDQoNCj5UaGUg b3RoZXIgKGV2ZW50IGxvb3ApIHNob3VsZCBiZSB0cnlpbmcgdG8gbG9jayB0aGUgc29ja2V0IGxv Y2sgZnJvbQ0KDQo+dmlyTmV0U29ja2V0RXZlbnRGcmVlKCkuIFNpbmNlIHRoZXNlIGFyZSB0d28g ZGlmZmVyZW50IHRocmVhZHMsIGVhY2ggb25lDQoNCj5vZiB0aGVtIGlzIG1vZGlmeWluZyB0aGUg c3RhdGUgb2YgdGhlIHNvY2tldCB3ZSBoYXZlIHRvIGhhdmUgdGhlbSB1c2UNCg0KPnRoZSBsb2Nr Lg0KDQo+DQoNCj4+IA0KDQo+PiAgICAgeW91IGNhbiBjaGVjayB0aGUgZGV0YWlsIHN0YWNrIHRy YWNlIGFuZCBvdXIgY29tbWVudHMgYWJvdXQgdGhlIGxvY2sncyBpc3N1ZSBpbiBmdW5jdGlvbiB2 aXJOZXRTb2NrZXRFdmVudEZyZWUgIGluIHRoZSBmb2xsb3dpbmcuDQoNCj4+IA0KDQo+PiANCg0K Pj4gDQoNCj4+IA0KDQo+PiAgID09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09ICANCg0KPj4gDQoNCj4+ICBkZWYgYWRkX3Rp bWVyKHNlbGYsIGludGVydmFsLCBjYiwgb3BhcXVlKToNCg0KPj4gDQoNCj4+ICAgICAgICAgdGlt ZXJJRCA9IHNlbGYubmV4dFRpbWVySUQgKyAxDQoNCj4+IA0KDQo+PiAgICAgICAgIHNlbGYubmV4 dFRpbWVySUQgPSBzZWxmLm5leHRUaW1lcklEICsgMQ0KDQo+PiANCg0KPj4gDQoNCj4+IA0KDQo+ PiANCg0KPj4gICAgICAgICBoID0gc2VsZi52aXJFdmVudExvb3BQdXJlVGltZXIodGltZXJJRCwg aW50ZXJ2YWwsIGNiLCBvcGFxdWUpDQoNCj4+IA0KDQo+PiAgICAgICAgIHNlbGYudGltZXJzLmFw cGVuZChoKQ0KDQo+PiANCg0KPj4gLSAgICAgICBzZWxmLnRpbWVyc19vcGFxdWVbdGltZXJJRF0g PSBvcGFxdWUNCg0KPj4gDQoNCj4+ICAgICAgICAgc2VsZi5pbnRlcnJ1cHQoKQ0KDQo+PiANCg0K Pj4gDQoNCj4+IA0KDQo+PiANCg0KPj4gICAgICAgICBkZWJ1ZygiQWRkIHRpbWVyICVkIGludGVy dmFsICVkIiAlICh0aW1lcklELCBpbnRlcnZhbCkpDQoNCj4+IA0KDQo+PiANCg0KPj4gDQoNCj4+ IA0KDQo+PiAgICAgICAgIHJldHVybiB0aW1lcklEDQoNCj4+IA0KDQo+PiANCg0KPj4gDQoNCj4+ IA0KDQo+PiANCg0KPj4gDQoNCj4+IA0KDQo+PiAgICAgIGRlZiByZW1vdmVfaGFuZGxlKHNlbGYs IGhhbmRsZUlEKToNCg0KPj4gDQoNCj4+ICAgICAgICAgIGhhbmRsZXMgPSBbXQ0KDQo+PiANCg0K Pj4gLSAgICAgICAgb3BhcXVlID0gTm9uZQ0KDQo+PiANCg0KPj4gICAgICAgICAgZm9yIGggaW4g c2VsZi5oYW5kbGVzOg0KDQo+PiANCg0KPj4gICAgICAgICAgICAgIGlmIGguZ2V0X2lkKCkgPT0g aGFuZGxlSUQ6DQoNCj4+IA0KDQo+PiAgICAgICAgICAgICAgICAgIHNlbGYucG9sbC51bnJlZ2lz dGVyKGguZ2V0X2ZkKCkpDQoNCj4+IA0KDQo+PiAtICAgICAgICAgICAgICAgIG9wYXF1ZSA9IHNl bGYub3BhcXVlc1toYW5kbGVJRF0NCg0KPj4gDQoNCj4+IC0gICAgICAgICAgICAgICAgZGVsIHNl bGYub3BhcXVlc1toYW5kbGVJRF0NCg0KPj4gDQoNCj4+ICAgICAgICAgICAgICAgICAgZGVidWco IlJlbW92ZSBoYW5kbGUgJWQgZmQgJWQiICUgKGhhbmRsZUlELCBoLmdldF9mZCgpKSkNCg0KPj4g DQoNCj4+ICAgICAgICAgICAgICBlbHNlOg0KDQo+PiANCg0KPj4gICAgICAgICAgICAgICAgICBo YW5kbGVzLmFwcGVuZChoKQ0KDQo+PiANCg0KPj4gICAgICAgICAgc2VsZi5oYW5kbGVzID0gaGFu ZGxlcw0KDQo+PiANCg0KPj4gICAgICAgICAgc2VsZi5pbnRlcnJ1cHQoKQ0KDQo+PiANCg0KPj4g LSAgICAgICAgcmV0dXJuIG9wYXF1ZQ0KDQo+PiANCg0KPj4gDQoNCj4+IA0KDQo+PiANCg0KPj4g ICAgICAjIFN0b3AgZmlyaW5nIHRoZSBwZXJpb2RpYyB0aW1lcg0KDQo+PiANCg0KPj4gICAgICBk ZWYgcmVtb3ZlX3RpbWVyKHNlbGYsIHRpbWVySUQpOg0KDQo+PiANCg0KPj4gICAgICAgICAgdGlt ZXJzID0gW10NCg0KPj4gDQoNCj4+IC0gICAgICAgIG9wYXF1ZSA9IE5vbmUNCg0KPj4gDQoNCj4+ ICAgICAgICAgIGZvciBoIGluIHNlbGYudGltZXJzOg0KDQo+PiANCg0KPj4gICAgICAgICAgICAg IGlmIGguZ2V0X2lkKCkgIT0gdGltZXJJRDoNCg0KPj4gDQoNCj4+ICAgICAgICAgICAgICAgICAg dGltZXJzLmFwcGVuZChoKQ0KDQo+PiANCg0KPj4gLSAgICAgICAgICAgIGVsc2U6DQoNCj4+IA0K DQo+PiAtICAgICAgICAgICAgICAgIG9wYXF1ZSA9IHNlbGYudGltZXJzX29wYXF1ZVt0aW1lcklE XQ0KDQo+PiANCg0KPj4gICAgICAgICAgICAgICAgICBkZWJ1ZygiUmVtb3ZlIHRpbWVyICVkIiAl IHRpbWVySUQpDQoNCj4+IA0KDQo+PiAgICAgICAgICBzZWxmLnRpbWVycyA9IHRpbWVycw0KDQo+ PiANCg0KPj4gICAgICAgICAgc2VsZi5pbnRlcnJ1cHQoKQ0KDQo+PiANCg0KPj4gLSAgICAgICAg cmV0dXJuIG9wYXF1ZQ0KDQo+DQoNCj4NCg0KPkkgZG9uJ3Qgc2VlIHRoaXMgY29kZSBhbnl3aGVy ZSBhbmQgdGh1cyBjYW5ub3QgcGVyZm9ybSB0aGUgY2hhbmdlcw0KDQo+eW91J3ZlIHN1Z2dlc3Rl ZC4gU29ycnkuIElzIHRoaXMgY3VycmVudCBnaXQgSEVBRD8NCg0KPg0KDQo+PiANCg0KPj4gPT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09DQoNCj4+IA0KDQo+PiANCg0KPj4gDQoNCj4+IA0KDQo+ PiANCg0KPj4+PiBPbiAwNy8xNS8yMDE3IDA1OjAwIFBNLCBQZW5nIEhhbyB3cm90ZToNCg0KPj4g DQoNCj4+Pj4gdmlyTmV0U29ja2V0UmVtb3ZlSU9DYWxsYmFjayBnZXQgc29jaydzIE9iamVjdExv Y2sgYW5kIHdpbGwgY2FsbA0KDQo+PiANCg0KPj4+PiB2aXJOZXRTb2NrZXRFdmVudEZyZWUuIHZp ck5ldFNvY2tldEV2ZW50RnJlZSBtYXkgYmUgZnJlZSBzb2NrDQoNCj4+IA0KDQo+Pj4+IG9iamVj dCBhbmQgdmlyTmV0U29ja2V0UmVtb3ZlSU9DYWxsYmFjayB3aWxsIGFjY2VzcyBhIG51bGwgcG9p bnRlcg0KDQo+PiANCg0KPj4+PiBpbiByZWxlYXNlIHNvY2sncyBPYmplY3RMb2NrLg0KDQo+PiAN Cg0KPj4+Pg0KDQo+PiANCg0KPj4+PiBTaWduZWQtb2ZmLWJ5OiBMaXUgWXVuIDxsaXUueXVuaEB6 dGUuY29tLmNuPg0KDQo+PiANCg0KPj4+PiBTaWduZWQtb2ZmLWJ5OiBQZW5nIEhhbyA8cGVuZy5o YW8yQHp0ZS5jb20uY24+DQoNCj4+IA0KDQo+Pj4+IC0tLQ0KDQo+PiANCg0KPj4+PiAgc3JjL3Jw Yy92aXJuZXRzb2NrZXQuYyB8IDYgKysrLS0tDQoNCj4+IA0KDQo+Pj4+ICAxIGZpbGUgY2hhbmdl ZCwgMyBpbnNlcnRpb25zKCspLCAzIGRlbGV0aW9ucygtKQ0KDQo+PiANCg0KPj4+Pg0KDQo+PiAN Cg0KPj4+DQoNCj4+IA0KDQo+Pj4gSSBkb24ndCB0aGluayB0aGlzIGNhbiB3b3JrLg0KDQo+PiAN Cg0KPj4+DQoNCj4+IA0KDQo+Pj4+IGRpZmYgLS1naXQgYS9zcmMvcnBjL3Zpcm5ldHNvY2tldC5j IGIvc3JjL3JwYy92aXJuZXRzb2NrZXQuYw0KDQo+PiANCg0KPj4+PiBpbmRleCBkMjI4YzhhLi44 YjU1MGU4IDEwMDY0NA0KDQo+PiANCg0KPj4+PiAtLS0gYS9zcmMvcnBjL3Zpcm5ldHNvY2tldC5j DQoNCj4+IA0KDQo+Pj4+ICsrKyBiL3NyYy9ycGMvdmlybmV0c29ja2V0LmMNCg0KPj4gDQoNCj4+ Pj4gQEAgLTIxNDAsMTQgKzIxNDAsMTIgQEAgc3RhdGljIHZvaWQgdmlyTmV0U29ja2V0RXZlbnRG cmVlKHZvaWQgKm9wYXF1ZSkNCg0KPj4gDQoNCj4+Pj4gICAgICB2aXJGcmVlQ2FsbGJhY2sgZmYN Cg0KPj4gDQoNCj4+Pj4gICAgICB2b2lkICplb3BhcXVlDQoNCj4+IA0KDQo+Pj4+ICANCg0KPj4g DQoNCj4+Pj4gLSAgICB2aXJPYmplY3RMb2NrKHNvY2spDQoNCj4+IA0KDQo+Pj4+ICAgICAgZmYg PSBzb2NrLT5mZg0KDQo+PiANCg0KPj4+PiAgICAgIGVvcGFxdWUgPSBzb2NrLT5vcGFxdWUNCg0K Pj4gDQoNCj4+Pj4gICAgICBzb2NrLT5mdW5jID0gTlVMTA0KDQo+PiANCg0KPj4+PiAgICAgIHNv Y2stPmZmID0gTlVMTA0KDQo+PiANCg0KPj4+PiAgICAgIHNvY2stPm9wYXF1ZSA9IE5VTEwNCg0K Pj4gDQoNCj4+Pj4gLSAgICB2aXJPYmplY3RVbmxvY2soc29jaykNCg0KPj4gDQoNCj4+Pg0KDQo+ PiANCg0KPj4+IEkgdGhpbmsgd2UgbmVlZCB0aGUgbG9jayBoZXJlLiBUaGlzIGZ1bmN0aW9uIGlz IGNhbGxlZCBmcm9tIHRoZSBldmVudA0KDQo+PiANCg0KPj4+IGxvb3AgdGhyZWFkLiBTbyBldmVu IGlmIHZpck5ldFNvY2tldFVwZGF0ZUlPQ2FsbGJhY2soKSBsb2NrcyB0aGUgQHNvY2tldA0KDQo+ PiANCg0KPj4+IHRoaXMgY29kZSBjYW4gc2VlIGl0IHVubG9ja2VkLiBPciBsb2NrZWQuIEJ1dCB0 aGUgY3J1Y2lhbCBwYXJ0IGlzIGl0J3MNCg0KPj4gDQoNCj4+PiBtb2RpZnlpbmcgdGhlIG9iamVj dCBhbmQgdGh1cyBzaG91bGQgaGF2ZSBsb2NrIGhlbGQuDQoNCj4+IA0KDQo+Pj4NCg0KPj4gDQoN Cj4+ICAgIEkgaGF2ZSBjaGVjayB0aGUgY29kZSAsIGluIGRlZmF1bHQgaW1wbGVtZW50YXRpb24g b2YgZXZlbnRQb2xsLCB2aXJFdmVudFBvbGxSdW5PbmNlIGFsd2F5cyBkaXNwYXRjaCBhbmQgY2xl YXIgaW4gb25lIHRocmVhZCBsb29wLA0KDQo+PiANCg0KPj4gc28sIHRoZSBsb2NrIGluIHRoZSB2 aXJOZXRTb2NrZXRFdmVudEZyZWUgbWF5IGJlIHVubmVzc2FyeS4NCg0KPj4gDQoNCj4+IA0KDQo+ PiANCg0KPj4gDQoNCj4+Pj4gLQ0KDQo+PiANCg0KPj4+PiArICANCg0KPj4gDQoNCj4+Pj4gICAg ICBpZiAoZmYpDQoNCj4+IA0KDQo+Pj4+ICAgICAgICAgIGZmKGVvcGFxdWUpDQoNCj4+IA0KDQo+ Pj4+ICANCg0KPj4gDQoNCj4+Pj4gQEAgLTIyMDcsNiArMjIwNSw3IEBAIHZvaWQgdmlyTmV0U29j a2V0VXBkYXRlSU9DYWxsYmFjayh2aXJOZXRTb2NrZXRQdHIgc29jaywNCg0KPj4gDQoNCj4+Pj4g IA0KDQo+PiANCg0KPj4+PiAgdm9pZCB2aXJOZXRTb2NrZXRSZW1vdmVJT0NhbGxiYWNrKHZpck5l dFNvY2tldFB0ciBzb2NrKQ0KDQo+PiANCg0KPj4+PiAgew0KDQo+PiANCg0KPj4+PiArICAgIHZp ck9iamVjdFJlZihzb2NrKQ0KDQo+PiANCg0KPj4gDQoNCj4+IA0KDQo+PiANCg0KPj4gVGhpcyBz aG91bGQgYmUgbWlzdGFrZSB3aGVuIGdlbmVyYXRlIHRoZSBwYXRjaC4gVGhlIGNvcnJlY3Qgb25l IGlzIA0KDQo+PiANCg0KPj4gICAgICArICAgIHZpck9iamVjdFVucmVmKHNvY2spDQoNCj4+IA0K DQo+Pj4+ICAgICAgdmlyT2JqZWN0TG9jayhzb2NrKQ0KDQo+PiANCg0KPj4+DQoNCj4+IA0KDQo+ Pj4gSSB0aGluayB0aGlzIGlzIHdoYXQgYWN0dWFsbHkgZml4ZXMgeW91ciBwcm9ibGVtLiBIb3dl dmVyLCBJIGFsc28gdGhpbmsNCg0KPj4gDQoNCj4+PiBpdCBpbnRyb2R1Y2VzIHVuZXZlbiByYXRp byBvZiByZWY6dW5yZWYgY2FsbHMuDQoNCj4+IA0KDQo+Pj4NCg0KPj4gDQoNCj4+Pj4gIA0KDQo+ PiANCg0KPj4+PiAgICAgIGlmIChzb2NrLT53YXRjaCA8IDApIHsNCg0KPj4gDQoNCj4+Pj4gQEAg LTIyMjAsNiArMjIxOSw3IEBAIHZvaWQgdmlyTmV0U29ja2V0UmVtb3ZlSU9DYWxsYmFjayh2aXJO ZXRTb2NrZXRQdHIgc29jaykNCg0KPj4gDQoNCj4+Pj4gICAgICBzb2NrLT53YXRjaCA9IC0xDQoN Cj4+IA0KDQo+Pj4+ICANCg0KPj4gDQoNCj4+Pj4gICAgICB2aXJPYmplY3RVbmxvY2soc29jaykN Cg0KPj4gDQoNCj4+Pj4gKyAgICB2aXJPYmplY3RSZWYoc29jaykNCg0KPj4gDQoNCj4+IA0KDQo+ PiANCg0KPj4gDQoNCj4+IA0KDQo+PiBUaGlzIHNob3VsZCBiZSBtaXN0YWtlIHdoZW4gZ2VuZXJh dGUgdGhlIHBhdGNoLiBUaGUgY29ycmVjdCBvbmUgaXMgDQoNCj4+IA0KDQo+PiANCg0KPj4gICAg ICArICAgIHZpck9iamVjdFVucmVmKHNvY2spDQoNCj4+IA0KDQo+Pj4NCg0KPj4gDQoNCj4+PiBJ dCBkZWZpbml0ZWx5IGRvZXMgc28gYmVjYXVzZSB5b3UgcmVmIHR3aWNlLiBBbnl3YXksIGRvIHlv dSBwZXJoYXBzIGhhdmUNCg0KPj4gDQoNCj4+PiBhIGJhY2t0cmFjZSB0byBzaGFyZT8NCg0KPj4g DQoNCj4NCg0KPkNhbiB5b3UgcHJvdmlkZSB0aGUgb3V0cHV0IG9mICd0IGEgYSBidCc/IEkgd29u ZGVyIGlmIHRoaXMgaXMgdGhlIG9ubHkNCg0KPnRocmVhZCAoYW5kIHRodXMgc29tZXRoaW5nIGxl ZnQgc29ja2V0IGxvY2tlZCkgb3Igd2UgaGF2ZSBzb21lIGRlYWRsb2NrDQoNCj5oZXJlLg0KDQog ICBmb2xsb3dpbmcgaXMgdGhlIGFsbCB0aHJlYWRzIGJhY3RyYWNlIGluZm9ybWF0aW9uLiBpdCBz aG93IG9ubHkgb25lIHRocmVhZCB0byBhY2NxdWlyZSB0aGUgbG9jay4NCg0KKGdkYikgaW5mbyB0 aHJlYWRzDQoNCiAgSWQgICBUYXJnZXQgSWQgICAgICAgICBGcmFtZQ0KDQogIDIgICAgVGhyZWFk IDB4N2ZmM2M2MzYyNzQwIChMV1AgMjAwNzgpIHNlbV93YWl0ICgpIGF0IC4uL25wdGwvc3lzZGVw cy91bml4L3N5c3YvbGludXgveDg2XzY0L3NlbV93YWl0LlM6ODUNCg0KKiAxICAgIFRocmVhZCAw eDdmZjNiNDhkNDcwMCAoTFdQIDIwMDgxKSBfX2xsbF9sb2NrX3dhaXQgKCkgYXQgLi4vbnB0bC9z eXNkZXBzL3VuaXgvc3lzdi9saW51eC94ODZfNjQvbG93bGV2ZWxsb2NrLlM6MTM1DQoNCihnZGIp IHRocmVhZCAyDQoNCltTd2l0Y2hpbmcgdG8gdGhyZWFkIDIgKFRocmVhZCAweDdmZjNjNjM2Mjc0 MCAoTFdQIDIwMDc4KSldDQoNCiMwICBzZW1fd2FpdCAoKSBhdCAuLi9ucHRsL3N5c2RlcHMvdW5p eC9zeXN2L2xpbnV4L3g4Nl82NC9zZW1fd2FpdC5TOjg1DQoNCjg1ICAgICAgICAgICAgICBtb3Zx ICAgICVyYXgsICVyY3gNCg0KKGdkYikgYnQNCg0KIzAgIHNlbV93YWl0ICgpIGF0IC4uL25wdGwv c3lzZGVwcy91bml4L3N5c3YvbGludXgveDg2XzY0L3NlbV93YWl0LlM6ODUNCg0KIzEgIDB4MDAw MDdmZjNjNWViMTdiNSBpbiBQeVRocmVhZF9hY3F1aXJlX2xvY2sgKCkgZnJvbSAvbGliNjQvbGli cHl0aG9uMi43LnNvLjEuMA0KDQojMiAgMHgwMDAwN2ZmM2M1ZTgwOGE2IGluIFB5RXZhbF9SZXN0 b3JlVGhyZWFkICgpIGZyb20gL2xpYjY0L2xpYnB5dGhvbjIuNy5zby4xLjANCg0KIzMgIDB4MDAw MDdmZjNiNTBmZTA4NiBpbiB0aW1lX3NsZWVwICgpIGZyb20gL3Vzci9saWI2NC9weXRob24yLjcv bGliLWR5bmxvYWQvdGltZW1vZHVsZS5zbw0KDQojNCAgMHgwMDAwN2ZmM2M1ZTg1YWE0IGluIFB5 RXZhbF9FdmFsRnJhbWVFeCAoKSBmcm9tIC9saWI2NC9saWJweXRob24yLjcuc28uMS4wDQoNCiM1 ICAweDAwMDA3ZmYzYzVlODcwYmQgaW4gUHlFdmFsX0V2YWxDb2RlRXggKCkgZnJvbSAvbGliNjQv bGlicHl0aG9uMi43LnNvLjEuMA0KDQojNiAgMHgwMDAwN2ZmM2M1ZTg1NzZmIGluIFB5RXZhbF9F dmFsRnJhbWVFeCAoKSBmcm9tIC9saWI2NC9saWJweXRob24yLjcuc28uMS4wDQoNCiM3ICAweDAw MDA3ZmYzYzVlODcwYmQgaW4gUHlFdmFsX0V2YWxDb2RlRXggKCkgZnJvbSAvbGliNjQvbGlicHl0 aG9uMi43LnNvLjEuMA0KDQojOCAgMHgwMDAwN2ZmM2M1ZTg3MWMyIGluIFB5RXZhbF9FdmFsQ29k ZSAoKSBmcm9tIC9saWI2NC9saWJweXRob24yLjcuc28uMS4wDQoNCiM5ICAweDAwMDA3ZmYzYzVl YTA1ZmYgaW4gcnVuX21vZCAoKSBmcm9tIC9saWI2NC9saWJweXRob24yLjcuc28uMS4wDQoNCiMx MCAweDAwMDA3ZmYzYzVlYTE3YmUgaW4gUHlSdW5fRmlsZUV4RmxhZ3MgKCkgZnJvbSAvbGliNjQv bGlicHl0aG9uMi43LnNvLjEuMA0KDQojMTEgMHgwMDAwN2ZmM2M1ZWEyYTQ5IGluIFB5UnVuX1Np bXBsZUZpbGVFeEZsYWdzICgpIGZyb20gL2xpYjY0L2xpYnB5dGhvbjIuNy5zby4xLjANCg0KIzEy IDB4MDAwMDdmZjNjNWViM2I5ZiBpbiBQeV9NYWluICgpIGZyb20gL2xpYjY0L2xpYnB5dGhvbjIu Ny5zby4xLjANCg0KIzEzIDB4MDAwMDdmZjNjNTBlMGIxNSBpbiBfX2xpYmNfc3RhcnRfbWFpbiAo bWFpbj0weDQwMDZmMCA8bWFpbj4sIGFyZ2M9MywgdWJwX2F2PTB4N2ZmY2E1OGVmNTI4LCBpbml0 PTxvcHRpbWl6ZWQgb3V0PiwgZmluaT08b3B0aW1pemVkIG91dD4sIHJ0bGRfZmluaT08b3B0aW1p emVkIG91dD4sIHN0YWNrX2VuZD0weDdmZmNhNThlZjUxOCkNCg0KICAgIGF0IGxpYmMtc3RhcnQu YzoyNzQNCg0KIzE0IDB4MDAwMDAwMDAwMDQwMDcyMSBpbiBfc3RhcnQgKCkNCg0KKGdkYikgdGhy ZWFkIDENCg0KW1N3aXRjaGluZyB0byB0aHJlYWQgMSAoVGhyZWFkIDB4N2ZmM2I0OGQ0NzAwIChM V1AgMjAwODEpKV0NCg0KIzAgIF9fbGxsX2xvY2tfd2FpdCAoKSBhdCAuLi9ucHRsL3N5c2RlcHMv dW5peC9zeXN2L2xpbnV4L3g4Nl82NC9sb3dsZXZlbGxvY2suUzoxMzUNCg0KMTM1ICAgICAyOiAg ICAgIG1vdmwgICAgJWVkeCwgJWVheA0KDQooZ2RiKSBidA0KDQojMCAgX19sbGxfbG9ja193YWl0 ICgpIGF0IC4uL25wdGwvc3lzZGVwcy91bml4L3N5c3YvbGludXgveDg2XzY0L2xvd2xldmVsbG9j ay5TOjEzNQ0KDQojMSAgMHgwMDAwN2ZmM2M1YjkyZDAyIGluIF9MX2xvY2tfNzkxICgpIGZyb20g L2xpYjY0L2xpYnB0aHJlYWQuc28uMA0KDQojMiAgMHgwMDAwN2ZmM2M1YjkyYzA4IGluIF9fR0lf X19wdGhyZWFkX211dGV4X2xvY2sgKG11dGV4PW11dGV4QGVudHJ5PTB4YTZiOWMwKSBhdCBwdGhy ZWFkX211dGV4X2xvY2suYzo2NA0KDQojMyAgMHgwMDAwN2ZmM2JlMzUxZTE1IGluIHZpck11dGV4 TG9jayAobT1tQGVudHJ5PTB4YTZiOWMwKSBhdCB1dGlsL3ZpcnRocmVhZC5jOjg5DQoNCiM0ICAw eDAwMDA3ZmYzYmUzMzM4YWUgaW4gdmlyT2JqZWN0TG9jayAoYW55b2JqPWFueW9iakBlbnRyeT0w eGE2YjliMCkgYXQgdXRpbC92aXJvYmplY3QuYzozMjMNCg0KIzUgIDB4MDAwMDdmZjNiZTQ3YTUy YyBpbiB2aXJOZXRTb2NrZXRFdmVudEZyZWUgKG9wYXF1ZT0weGE2YjliMCkgYXQgcnBjL3Zpcm5l dHNvY2tldC5jOjIxMzQNCg0KIzYgIDB4MDAwMDdmZjNiZTgyYWY4NyBpbiBsaWJ2aXJ0X3ZpckV2 ZW50UmVtb3ZlSGFuZGxlRnVuYyAod2F0Y2g9PG9wdGltaXplZCBvdXQ+KSBhdCBsaWJ2aXJ0LW92 ZXJyaWRlLmM6NTQ5Ng0KDQojNyAgMHgwMDAwN2ZmM2JlNDdkYzY5IGluIHZpck5ldFNvY2tldFJl bW92ZUlPQ2FsbGJhY2sgKHNvY2s9MHhhNmI5YjApIGF0IHJwYy92aXJuZXRzb2NrZXQuYzoyMjEy DQoNCiM4ICAweDAwMDA3ZmYzYmU0NjlkNzYgaW4gdmlyTmV0Q2xpZW50TWFya0Nsb3NlIChjbGll bnQ9MHhhNmJjYjAsIHJlYXNvbj0wKSBhdCBycGMvdmlybmV0Y2xpZW50LmM6Nzc5DQoNCiM5ICAw eDAwMDA3ZmYzYmU0NmEwZWIgaW4gdmlyTmV0Q2xpZW50SW5jb21pbmdFdmVudCAoc29jaz0weGE2 YjliMCwgZXZlbnRzPTksIG9wYXF1ZT0weGE2YmNiMCkgYXQgcnBjL3Zpcm5ldGNsaWVudC5jOjE5 ODUNCg0KIzEwIDB4MDAwMDdmZjNiZTgxZTM0NyBpbiBsaWJ2aXJ0X3ZpckV2ZW50SW52b2tlSGFu ZGxlQ2FsbGJhY2sgKHNlbGY9PG9wdGltaXplZCBvdXQ+LCBhcmdzPTxvcHRpbWl6ZWQgb3V0Pikg YXQgbGlidmlydC1vdmVycmlkZS5jOjU3MTgNCg0KIzExIDB4MDAwMDdmZjNjNWU4NWFhNCBpbiBQ eUV2YWxfRXZhbEZyYW1lRXggKCkgZnJvbSAvbGliNjQvbGlicHl0aG9uMi43LnNvLjEuMA0KDQoj MTIgMHgwMDAwN2ZmM2M1ZTg3MGJkIGluIFB5RXZhbF9FdmFsQ29kZUV4ICgpIGZyb20gL2xpYjY0 L2xpYnB5dGhvbjIuNy5zby4xLjANCg0KIzEzIDB4MDAwMDdmZjNjNWU4NTc2ZiBpbiBQeUV2YWxf RXZhbEZyYW1lRXggKCkgZnJvbSAvbGliNjQvbGlicHl0aG9uMi43LnNvLjEuMA0KDQojMTQgMHgw MDAwN2ZmM2M1ZTg1ODYwIGluIFB5RXZhbF9FdmFsRnJhbWVFeCAoKSBmcm9tIC9saWI2NC9saWJw eXRob24yLjcuc28uMS4wDQoNCiMxNSAweDAwMDA3ZmYzYzVlODU4NjAgaW4gUHlFdmFsX0V2YWxG cmFtZUV4ICgpIGZyb20gL2xpYjY0L2xpYnB5dGhvbjIuNy5zby4xLjANCg0KIzE2IDB4MDAwMDdm ZjNjNWU4NTg2MCBpbiBQeUV2YWxfRXZhbEZyYW1lRXggKCkgZnJvbSAvbGliNjQvbGlicHl0aG9u Mi43LnNvLjEuMA0KDQojMTcgMHgwMDAwN2ZmM2M1ZTg3MGJkIGluIFB5RXZhbF9FdmFsQ29kZUV4 ICgpIGZyb20gL2xpYjY0L2xpYnB5dGhvbjIuNy5zby4xLjANCg0KIzE4IDB4MDAwMDdmZjNjNWUx NDA1ZCBpbiBmdW5jdGlvbl9jYWxsICgpIGZyb20gL2xpYjY0L2xpYnB5dGhvbjIuNy5zby4xLjAN Cg0KIzE5IDB4MDAwMDdmZjNjNWRlZjBiMyBpbiBQeU9iamVjdF9DYWxsICgpIGZyb20gL2xpYjY0 L2xpYnB5dGhvbjIuNy5zby4xLjANCg0KIzIwIDB4MDAwMDdmZjNjNWU4MjJmNyBpbiBQeUV2YWxf RXZhbEZyYW1lRXggKCkgZnJvbSAvbGliNjQvbGlicHl0aG9uMi43LnNvLjEuMA0KDQojMjEgMHgw MDAwN2ZmM2M1ZTg1ODYwIGluIFB5RXZhbF9FdmFsRnJhbWVFeCAoKSBmcm9tIC9saWI2NC9saWJw eXRob24yLjcuc28uMS4wDQoNCiMyMiAweDAwMDA3ZmYzYzVlODU4NjAgaW4gUHlFdmFsX0V2YWxG cmFtZUV4ICgpIGZyb20gL2xpYjY0L2xpYnB5dGhvbjIuNy5zby4xLjANCg0KIzIzIDB4MDAwMDdm ZjNjNWU4NzBiZCBpbiBQeUV2YWxfRXZhbENvZGVFeCAoKSBmcm9tIC9saWI2NC9saWJweXRob24y Ljcuc28uMS4wDQoNCiMyNCAweDAwMDA3ZmYzYzVlMTNmNjggaW4gZnVuY3Rpb25fY2FsbCAoKSBm cm9tIC9saWI2NC9saWJweXRob24yLjcuc28uMS4wDQoNCiMyNSAweDAwMDA3ZmYzYzVkZWYwYjMg aW4gUHlPYmplY3RfQ2FsbCAoKSBmcm9tIC9saWI2NC9saWJweXRob24yLjcuc28uMS4wDQoNCiMy NiAweDAwMDA3ZmYzYzVkZmUwYTUgaW4gaW5zdGFuY2VtZXRob2RfY2FsbCAoKSBmcm9tIC9saWI2 NC9saWJweXRob24yLjcuc28uMS4wDQoNCiMyNyAweDAwMDA3ZmYzYzVkZWYwYjMgaW4gUHlPYmpl Y3RfQ2FsbCAoKSBmcm9tIC9saWI2NC9saWJweXRob24yLjcuc28uMS4wDQoNCiMyOCAweDAwMDA3 ZmYzYzVlODBmMDcgaW4gUHlFdmFsX0NhbGxPYmplY3RXaXRoS2V5d29yZHMgKCkgZnJvbSAvbGli NjQvbGlicHl0aG9uMi43LnNvLjEuMA0KDQojMjkgMHgwMDAwN2ZmM2M1ZWI1ODQyIGluIHRfYm9v dHN0cmFwICgpIGZyb20gL2xpYjY0L2xpYnB5dGhvbjIuNy5zby4xLjANCg0KIzMwIDB4MDAwMDdm ZjNjNWI5MGRjNSBpbiBzdGFydF90aHJlYWQgKGFyZz0weDdmZjNiNDhkNDcwMCkgYXQgcHRocmVh ZF9jcmVhdGUuYzozMDgNCg0KIzMxIDB4MDAwMDdmZjNjNTFiNTIxZCBpbiBjbG9uZSAoKSBhdCAu Li9zeXNkZXBzL3VuaXgvc3lzdi9saW51eC94ODZfNjQvY2xvbmUuUzoxMTMNCg0KKGdkYikNCg0K Pg0KDQo+PiAgICAgDQoNCj4+IA0KDQo+PiAjMCAgX19sbGxfbG9ja193YWl0ICgpDQoNCj4+IA0K DQo+PiAgICAgYXQgLi4vbnB0bC9zeXNkZXBzL3VuaXgvc3lzdi9saW51eC94ODZfNjQvbG93bGV2 ZWxsb2NrLlM6MTM1DQoNCj4+IA0KDQo+PiAjMSAgMHgwMDAwN2ZkZTYyMDdjZDAyIGluIF9MX2xv Y2tfNzkxICgpIGZyb20gL2xpYjY0L2xpYnB0aHJlYWQuc28uMA0KDQo+PiANCg0KPj4gIzIgIDB4 MDAwMDdmZGU2MjA3Y2MwOCBpbiBfX0dJX19fcHRocmVhZF9tdXRleF9sb2NrICgNCg0KPj4gDQoN Cj4+ICAgICBtdXRleD1tdXRleEBlbnRyeT0weDExOWMzZTApIGF0IHB0aHJlYWRfbXV0ZXhfbG9j ay5jOjY0DQoNCj4+IA0KDQo+PiAjMyAgMHgwMDAwN2ZkZTVhOTdlZTE1IGluIHZpck11dGV4TG9j ayAobT1tQGVudHJ5PTB4MTE5YzNlMCkNCg0KPj4gDQoNCj4+ICAgICBhdCB1dGlsL3ZpcnRocmVh ZC5jOjg5DQoNCj4+IA0KDQo+PiAjNCAgMHgwMDAwN2ZkZTVhOTYwOGFlIGluIHZpck9iamVjdExv Y2sgKGFueW9iaj1hbnlvYmpAZW50cnk9MHgxMTljM2QwKQ0KDQo+PiANCg0KPj4gICAgIGF0IHV0 aWwvdmlyb2JqZWN0LmM6MzIzDQoNCj4+IA0KDQo+PiAjNSAgMHgwMDAwN2ZkZTVhYWE3NTJjIGlu IHZpck5ldFNvY2tldEV2ZW50RnJlZSAob3BhcXVlPTB4MTE5YzNkMCkNCg0KPj4gDQoNCj4+ICAg ICBhdCBycGMvdmlybmV0c29ja2V0LmM6MjEzNA0KDQo+PiANCg0KPj4gIzYgIDB4MDAwMDdmZGU1 YWU1N2Y4NyBpbiBsaWJ2aXJ0X3ZpckV2ZW50UmVtb3ZlSGFuZGxlRnVuYyAoDQoNCj4+IA0KDQo+ PiAgICAgd2F0Y2g9PG9wdGltaXplZCBvdXQ+KSBhdCBsaWJ2aXJ0LW92ZXJyaWRlLmM6NTQ5Ng0K DQo+PiANCg0KPj4gIzcgIDB4MDAwMDdmZGU1YWFhYWM2OSBpbiB2aXJOZXRTb2NrZXRSZW1vdmVJ T0NhbGxiYWNrIChzb2NrPTB4MTE5YzNkMCkNCg0KPj4gDQoNCj4+ICAgICBhdCBycGMvdmlybmV0 c29ja2V0LmM6MjIxMg0KDQo+PiANCg0KPj4gIzggIDB4MDAwMDdmZGU1YWE5NmQ3NiBpbiB2aXJO ZXRDbGllbnRNYXJrQ2xvc2UgKGNsaWVudD0weDExOWM2NTAsIHJlYXNvbj0wKQ0KDQo+PiANCg0K Pj4gICAgIGF0IHJwYy92aXJuZXRjbGllbnQuYzo3NzkNCg0KPj4gDQoNCj4+ICM5ICAweDAwMDA3 ZmRlNWFhOTcwZWIgaW4gdmlyTmV0Q2xpZW50SW5jb21pbmdFdmVudCAoc29jaz0weDExOWMzZDAs IGV2ZW50cz05LA0KDQo+PiANCg0KPj4gICAgIG9wYXF1ZT0weDExOWM2NTApIGF0IHJwYy92aXJu ZXRjbGllbnQuYzoxOTg1DQoNCj4+IA0KDQo+PiAjMTAgMHgwMDAwN2ZkZTVhZTRiMzQ3IGluIGxp YnZpcnRfdmlyRXZlbnRJbnZva2VIYW5kbGVDYWxsYmFjayAoDQoNCj4+IA0KDQo+PiAgICAgc2Vs Zj08b3B0aW1pemVkIG91dD4sIGFyZ3M9PG9wdGltaXplZCBvdXQ+KSBhdCBsaWJ2aXJ0LW92ZXJy aWRlLmM6NTcxOA0KDQo+PiANCg0KPj4gIzExIDB4MDAwMDdmZGU2MjM2ZmFhNCBpbiBQeUV2YWxf RXZhbEZyYW1lRXggKCkgZnJvbSAvbGliNjQvbGlicHl0aG9uMi43LnNvLjEuMA0KDQo+PiANCg0K Pj4gIzEyIDB4MDAwMDdmZGU2MjM3MTBiZCBpbiBQeUV2YWxfRXZhbENvZGVFeCAoKSBmcm9tIC9s aWI2NC9saWJweXRob24yLjcuc28uMS4wDQoNCj4+IA0KDQo+PiAtLS1UeXBlIDxyZXR1cm4+IHRv IGNvbnRpbnVlLCBvciBxIDxyZXR1cm4+IHRvIHF1aXQtLS0NCg0KPj4gDQoNCj4+ICMxMyAweDAw MDA3ZmRlNjIzNmY3NmYgaW4gUHlFdmFsX0V2YWxGcmFtZUV4ICgpIGZyb20gL2xpYjY0L2xpYnB5 dGhvbjIuNy5zby4xLjANCg0KPj4gDQoNCj4+ICMxNCAweDAwMDA3ZmRlNjIzNmY4NjAgaW4gUHlF dmFsX0V2YWxGcmFtZUV4ICgpIGZyb20gL2xpYjY0L2xpYnB5dGhvbjIuNy5zby4xLjANCg0KPj4g DQoNCj4+ICMxNSAweDAwMDA3ZmRlNjIzNmY4NjAgaW4gUHlFdmFsX0V2YWxGcmFtZUV4ICgpIGZy b20gL2xpYjY0L2xpYnB5dGhvbjIuNy5zby4xLjANCg0KPj4gDQoNCj4+ICMxNiAweDAwMDA3ZmRl NjIzNmY4NjAgaW4gUHlFdmFsX0V2YWxGcmFtZUV4ICgpIGZyb20gL2xpYjY0L2xpYnB5dGhvbjIu Ny5zby4xLjANCg0KPj4gDQoNCj4+ICMxNyAweDAwMDA3ZmRlNjIzNzEwYmQgaW4gUHlFdmFsX0V2 YWxDb2RlRXggKCkgZnJvbSAvbGliNjQvbGlicHl0aG9uMi43LnNvLjEuMA0KDQo+PiANCg0KPj4g IzE4IDB4MDAwMDdmZGU2MjJmZTA1ZCBpbiBmdW5jdGlvbl9jYWxsICgpIGZyb20gL2xpYjY0L2xp YnB5dGhvbjIuNy5zby4xLjANCg0KPj4gDQoNCj4+ICMxOSAweDAwMDA3ZmRlNjIyZDkwYjMgaW4g UHlPYmplY3RfQ2FsbCAoKSBmcm9tIC9saWI2NC9saWJweXRob24yLjcuc28uMS4wDQoNCj4+IA0K DQo+PiAjMjAgMHgwMDAwN2ZkZTYyMzZjMmY3IGluIFB5RXZhbF9FdmFsRnJhbWVFeCAoKSBmcm9t IC9saWI2NC9saWJweXRob24yLjcuc28uMS4wDQoNCj4+IA0KDQo+PiAjMjEgMHgwMDAwN2ZkZTYy MzZmODYwIGluIFB5RXZhbF9FdmFsRnJhbWVFeCAoKSBmcm9tIC9saWI2NC9saWJweXRob24yLjcu c28uMS4wDQoNCj4+IA0KDQo+PiAjMjIgMHgwMDAwN2ZkZTYyMzZmODYwIGluIFB5RXZhbF9FdmFs RnJhbWVFeCAoKSBmcm9tIC9saWI2NC9saWJweXRob24yLjcuc28uMS4wDQoNCj4+IA0KDQo+PiAj MjMgMHgwMDAwN2ZkZTYyMzcxMGJkIGluIFB5RXZhbF9FdmFsQ29kZUV4ICgpIGZyb20gL2xpYjY0 L2xpYnB5dGhvbjIuNy5zby4xLjANCg0KPj4gDQoNCj4+ICMyNCAweDAwMDA3ZmRlNjIyZmRmNjgg aW4gZnVuY3Rpb25fY2FsbCAoKSBmcm9tIC9saWI2NC9saWJweXRob24yLjcuc28uMS4wDQoNCj4+ IA0KDQo+PiAjMjUgMHgwMDAwN2ZkZTYyMmQ5MGIzIGluIFB5T2JqZWN0X0NhbGwgKCkgZnJvbSAv bGliNjQvbGlicHl0aG9uMi43LnNvLjEuMA0KDQo+PiANCg0KPj4gIzI2IDB4MDAwMDdmZGU2MjJl ODBhNSBpbiBpbnN0YW5jZW1ldGhvZF9jYWxsICgpDQoNCj4+IA0KDQo+PiANCg0KPg0KDQo+TWlj aGFs
On 07/22/2017 05:07 AM, liu.yunh@zte.com.cn wrote:
On 07/21/2017 08:20 AM, liu.yunh@zte.com.cn wrote:
Hi Michal,
This problem is triggerred by libvirt python's example event-test.py. the original examples has resouce leak issue
at the remove_handle and remove_timer.
with "python -u event-test.py" run this example and "systemctl restart libvirtd.service" will trigger resource leak problem.
with lsof -p <event-test.pid> can see socket handler's number increased , after restart libvirtd.serivce each time.
This is interesting. When I try this out, the python script just gets
disconnected and never connects back. So I don't see any number (FD)
getting increased.
we are evaluating the event driven framework in the event-test.py example. because it's only illurstrate one shot connection, we modify virEventLoopPureStart with a loop
to allow the thread reconnect to libvirtd.service. the original example as you seen, once the libvirtd.service was restart, the example stop running. so you also need do little modification
to allow the thread reconnect to libvirtd.service. the resource leak problem will be seen after the modification.
Ah, so I couldn't have applied that patch, because there's nothing to apply it to. Is your code available somewhere so that I can do the changes? Also, is it possible that there's a problem in your code?
<trim/>
Can you provide the output of 't a a bt'? I wonder if this is the only
thread (and thus something left socket locked) or we have some deadlock
here.
following is the all threads bactrace information. it show only one thread to accquire the lock.
(gdb) info threads
Id Target Id Frame
2 Thread 0x7ff3c6362740 (LWP 20078) sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
* 1 Thread 0x7ff3b48d4700 (LWP 20081) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
(gdb) thread 2
[Switching to thread 2 (Thread 0x7ff3c6362740 (LWP 20078))]
#0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
85 movq %rax, %rcx
(gdb) bt
#0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
#1 0x00007ff3c5eb17b5 in PyThread_acquire_lock () from /lib64/libpython2.7.so.1.0
#2 0x00007ff3c5e808a6 in PyEval_RestoreThread () from /lib64/libpython2.7.so.1.0
#3 0x00007ff3b50fe086 in time_sleep () from /usr/lib64/python2.7/lib-dynload/timemodule.so
#4 0x00007ff3c5e85aa4 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#5 0x00007ff3c5e870bd in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
#6 0x00007ff3c5e8576f in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#7 0x00007ff3c5e870bd in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
#8 0x00007ff3c5e871c2 in PyEval_EvalCode () from /lib64/libpython2.7.so.1.0
#9 0x00007ff3c5ea05ff in run_mod () from /lib64/libpython2.7.so.1.0
#10 0x00007ff3c5ea17be in PyRun_FileExFlags () from /lib64/libpython2.7.so.1.0
#11 0x00007ff3c5ea2a49 in PyRun_SimpleFileExFlags () from /lib64/libpython2.7.so.1.0
#12 0x00007ff3c5eb3b9f in Py_Main () from /lib64/libpython2.7.so.1.0
#13 0x00007ff3c50e0b15 in __libc_start_main (main=0x4006f0 <main>, argc=3, ubp_av=0x7ffca58ef528, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffca58ef518)
at libc-start.c:274
#14 0x0000000000400721 in _start ()
(gdb) thread 1
[Switching to thread 1 (Thread 0x7ff3b48d4700 (LWP 20081))]
#0 __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
135 2: movl %edx, %eax
(gdb) bt
#0 __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1 0x00007ff3c5b92d02 in _L_lock_791 () from /lib64/libpthread.so.0
#2 0x00007ff3c5b92c08 in __GI___pthread_mutex_lock (mutex=mutex@entry=0xa6b9c0) at pthread_mutex_lock.c:64
#3 0x00007ff3be351e15 in virMutexLock (m=m@entry=0xa6b9c0) at util/virthread.c:89
#4 0x00007ff3be3338ae in virObjectLock (anyobj=anyobj@entry=0xa6b9b0) at util/virobject.c:323
#5 0x00007ff3be47a52c in virNetSocketEventFree (opaque=0xa6b9b0) at rpc/virnetsocket.c:2134
#6 0x00007ff3be82af87 in libvirt_virEventRemoveHandleFunc (watch=<optimized out>) at libvirt-override.c:5496
#7 0x00007ff3be47dc69 in virNetSocketRemoveIOCallback (sock=0xa6b9b0) at rpc/virnetsocket.c:2212
#8 0x00007ff3be469d76 in virNetClientMarkClose (client=0xa6bcb0, reason=0) at rpc/virnetclient.c:779
#9 0x00007ff3be46a0eb in virNetClientIncomingEvent (sock=0xa6b9b0, events=9, opaque=0xa6bcb0) at rpc/virnetclient.c:1985
#10 0x00007ff3be81e347 in libvirt_virEventInvokeHandleCallback (self=<optimized out>, args=<optimized out>) at libvirt-override.c:5718
#11 0x00007ff3c5e85aa4 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#12 0x00007ff3c5e870bd in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
#13 0x00007ff3c5e8576f in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#14 0x00007ff3c5e85860 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#15 0x00007ff3c5e85860 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#16 0x00007ff3c5e85860 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#17 0x00007ff3c5e870bd in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
#18 0x00007ff3c5e1405d in function_call () from /lib64/libpython2.7.so.1.0
#19 0x00007ff3c5def0b3 in PyObject_Call () from /lib64/libpython2.7.so.1.0
#20 0x00007ff3c5e822f7 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#21 0x00007ff3c5e85860 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#22 0x00007ff3c5e85860 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#23 0x00007ff3c5e870bd in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
#24 0x00007ff3c5e13f68 in function_call () from /lib64/libpython2.7.so.1.0
#25 0x00007ff3c5def0b3 in PyObject_Call () from /lib64/libpython2.7.so.1.0
#26 0x00007ff3c5dfe0a5 in instancemethod_call () from /lib64/libpython2.7.so.1.0
#27 0x00007ff3c5def0b3 in PyObject_Call () from /lib64/libpython2.7.so.1.0
#28 0x00007ff3c5e80f07 in PyEval_CallObjectWithKeywords () from /lib64/libpython2.7.so.1.0
#29 0x00007ff3c5eb5842 in t_bootstrap () from /lib64/libpython2.7.so.1.0
#30 0x00007ff3c5b90dc5 in start_thread (arg=0x7ff3b48d4700) at pthread_create.c:308
#31 0x00007ff3c51b521d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
(gdb)
Okay, so there is no deadlock in sense that two threads holding two locks and fight for the other ones. However, this looks like somebody left the socket locked (e.g. a bug in your code?). Alternatively, socket might have been freed and thus subsequent lock attempt just hangs (as a result of undefined behaviour). Anyway, unless I can reproduce the problem I am hesitant to merge the patch (esp. if it doesn't look right). Or can you provide a small reproducer? Michal
participants (3)
-
liu.yunh＠zte.com.cn -
Michal Privoznik -
Peng Hao