[libvirt] xen/libvirt freeze while attching network-device to vm´s - question about provided patches
Hello everybody, a colleague of me described last november a problem that we have with libvirt and xen https://www.redhat.com/archives/libvir-list/2015-November/msg00130.html Jim Fehlig provided for this problem a patch a few days later: https://www.redhat.com/archives/libvir-list/2015-November/msg00520.html The provided patch work great and we have no problems until we applied them. Now we a in the situation that we want to migrate to libvirt 2.1, and we saw the same problem that we have last year, again. After we take a look in the source code, we determine that the patches are didn´t go in the source code. So we modify the patches in that way, that we could apply them again, and all looks good. My question, is their any special reason why this patches didn´t go in the source code? I see same posts on the maillinglist, but absolute statement for or again the patches from Jim Fehlig. all the best guido ____________ Virus checked by G Data MailSecurity Version: AVA 25.8195 dated 08.09.2016 Virus news: www.antiviruslab.com
On Thu, 2016-09-08 at 14:04 +0000, Guido.Rossmueller@gdata.de wrote:
Hello everybody,
a colleague of me described last november a problem that we have with libvirt and xen
https://www.redhat.com/archives/libvir-list/2015-November/msg00130.html
Jim Fehlig provided for this problem a patch a few days later:
https://www.redhat.com/archives/libvir-list/2015-November/msg00520.html
The provided patch work great and we have no problems until we applied them..
Now we a in the situation that we want to migrate to libvirt 2.1, and we saw the same problem that we have last year, again. After we take a look in the source code, we determine that the patches are didn´t go in the source code. So we modify the patches in that way, that we could apply them again, and all looks good.
My question, is their any special reason why this patches didn´t go in the source code?
I see same posts on the maillinglist, but absolute statement for or again the patches from Jim Fehlig.
Jim said he would submit a v2 to address some concerns raised during review[1]; later, he reported some issues he'd run into[2] while working on it. So it looks to me like the code was not yet ready for merge, which would explain it being tagged as RFC. And then it probably just slipped through the cracks :) CCing Jim for insights on the current status of the patch. [1] https://www.redhat.com/archives/libvir-list/2015-November/msg00958.html [2] https://www.redhat.com/archives/libvir-list/2015-November/msg00974.html -- Andrea Bolognani / Red Hat / Virtualization
On 09/09/2016 09:43 AM, Andrea Bolognani wrote:
On Thu, 2016-09-08 at 14:04 +0000, Guido.Rossmueller@gdata.de wrote:
Hello everybody,
a colleague of me described last november a problem that we have with libvirt and xen
https://www.redhat.com/archives/libvir-list/2015-November/msg00130.html
Jim Fehlig provided for this problem a patch a few days later:
https://www.redhat.com/archives/libvir-list/2015-November/msg00520.html
The provided patch work great and we have no problems until we applied them..
Now we a in the situation that we want to migrate to libvirt 2.1, and we saw the same problem that we have last year, again. After we take a look in the source code, we determine that the patches are didn´t go in the source code. So we modify the patches in that way, that we could apply them again, and all looks good.
My question, is their any special reason why this patches didn´t go in the source code?
I see same posts on the maillinglist, but absolute statement for or again the patches from Jim Fehlig. Jim said he would submit a v2 to address some concerns raised during review[1]; later, he reported some issues he'd run into[2] while working on it.
I had planned to wait a few days to see if Ian had any thoughts on the libxl side. I'm not familiar with the code and would need some time to understand it before debugging further. But I've never got back to that work :-(.
So it looks to me like the code was not yet ready for merge, which would explain it being tagged as RFC. And then it probably just slipped through the cracks :)
Sort of. It is an item that is fermenting on my todo list.
CCing Jim for insights on the current status of the patch.
I still have a branch in my local git tree :-). It needs rebased and tested against Xen 4.7 (and probably xen.git master).
[1] https://www.redhat.com/archives/libvir-list/2015-November/msg00958.html [2] https://www.redhat.com/archives/libvir-list/2015-November/msg00974.html
Guido, With the rebased version of the patch, do you notice the issue described in [2] during your testing? To trigger it, I used a hacky script that created, shutdown, destroyed, saved, and restored many domains in parallel. What version of Xen are you using? Regards, Jim
Hi Jim, thanks for your response ans sorry for my late answer, i´m sorry, i don´t really understand what you are missing in [2], or from where in the log-output you see that an shutdown event is missing. on our test-system the output looks normaly as follow: 2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1719:devices_destroy_cb: forked pid 14987 for destroy of domain 1375 2016-09-20 15:00:38 CEST libxl: debug: libxl_event.c:576:watchfd_callback: watch w=0x7fe63c4b9180 wpath=@releaseDomain token=3/0: event epath=@releaseDomain 2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1207:domain_death_xswatch_callback: [evg=0x7fe63c3ac070:0] nentries=200 rc=14 0..1388 2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1218:domain_death_xswatch_callback: [evg=0x7fe63c3ac070:0] got=domaininfos[0] got->domain=0 2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1245:domain_death_xswatch_callback: exists shutdown_reported=0 dominf.flags=ffff0020 2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1218:domain_death_xswatch_callback: [evg=0x7fe63c1564d0:1374] got=domaininfos[0] got->domain=0 2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1218:domain_death_xswatch_callback: [evg=0x7fe63c1564d0:1374] got=domaininfos[1] got->domain=1374 2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1245:domain_death_xswatch_callback: exists shutdown_reported=0 dominf.flags=ffff0022 2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1218:domain_death_xswatch_callback: [evg=0x7fe67c0be740:1375] got=domaininfos[1] got->domain=1374 2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1218:domain_death_xswatch_callback: [evg=0x7fe67c0be740:1375] got=domaininfos[2] got->domain=1376 2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1163:domain_death_occurred: missing from list 2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1218:domain_death_xswatch_callback: [evg=0x7fe67c0c4cd0:1376] got=domaininfos[2] got->domain=1376 2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1245:domain_death_xswatch_callback: exists shutdown_reported=0 dominf.flags=ffff0012 2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1218:domain_death_xswatch_callback: [evg=0x7fe63c4937e0:1378] got=domaininfos[2] got->domain=1376 2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1218:domain_death_xswatch_callback: [evg=0x7fe63c4937e0:1378] got=domaininfos[3] got->domain=1378 2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1245:domain_death_xswatch_callback: exists shutdown_reported=0 dominf.flags=ffff0012 2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1218:domain_death_xswatch_callback: [evg=0x7fe68407e8d0:1379] got=domaininfos[3] got->domain=1378 2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1218:domain_death_xswatch_callback: [evg=0x7fe68407e8d0:1379] got=domaininfos[4] got->domain=1379 2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1245:domain_death_xswatch_callback: exists shutdown_reported=0 dominf.flags=ffff0012 2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1218:domain_death_xswatch_callback: [evg=0x7fe63c497090:1380] got=domaininfos[4] got->domain=1379 2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1218:domain_death_xswatch_callback: [evg=0x7fe63c497090:1380] got=domaininfos[5] got->domain=1380 2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1245:domain_death_xswatch_callback: exists shutdown_reported=0 dominf.flags=ffff0012 2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1218:domain_death_xswatch_callback: [evg=0x7fe6800c1bc0:1381] got=domaininfos[5] got->domain=1380 2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1218:domain_death_xswatch_callback: [evg=0x7fe6800c1bc0:1381] got=domaininfos[6] got->domain=1381 2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1245:domain_death_xswatch_callback: exists shutdown_reported=0 dominf.flags=ffff0012 2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1218:domain_death_xswatch_callback: [evg=0x7fe6780a2c90:1382] got=domaininfos[6] got->domain=1381 2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1218:domain_death_xswatch_callback: [evg=0x7fe6780a2c90:1382] got=domaininfos[7] got->domain=1382 2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1245:domain_death_xswatch_callback: exists shutdown_reported=0 dominf.flags=ffff0022 2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1218:domain_death_xswatch_callback: [evg=0x7fe63c4c5850:1383] got=domaininfos[7] got->domain=1382 2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1218:domain_death_xswatch_callback: [evg=0x7fe63c4c5850:1383] got=domaininfos[8] got->domain=1383 2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1245:domain_death_xswatch_callback: exists shutdown_reported=0 dominf.flags=ffff0012 2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1218:domain_death_xswatch_callback: [evg=0x7fe684080480:1384] got=domaininfos[8] got->domain=1383 2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1218:domain_death_xswatch_callback: [evg=0x7fe684080480:1384] got=domaininfos[9] got->domain=1384 2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1245:domain_death_xswatch_callback: exists shutdown_reported=0 dominf.flags=ffff0012 2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1218:domain_death_xswatch_callback: [evg=0x7fe6800b6d50:1385] got=domaininfos[9] got->domain=1384 2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1218:domain_death_xswatch_callback: [evg=0x7fe6800b6d50:1385] got=domaininfos[10] got->domain=1385 2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1245:domain_death_xswatch_callback: exists shutdown_reported=0 dominf.flags=ffff0012 2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1218:domain_death_xswatch_callback: [evg=0x7fe6800a7440:1386] got=domaininfos[10] got->domain=1385 2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1218:domain_death_xswatch_callback: [evg=0x7fe6800a7440:1386] got=domaininfos[11] got->domain=1386 2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1245:domain_death_xswatch_callback: exists shutdown_reported=0 dominf.flags=ffff0002 2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1218:domain_death_xswatch_callback: [evg=0x7fe6800b74a0:1387] got=domaininfos[11] got->domain=1386 2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1218:domain_death_xswatch_callback: [evg=0x7fe6800b74a0:1387] got=domaininfos[12] got->domain=1387 2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1245:domain_death_xswatch_callback: exists shutdown_reported=0 dominf.flags=ffff0022 2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1218:domain_death_xswatch_callback: [evg=0x7fe63c1550d0:1388] got=domaininfos[12] got->domain=1387 2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1218:domain_death_xswatch_callback: [evg=0x7fe63c1550d0:1388] got=domaininfos[13] got->domain=1388 2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1245:domain_death_xswatch_callback: exists shutdown_reported=0 dominf.flags=ffff0012 2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1211:domain_death_xswatch_callback: [evg=0] all reported 2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1274:domain_death_xswatch_callback: domain death search done 2016-09-20 15:00:38 CEST libxl: debug: libxl_event.c:1874:libxl__ao_complete: ao 0x7fe6340145c0: complete, rc=0 2016-09-20 15:00:38 CEST libxl: debug: libxl_event.c:1843:libxl__ao__destroy: ao 0x7fe6340145c0: destroy the output is from an fedora23 with xen-4.6.1 and libvirt-2.1.0 build with the patches, which you provide last year. And it has done 1400 vm restores, at the moment, that i write here. all the best guido ________________________________________ Von: Jim Fehlig [jfehlig@suse.com] Gesendet: Freitag, 16. September 2016 22:28 An: Andrea Bolognani; Rossmueller, Guido; libvir-list@redhat.com Cc: ustermann78@web.de Betreff: Re: [libvirt] xen/libvirt freeze while attching network-device to vm´s - question about provided patches On 09/09/2016 09:43 AM, Andrea Bolognani wrote:
On Thu, 2016-09-08 at 14:04 +0000, Guido.Rossmueller@gdata.de wrote:
Hello everybody,
a colleague of me described last november a problem that we have with libvirt and xen
https://www.redhat.com/archives/libvir-list/2015-November/msg00130.html
Jim Fehlig provided for this problem a patch a few days later:
https://www.redhat.com/archives/libvir-list/2015-November/msg00520.html
The provided patch work great and we have no problems until we applied them..
Now we a in the situation that we want to migrate to libvirt 2.1, and we saw the same problem that we have last year, again. After we take a look in the source code, we determine that the patches are didn´t go in the source code. So we modify the patches in that way, that we could apply them again, and all looks good.
My question, is their any special reason why this patches didn´t go in the source code?
I see same posts on the maillinglist, but absolute statement for or again the patches from Jim Fehlig. Jim said he would submit a v2 to address some concerns raised during review[1]; later, he reported some issues he'd run into[2] while working on it.
I had planned to wait a few days to see if Ian had any thoughts on the libxl side. I'm not familiar with the code and would need some time to understand it before debugging further. But I've never got back to that work :-(.
So it looks to me like the code was not yet ready for merge, which would explain it being tagged as RFC. And then it probably just slipped through the cracks :)
Sort of. It is an item that is fermenting on my todo list.
CCing Jim for insights on the current status of the patch.
I still have a branch in my local git tree :-). It needs rebased and tested against Xen 4.7 (and probably xen.git master).
[1] https://www.redhat.com/archives/libvir-list/2015-November/msg00958.html [2] https://www.redhat.com/archives/libvir-list/2015-November/msg00974.html
Guido, With the rebased version of the patch, do you notice the issue described in [2] during your testing? To trigger it, I used a hacky script that created, shutdown, destroyed, saved, and restored many domains in parallel. What version of Xen are you using? Regards, Jim ____________ Virus checked by G Data MailSecurity Version: AVA 25.8356 dated 20.09.2016 Virus news: www.antiviruslab.com
Guido.Rossmueller@gdata.de wrote:
Hi Jim,
thanks for your response ans sorry for my late answer,
i´m sorry, i don´t really understand what you are missing in [2], or from where in the log-output you see that an shutdown event is missing.
Well, during my testing of the patch I found that on occasion libxl was not delivering a shutdown event to libvirt. But after re-reading the patch thread I'm not sure how I mis-interpreted Ian Campbell's suggestion here https://www.redhat.com/archives/libvir-list/2015-November/msg00851.html I've posted a much simpler patch implementing Ian's suggestion, which so far is working great in my testing https://www.redhat.com/archives/libvir-list/2016-September/msg00970.html Can you please test that patch in your environment and report back? Thanks! Regards, Jim
Hello, we have tested the provided patches from Jim Fehlig. Our testsetup based on an Fedora 23 with xen 4.6.1 and libvirt 2.1 The testsystem restores 15 vm´s (WindowsXP , HVM) in parallel, attached a block-device to each vm and an nic, run´s the vm 60 sec and then destroy them. The testsystem did 18150 restores and we observe no problems. so from our side it looks realy good and we would embrace, if the patches would be include in the libvirt code. all the best guido ________________________________________ Von: Jim Fehlig [jfehlig@suse.com] Gesendet: Mittwoch, 21. September 2016 23:36 An: Rossmueller, Guido Cc: abologna@redhat.com; libvir-list@redhat.com; ustermann78@web.de Betreff: Re: AW: [libvirt] xen/libvirt freeze while attching network-device to vm´s - question about provided patches Guido.Rossmueller@gdata.de wrote:
Hi Jim,
thanks for your response ans sorry for my late answer,
i´m sorry, i don´t really understand what you are missing in [2], or from where in the log-output you see that an shutdown event is missing.
Well, during my testing of the patch I found that on occasion libxl was not delivering a shutdown event to libvirt. But after re-reading the patch thread I'm not sure how I mis-interpreted Ian Campbell's suggestion here https://www.redhat.com/archives/libvir-list/2015-November/msg00851.html I've posted a much simpler patch implementing Ian's suggestion, which so far is working great in my testing https://www.redhat.com/archives/libvir-list/2016-September/msg00970.html Can you please test that patch in your environment and report back? Thanks! Regards, Jim ____________ Virus checked by G Data MailSecurity Version: AVA 25.8447 dated 28.09.2016 Virus news: www.antiviruslab.com
On 09/28/2016 07:56 AM, Guido.Rossmueller@gdata.de wrote:
Hello,
we have tested the provided patches from Jim Fehlig. Our testsetup based on an Fedora 23 with xen 4.6.1 and libvirt 2.1
The testsystem restores 15 vm´s (WindowsXP , HVM) in parallel, attached a block-device to each vm and an nic, run´s the vm 60 sec and then destroy them.
The testsystem did 18150 restores and we observe no problems.
so from our side it looks realy good and we would embrace, if the patches would be include in the libvirt code.
Just to be clear, the patch you tested is this one right? https://www.redhat.com/archives/libvir-list/2016-September/msg00970.html At least I hope that is the patch you tested :-). I've already pushed it to libvirt.git and it will be included in the upcoming 2.3.0 release. Regards, Jim
Hi Jim, yes, i tested the patch https://www.redhat.com/archives/libvir-list/2016-September/msg00970.html all the best guido ________________________________________ Von: Jim Fehlig [jfehlig@suse.com] Gesendet: Mittwoch, 28. September 2016 16:08 An: Rossmueller, Guido Cc: abologna@redhat.com; libvir-list@redhat.com; ustermann78@web.de Betreff: Re: AW: AW: [libvirt] xen/libvirt freeze while attching network-device to vm´s - question about provided patches On 09/28/2016 07:56 AM, Guido.Rossmueller@gdata.de wrote:
Hello,
we have tested the provided patches from Jim Fehlig. Our testsetup based on an Fedora 23 with xen 4.6.1 and libvirt 2.1
The testsystem restores 15 vm´s (WindowsXP , HVM) in parallel, attached a block-device to each vm and an nic, run´s the vm 60 sec and then destroy them.
The testsystem did 18150 restores and we observe no problems.
so from our side it looks realy good and we would embrace, if the patches would be include in the libvirt code.
Just to be clear, the patch you tested is this one right? https://www.redhat.com/archives/libvir-list/2016-September/msg00970.html At least I hope that is the patch you tested :-). I've already pushed it to libvirt.git and it will be included in the upcoming 2.3.0 release. Regards, Jim ____________ Virus checked by G Data MailSecurity Version: AVA 25.8447 dated 28.09.2016 Virus news: www.antiviruslab.com
On 09/28/2016 08:15 AM, Guido.Rossmueller@gdata.de wrote:
Hi Jim,
yes, i tested the patch https://www.redhat.com/archives/libvir-list/2016-September/msg00970.html
Awesome! Thanks for testing and providing feedback. Regards, Jim
participants (3)
-
Andrea Bolognani -
Guido.Rossmueller＠gdata.de -
Jim Fehlig