[libvirt] xen/libvirt freeze while attching network-device to vm´s - question about provided patches
9:04 a.m.
Hello everybody,
a colleague of me described last november a problem that we have with libvirt and xen
https://www.redhat.com/archives/libvir-list/2015-November/msg00130.html
Jim Fehlig provided for this problem a patch a few days later:
https://www.redhat.com/archives/libvir-list/2015-November/msg00520.html
The provided patch work great and we have no problems until we applied them.
Now we a in the situation that we want to migrate to libvirt 2.1, and we saw the same
problem that we have last year, again.
After we take a look in the source code, we determine that the patches are didn´t go in
the source code. So we modify the patches in that way, that we could apply them again, and
all looks good.
My question, is their any special reason why this patches didn´t go in the source code?
I see same posts on the maillinglist, but absolute statement for or again the patches from
Jim Fehlig.
all the best guido
____________
Virus checked by G Data MailSecurity
Version: AVA 25.8195 dated 08.09.2016
Virus news: www.antiviruslab.com
10:43 a.m.
On Thu, 2016-09-08 at 14:04 +0000, Guido.Rossmueller(a)gdata.de wrote:
Hello everybody,
a colleague of me described last november a problem that we have with libvirt and xen
https://www.redhat.com/archives/libvir-list/2015-November/msg00130.html
Jim Fehlig provided for this problem a patch a few days later:
https://www.redhat.com/archives/libvir-list/2015-November/msg00520.html
The provided patch work great and we have no problems until we applied them..
Now we a in the situation that we want to migrate to libvirt 2.1, and we saw the same
problem that we have last year, again.
After we take a look in the source code, we determine that the patches are didn´t go
in the source code. So we modify the patches in that way, that we could apply them
again, and all looks good.
My question, is their any special reason why this patches didn´t go in the source code?
I see same posts on the maillinglist, but absolute statement for or again the patches
from Jim Fehlig.
Jim said he would submit a v2 to address some concerns raised during
review[1]; later, he reported some issues he'd run into[2] while
working on it.
So it looks to me like the code was not yet ready for merge, which
would explain it being tagged as RFC. And then it probably just
slipped through the cracks :)
CCing Jim for insights on the current status of the patch.
[1] https://www.redhat.com/archives/libvir-list/2015-November/msg00958.html
[2] https://www.redhat.com/archives/libvir-list/2015-November/msg00974.html
--
Andrea Bolognani / Red Hat / Virtualization
3:28 p.m.
On 09/09/2016 09:43 AM, Andrea Bolognani wrote:
On Thu, 2016-09-08 at 14:04 +0000, Guido.Rossmueller(a)gdata.de wrote:
> Hello everybody,
>
> a colleague of me described last november a problem that we have with libvirt and
xen
>
> https://www.redhat.com/archives/libvir-list/2015-November/msg00130.html
>
> Jim Fehlig provided for this problem a patch a few days later:
>
> https://www.redhat.com/archives/libvir-list/2015-November/msg00520.html
>
> The provided patch work great and we have no problems until we applied them..
>
> Now we a in the situation that we want to migrate to libvirt 2.1, and we saw the
same
> problem that we have last year, again.
> After we take a look in the source code, we determine that the patches are didn´t go
> in the source code. So we modify the patches in that way, that we could apply them
> again, and all looks good.
>
> My question, is their any special reason why this patches didn´t go in the source
code?
>
> I see same posts on the maillinglist, but absolute statement for or again the
patches
> from Jim Fehlig.
Jim said he would submit a v2 to address some concerns raised during
review[1]; later, he reported some issues he'd run into[2] while
working on it.
I had planned to wait a few days to see if Ian had any thoughts on the libxl
side. I'm not familiar with the code and would need some time to understand it
before debugging further. But I've never got back to that work :-(.
So it looks to me like the code was not yet ready for merge, which
would explain it being tagged as RFC. And then it probably just
slipped through the cracks :)
Sort of. It is an item that is fermenting on my todo list.
CCing Jim for insights on the current status of the patch.
I still have a branch in my local git tree :-). It needs rebased and tested
against Xen 4.7 (and probably xen.git master).
Guido,
With the rebased version of the patch, do you notice the issue described in [2]
during your testing? To trigger it, I used a hacky script that created,
shutdown, destroyed, saved, and restored many domains in parallel.
What version of Xen are you using?
Regards,
Jim
8:17 a.m.
Hi Jim,
thanks for your response ans sorry for my late answer,
i´m sorry, i don´t really understand what you are missing in [2], or from where in the
log-output you see that an
shutdown event is missing.
on our test-system the output looks normaly as follow:
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1719:devices_destroy_cb: forked pid 14987
for destroy of domain 1375
2016-09-20 15:00:38 CEST libxl: debug: libxl_event.c:576:watchfd_callback: watch
w=0x7fe63c4b9180 wpath=@releaseDomain token=3/0: event epath=@releaseDomain
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1207:domain_death_xswatch_callback:
[evg=0x7fe63c3ac070:0] nentries=200 rc=14 0..1388
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1218:domain_death_xswatch_callback:
[evg=0x7fe63c3ac070:0] got=domaininfos[0] got->domain=0
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1245:domain_death_xswatch_callback: exists
shutdown_reported=0 dominf.flags=ffff0020
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1218:domain_death_xswatch_callback:
[evg=0x7fe63c1564d0:1374] got=domaininfos[0] got->domain=0
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1218:domain_death_xswatch_callback:
[evg=0x7fe63c1564d0:1374] got=domaininfos[1] got->domain=1374
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1245:domain_death_xswatch_callback: exists
shutdown_reported=0 dominf.flags=ffff0022
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1218:domain_death_xswatch_callback:
[evg=0x7fe67c0be740:1375] got=domaininfos[1] got->domain=1374
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1218:domain_death_xswatch_callback:
[evg=0x7fe67c0be740:1375] got=domaininfos[2] got->domain=1376
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1163:domain_death_occurred: missing from
list
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1218:domain_death_xswatch_callback:
[evg=0x7fe67c0c4cd0:1376] got=domaininfos[2] got->domain=1376
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1245:domain_death_xswatch_callback: exists
shutdown_reported=0 dominf.flags=ffff0012
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1218:domain_death_xswatch_callback:
[evg=0x7fe63c4937e0:1378] got=domaininfos[2] got->domain=1376
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1218:domain_death_xswatch_callback:
[evg=0x7fe63c4937e0:1378] got=domaininfos[3] got->domain=1378
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1245:domain_death_xswatch_callback: exists
shutdown_reported=0 dominf.flags=ffff0012
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1218:domain_death_xswatch_callback:
[evg=0x7fe68407e8d0:1379] got=domaininfos[3] got->domain=1378
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1218:domain_death_xswatch_callback:
[evg=0x7fe68407e8d0:1379] got=domaininfos[4] got->domain=1379
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1245:domain_death_xswatch_callback: exists
shutdown_reported=0 dominf.flags=ffff0012
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1218:domain_death_xswatch_callback:
[evg=0x7fe63c497090:1380] got=domaininfos[4] got->domain=1379
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1218:domain_death_xswatch_callback:
[evg=0x7fe63c497090:1380] got=domaininfos[5] got->domain=1380
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1245:domain_death_xswatch_callback: exists
shutdown_reported=0 dominf.flags=ffff0012
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1218:domain_death_xswatch_callback:
[evg=0x7fe6800c1bc0:1381] got=domaininfos[5] got->domain=1380
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1218:domain_death_xswatch_callback:
[evg=0x7fe6800c1bc0:1381] got=domaininfos[6] got->domain=1381
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1245:domain_death_xswatch_callback: exists
shutdown_reported=0 dominf.flags=ffff0012
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1218:domain_death_xswatch_callback:
[evg=0x7fe6780a2c90:1382] got=domaininfos[6] got->domain=1381
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1218:domain_death_xswatch_callback:
[evg=0x7fe6780a2c90:1382] got=domaininfos[7] got->domain=1382
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1245:domain_death_xswatch_callback: exists
shutdown_reported=0 dominf.flags=ffff0022
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1218:domain_death_xswatch_callback:
[evg=0x7fe63c4c5850:1383] got=domaininfos[7] got->domain=1382
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1218:domain_death_xswatch_callback:
[evg=0x7fe63c4c5850:1383] got=domaininfos[8] got->domain=1383
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1245:domain_death_xswatch_callback: exists
shutdown_reported=0 dominf.flags=ffff0012
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1218:domain_death_xswatch_callback:
[evg=0x7fe684080480:1384] got=domaininfos[8] got->domain=1383
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1218:domain_death_xswatch_callback:
[evg=0x7fe684080480:1384] got=domaininfos[9] got->domain=1384
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1245:domain_death_xswatch_callback: exists
shutdown_reported=0 dominf.flags=ffff0012
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1218:domain_death_xswatch_callback:
[evg=0x7fe6800b6d50:1385] got=domaininfos[9] got->domain=1384
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1218:domain_death_xswatch_callback:
[evg=0x7fe6800b6d50:1385] got=domaininfos[10] got->domain=1385
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1245:domain_death_xswatch_callback: exists
shutdown_reported=0 dominf.flags=ffff0012
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1218:domain_death_xswatch_callback:
[evg=0x7fe6800a7440:1386] got=domaininfos[10] got->domain=1385
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1218:domain_death_xswatch_callback:
[evg=0x7fe6800a7440:1386] got=domaininfos[11] got->domain=1386
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1245:domain_death_xswatch_callback: exists
shutdown_reported=0 dominf.flags=ffff0002
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1218:domain_death_xswatch_callback:
[evg=0x7fe6800b74a0:1387] got=domaininfos[11] got->domain=1386
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1218:domain_death_xswatch_callback:
[evg=0x7fe6800b74a0:1387] got=domaininfos[12] got->domain=1387
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1245:domain_death_xswatch_callback: exists
shutdown_reported=0 dominf.flags=ffff0022
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1218:domain_death_xswatch_callback:
[evg=0x7fe63c1550d0:1388] got=domaininfos[12] got->domain=1387
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1218:domain_death_xswatch_callback:
[evg=0x7fe63c1550d0:1388] got=domaininfos[13] got->domain=1388
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1245:domain_death_xswatch_callback: exists
shutdown_reported=0 dominf.flags=ffff0012
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1211:domain_death_xswatch_callback: [evg=0]
all reported
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1274:domain_death_xswatch_callback: domain
death search done
2016-09-20 15:00:38 CEST libxl: debug: libxl_event.c:1874:libxl__ao_complete: ao
0x7fe6340145c0: complete, rc=0
2016-09-20 15:00:38 CEST libxl: debug: libxl_event.c:1843:libxl__ao__destroy: ao
0x7fe6340145c0: destroy
the output is from an fedora23 with xen-4.6.1 and libvirt-2.1.0 build with the patches,
which you provide last year.
And it has done 1400 vm restores, at the moment, that i write here.
all the best
guido
________________________________________
Von: Jim Fehlig [jfehlig(a)suse.com]
Gesendet: Freitag, 16. September 2016 22:28
An: Andrea Bolognani; Rossmueller, Guido; libvir-list(a)redhat.com
Cc: ustermann78(a)web.de
Betreff: Re: [libvirt] xen/libvirt freeze while attching network-device to vm´s - question
about provided patches
On 09/09/2016 09:43 AM, Andrea Bolognani wrote:
On Thu, 2016-09-08 at 14:04 +0000, Guido.Rossmueller(a)gdata.de wrote:
> Hello everybody,
>
> a colleague of me described last november a problem that we have with libvirt and
xen
>
> https://www.redhat.com/archives/libvir-list/2015-November/msg00130.html
>
> Jim Fehlig provided for this problem a patch a few days later:
>
> https://www.redhat.com/archives/libvir-list/2015-November/msg00520.html
>
> The provided patch work great and we have no problems until we applied them..
>
> Now we a in the situation that we want to migrate to libvirt 2.1, and we saw the
same
> problem that we have last year, again.
> After we take a look in the source code, we determine that the patches are didn´t go
> in the source code. So we modify the patches in that way, that we could apply them
> again, and all looks good.
>
> My question, is their any special reason why this patches didn´t go in the source
code?
>
> I see same posts on the maillinglist, but absolute statement for or again the
patches
> from Jim Fehlig.
Jim said he would submit a v2 to address some concerns raised during
review[1]; later, he reported some issues he'd run into[2] while
working on it.
I had planned to wait a few days to see if Ian had any thoughts on the libxl
side. I'm not familiar with the code and would need some time to understand it
before debugging further. But I've never got back to that work :-(.
So it looks to me like the code was not yet ready for merge, which
would explain it being tagged as RFC. And then it probably just
slipped through the cracks :)
Sort of. It is an item that is fermenting on my todo list.
CCing Jim for insights on the current status of the patch.
I still have a branch in my local git tree :-). It needs rebased and tested
against Xen 4.7 (and probably xen.git master).
[1] https://www.redhat.com/archives/libvir-list/2015-November/msg00958.html
[2] https://www.redhat.com/archives/libvir-list/2015-November/msg00974.html
Guido,
With the rebased version of the patch, do you notice the issue described in [2]
during your testing? To trigger it, I used a hacky script that created,
shutdown, destroyed, saved, and restored many domains in parallel.
What version of Xen are you using?
Regards,
Jim
____________
Virus checked by G Data MailSecurity
Version: AVA 25.8356 dated 20.09.2016
Virus news: www.antiviruslab.com
4:36 p.m.
Guido.Rossmueller(a)gdata.de wrote:
Hi Jim,
thanks for your response ans sorry for my late answer,
i´m sorry, i don´t really understand what you are missing in [2], or from where in the
log-output you see that an
shutdown event is missing.
Well, during my testing of the patch I found that on occasion libxl was not
delivering a shutdown event to libvirt. But after re-reading the patch thread
I'm not sure how I mis-interpreted Ian Campbell's suggestion here
https://www.redhat.com/archives/libvir-list/2015-November/msg00851.html
I've posted a much simpler patch implementing Ian's suggestion, which so far is
working great in my testing
https://www.redhat.com/archives/libvir-list/2016-September/msg00970.html
Can you please test that patch in your environment and report back? Thanks!
Regards,
Jim
8:56 a.m.
Hello,
we have tested the provided patches from Jim Fehlig.
Our testsetup based on an Fedora 23 with xen 4.6.1 and libvirt 2.1
The testsystem restores 15 vm´s (WindowsXP , HVM) in parallel, attached a block-device to
each vm and an nic, run´s the vm 60 sec and then destroy them.
The testsystem did 18150 restores and we observe no problems.
so from our side it looks realy good and we would embrace, if the patches would be include
in the libvirt code.
all the best
guido
________________________________________
Von: Jim Fehlig [jfehlig(a)suse.com]
Gesendet: Mittwoch, 21. September 2016 23:36
An: Rossmueller, Guido
Cc: abologna(a)redhat.com; libvir-list(a)redhat.com; ustermann78(a)web.de
Betreff: Re: AW: [libvirt] xen/libvirt freeze while attching network-device to vm´s -
question about provided patches
Guido.Rossmueller(a)gdata.de wrote:
Hi Jim,
thanks for your response ans sorry for my late answer,
i´m sorry, i don´t really understand what you are missing in [2], or from where in the
log-output you see that an
shutdown event is missing.
Well, during my testing of the patch I found that on occasion libxl was not
delivering a shutdown event to libvirt. But after re-reading the patch thread
I'm not sure how I mis-interpreted Ian Campbell's suggestion here
https://www.redhat.com/archives/libvir-list/2015-November/msg00851.html
I've posted a much simpler patch implementing Ian's suggestion, which so far is
working great in my testing
https://www.redhat.com/archives/libvir-list/2016-September/msg00970.html
Can you please test that patch in your environment and report back? Thanks!
Regards,
Jim
____________
Virus checked by G Data MailSecurity
Version: AVA 25.8447 dated 28.09.2016
Virus news: www.antiviruslab.com
9:08 a.m.
On 09/28/2016 07:56 AM, Guido.Rossmueller(a)gdata.de wrote:
Hello,
we have tested the provided patches from Jim Fehlig.
Our testsetup based on an Fedora 23 with xen 4.6.1 and libvirt 2.1
The testsystem restores 15 vm´s (WindowsXP , HVM) in parallel, attached a block-device to
each vm and an nic, run´s the vm 60 sec and then destroy them.
The testsystem did 18150 restores and we observe no problems.
so from our side it looks realy good and we would embrace, if the patches would be
include in the libvirt code.
Just to be clear, the patch you tested is this one right?
https://www.redhat.com/archives/libvir-list/2016-September/msg00970.html
At least I hope that is the patch you tested :-). I've already pushed it to
libvirt.git and it will be included in the upcoming 2.3.0 release.
Regards,
Jim
9:15 a.m.
Hi Jim,
yes, i tested the patch
https://www.redhat.com/archives/libvir-list/2016-September/msg00970.html
all the best guido
________________________________________
Von: Jim Fehlig [jfehlig(a)suse.com]
Gesendet: Mittwoch, 28. September 2016 16:08
An: Rossmueller, Guido
Cc: abologna(a)redhat.com; libvir-list(a)redhat.com; ustermann78(a)web.de
Betreff: Re: AW: AW: [libvirt] xen/libvirt freeze while attching network-device to vm´s -
question about provided patches
On 09/28/2016 07:56 AM, Guido.Rossmueller(a)gdata.de wrote:
Hello,
we have tested the provided patches from Jim Fehlig.
Our testsetup based on an Fedora 23 with xen 4.6.1 and libvirt 2.1
The testsystem restores 15 vm´s (WindowsXP , HVM) in parallel, attached a block-device to
each vm and an nic, run´s the vm 60 sec and then destroy them.
The testsystem did 18150 restores and we observe no problems.
so from our side it looks realy good and we would embrace, if the patches would be
include in the libvirt code.
Just to be clear, the patch you tested is this one right?
https://www.redhat.com/archives/libvir-list/2016-September/msg00970.html
At least I hope that is the patch you tested :-). I've already pushed it to
libvirt.git and it will be included in the upcoming 2.3.0 release.
Regards,
Jim
____________
Virus checked by G Data MailSecurity
Version: AVA 25.8447 dated 28.09.2016
Virus news: www.antiviruslab.com
9:21 a.m.
On 09/28/2016 08:15 AM, Guido.Rossmueller(a)gdata.de wrote:
Hi Jim,
yes, i tested the patch
https://www.redhat.com/archives/libvir-list/2016-September/msg00970.html
Awesome! Thanks for testing and providing feedback.
Regards,
Jim
2977
days inactive
2997
days old
8 comments
3 participants
participants (3)
-
Andrea Bolognani -
Guido.Rossmueller＠gdata.de -
Jim Fehlig