As suggested yesterday, I have just tagged the release candidate 1 in git, and pushed signed tarbal and rpms to the usual place: ftp://libvirt.org/libvirt/ seems to work fine with my limited testing, and https://ci.centos.org/view/libvirt/ is all green (except for virt-viewer-master-rpm ?)

so things looks pretty good for me but please try it out on different systems and OSes. If everything goes well I will push rc2 on Tuesday targetting Thursday for the final release (or Friday if I get stuck in travels). thanks in advance for trying it out !

On Sat, 2018-07-28 at 21:56 +0800, Daniel Veillard wrote: >

Unfortunately I've spotted an issue during my testing of rc1 today: with the libvirt_guest NSS module enabled, Evolution would crash a few seconds after being started. Here's the stack trace: #0 0x00007fffe7b69ba5 in json_object_iter_next () at /lib64/libjson-glib-1.0.so.0 #1 0x00007fffad8e757b in virJSONValueFromJansson () at /lib64/libnss_libvirt_guest.so.2 #2 0x00007fffad8e75d8 in virJSONValueFromJansson () at /lib64/libnss_libvirt_guest.so.2 #3 0x00007fffad8e8994 in virJSONValueFromString () at /lib64/libnss_libvirt_guest.so.2 #4 0x00007fffad8ecb5a in virMacMapNew () at /lib64/libnss_libvirt_guest.so.2 #5 0x00007fffad8cc140 in findLease () at /lib64/libnss_libvirt_guest.so.2 #6 0x00007fffad8ccb1c in _nss_libvirt_guest_gethostbyname4_r () at /lib64/libnss_libvirt_guest.so.2 #7 0x00007fffeb2599d2 in gaih_inet.constprop () at /lib64/libc.so.6 #8 0x00007fffeb25aab4 in getaddrinfo () at /lib64/libc.so.6 #9 0x00007ffff1d41a04 in do_lookup_by_name () at /lib64/libgio-2.0.so.0 #10 0x00007ffff1d3e937 in g_task_thread_pool_thread () at /lib64/libgio-2.0.so.0 #11 0x00007ffff5c39933 in g_thread_pool_thread_proxy () at /lib64/libglib-2.0.so.0 #12 0x00007ffff5c38f2a in g_thread_proxy () at /lib64/libglib-2.0.so.0 #13 0x00007ffff6314594 in start_thread () at /lib64/libpthread.so.0 #14 0x00007fffeb2700df in clone () at /lib64/libc.so.6 I've talked about it with a few colleagues and we believe the issue to be caused by jansson and json-glib both exporting a symbol called json_object_iter_next: Evolution itself (indirectly?) links against the latter library, so when the libvirt_guest NSS module is loaded and attempts to process JSON using the former, it picks up the wrong implementation, leading to a crash. gnome-boxes also crashes with the same stack trace.

It seems like a similar issue could affect any application linking both to libvirt and json-glib, regardless of whether or not the NSS plugin has been enabled, which is of course pretty bad.

Unfortunately, I don't have any bright ideas on how to solve this, so anyone who might: please step forward! We're just a few days away from the next release, and if we can't figure out a way around this soon I'm afraid the only reasonable course of action would be to (temporarily) revert the switch from yajl to jansson.

On Tue, Jul 31, 2018 at 10:26:41AM +0200, Michal Privoznik wrote: > On 07/30/2018 05:20 PM, Andrea Bolognani wrote: > > On Sat, 2018-07-28 at 21:56 +0800, Daniel Veillard wrote: > > > > > > > > Unfortunately I've spotted an issue during my testing of rc1 today: > > with the libvirt_guest NSS module enabled, Evolution would crash a > > few seconds after being started. Here's the stack trace: > > > > #0 0x00007fffe7b69ba5 in json_object_iter_next () at /lib64/libjson-glib-1.0.so.0 > > #1 0x00007fffad8e757b in virJSONValueFromJansson () at /lib64/libnss_libvirt_guest.so.2 > > #2 0x00007fffad8e75d8 in virJSONValueFromJansson () at /lib64/libnss_libvirt_guest.so.2 > > #3 0x00007fffad8e8994 in virJSONValueFromString () at /lib64/libnss_libvirt_guest.so.2 > > #4 0x00007fffad8ecb5a in virMacMapNew () at /lib64/libnss_libvirt_guest.so.2 > > #5 0x00007fffad8cc140 in findLease () at /lib64/libnss_libvirt_guest.so.2 > > #6 0x00007fffad8ccb1c in _nss_libvirt_guest_gethostbyname4_r () at /lib64/libnss_libvirt_guest.so.2 > > #7 0x00007fffeb2599d2 in gaih_inet.constprop () at /lib64/libc.so.6 > > #8 0x00007fffeb25aab4 in getaddrinfo () at /lib64/libc.so.6 > > #9 0x00007ffff1d41a04 in do_lookup_by_name () at /lib64/libgio-2.0.so.0 > > #10 0x00007ffff1d3e937 in g_task_thread_pool_thread () at /lib64/libgio-2.0.so.0 > > #11 0x00007ffff5c39933 in g_thread_pool_thread_proxy () at /lib64/libglib-2.0.so.0 > > #12 0x00007ffff5c38f2a in g_thread_proxy () at /lib64/libglib-2.0.so.0 > > #13 0x00007ffff6314594 in start_thread () at /lib64/libpthread.so.0 > > #14 0x00007fffeb2700df in clone () at /lib64/libc.so.6 > > > > I've talked about it with a few colleagues and we believe the issue > > to be caused by jansson and json-glib both exporting a symbol called > > json_object_iter_next: Evolution itself (indirectly?) links against For the libraries installed on my Fedora 28, objdump shows json_object_iter_next as the only conflicting symbol, maybe we can get away with using a different iterator as a quick fix. > > the latter library, so when the libvirt_guest NSS module is loaded > > and attempts to process JSON using the former, it picks up the wrong > > implementation, leading to a crash. gnome-boxes also crashes with > > the same stack trace. > > Worse. querying gentoo portage I've found some important packages > requiring json-glib: > > x11-libs/gtk > gnome-base/gnome-shell > > So once users of these app update to latest libvirt they will see the > crashes. > > > > > It seems like a similar issue could affect any application linking > > both to libvirt and json-glib, regardless of whether or not the NSS > > plugin has been enabled, which is of course pretty bad. > > Yes, any application can crash. > > > > > Unfortunately, I don't have any bright ideas on how to solve this, > > so anyone who might: please step forward! We're just a few days > > away from the next release, and if we can't figure out a way around > > this soon I'm afraid the only reasonable course of action would be > > to (temporarily) revert the switch from yajl to jansson. > > > > Well, what if we linked with jansson statically? I'm not sure if it is > possible (and have no idea how to achieve that), but what if our dynamic > libraries we produce already contained jansson and thus linker would not > even try to resolve json_* symbols. For the client library, we can just compile out JSON - it should not be needed for anything. And we can generate the data for libvirt_nss in some simpler format.

On 07/30/2018 05:20 PM, Andrea Bolognani wrote: > On Sat, 2018-07-28 at 21:56 +0800, Daniel Veillard wrote: >> > > Unfortunately I've spotted an issue during my testing of rc1 today: > with the libvirt_guest NSS module enabled, Evolution would crash a > few seconds after being started. Here's the stack trace: > > #0 0x00007fffe7b69ba5 in json_object_iter_next () at /lib64/libjson-glib-1.0.so.0 > #1 0x00007fffad8e757b in virJSONValueFromJansson () at /lib64/libnss_libvirt_guest.so.2 > #2 0x00007fffad8e75d8 in virJSONValueFromJansson () at /lib64/libnss_libvirt_guest.so.2 > #3 0x00007fffad8e8994 in virJSONValueFromString () at /lib64/libnss_libvirt_guest.so.2 > #4 0x00007fffad8ecb5a in virMacMapNew () at /lib64/libnss_libvirt_guest.so.2 > #5 0x00007fffad8cc140 in findLease () at /lib64/libnss_libvirt_guest.so.2 > #6 0x00007fffad8ccb1c in _nss_libvirt_guest_gethostbyname4_r () at /lib64/libnss_libvirt_guest.so.2 > #7 0x00007fffeb2599d2 in gaih_inet.constprop () at /lib64/libc.so.6 > #8 0x00007fffeb25aab4 in getaddrinfo () at /lib64/libc.so.6 > #9 0x00007ffff1d41a04 in do_lookup_by_name () at /lib64/libgio-2.0.so.0 > #10 0x00007ffff1d3e937 in g_task_thread_pool_thread () at /lib64/libgio-2.0.so.0 > #11 0x00007ffff5c39933 in g_thread_pool_thread_proxy () at /lib64/libglib-2.0.so.0 > #12 0x00007ffff5c38f2a in g_thread_proxy () at /lib64/libglib-2.0.so.0 > #13 0x00007ffff6314594 in start_thread () at /lib64/libpthread.so.0 > #14 0x00007fffeb2700df in clone () at /lib64/libc.so.6 > > I've talked about it with a few colleagues and we believe the issue > to be caused by jansson and json-glib both exporting a symbol called > json_object_iter_next: Evolution itself (indirectly?) links against > the latter library, so when the libvirt_guest NSS module is loaded > and attempts to process JSON using the former, it picks up the wrong > implementation, leading to a crash. gnome-boxes also crashes with > the same stack trace. Worse. querying gentoo portage I've found some important packages requiring json-glib: x11-libs/gtk gnome-base/gnome-shell So once users of these app update to latest libvirt they will see the crashes. > > It seems like a similar issue could affect any application linking > both to libvirt and json-glib, regardless of whether or not the NSS > plugin has been enabled, which is of course pretty bad. Yes, any application can crash. > > Unfortunately, I don't have any bright ideas on how to solve this, > so anyone who might: please step forward! We're just a few days > away from the next release, and if we can't figure out a way around > this soon I'm afraid the only reasonable course of action would be > to (temporarily) revert the switch from yajl to jansson. > Well, what if we linked with jansson statically? I'm not sure if it is possible (and have no idea how to achieve that), but what if our dynamic libraries we produce already contained jansson and thus linker would not even try to resolve json_* symbols.

Michal -- libvir-list mailing list libvir-list(a)redhat.com https://www.redhat.com/mailman/listinfo/libvir-list

The same would happen if libvirt used `json-glib` and some application linking with libvirt would start using jansson (and also use some specific functions). Not that we were guarded against that now.  I'm not saying the release can go on, of course not, just that the ultimate fix is not something *we* should do.

It seems like a similar issue could affect any application linking both to libvirt and json-glib, regardless of whether or not the NSS plugin has been enabled, which is of course pretty bad. Unfortunately, I don't have any bright ideas on how to solve this, so anyone who might: please step forward! We're just a few days away from the next release, and if we can't figure out a way around this soon I'm afraid the only reasonable course of action would be to (temporarily) revert the switch from yajl to jansson.

It is out, tagged in git and with signed tarball and rpms at the usual place: ftp://libvirt.org/libvirt/ in my limited testing it works but we have that pending issue raised by Andrea. worse case someone reverse the patches and allows to build against the old lib. Please give it some testing, I will watch the commits and if there isn't any solution 2 days from now I will postpone the GA by a couple of days. Hopefully that won't be needed :-) thanks in advance, Daniel -- Daniel Veillard | Red Hat Developers Tools http://developer.redhat.com/ veillard(a)redhat.com | libxml Gnome XML XSLT toolkit http://xmlsoft.org/ http://veillard.com/ | virtualization library http://libvirt.org/ -- libvir-list mailing list libvir-list(a)redhat.com https://www.redhat.com/mailman/listinfo/libvir-list

I have not yet had the time to figure out what goes wrong, any ideas are welcome.

On 07/31/2018 03:54 PM, Bjoern Walk wrote: > Bjoern Walk <bwalk(a)linux.ibm.com&gt; [2018-07-31, 03:16PM +0200]: >> I have not yet had the time to figure out what goes wrong, any ideas are welcome. > > Ah, classic. The mocked virRandomBytes function is not endian-agnostic, > generating a different interface name as expected by the test. > Ugrh. Are you working on a fix or should I give it a try?

Michal -- libvir-list mailing list libvir-list(a)redhat.com https://www.redhat.com/mailman/listinfo/libvir-list

On Tue, Jul 31, 2018 at 03:54:43PM +0200, Bjoern Walk wrote: > Bjoern Walk <bwalk(a)linux.ibm.com&gt; [2018-07-31, 03:16PM +0200]: > > I have not yet had the time to figure out what goes wrong, any ideas are welcome. > > Ah, classic. The mocked virRandomBytes function is not endian-agnostic, > generating a different interface name as expected by the test. I'm not seeing why virRandomBytes is affected by endian-ness. It is simply populating an array of bytes, so there's no endin issues to consider there.

Can you elaborate on the actual error messages you are getting from the tests, and what aspect makes you think virRandomBytes is the problem ?

Seems more likely that it is something higher up the call stack.

Regards, Daniel

On Wed, Aug 01, 2018 at 01:41:48PM +0200, Bjoern Walk wrote: > Daniel P. Berrangé <berrange(a)redhat.com&gt; [2018-08-01, 11:51AM +0100]: > > On Tue, Jul 31, 2018 at 03:54:43PM +0200, Bjoern Walk wrote: > > > Bjoern Walk <bwalk(a)linux.ibm.com&gt; [2018-07-31, 03:16PM +0200]: > > > > I have not yet had the time to figure out what goes wrong, any ideas are welcome. > > > > > > Ah, classic. The mocked virRandomBytes function is not endian-agnostic, > > > generating a different interface name as expected by the test. > > > > I'm not seeing why virRandomBytes is affected by endian-ness. It is simply > > populating an array of bytes, so there's no endin issues to consider there. > > Ahem, we are writing linearly to a byte array, of course this is > dependend on the endianness of the architecture. The actual problem is > that the mocked version does _not_ provide a random value, but a > deterministic one, which is byte order reversed on big endianness. There's no concept of reversed byte order when you're accessing an array of bytes. If you write elements 0, 1, 2, ...7, etc and then the caller reads elements 0, 1,2, ..., 7 everything is fine. Endianness only comes into play if you take that array of bytes and then cast/assign it to a larger type were endianess is relevant (eg int16/int32/int64) eg char bytes[8]; virRandomBytes(bytes, 8); uint64_t val = (uint64_t)bytes; the problem in such a case isn't virRandomBytes, it is the cast to the larger sized integer type. > > Can you elaborate on the actual error messages you are getting from the > > tests, and what aspect makes you think virRandomBytes is the problem ? > > The name of the interface is wrong compared to what is explicitly > expected in the test case: > > 200 if (virAsprintf(&temp_ifacename, > (gdb) > 205 VIR_DEBUG("Attempting to create interface '%s' with IQN '%s'", > (gdb) p temp_ifacename > $1 = 0x1014320 "libvirt-iface-04050607" > > > Seems more likely that it is something higher up the call stack. That looks like it uses virRandomBits rather than virRandomBytes.

On Tue, Jul 31, 2018 at 12:19:28PM +0800, Daniel Veillard wrote: > It is out, tagged in git and with signed tarball and rpms at the > usual place: > > ftp://libvirt.org/libvirt/ > > in my limited testing it works but we have that pending issue > raised by Andrea. worse case someone reverse the patches and > allows to build against the old lib. > > Please give it some testing, I will watch the commits and > if there isn't any solution 2 days from now I will postpone the > GA by a couple of days. Hopefully that won't be needed :-) > Since commit 50edca1331298bfcb2622e8fe588d493aff9ab68 qemu: monitor: Add the 'query-nodes' argument for query-blockstats https://libvirt.org/git/?p=libvirt.git;a=commitdiff;h=50edca133129 'virsh domblkstat' shows nothing for me with QEMU 2.9.0 (works with v3.0.0-rc0-66-gccf02d73d1)

On 07/31/2018 07:19 PM, Ján Tomko wrote: > And I'm still unsure about leaving in > commit 55ce65646348884656fd7bf3f109ebf8f7603494 >    qemu: Use the correct vm def on cold attach > https://libvirt.org/git/?p=libvirt.git;a=commitdiff;h=55ce6564634 > Which means attach-device --live --config will attach an interface > with a different MAC address in live and persistent definition. Laine? > Well, It's not only MAC address that can change. The device address might change too. This points to a broader problem. When we are parsing a device XML we fill in the blanks in postParse callbacks. However, those look only at either live or at inactive XML. Not at both at the same time. So how can we fill in the blanks that would be valid for both XMLs?

On Wed, Aug 01, 2018 at 12:40:14 +0200, Michal Privoznik wrote: > On 07/31/2018 07:19 PM, Ján Tomko wrote: >> And I'm still unsure about leaving in >> commit 55ce65646348884656fd7bf3f109ebf8f7603494 >>    qemu: Use the correct vm def on cold attach >> https://libvirt.org/git/?p=libvirt.git;a=commitdiff;h=55ce6564634 >> Which means attach-device --live --config will attach an interface >> with a different MAC address in live and persistent definition. Laine? >> > > Well, It's not only MAC address that can change. The device address > might change too. This points to a broader problem. When we are parsing > a device XML we fill in the blanks in postParse callbacks. However, > those look only at either live or at inactive XML. Not at both at the > same time. So how can we fill in the blanks that would be valid for both > XMLs? We can fill in the definition and then copy it and validate it afterwards. That way the blanks are filled and we can then validate that it fits into the other definition. The description here sounds that in this release we made things worse than it was before though.

(Let's see now.... which people were the ones who believe it's a huge insult to Cc them in list replies, and which are the ones who appreciate the Cc and use it as a flag to raise the visibility of the message in their mail client ..... ?????. Ah, I give up, just leaving in the Cc's and let the complaints fly)(For the record, I like explicit Cc's when you want to make sure I see something - that sends a copy to my phone, which can be mildly annoying if it's a series of 40 patches, but also very effective in getting my attention :-P) On 08/01/2018 06:40 AM, Michal Privoznik wrote: > On 07/31/2018 07:19 PM, Ján Tomko wrote: >> And I'm still unsure about leaving in >> commit 55ce65646348884656fd7bf3f109ebf8f7603494 >>    qemu: Use the correct vm def on cold attach >> https://libvirt.org/git/?p=libvirt.git;a=commitdiff;h=55ce6564634 >> Which means attach-device --live --config will attach an interface >> with a different MAC address in live and persistent definition. Laine? (Let's see, you said my name once in the Cc, and once here. You really need to say it *three* times to get my attention ala Beetlejuice or Biggy Smalls. (and it helps to recite it into a mirror in a dark room, apparently)). >> > Well, It's not only MAC address that can change. The device address > might change too. This points to a broader problem. When we are parsing > a device XML we fill in the blanks in postParse callbacks. However, > those look only at either live or at inactive XML. Not at both at the > same time. So how can we fill in the blanks that would be valid for both > XMLs? Yeah, this has been a problem for a long time (it's also problematic for, e.g. the index of PCI controllers, which can cascade into differences in the PCI addresses of devices that are connected to those controllers; similar problem for other types of controllers of course). I think I even tried fixing it once, but the solution was inadequate. The problem comes when someone uses a mixture of --live only, --config only, and --live+--config device attaches/detaches. At the base of the problem is the fact that the live and config domaindef objects are stored completely independent of each other, and the code that assigns PCI addresses "seeds" the list of in-use address from a single domaindef object. Perhaps we need to merge these into a single object where each device is marked as "live", "config" or "both" - then any new device added would be guaranteed to not conflict with any existing device in live or config - the unified device list would be used to determine what addresses are used, and the new device would just be added once to one domaindef object (just with live and config flags set) so there would be no need to copy it. Or, well..., after about 30 seconds of thought I realize this would lead to different behavior if someone e.g. deleted a currently-active device only from the config, then wanted to add that same device back again - the re-added device would end up with a different PCI address. So maybe that isn't the right solution either. At any rate, there is no perfect solution in sight for the current release, so the question is whether the new (bad) behavior is better or worse than the old (also bad) behavior. My understanding is that the old behavior could lead to a config that had two devices at the same PCI address, which is definitely undesirable. The new behavior could lead to the PCI address of a newly-added device being different the next time the guest is shutdown and restarted. I would tend to prefer the latter, with the caveat that this new behavior provides a config that works (from libvirt's domain parsing POV), but might create a strange error in the guest that would be extremely difficult to troubleshoot (especially 6 months from now after we've all forgotten about this patch (and forgotten about the idea that a more complete fix was needed). So I'm undecided about my opinion. And when undecided I tend toward inaction. Now *that's* helpful, isn't it?

On Wed, Aug 01, 2018 at 17:44:56 -0400, John Ferlan wrote: > On 08/01/2018 04:44 PM, Laine Stump wrote: [...] >> At any rate, there is no perfect solution in sight for the current >> release, so the question is whether the new (bad) behavior is better or >> worse than the old (also bad) behavior. My understanding is that the old >> behavior could lead to a config that had two devices at the same PCI >> address, which is definitely undesirable. The new behavior could lead to >> the PCI address of a newly-added device being different the next time >> the guest is shutdown and restarted. I would tend to prefer the latter, >> with the caveat that this new behavior provides a config that works >> (from libvirt's domain parsing POV), but might create a strange error in >> the guest that would be extremely difficult to troubleshoot (especially >> 6 months from now after we've all forgotten about this patch (and >> forgotten about the idea that a more complete fix was needed). >> >> So I'm undecided about my opinion. And when undecided I tend toward >> inaction. Now *that's* helpful, isn't it? >> > > Laine is stumped ;-). > > I'm still stumped over what strange error could be created. How is the > virtual {PCI|SCSI} address changing any different from "real hardware" > if someone adds something new into their physical system? Or the other Well, the issue is that you issue an API to add the device, which in real life would translate into plugging it into the machine. Afterwards you turn the machine off and back on. Without any API (or physical contact) you expect that the hardware will not move places by itself.

If the API call includes both AFFECT_LIVE and AFFECT_CONFIG we should make sure that the device plugged in is exactly the same. If they are issued separately we don't care at all though. Btw, having this analogy, specifying only AFFECT_CONFIG is probably similar to putting a post-it note on top of the power button for the next person to attach the hardware prior to powering it up.

> fun adventure, a physical move from location A to location B where > someone "forgot" to properly label what went where and upon reassembly > things are ordered differently. > > Consider some (perhaps not so) long ago SCSI devices where you'd need to > grab a small, sharp, pin like object to toggle switches to define the > address for the device when it got plugged in. Note that in that era it would be very bad in some cases when the hardware would change the jumper configuration by itself as sometimes the drivers could not autoconfigure. The addresses were put in config files.