[libvirt] genprotocol.pl generates unstable output
If the same source gets built twice the resulting files may differ. One example is this: ... [ 305s] /usr/bin/perl -w ./rpc/genprotocol.pl /usr/bin/rpcgen -c \ [ 305s] remote/remote_protocol.x ./remote/remote_protocol.c ... [ 577s] /usr/src/debug/libvirt-20150929T082652.68572de/src/remote/remote_protocol.c differs (C source, ASCII text) [ 577s] --- old//usr/src/debug/libvirt-20150929T082652.68572de/src/remote/remote_protocol.c 2015-09-30 11:34:31.000000000 +0000 [ 577s] +++ new//usr/src/debug/libvirt-20150929T082652.68572de/src/remote/remote_protocol.c 2015-09-30 14:57:39.000000000 +0000 [ 577s] @@ -2084,8 +2084,8 @@ [ 577s] bool_t [ 577s] xdr_remote_domain_get_vcpus_ret (XDR *xdrs, remote_domain_get_vcpus_ret *objp) [ 577s] { [ 577s] - char **objp_cpp1 = (char **) (void *) &objp->cpumaps.cpumaps_val; [ 577s] char **objp_cpp0 = (char **) (void *) &objp->info.info_val; [ 577s] + char **objp_cpp1 = (char **) (void *) &objp->cpumaps.cpumaps_val; [ 577s] [ 577s] if (!xdr_array (xdrs, objp_cpp0, (u_int *) &objp->info.info_len, REMOTE_VCPUINFO_MAX, [ 577s] sizeof (remote_vcpu_info), (xdrproc_t) xdr_remote_vcpu_info)) ... The input is like: bool_t xdr_remote_domain_get_vcpus_ret (XDR *xdrs, remote_domain_get_vcpus_ret *objp) { register int32_t *buf; if (!xdr_array (xdrs, (char **)&objp->info.info_val, (u_int *) &objp->info.info_len, REMOTE_VCPUINFO_MAX, sizeof (remote_vcpu_info), (xdrproc_t) xdr_remote_vcpu_info)) return FALSE; if (!xdr_bytes (xdrs, (char **)&objp->cpumaps.cpumaps_val, (u_int *) &objp->cpumaps.cpumaps_len, REMOTE_CPUMAPS_MAX)) return FALSE; return TRUE; } Looks like the perl script transforms the pointers, but it does not sort them to enforce an order. I think its the map() which does it, but dont know enough perl to be sure. How can this be fixed? Olaf
On Thu, Oct 01, 2015 at 08:44:55AM +0200, Olaf Hering wrote:
If the same source gets built twice the resulting files may differ. One example is this:
... [ 305s] /usr/bin/perl -w ./rpc/genprotocol.pl /usr/bin/rpcgen -c \ [ 305s] remote/remote_protocol.x ./remote/remote_protocol.c ... [ 577s] /usr/src/debug/libvirt-20150929T082652.68572de/src/remote/remote_protocol.c differs (C source, ASCII text) [ 577s] --- old//usr/src/debug/libvirt-20150929T082652.68572de/src/remote/remote_protocol.c 2015-09-30 11:34:31.000000000 +0000 [ 577s] +++ new//usr/src/debug/libvirt-20150929T082652.68572de/src/remote/remote_protocol.c 2015-09-30 14:57:39.000000000 +0000 [ 577s] @@ -2084,8 +2084,8 @@ [ 577s] bool_t [ 577s] xdr_remote_domain_get_vcpus_ret (XDR *xdrs, remote_domain_get_vcpus_ret *objp) [ 577s] { [ 577s] - char **objp_cpp1 = (char **) (void *) &objp->cpumaps.cpumaps_val; [ 577s] char **objp_cpp0 = (char **) (void *) &objp->info.info_val; [ 577s] + char **objp_cpp1 = (char **) (void *) &objp->cpumaps.cpumaps_val; [ 577s]
I don't see the problem, these pointers are not depending on each other. Or are you saying that it leaves your tree dirty? Because that has not happened to me and I believe others did not have a problem with it either, otherwise this would happen a LOT of times.
[ 577s] if (!xdr_array (xdrs, objp_cpp0, (u_int *) &objp->info.info_len, REMOTE_VCPUINFO_MAX, [ 577s] sizeof (remote_vcpu_info), (xdrproc_t) xdr_remote_vcpu_info)) ...
The input is like:
bool_t xdr_remote_domain_get_vcpus_ret (XDR *xdrs, remote_domain_get_vcpus_ret *objp) { register int32_t *buf;
if (!xdr_array (xdrs, (char **)&objp->info.info_val, (u_int *) &objp->info.info_len, REMOTE_VCPUINFO_MAX, sizeof (remote_vcpu_info), (xdrproc_t) xdr_remote_vcpu_info)) return FALSE; if (!xdr_bytes (xdrs, (char **)&objp->cpumaps.cpumaps_val, (u_int *) &objp->cpumaps.cpumaps_len, REMOTE_CPUMAPS_MAX)) return FALSE; return TRUE; }
Looks like the perl script transforms the pointers, but it does not sort them to enforce an order. I think its the map() which does it, but dont know enough perl to be sure. How can this be fixed?
I don't know that much about perl myself and I don't quite get it from the genprotocol.pl, but you could sort the output before printing, for sure. But I'm still more curious about how come that causes you something that's not happening for me.
Olaf
-- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
On Wed, Oct 14, Martin Kletzander wrote:
On Thu, Oct 01, 2015 at 08:44:55AM +0200, Olaf Hering wrote:
If the same source gets built twice the resulting files may differ.
I don't see the problem, these pointers are not depending on each other. Or are you saying that it leaves your tree dirty? Because that has not happened to me and I believe others did not have a problem with it either, otherwise this would happen a LOT of times.
'built twice' means 'build same source on different hosts at different times'. I want to get reproducible build results for binary packages.
Looks like the perl script transforms the pointers, but it does not sort them to enforce an order. I think its the map() which does it, but dont know enough perl to be sure. How can this be fixed?
I don't know that much about perl myself and I don't quite get it from the genprotocol.pl, but you could sort the output before printing, for sure. But I'm still more curious about how come that causes you something that's not happening for me.
I'm sure it returns reproducible results on localhost. But even if it did not noone would notice because if the file happens to differ it will just get rebuilt. Olaf
On Thu, Oct 15, 2015 at 09:59:24AM +0200, Olaf Hering wrote:
On Wed, Oct 14, Martin Kletzander wrote:
On Thu, Oct 01, 2015 at 08:44:55AM +0200, Olaf Hering wrote:
If the same source gets built twice the resulting files may differ.
I don't see the problem, these pointers are not depending on each other. Or are you saying that it leaves your tree dirty? Because that has not happened to me and I believe others did not have a problem with it either, otherwise this would happen a LOT of times.
'built twice' means 'build same source on different hosts at different times'. I want to get reproducible build results for binary packages.
That makes total sense. It should be mostly just a case of adding calls to the perl sort function in various places in the code generator. Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On Thu, Oct 15, 2015 at 09:59:24AM +0200, Olaf Hering wrote:
On Wed, Oct 14, Martin Kletzander wrote:
On Thu, Oct 01, 2015 at 08:44:55AM +0200, Olaf Hering wrote:
If the same source gets built twice the resulting files may differ.
I don't see the problem, these pointers are not depending on each other. Or are you saying that it leaves your tree dirty? Because that has not happened to me and I believe others did not have a problem with it either, otherwise this would happen a LOT of times.
'built twice' means 'build same source on different hosts at different times'. I want to get reproducible build results for binary packages.
Oh, so if you're wondering about reproducibility, then it's a different thing, I understand that. And libvirt is not a good one with regards to reproducible builds. So we should be able to fix that by sorting the output. I'll see once more how to do that in genprotocol.pl, but I don't promise anything. Feel free to also submit even a partial solution as a patch to the list.
Looks like the perl script transforms the pointers, but it does not sort them to enforce an order. I think its the map() which does it, but dont know enough perl to be sure. How can this be fixed?
I don't know that much about perl myself and I don't quite get it from the genprotocol.pl, but you could sort the output before printing, for sure. But I'm still more curious about how come that causes you something that's not happening for me.
I'm sure it returns reproducible results on localhost. But even if it did not noone would notice because if the file happens to differ it will just get rebuilt.
Olaf
On Thu, Oct 15, Martin Kletzander wrote:
Oh, so if you're wondering about reproducibility, then it's a different thing, I understand that. And libvirt is not a good one with regards to reproducible builds.
libvirt is perfect. Its just that little detail which pops up ever other day. Olaf
participants (3)
-
Daniel P. Berrange -
Martin Kletzander -
Olaf Hering