Hi everyone, commit cc3d52e[1] has apparently broken 'make dist' for at least some platforms: Fedora 20 is fine[2], but both Fedora 21 and rawhide[3][4] are currently failing. My laptop is running Fedora 22 and can make dist just fine, which is of course very confusing. The issue is caused by the following tar error: tar: libvirt-1.2.18/tests/nodeinfodata/linux-deconfigured-cpus/ \ node/node0/cpu16/topology/core_siblings_list: \ link name is too long; not dumped There seem to be basically two ways to solve the failure: 1. rename the test case, making it shorter; 2. bump tar format from ustar, which has limitations on the length of file names, to posix/pax, which has no such limitations. I'm not a fan of the former option because short names are usually very opaque, see tests/nodeinfodata/linux-test[0-8] for some perfect examples. On the other hand, I'm not sure whether bumping the tar version would affect portability in a significant way: the automake documentation[5] reports that this format is very young and should probably be restricted to packages that target only very modern platforms but the GNU tar documentation[6], while containing some warnings as well, suggest that this format is going to become the default. Any suggestions on how to proceed? Cheers. [1] https://www.redhat.com/archives/libvir-list/2015-July/msg00662.html [2] https://ci.centos.org/view/libvirt-project/job/libvirt-daemon -rpm/systems=libvirt-fedora-20/327/ [3] https://ci.centos.org/view/libvirt-project/job/libvirt-daemon -rpm/systems=libvirt-fedora-21/327/ [4] https://ci.centos.org/view/libvirt-project/job/libvirt-daemon -rpm/systems=libvirt-fedora-rawhide/327/ [5] https://www.gnu.org/software/automake/manual/html_node/List-of -Automake-options.html [6] http://www.gnu.org/software/tar/manual/html_chapter/tar_8.html -- Andrea Bolognani Software Engineer - Virtualization Team
On Wed, Jul 22, 2015 at 04:44:56PM +0200, Andrea Bolognani wrote:
Hi everyone,
commit cc3d52e[1] has apparently broken 'make dist' for at least some platforms: Fedora 20 is fine[2], but both Fedora 21 and rawhide[3][4] are currently failing. My laptop is running Fedora 22 and can make dist just fine, which is of course very confusing.
The issue is caused by the following tar error:
tar: libvirt-1.2.18/tests/nodeinfodata/linux-deconfigured-cpus/ \ node/node0/cpu16/topology/core_siblings_list: \ link name is too long; not dumped
There seem to be basically two ways to solve the failure:
1. rename the test case, making it shorter;
2. bump tar format from ustar, which has limitations on the length of file names, to posix/pax, which has no such limitations.
I'm not a fan of the former option because short names are usually very opaque, see tests/nodeinfodata/linux-test[0-8] for some perfect examples.
On the other hand, I'm not sure whether bumping the tar version would affect portability in a significant way: the automake documentation[5] reports that
this format is very young and should probably be restricted to packages that target only very modern platforms
but the GNU tar documentation[6], while containing some warnings as well, suggest that this format is going to become the default.
Any suggestions on how to proceed?
We've hit this many times before and always just shorten the file names in GIT to avoid it. Really we could do with adding a check for filename length to make check to avoid this again. Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On Wed, 2015-07-22 at 15:58 +0100, Daniel P. Berrange wrote:
Any suggestions on how to proceed?
We've hit this many times before and always just shorten the file names in GIT to avoid it.
Really we could do with adding a check for filename length to make check to avoid this again.
Might not be trivial, as limits are apparently different for symbolic links and regular files, nevermind the fact that very similar versions of tar (eg. Fedora 22 and Fedora rawhide) don't behave the same. Anyway, if bumping the tar format is not acceptable, I guess the only option is to shorten the name of the test case. Cheers. -- Andrea Bolognani Software Engineer - Virtualization Team
On Wed, Jul 22, 2015 at 05:49:20PM +0200, Andrea Bolognani wrote:
On Wed, 2015-07-22 at 15:58 +0100, Daniel P. Berrange wrote:
Any suggestions on how to proceed?
We've hit this many times before and always just shorten the file names in GIT to avoid it.
Really we could do with adding a check for filename length to make check to avoid this again.
Might not be trivial, as limits are apparently different for symbolic links and regular files, nevermind the fact that very similar versions of tar (eg. Fedora 22 and Fedora rawhide) don't behave the same.
It isn't that hard. The max symlink length is 100.
Anyway, if bumping the tar format is not acceptable, I guess the only option is to shorten the name of the test case.
Cheers.
-- Andrea Bolognani Software Engineer - Virtualization Team
Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On Wed, Jul 22, 2015 at 04:59:35PM +0100, Daniel P. Berrange wrote:
On Wed, Jul 22, 2015 at 05:49:20PM +0200, Andrea Bolognani wrote:
On Wed, 2015-07-22 at 15:58 +0100, Daniel P. Berrange wrote:
Any suggestions on how to proceed?
We've hit this many times before and always just shorten the file names in GIT to avoid it.
Really we could do with adding a check for filename length to make check to avoid this again.
Might not be trivial, as limits are apparently different for symbolic links and regular files, nevermind the fact that very similar versions of tar (eg. Fedora 22 and Fedora rawhide) don't behave the same.
It isn't that hard. The max symlink length is 100.
Sorry, replied too soon. See next mail. Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On Wed, Jul 22, 2015 at 05:49:20PM +0200, Andrea Bolognani wrote:
On Wed, 2015-07-22 at 15:58 +0100, Daniel P. Berrange wrote:
Any suggestions on how to proceed?
We've hit this many times before and always just shorten the file names in GIT to avoid it.
Really we could do with adding a check for filename length to make check to avoid this again.
Might not be trivial, as limits are apparently different for symbolic links and regular files, nevermind the fact that very similar versions of tar (eg. Fedora 22 and Fedora rawhide) don't behave the same.
If we assume 'libvirt-X.Y.Z/' (with double-digits for each version field) as a prefix, that's upto 17 characters consumed. With a limit of 100 chars for symlinks, we should simply make sure all our file names are 83 characters or less $ ./build-aux/vc-list-files | perl -e 'while (<>) { die "filename $_ too long" if length($_) > 83 }' filename tests/cputestdata/x86-host-Haswell-noTSX+Haswell-noTSX,haswell,Haswell-noTSX-result.xml too long at -e line 1, <> line 1280. Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On Wed, 2015-07-22 at 17:01 +0100, Daniel P. Berrange wrote:
Might not be trivial, as limits are apparently different for symbolic links and regular files, nevermind the fact that very similar versions of tar (eg. Fedora 22 and Fedora rawhide) don't behave the same.
If we assume 'libvirt-X.Y.Z/' (with double-digits for each version field) as a prefix, that's upto 17 characters consumed. With a limit of 100 chars for symlinks, we should simply make sure all our file names are 83 characters or less
$ ./build-aux/vc-list-files | perl -e 'while (<>) { die "filename $_ too long" if length($_) > 83 }' filename tests/cputestdata/x86-host-Haswell-noTSX+Haswell -noTSX,haswell,Haswell-noTSX-result.xml too long at -e line 1, <> line 1280.
Unfortunately, that script[1] doesn't catch any of the files that are breaking the build, and prints out a bunch of false positives instead. That's because vc-list-files explicitly filters out symbolic links; even if it didn't, you would only get the name of the actual symbolic link in the output, for example tests/nodeinfodata/linux-deconfigured-cpus/node/node17/cpu120 which is still way shorter than 83 characters and as such wouldn't trigger the error condition. The problem is that, IIUC, all symbolic links are converted to hard links when creating the tarball and if the path to the file requiring this conversion has length >100, like libvirt-1.2.18/tests/nodeinfodata/linux-deconfigured-cpus/ \ node/node17/cpu120/topology/thread_siblings then we get into trouble. Cheers. [1] with the call to die() replaced with a print() so that it doesn't stop at the first error -- Andrea Bolognani Software Engineer - Virtualization Team
On 07/22/2015 08:44 AM, Andrea Bolognani wrote:
1. rename the test case, making it shorter;
Works if we can do it, but I agree we should add a syntax check to enforce it.
2. bump tar format from ustar, which has limitations on the length of file names, to posix/pax, which has no such limitations.
Might be doable, if RHEL 5 autoconf/automake/tar supports it. Here's where we switched to ustar: https://www.redhat.com/archives/libvir-list/2012-April/msg01435.html commit 75ad8b67 and even then, we ran into problems with too-long symlinks (which are processed differently than regular files). -- Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
On Wed, Jul 22, 2015 at 09:57:02 -0600, Eric Blake wrote:
On 07/22/2015 08:44 AM, Andrea Bolognani wrote:
1. rename the test case, making it shorter;
Works if we can do it, but I agree we should add a syntax check to enforce it.
2. bump tar format from ustar, which has limitations on the length of file names, to posix/pax, which has no such limitations.
Might be doable, if RHEL 5 autoconf/automake/tar supports it. Here's where we switched to ustar:
Do we actually really think that anybody would ever use a very recent libvirt with a very old operating system? I think we should draw a line at some point. </rant> Also RHEL-5 tar can use --format=posix: [root@rhel-5 ~]# cat /etc/redhat-release Red Hat Enterprise Linux Server release 5.11 (Tikanga) [root@rhel-5 ~]# tar --format=posix -cvvf /tmp/garbage /etc/redhat-release tar: Removing leading `/' from member names -rw-r--r-- root/root 55 2014-08-12 07:07:17 /etc/redhat-release [root@rhel-5 ~]# tar --format=pax -cvvf /tmp/garbage /etc/redhat-release tar: Removing leading `/' from member names -rw-r--r-- root/root 55 2014-08-12 07:07:17 /etc/redhat-release Peter
This issue appeared again: tar: libvirt-1.3.6/tests/virhostcpudata/linux-deconf-cpus/node/node17/cpu136/topology/thread_siblings_list: link name is too long; not dumped tar: libvirt-1.3.6/tests/virhostcpudata/linux-deconf-cpus/node/node17/cpu128/topology/thread_siblings_list: link name is too long; not dumped tar: libvirt-1.3.6/tests/virhostcpudata/linux-deconf-cpus/node/node17/cpu120/topology/thread_siblings_list: link name is too long; not dumped tar: libvirt-1.3.6/tests/virhostcpudata/linux-deconf-cpus/node/node16/cpu112/topology/thread_siblings_list: link name is too long; not dumped tar: libvirt-1.3.6/tests/virhostcpudata/linux-deconf-cpus/node/node16/cpu104/topology/thread_siblings_list: link name is too long; not dumped Affects fedora 23 but not fedora 24 for whatever reason On 07/22/2015 11:57 AM, Eric Blake wrote:
On 07/22/2015 08:44 AM, Andrea Bolognani wrote:
1. rename the test case, making it shorter;
Works if we can do it, but I agree we should add a syntax check to enforce it.
2. bump tar format from ustar, which has limitations on the length of file names, to posix/pax, which has no such limitations.
Might be doable, if RHEL 5 autoconf/automake/tar supports it. Here's where we switched to ustar: https://www.redhat.com/archives/libvir-list/2012-April/msg01435.html commit 75ad8b67
and even then, we ran into problems with too-long symlinks (which are processed differently than regular files).
Since RHEL5 support has been dropped for a while now, maybe it's time to revisit changing the tar format - Cole
On Sat, Jun 11, 2016 at 10:14:11AM -0400, Cole Robinson wrote:
This issue appeared again:
On 07/22/2015 11:57 AM, Eric Blake wrote:
On 07/22/2015 08:44 AM, Andrea Bolognani wrote:
1. rename the test case, making it shorter;
Works if we can do it, but I agree we should add a syntax check to enforce it.
2. bump tar format from ustar, which has limitations on the length of file names, to posix/pax, which has no such limitations.
Might be doable, if RHEL 5 autoconf/automake/tar supports it. Here's where we switched to ustar: https://www.redhat.com/archives/libvir-list/2012-April/msg01435.html commit 75ad8b67
and even then, we ran into problems with too-long symlinks (which are processed differently than regular files).
Since RHEL5 support has been dropped for a while now, maybe it's time to revisit changing the tar format
Yep, IIUC we should be fine for require pax support for the vintage of Linux we required. *BSD should be fine too, so IIUC, their tar version uses libarchive which supports pax. Windows has 7-zip which can do pax and of course cygwin. Finally OS-X has the pax command and support in the apple archive utility. So I think we're be fine to require it. While, we're changing this, I think we should probably take the opportunity to also switch over to using 'xz' as our compression format, instead of gz. Consider the 1.3.5 release compressed with different formats: 35109092 libvirt-1.3.5.tar.gz 25573966 libvirt-1.3.5.tar.bz2 12112612 libvirt-1.3.5.tar.xz Those results seem pretty compelling to me :-) Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On Mon, 2016-06-13 at 09:57 +0100, Daniel P. Berrange wrote:
Since RHEL5 support has been dropped for a while now, maybe it's time to revisit changing the tar format
Yep, IIUC we should be fine for require pax support for the vintage of Linux we required. *BSD should be fine too, so IIUC, their tar version uses libarchive which supports pax. Windows has 7-zip which can do pax and of course cygwin. Finally OS-X has the pax command and support in the apple archive utility.
So I think we're be fine to require it.
While, we're changing this, I think we should probably take the opportunity to also switch over to using 'xz' as our compression format, instead of gz. Consider the 1.3.5 release compressed with different formats:
35109092 libvirt-1.3.5.tar.gz 25573966 libvirt-1.3.5.tar.bz2 12112612 libvirt-1.3.5.tar.xz
Those results seem pretty compelling to me :-)
xz compression sure takes a lot of time! I vote yes to both, though :) -- Andrea Bolognani Software Engineer - Virtualization Team
On 13.06.2016 11:46, Andrea Bolognani wrote:
On Mon, 2016-06-13 at 09:57 +0100, Daniel P. Berrange wrote:
Since RHEL5 support has been dropped for a while now, maybe it's time to revisit changing the tar format
Yep, IIUC we should be fine for require pax support for the vintage of Linux we required. *BSD should be fine too, so IIUC, their tar version uses libarchive which supports pax. Windows has 7-zip which can do pax and of course cygwin. Finally OS-X has the pax command and support in the apple archive utility.
So I think we're be fine to require it.
While, we're changing this, I think we should probably take the opportunity to also switch over to using 'xz' as our compression format, instead of gz. Consider the 1.3.5 release compressed with different formats:
35109092 libvirt-1.3.5.tar.gz 25573966 libvirt-1.3.5.tar.bz2 12112612 libvirt-1.3.5.tar.xz
Those results seem pretty compelling to me :-)
xz compression sure takes a lot of time!
Maybe it does, but it's done just once, while decompression is done multiple times. So I think we can switch to xz. In fact, I'd be okay with nothing but xz. But will this solve the issue? I mean, the problem that Cole is seeing (and I'm too) with too long path names. Isn't tar the origin of it? Because if it is, I fear that changing compression algorithm won't help much. Michal
On Wed, Jun 15, 2016 at 01:25:35PM +0200, Michal Privoznik wrote:
On 13.06.2016 11:46, Andrea Bolognani wrote:
On Mon, 2016-06-13 at 09:57 +0100, Daniel P. Berrange wrote:
Since RHEL5 support has been dropped for a while now, maybe it's time to revisit changing the tar format
Yep, IIUC we should be fine for require pax support for the vintage of Linux we required. *BSD should be fine too, so IIUC, their tar version uses libarchive which supports pax. Windows has 7-zip which can do pax and of course cygwin. Finally OS-X has the pax command and support in the apple archive utility.
So I think we're be fine to require it.
While, we're changing this, I think we should probably take the opportunity to also switch over to using 'xz' as our compression format, instead of gz. Consider the 1.3.5 release compressed with different formats:
35109092 libvirt-1.3.5.tar.gz 25573966 libvirt-1.3.5.tar.bz2 12112612 libvirt-1.3.5.tar.xz
Those results seem pretty compelling to me :-)
xz compression sure takes a lot of time!
Maybe it does, but it's done just once, while decompression is done multiple times. So I think we can switch to xz. In fact, I'd be okay with nothing but xz.
But will this solve the issue? I mean, the problem that Cole is seeing (and I'm too) with too long path names. Isn't tar the origin of it? Because if it is, I fear that changing compression algorithm won't help much.
We already agreed that we should switch to the pax format, I'm just saying, we should *also* switch to xz compression. Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
participants (6)
-
Andrea Bolognani -
Cole Robinson -
Daniel P. Berrange -
Eric Blake -
Michal Privoznik -
Peter Krempa