9:44 a.m.
Hi everyone,
commit cc3d52e[1] has apparently broken 'make dist' for at least some
platforms: Fedora 20 is fine[2], but both Fedora 21 and rawhide[3][4]
are currently failing. My laptop is running Fedora 22 and can make
dist just fine, which is of course very confusing.
The issue is caused by the following tar error:
tar: libvirt-1.2.18/tests/nodeinfodata/linux-deconfigured-cpus/ \
node/node0/cpu16/topology/core_siblings_list: \
link name is too long; not dumped
There seem to be basically two ways to solve the failure:
1. rename the test case, making it shorter;
2. bump tar format from ustar, which has limitations on the length
of file names, to posix/pax, which has no such limitations.
I'm not a fan of the former option because short names are usually very
opaque, see tests/nodeinfodata/linux-test[0-8] for some perfect
examples.
On the other hand, I'm not sure whether bumping the tar version would
affect portability in a significant way: the automake documentation[5]
reports that
this format is very young and should probably be restricted to
packages that target only very modern platforms
but the GNU tar documentation[6], while containing some warnings as
well, suggest that this format is going to become the default.
Any suggestions on how to proceed?
Cheers.
[1] https://www.redhat.com/archives/libvir-list/2015-July/msg00662.html
[2] https://ci.centos.org/view/libvirt-project/job/libvirt-daemon
-rpm/systems=libvirt-fedora-20/327/
[3] https://ci.centos.org/view/libvirt-project/job/libvirt-daemon
-rpm/systems=libvirt-fedora-21/327/
[4] https://ci.centos.org/view/libvirt-project/job/libvirt-daemon
-rpm/systems=libvirt-fedora-rawhide/327/
[5] https://www.gnu.org/software/automake/manual/html_node/List-of
-Automake-options.html
[6] http://www.gnu.org/software/tar/manual/html_chapter/tar_8.html
--
Andrea Bolognani
Software Engineer - Virtualization Team
9:58 a.m.
On Wed, Jul 22, 2015 at 04:44:56PM +0200, Andrea Bolognani wrote:
Hi everyone,
commit cc3d52e[1] has apparently broken 'make dist' for at least some
platforms: Fedora 20 is fine[2], but both Fedora 21 and rawhide[3][4]
are currently failing. My laptop is running Fedora 22 and can make
dist just fine, which is of course very confusing.
The issue is caused by the following tar error:
tar: libvirt-1.2.18/tests/nodeinfodata/linux-deconfigured-cpus/ \
node/node0/cpu16/topology/core_siblings_list: \
link name is too long; not dumped
There seem to be basically two ways to solve the failure:
1. rename the test case, making it shorter;
2. bump tar format from ustar, which has limitations on the length
of file names, to posix/pax, which has no such limitations.
I'm not a fan of the former option because short names are usually very
opaque, see tests/nodeinfodata/linux-test[0-8] for some perfect
examples.
On the other hand, I'm not sure whether bumping the tar version would
affect portability in a significant way: the automake documentation[5]
reports that
this format is very young and should probably be restricted to
packages that target only very modern platforms
but the GNU tar documentation[6], while containing some warnings as
well, suggest that this format is going to become the default.
Any suggestions on how to proceed?
We've hit this many times before and always just shorten the file
names in GIT to avoid it.
Really we could do with adding a check for filename length to
make check to avoid this again.
Regards,
Daniel
--
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org -o- http://virt-manager.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
10:49 a.m.
On Wed, 2015-07-22 at 15:58 +0100, Daniel P. Berrange wrote:
> Any suggestions on how to proceed?
We've hit this many times before and always just shorten the file
names in GIT to avoid it.
Really we could do with adding a check for filename length to
make check to avoid this again.
Might not be trivial, as limits are apparently different for symbolic
links and regular files, nevermind the fact that very similar versions
of tar (eg. Fedora 22 and Fedora rawhide) don't behave the same.
Anyway, if bumping the tar format is not acceptable, I guess the only
option is to shorten the name of the test case.
Cheers.
--
Andrea Bolognani
Software Engineer - Virtualization Team
10:59 a.m.
On Wed, Jul 22, 2015 at 05:49:20PM +0200, Andrea Bolognani wrote:
On Wed, 2015-07-22 at 15:58 +0100, Daniel P. Berrange wrote:
>
> > Any suggestions on how to proceed?
>
> We've hit this many times before and always just shorten the file
> names in GIT to avoid it.
>
> Really we could do with adding a check for filename length to
> make check to avoid this again.
Might not be trivial, as limits are apparently different for symbolic
links and regular files, nevermind the fact that very similar versions
of tar (eg. Fedora 22 and Fedora rawhide) don't behave the same.
It isn't that hard. The max symlink length is 100.
Anyway, if bumping the tar format is not acceptable, I guess the only
option is to shorten the name of the test case.
Cheers.
--
Andrea Bolognani
Software Engineer - Virtualization Team
Regards,
Daniel
--
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org -o- http://virt-manager.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
11:02 a.m.
On Wed, Jul 22, 2015 at 04:59:35PM +0100, Daniel P. Berrange wrote:
On Wed, Jul 22, 2015 at 05:49:20PM +0200, Andrea Bolognani wrote:
> On Wed, 2015-07-22 at 15:58 +0100, Daniel P. Berrange wrote:
> >
> > > Any suggestions on how to proceed?
> >
> > We've hit this many times before and always just shorten the file
> > names in GIT to avoid it.
> >
> > Really we could do with adding a check for filename length to
> > make check to avoid this again.
>
> Might not be trivial, as limits are apparently different for symbolic
> links and regular files, nevermind the fact that very similar versions
> of tar (eg. Fedora 22 and Fedora rawhide) don't behave the same.
It isn't that hard. The max symlink length is 100.
Sorry, replied too soon. See next mail.
Regards,
Daniel
--
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org -o- http://virt-manager.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
11:01 a.m.
On Wed, Jul 22, 2015 at 05:49:20PM +0200, Andrea Bolognani wrote:
On Wed, 2015-07-22 at 15:58 +0100, Daniel P. Berrange wrote:
>
> > Any suggestions on how to proceed?
>
> We've hit this many times before and always just shorten the file
> names in GIT to avoid it.
>
> Really we could do with adding a check for filename length to
> make check to avoid this again.
Might not be trivial, as limits are apparently different for symbolic
links and regular files, nevermind the fact that very similar versions
of tar (eg. Fedora 22 and Fedora rawhide) don't behave the same.
If we assume 'libvirt-X.Y.Z/' (with double-digits for each version field)
as a prefix, that's upto 17 characters consumed. With a limit of 100 chars
for symlinks, we should simply make sure all our file names are 83 characters
or less
$ ./build-aux/vc-list-files | perl -e 'while (<>) { die "filename $_ too
long" if length($_) > 83 }'
filename
tests/cputestdata/x86-host-Haswell-noTSX+Haswell-noTSX,haswell,Haswell-noTSX-result.xml
too long at -e line 1, <> line 1280.
Regards,
Daniel
--
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org -o- http://virt-manager.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
2:40 a.m.
On Wed, 2015-07-22 at 17:01 +0100, Daniel P. Berrange wrote:
> Might not be trivial, as limits are apparently different for
> symbolic
> links and regular files, nevermind the fact that very similar
> versions
> of tar (eg. Fedora 22 and Fedora rawhide) don't behave the same.
If we assume 'libvirt-X.Y.Z/' (with double-digits for each version
field)
as a prefix, that's upto 17 characters consumed. With a limit of 100
chars
for symlinks, we should simply make sure all our file names are 83
characters
or less
$ ./build-aux/vc-list-files | perl -e 'while (<>) { die "filename $_
too long" if length($_) > 83 }'
filename tests/cputestdata/x86-host-Haswell-noTSX+Haswell
-noTSX,haswell,Haswell-noTSX-result.xml
too long at -e line 1, <> line 1280.
Unfortunately, that script[1] doesn't catch any of the files
that are breaking the build, and prints out a bunch of false
positives instead.
That's because vc-list-files explicitly filters out symbolic
links; even if it didn't, you would only get the name of the
actual symbolic link in the output, for example
tests/nodeinfodata/linux-deconfigured-cpus/node/node17/cpu120
which is still way shorter than 83 characters and as such
wouldn't trigger the error condition.
The problem is that, IIUC, all symbolic links are converted
to hard links when creating the tarball and if the path to
the file requiring this conversion has length >100, like
libvirt-1.2.18/tests/nodeinfodata/linux-deconfigured-cpus/ \
node/node17/cpu120/topology/thread_siblings
then we get into trouble.
Cheers.
[1] with the call to die() replaced with a print() so that it
doesn't stop at the first error
--
Andrea Bolognani
Software Engineer - Virtualization Team
10:57 a.m.
On 07/22/2015 08:44 AM, Andrea Bolognani wrote:
1. rename the test case, making it shorter;
Works if we can do it, but I agree we should add a syntax check to
enforce it.
2. bump tar format from ustar, which has limitations on the length
of file names, to posix/pax, which has no such limitations.
Might be doable, if RHEL 5 autoconf/automake/tar supports it. Here's
where we switched to ustar:
https://www.redhat.com/archives/libvir-list/2012-April/msg01435.html
commit 75ad8b67
and even then, we ran into problems with too-long symlinks (which are
processed differently than regular files).
--
Eric Blake eblake redhat com +1-919-301-3266
Libvirt virtualization library http://libvirt.org
4:44 a.m.
On Wed, Jul 22, 2015 at 09:57:02 -0600, Eric Blake wrote:
On 07/22/2015 08:44 AM, Andrea Bolognani wrote:
> 1. rename the test case, making it shorter;
Works if we can do it, but I agree we should add a syntax check to
enforce it.
>
> 2. bump tar format from ustar, which has limitations on the length
> of file names, to posix/pax, which has no such limitations.
Might be doable, if RHEL 5 autoconf/automake/tar supports it. Here's
where we switched to ustar:
Do we actually really think that anybody would ever use a very recent
libvirt with a very old operating system? I think we should draw a line
at some point. </rant>
Also RHEL-5 tar can use --format=posix:
[root@rhel-5 ~]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 5.11 (Tikanga)
[root@rhel-5 ~]# tar --format=posix -cvvf /tmp/garbage /etc/redhat-release
tar: Removing leading `/' from member names
-rw-r--r-- root/root 55 2014-08-12 07:07:17 /etc/redhat-release
[root@rhel-5 ~]# tar --format=pax -cvvf /tmp/garbage /etc/redhat-release
tar: Removing leading `/' from member names
-rw-r--r-- root/root 55 2014-08-12 07:07:17 /etc/redhat-release
Peter
9:14 a.m.
This issue appeared again:
tar:
libvirt-1.3.6/tests/virhostcpudata/linux-deconf-cpus/node/node17/cpu136/topology/thread_siblings_list:
link name is too long; not dumped
tar:
libvirt-1.3.6/tests/virhostcpudata/linux-deconf-cpus/node/node17/cpu128/topology/thread_siblings_list:
link name is too long; not dumped
tar:
libvirt-1.3.6/tests/virhostcpudata/linux-deconf-cpus/node/node17/cpu120/topology/thread_siblings_list:
link name is too long; not dumped
tar:
libvirt-1.3.6/tests/virhostcpudata/linux-deconf-cpus/node/node16/cpu112/topology/thread_siblings_list:
link name is too long; not dumped
tar:
libvirt-1.3.6/tests/virhostcpudata/linux-deconf-cpus/node/node16/cpu104/topology/thread_siblings_list:
link name is too long; not dumped
Affects fedora 23 but not fedora 24 for whatever reason
On 07/22/2015 11:57 AM, Eric Blake wrote:
On 07/22/2015 08:44 AM, Andrea Bolognani wrote:
> 1. rename the test case, making it shorter;
Works if we can do it, but I agree we should add a syntax check to
enforce it.
>
> 2. bump tar format from ustar, which has limitations on the length
> of file names, to posix/pax, which has no such limitations.
Might be doable, if RHEL 5 autoconf/automake/tar supports it. Here's
where we switched to ustar:
https://www.redhat.com/archives/libvir-list/2012-April/msg01435.html
commit 75ad8b67
and even then, we ran into problems with too-long symlinks (which are
processed differently than regular files).
Since RHEL5 support has been dropped for a while now, maybe it's time to
revisit changing the tar format
- Cole
3:57 a.m.
On Sat, Jun 11, 2016 at 10:14:11AM -0400, Cole Robinson wrote:
This issue appeared again:
On 07/22/2015 11:57 AM, Eric Blake wrote:
> On 07/22/2015 08:44 AM, Andrea Bolognani wrote:
>> 1. rename the test case, making it shorter;
>
> Works if we can do it, but I agree we should add a syntax check to
> enforce it.
>
>>
>> 2. bump tar format from ustar, which has limitations on the length
>> of file names, to posix/pax, which has no such limitations.
>
> Might be doable, if RHEL 5 autoconf/automake/tar supports it. Here's
> where we switched to ustar:
> https://www.redhat.com/archives/libvir-list/2012-April/msg01435.html
> commit 75ad8b67
>
> and even then, we ran into problems with too-long symlinks (which are
> processed differently than regular files).
Since RHEL5 support has been dropped for a while now, maybe it's time to
revisit changing the tar format
Yep, IIUC we should be fine for require pax support for the vintage of
Linux we required. *BSD should be fine too, so IIUC, their tar version
uses libarchive which supports pax. Windows has 7-zip which can do pax
and of course cygwin. Finally OS-X has the pax command and support in
the apple archive utility.
So I think we're be fine to require it.
While, we're changing this, I think we should probably take the opportunity
to also switch over to using 'xz' as our compression format, instead of gz.
Consider the 1.3.5 release compressed with different formats:
35109092 libvirt-1.3.5.tar.gz
25573966 libvirt-1.3.5.tar.bz2
12112612 libvirt-1.3.5.tar.xz
Those results seem pretty compelling to me :-)
Regards,
Daniel
--
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org -o- http://virt-manager.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
4:46 a.m.
On Mon, 2016-06-13 at 09:57 +0100, Daniel P. Berrange wrote:
> Since RHEL5 support has been dropped for a while now, maybe
it's time to
> revisit changing the tar format
Yep, IIUC we should be fine for require pax support for the vintage of
Linux we required. *BSD should be fine too, so IIUC, their tar version
uses libarchive which supports pax. Windows has 7-zip which can do pax
and of course cygwin. Finally OS-X has the pax command and support in
the apple archive utility.
So I think we're be fine to require it.
While, we're changing this, I think we should probably take the opportunity
to also switch over to using 'xz' as our compression format, instead of gz.
Consider the 1.3.5 release compressed with different formats:
35109092 libvirt-1.3.5.tar.gz
25573966 libvirt-1.3.5.tar.bz2
12112612 libvirt-1.3.5.tar.xz
Those results seem pretty compelling to me :-)
xz compression sure takes a lot of time!
I vote yes to both, though :)
--
Andrea Bolognani
Software Engineer - Virtualization Team
6:25 a.m.
On 13.06.2016 11:46, Andrea Bolognani wrote:
On Mon, 2016-06-13 at 09:57 +0100, Daniel P. Berrange wrote:
>> Since RHEL5 support has been dropped for a while now, maybe it's time to
>> revisit changing the tar format
>
> Yep, IIUC we should be fine for require pax support for the vintage of
> Linux we required. *BSD should be fine too, so IIUC, their tar version
> uses libarchive which supports pax. Windows has 7-zip which can do pax
> and of course cygwin. Finally OS-X has the pax command and support in
> the apple archive utility.
>
> So I think we're be fine to require it.
>
> While, we're changing this, I think we should probably take the opportunity
> to also switch over to using 'xz' as our compression format, instead of gz.
> Consider the 1.3.5 release compressed with different formats:
>
> 35109092 libvirt-1.3.5.tar.gz
> 25573966 libvirt-1.3.5.tar.bz2
> 12112612 libvirt-1.3.5.tar.xz
>
> Those results seem pretty compelling to me :-)
xz compression sure takes a lot of time!
Maybe it does, but it's done just once, while decompression is done
multiple times. So I think we can switch to xz. In fact, I'd be okay
with nothing but xz.
But will this solve the issue? I mean, the problem that Cole is seeing
(and I'm too) with too long path names. Isn't tar the origin of it?
Because if it is, I fear that changing compression algorithm won't help
much.
Michal
6:29 a.m.
On Wed, Jun 15, 2016 at 01:25:35PM +0200, Michal Privoznik wrote:
On 13.06.2016 11:46, Andrea Bolognani wrote:
> On Mon, 2016-06-13 at 09:57 +0100, Daniel P. Berrange wrote:
>>> Since RHEL5 support has been dropped for a while now, maybe it's time
to
>>> revisit changing the tar format
>>
>> Yep, IIUC we should be fine for require pax support for the vintage of
>> Linux we required. *BSD should be fine too, so IIUC, their tar version
>> uses libarchive which supports pax. Windows has 7-zip which can do pax
>> and of course cygwin. Finally OS-X has the pax command and support in
>> the apple archive utility.
>>
>> So I think we're be fine to require it.
>>
>> While, we're changing this, I think we should probably take the opportunity
>> to also switch over to using 'xz' as our compression format, instead of
gz.
>> Consider the 1.3.5 release compressed with different formats:
>>
>> 35109092 libvirt-1.3.5.tar.gz
>> 25573966 libvirt-1.3.5.tar.bz2
>> 12112612 libvirt-1.3.5.tar.xz
>>
>> Those results seem pretty compelling to me :-)
>
> xz compression sure takes a lot of time!
Maybe it does, but it's done just once, while decompression is done
multiple times. So I think we can switch to xz. In fact, I'd be okay
with nothing but xz.
But will this solve the issue? I mean, the problem that Cole is seeing
(and I'm too) with too long path names. Isn't tar the origin of it?
Because if it is, I fear that changing compression algorithm won't help
much.
We already agreed that we should switch to the pax format, I'm just
saying, we should *also* switch to xz compression.
Regards,
Daniel
--
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org -o- http://virt-manager.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
2895
days inactive
3224
days old
13 comments
6 participants
participants (6)
-
Andrea Bolognani -
Cole Robinson -
Daniel P. Berrange -
Eric Blake -
Michal Privoznik -
Peter Krempa