[libvirt] [PATCH 00/11] scripts: convert many scripts from perl to python
Less of use re 'eval'; ''=~('(?{'.( ('`')| '%').('['^'-'). ('`'|'!'). ('`'|',').'"\\$~=' .('['^'+') .('`'| '/').('['^'+').'||'. "'"."'".';'.('`'|'/' ).('['^'+').('`'|'%'). ('`'|'.').('\\$%;').( '['^'"').(',!-~,#,,').( '['^'(').',(.).,\\' .'$+,'.('`'|"'").('['^'(') .',\\$~&&(\\$' .'_='.('['^')').('`'|('%')).( '['^'-').('`'| '%').('['^')').('['^'(').(('`')| '%').')'.("\`"| '&').('`'|'/').('['^"\)").'\\@~='.( '`'|"'").("\["^ ')').('`'|'%').('['^'+').('\\$|--,('). "'"."'".(')').( '['^'#').('^'^('`'|'/')).(':'&'=').',<'. ('^'^('`'|'.') ).'>;\\@;='.('`'|'-').('`'|'!').('['^'+') .'~~'.('['^')' ).('`'|'%').('['^'-').('`'|'%').('['^')'). ('['^'(').('`'|'%').','.('['^')').('`'|'%').('['^'-').('`' |'%').('['^')').('['^'(').('`'|'%').'\\@~;'.('`'|'-').('`'| '!').('['^'+').'\\{'.('['^'(').('['^'"').('['^'(').(('[')^ '/').('`'|'%').('`'|'-').'\\$^'.('`'^'/').'=~'.('{'^"\,").( '`'|')').('`'|'.').'?'.('`'^'#').('`'^',').('{'^'(').(':'). "'".('`'|'#').('`'|',').('`'|'%').('`'|'!').('['^')')."'". ';(\\$-=\\$_%'.('^'^('`'|'-')).')||(--\\$|,'.('`'|'-' ).( '`'|'!').('['^'+').'\\$_='.('['^')').('`'|'%').('[' ^(( '-'))).('`'|'%').('['^')').('['^'(').('`' |('%')). ',' .'\\@~,\\@;);'.('['^'+').('['^(')')).( '`'|')' ).( "\`"| '.').('['^'/').'\\$\\"'.("\["^ ('#')). '(' .'\\$=/'.('^'^('`'|'-')).'*'. (('`')| '!' ).("\`"| '"').('['^ "\("). '\\$|' .+ ('*').( '^'^('`' |',')) .'-\\' .+ '$-),'. '\\$_,'. '\\$' .'/'. ( ('`')| ('&')).( '`'| '/') .('['^ ')').'\\' .'$' .'-' .'&'. (('^')^( '`'| '/') ).'?' .'\\@;' .':' .''. '\\' .'@~;' .''. ('[' ^'(' ).( '`'| ',') .''. ((( '`' ))| '%' ).( '`' |(( '%' ))) .+( '[' ^(( '+' ))) .+ (( '!')). (( (( '\\') )) ). '$%\\}'. (( ((( '\\' ))))) .+ '$' .'%..' .''. ((( '^') )^("\`"| '/' )).( "\^"^( ('`')| ('/'))). '"})'); More ###### ## ######### ## ###### ##### #################### ############### ######## # ####### # ######## # ####### ## ######## ### ####### #### ####### ###### ##### ####### ############ ##### ######## ################## ###### ###################### ################# ################# ############# ########### ####### Daniel P. Berrangé (11): cfg.mk: fix comment detection for python semicolon check docs: rewrite hvsupport.html page generator in python docs: rewrite ACL permissions checker in Python docs: rewrite symfile sorting checker in Python docs: rewrite symfile library checker in Python docs: rewrite augest test generator in Python docs: rewrite po file minimizer in Python docs: rewrite duplicate header checker in Python docs: rewrite whitespace checker in Python docs: rewrite mock inline checker in Python docs: rewrite header ifdef checker in Python Makefile.am | 12 +- build-aux/augeas-gentest.pl | 60 ---- build-aux/augeas-gentest.py | 72 ++++ build-aux/check-spacing.pl | 198 ---------- build-aux/check-spacing.py | 204 +++++++++++ build-aux/header-ifdef.pl | 182 ---------- build-aux/header-ifdef.py | 206 +++++++++++ build-aux/minimize-po.pl | 37 -- build-aux/minimize-po.py | 60 ++++ build-aux/mock-noinline.pl | 75 ---- build-aux/mock-noinline.py | 88 +++++ build-aux/prohibit-duplicate-header.pl | 26 -- build-aux/prohibit-duplicate-header.py | 54 +++ cfg.mk | 10 +- docs/Makefile.am | 6 +- docs/hvsupport.pl | 458 ----------------------- docs/hvsupport.py | 479 +++++++++++++++++++++++++ po/Makefile.am | 2 +- src/Makefile.am | 16 +- src/check-aclperms.pl | 73 ---- src/check-aclperms.py | 77 ++++ src/check-symfile.pl | 70 ---- src/check-symfile.py | 80 +++++ src/check-symsorting.pl | 106 ------ src/check-symsorting.py | 112 ++++++ 25 files changed, 1455 insertions(+), 1308 deletions(-) delete mode 100755 build-aux/augeas-gentest.pl create mode 100755 build-aux/augeas-gentest.py delete mode 100755 build-aux/check-spacing.pl create mode 100755 build-aux/check-spacing.py delete mode 100644 build-aux/header-ifdef.pl create mode 100644 build-aux/header-ifdef.py delete mode 100755 build-aux/minimize-po.pl create mode 100755 build-aux/minimize-po.py delete mode 100644 build-aux/mock-noinline.pl create mode 100644 build-aux/mock-noinline.py delete mode 100644 build-aux/prohibit-duplicate-header.pl create mode 100644 build-aux/prohibit-duplicate-header.py delete mode 100755 docs/hvsupport.pl create mode 100755 docs/hvsupport.py delete mode 100755 src/check-aclperms.pl create mode 100755 src/check-aclperms.py delete mode 100755 src/check-symfile.pl create mode 100755 src/check-symfile.py delete mode 100755 src/check-symsorting.pl create mode 100755 src/check-symsorting.py -- 2.21.0
The pattern ^[^#].*\;$$ Was attempting to detect any trailing ';' in python code which was not in a comment. This does not allow for the comment '#' character to be indented with whitespace. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- cfg.mk | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cfg.mk b/cfg.mk index 1f29729949..d10dfa16a1 100644 --- a/cfg.mk +++ b/cfg.mk @@ -815,7 +815,7 @@ sc_require_enum_last_marker: # In Python files we don't want to end lines with a semicolon like in C sc_prohibit_semicolon_at_eol_in_python: - @prohibit='^[^#].*\;$$' \ + @prohibit='^[^#]*\;$$' \ in_vc_files='\.py$$' \ halt='python does not require to end lines with a semicolon' \ $(_sc_search_regexp) -- 2.21.0
On 9/5/19 5:21 AM, Daniel P. Berrangé wrote:
The pattern
^[^#].*\;$$
Was attempting to detect any trailing ';' in python code which was not in a comment. This does not allow for the comment '#' character to be indented with whitespace.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- cfg.mk | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/cfg.mk b/cfg.mk index 1f29729949..d10dfa16a1 100644 --- a/cfg.mk +++ b/cfg.mk @@ -815,7 +815,7 @@ sc_require_enum_last_marker:
# In Python files we don't want to end lines with a semicolon like in C sc_prohibit_semicolon_at_eol_in_python: - @prohibit='^[^#].*\;$$' \ + @prohibit='^[^#]*\;$$' \
But this new pattern does not prohibit: ch = '#'; I think you want: '^[ \t]*[^#].*\;$$' to flag all lines that have any amount of leading whitespace, where the first non-whitespace is not #, and which end in ;. -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
On Thu, Sep 05, 2019 at 08:57:43AM -0500, Eric Blake wrote:
On 9/5/19 5:21 AM, Daniel P. Berrangé wrote:
The pattern
^[^#].*\;$$
Was attempting to detect any trailing ';' in python code which was not in a comment. This does not allow for the comment '#' character to be indented with whitespace.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- cfg.mk | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/cfg.mk b/cfg.mk index 1f29729949..d10dfa16a1 100644 --- a/cfg.mk +++ b/cfg.mk @@ -815,7 +815,7 @@ sc_require_enum_last_marker:
# In Python files we don't want to end lines with a semicolon like in C sc_prohibit_semicolon_at_eol_in_python: - @prohibit='^[^#].*\;$$' \ + @prohibit='^[^#]*\;$$' \
But this new pattern does not prohibit:
ch = '#';
I think you want:
'^[ \t]*[^#].*\;$$'
to flag all lines that have any amount of leading whitespace, where the first non-whitespace is not #, and which end in ;.
Unfortunately this doesn't work when face with a line # foo(); The '[ \t]*' matches 0 times. '[^#]' matches on the first leading whitespace character. '.*' then matches ' # foo()'. So we get a bogus warning. This does make me realize how to fix it though - we need to force the '[ \t]*' to always be matched against leading space with: '^[ \t]*[^# \t].*\;$$' This will still give a bogus warning though if the comment is following after a valid statement eg some call # foo(); Luckily we don't have any examples of this pattern, so I could leave it to some other sucker to fix when they need that. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
As part of an goal to eliminate Perl from libvirt build tools, rewrite the hvsupport.pl tool in Python. This was a straight conversion, manually going line-by-line to change the syntax from Perl to Python. Thus the overall structure of the file and approach is the same. The new impl generates byte-for-byte identical output to the old impl. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- docs/Makefile.am | 6 +- docs/hvsupport.pl | 458 -------------------------------------------- docs/hvsupport.py | 479 ++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 482 insertions(+), 461 deletions(-) delete mode 100755 docs/hvsupport.pl create mode 100755 docs/hvsupport.py diff --git a/docs/Makefile.am b/docs/Makefile.am index 1cdb584b0b..f7aba5499d 100644 --- a/docs/Makefile.am +++ b/docs/Makefile.am @@ -194,7 +194,7 @@ EXTRA_DIST= \ $(internals_html_in) $(internals_html) $(fonts) \ $(kbase_html_in) $(kbase_html) \ aclperms.htmlinc \ - hvsupport.pl \ + hvsupport.py \ $(schema_DATA) acl_generated = aclperms.htmlinc @@ -230,11 +230,11 @@ web: $(dot_html) $(internals_html) $(kbase_html) \ hvsupport.html: $(srcdir)/hvsupport.html.in -$(srcdir)/hvsupport.html.in: $(srcdir)/hvsupport.pl $(api_DATA) \ +$(srcdir)/hvsupport.html.in: $(srcdir)/hvsupport.py $(api_DATA) \ $(top_srcdir)/src/libvirt_public.syms \ $(top_srcdir)/src/libvirt_qemu.syms $(top_srcdir)/src/libvirt_lxc.syms \ $(top_srcdir)/src/driver.h - $(AM_V_GEN)$(PERL) $(srcdir)/hvsupport.pl $(top_srcdir)/src > $@ \ + $(AM_V_GEN)$(PYTHON) $(srcdir)/hvsupport.py $(top_srcdir)/src > $@ \ || { rm $@ && exit 1; } news.html.in: \ diff --git a/docs/hvsupport.pl b/docs/hvsupport.pl deleted file mode 100755 index 494b8a27ec..0000000000 --- a/docs/hvsupport.pl +++ /dev/null @@ -1,458 +0,0 @@ -#!/usr/bin/env perl - -use strict; -use warnings; - -use File::Find; - -die "syntax: $0 SRCDIR\n" unless int(@ARGV) == 1; - -my $srcdir = shift @ARGV; - -my $symslibvirt = "$srcdir/libvirt_public.syms"; -my $symsqemu = "$srcdir/libvirt_qemu.syms"; -my $symslxc = "$srcdir/libvirt_lxc.syms"; -my @drivertable = ( - "$srcdir/driver-hypervisor.h", - "$srcdir/driver-interface.h", - "$srcdir/driver-network.h", - "$srcdir/driver-nodedev.h", - "$srcdir/driver-nwfilter.h", - "$srcdir/driver-secret.h", - "$srcdir/driver-state.h", - "$srcdir/driver-storage.h", - "$srcdir/driver-stream.h", - ); - -my %groupheaders = ( - "virHypervisorDriver" => "Hypervisor APIs", - "virNetworkDriver" => "Virtual Network APIs", - "virInterfaceDriver" => "Host Interface APIs", - "virNodeDeviceDriver" => "Host Device APIs", - "virStorageDriver" => "Storage Pool APIs", - "virSecretDriver" => "Secret APIs", - "virNWFilterDriver" => "Network Filter APIs", - ); - - -my @srcs; -find({ - wanted => sub { - if (m!$srcdir/.*/\w+_(driver|common|tmpl|monitor|hal|udev)\.c$!) { - push @srcs, $_ if $_ !~ /vbox_driver\.c/; - } - }, no_chdir => 1}, $srcdir); - -# Map API functions to the header and documentation files they're in -# so that we can generate proper hyperlinks to their documentation. -# -# The function names are grep'd from the XML output of apibuild.py. -sub getAPIFilenames { - my $filename = shift; - - my %files; - my $line; - - open FILE, "<", $filename or die "cannot read $filename: $!"; - - while (defined($line = <FILE>)) { - if ($line =~ /function name='([^']+)' file='([^']+)'/) { - $files{$1} = $2; - } - } - - close FILE; - - if (keys %files == 0) { - die "No functions found in $filename. Has the apibuild.py output changed?"; - } - return \%files; -} - -sub parseSymsFile { - my $apisref = shift; - my $prefix = shift; - my $filename = shift; - my $xmlfilename = shift; - - my $line; - my $vers; - my $prevvers; - - my $filenames = getAPIFilenames($xmlfilename); - - open FILE, "<$filename" - or die "cannot read $filename: $!"; - - while (defined($line = <FILE>)) { - chomp $line; - next if $line =~ /^\s*#/; - next if $line =~ /^\s*$/; - next if $line =~ /^\s*(global|local):/; - if ($line =~ /^\s*${prefix}_(\d+\.\d+\.\d+)\s*{\s*$/) { - if (defined $vers) { - die "malformed syms file"; - } - $vers = $1; - } elsif ($line =~ /\s*}\s*;\s*$/) { - if (defined $prevvers) { - die "malformed syms file"; - } - $prevvers = $vers; - $vers = undef; - } elsif ($line =~ /\s*}\s*${prefix}_(\d+\.\d+\.\d+)\s*;\s*$/) { - if ($1 ne $prevvers) { - die "malformed syms file $1 != $vers"; - } - $prevvers = $vers; - $vers = undef; - } elsif ($line =~ /\s*(\w+)\s*;\s*$/) { - $$apisref{$1} = {}; - $$apisref{$1}->{vers} = $vers; - $$apisref{$1}->{file} = $$filenames{$1}; - } else { - die "unexpected data $line\n"; - } - } - - close FILE; -} - -my %apis; -# Get the list of all public APIs and their corresponding version -parseSymsFile(\%apis, "LIBVIRT", $symslibvirt, "$srcdir/../docs/libvirt-api.xml"); - -# And the same for the QEMU specific APIs -parseSymsFile(\%apis, "LIBVIRT_QEMU", $symsqemu, "$srcdir/../docs/libvirt-qemu-api.xml"); - -# And the same for the LXC specific APIs -parseSymsFile(\%apis, "LIBVIRT_LXC", $symslxc, "$srcdir/../docs/libvirt-lxc-api.xml"); - - -# Some special things which aren't public APIs, -# but we want to report -$apis{virConnectSupportsFeature}->{vers} = "0.3.2"; -$apis{virDomainMigratePrepare}->{vers} = "0.3.2"; -$apis{virDomainMigratePerform}->{vers} = "0.3.2"; -$apis{virDomainMigrateFinish}->{vers} = "0.3.2"; -$apis{virDomainMigratePrepare2}->{vers} = "0.5.0"; -$apis{virDomainMigrateFinish2}->{vers} = "0.5.0"; -$apis{virDomainMigratePrepareTunnel}->{vers} = "0.7.2"; - -$apis{virDomainMigrateBegin3}->{vers} = "0.9.2"; -$apis{virDomainMigratePrepare3}->{vers} = "0.9.2"; -$apis{virDomainMigratePrepareTunnel3}->{vers} = "0.9.2"; -$apis{virDomainMigratePerform3}->{vers} = "0.9.2"; -$apis{virDomainMigrateFinish3}->{vers} = "0.9.2"; -$apis{virDomainMigrateConfirm3}->{vers} = "0.9.2"; - -$apis{virDomainMigrateBegin3Params}->{vers} = "1.1.0"; -$apis{virDomainMigratePrepare3Params}->{vers} = "1.1.0"; -$apis{virDomainMigratePrepareTunnel3Params}->{vers} = "1.1.0"; -$apis{virDomainMigratePerform3Params}->{vers} = "1.1.0"; -$apis{virDomainMigrateFinish3Params}->{vers} = "1.1.0"; -$apis{virDomainMigrateConfirm3Params}->{vers} = "1.1.0"; - - - -# Now we want to get the mapping between public APIs -# and driver struct fields. This lets us later match -# update the driver impls with the public APis. - -my $line; - -# Group name -> hash of APIs { fields -> api name } -my %groups; -my $ingrp; -foreach my $drivertable (@drivertable) { - open FILE, "<$drivertable" - or die "cannot read $drivertable: $!"; - - while (defined($line = <FILE>)) { - if ($line =~ /struct _(vir\w*Driver)/) { - my $grp = $1; - if ($grp ne "virStateDriver" && - $grp ne "virStreamDriver") { - $ingrp = $grp; - $groups{$ingrp} = { apis => {}, drivers => {} }; - } - } elsif ($ingrp) { - if ($line =~ /^\s*vir(?:Drv)(\w+)\s+(\w+);\s*$/) { - my $field = $2; - my $name = $1; - - my $api; - if (exists $apis{"vir$name"}) { - $api = "vir$name"; - } elsif ($name =~ /\w+(Open|Close|URIProbe)/) { - next; - } else { - die "driver $name does not have a public API"; - } - $groups{$ingrp}->{apis}->{$field} = $api; - } elsif ($line =~ /};/) { - $ingrp = undef; - } - } - } - - close FILE; -} - - -# Finally, we read all the primary driver files and extract -# the driver API tables from each one. - -foreach my $src (@srcs) { - open FILE, "<$src" or - die "cannot read $src: $!"; - - my $groups_regex = join("|", keys %groups); - $ingrp = undef; - my $impl; - while (defined($line = <FILE>)) { - if (!$ingrp) { - # skip non-matching lines early to save time - next if not $line =~ /$groups_regex/; - - if ($line =~ /^\s*(?:static\s+)?($groups_regex)\s+(\w+)\s*=\s*{/ || - $line =~ /^\s*(?:static\s+)?($groups_regex)\s+NAME\(\w+\)\s*=\s*{/) { - $ingrp = $1; - $impl = $src; - - if ($impl =~ m,.*/node_device_(\w+)\.c,) { - $impl = $1; - } else { - $impl =~ s,.*/(\w+?)_((\w+)_)?(\w+)\.c,$1,; - } - - if ($groups{$ingrp}->{drivers}->{$impl}) { - die "Group $ingrp already contains $impl"; - } - - $groups{$ingrp}->{drivers}->{$impl} = {}; - } - - } else { - if ($line =~ m!\s*\.(\w+)\s*=\s*(\w+)\s*,?\s*(?:/\*\s*(\d+\.\d+\.\d+)\s*(?:-\s*(\d+\.\d+\.\d+))?\s*\*/\s*)?$!) { - my $api = $1; - my $meth = $2; - my $vers = $3; - my $deleted = $4; - - next if $api eq "no" || $api eq "name"; - - if ($meth eq "NULL" && !defined $deleted) { - die "Method impl for $api is NULL, but no deleted version is provided"; - } - if ($meth ne "NULL" && defined $deleted) { - die "Method impl for $api is non-NULL, but deleted version is provided"; - } - - die "Method $meth in $src is missing version" unless defined $vers || $api eq "connectURIProbe"; - - if (!exists($groups{$ingrp}->{apis}->{$api})) { - next if $api =~ /\w(Open|Close|URIProbe)/; - - die "Found unexpected method $api in $ingrp\n"; - } - - $groups{$ingrp}->{drivers}->{$impl}->{$api} = {}; - $groups{$ingrp}->{drivers}->{$impl}->{$api}->{vers} = $vers; - $groups{$ingrp}->{drivers}->{$impl}->{$api}->{deleted} = $deleted; - if ($api eq "domainMigratePrepare" || - $api eq "domainMigratePrepare2" || - $api eq "domainMigratePrepare3") { - if (!$groups{$ingrp}->{drivers}->{$impl}->{"domainMigrate"}) { - $groups{$ingrp}->{drivers}->{$impl}->{"domainMigrate"} = {}; - $groups{$ingrp}->{drivers}->{$impl}->{"domainMigrate"}->{vers} = $vers; - } - } - - } elsif ($line =~ /}/) { - $ingrp = undef; - } - } - } - - close FILE; -} - - -# The '.open' driver method is used for 3 public APIs, so we -# have a bit of manual fixup todo with the per-driver versioning -# and support matrix - -$groups{virHypervisorDriver}->{apis}->{"openAuth"} = "virConnectOpenAuth"; -$groups{virHypervisorDriver}->{apis}->{"openReadOnly"} = "virConnectOpenReadOnly"; -$groups{virHypervisorDriver}->{apis}->{"domainMigrate"} = "virDomainMigrate"; - -my $openAuthVers = (0 * 1000 * 1000) + (4 * 1000) + 0; - -foreach my $drv (keys %{$groups{"virHypervisorDriver"}->{drivers}}) { - my $openVersStr = $groups{"virHypervisorDriver"}->{drivers}->{$drv}->{"connectOpen"}->{vers}; - my $openVers; - if ($openVersStr =~ /(\d+)\.(\d+)\.(\d+)/) { - $openVers = ($1 * 1000 * 1000) + ($2 * 1000) + $3; - } - - # virConnectOpenReadOnly always matches virConnectOpen version - $groups{"virHypervisorDriver"}->{drivers}->{$drv}->{"connectOpenReadOnly"} = - $groups{"virHypervisorDriver"}->{drivers}->{$drv}->{"connectOpen"}; - - $groups{"virHypervisorDriver"}->{drivers}->{$drv}->{"connectOpenAuth"} = {}; - - # virConnectOpenAuth is always 0.4.0 if the driver existed - # before this time, otherwise it matches the version of - # the driver's virConnectOpen entry - if ($openVersStr eq "Y" || - $openVers >= $openAuthVers) { - $groups{"virHypervisorDriver"}->{drivers}->{$drv}->{"connectOpenAuth"}->{vers} = $openVersStr; - } else { - $groups{"virHypervisorDriver"}->{drivers}->{$drv}->{"connectOpenAuth"}->{vers} = "0.4.0"; - } -} - - -# Another special case for the virDomainCreateLinux which was replaced -# with virDomainCreateXML -$groups{virHypervisorDriver}->{apis}->{"domainCreateLinux"} = "virDomainCreateLinux"; - -my $createAPIVers = (0 * 1000 * 1000) + (0 * 1000) + 3; - -foreach my $drv (keys %{$groups{"virHypervisorDriver"}->{drivers}}) { - my $createVersStr = $groups{"virHypervisorDriver"}->{drivers}->{$drv}->{"domainCreateXML"}->{vers}; - next unless defined $createVersStr; - my $createVers; - if ($createVersStr =~ /(\d+)\.(\d+)\.(\d+)/) { - $createVers = ($1 * 1000 * 1000) + ($2 * 1000) + $3; - } - - $groups{"virHypervisorDriver"}->{drivers}->{$drv}->{"domainCreateLinux"} = {}; - - # virCreateLinux is always 0.0.3 if the driver existed - # before this time, otherwise it matches the version of - # the driver's virCreateXML entry - if ($createVersStr eq "Y" || - $createVers >= $createAPIVers) { - $groups{"virHypervisorDriver"}->{drivers}->{$drv}->{"domainCreateLinux"}->{vers} = $createVersStr; - } else { - $groups{"virHypervisorDriver"}->{drivers}->{$drv}->{"domainCreateLinux"}->{vers} = "0.0.3"; - } -} - - -# Finally we generate the HTML file with the tables - -print <<EOF; -<?xml version="1.0" encoding="UTF-8"?> -<!DOCTYPE html> -<html xmlns="http://www.w3.org/1999/xhtml"> -<body class="hvsupport"> -<h1>libvirt API support matrix</h1> - -<ul id="toc"></ul> - -<p> -This page documents which <a href="html/">libvirt calls</a> work on -which libvirt drivers / hypervisors, and which version the API appeared -in. If a hypervisor driver later dropped support for the API, the version -when it was removed is also mentioned (highlighted in -<span class="removedhv">dark red</span>). -</p> - -EOF - - foreach my $grp (sort { $a cmp $b } keys %groups) { - print "<h2><a id=\"$grp\">", $groupheaders{$grp}, "</a></h2>\n"; - print <<EOF; -<table class="top_table"> -<thead> -<tr> -<th>API</th> -<th>Version</th> -EOF - - foreach my $drv (sort { $a cmp $b } keys %{$groups{$grp}->{drivers}}) { - print " <th>$drv</th>\n"; - } - - print <<EOF; -</tr> -</thead> -<tbody> -EOF - - my $row = 0; - foreach my $field (sort { - $groups{$grp}->{apis}->{$a} - cmp - $groups{$grp}->{apis}->{$b} - } keys %{$groups{$grp}->{apis}}) { - my $api = $groups{$grp}->{apis}->{$field}; - my $vers = $apis{$api}->{vers}; - my $htmlgrp = $apis{$api}->{file}; - print <<EOF; -<tr> -<td> -EOF - - if (defined $htmlgrp) { - print <<EOF; -<a href=\"html/libvirt-$htmlgrp.html#$api\">$api</a> -EOF - - } else { - print $api; - } - print <<EOF; -</td> -<td>$vers</td> -EOF - - foreach my $drv (sort {$a cmp $b } keys %{$groups{$grp}->{drivers}}) { - print "<td>"; - if (exists $groups{$grp}->{drivers}->{$drv}->{$field}) { - if ($groups{$grp}->{drivers}->{$drv}->{$field}->{vers}) { - print $groups{$grp}->{drivers}->{$drv}->{$field}->{vers}; - } - if ($groups{$grp}->{drivers}->{$drv}->{$field}->{deleted}) { - print " - <span class=\"removedhv\">", $groups{$grp}->{drivers}->{$drv}->{$field}->{deleted}, "</span>"; - } - } - print "</td>\n"; - } - - print <<EOF; -</tr> -EOF - - $row++; - if (($row % 15) == 0) { - print <<EOF; -<tr> -<th>API</th> -<th>Version</th> -EOF - - foreach my $drv (sort { $a cmp $b } keys %{$groups{$grp}->{drivers}}) { - print " <th>$drv</th>\n"; - } - - print <<EOF; -</tr> -EOF - } - - } - - print <<EOF; -</tbody> -</table> -EOF -} - -print <<EOF; -</body> -</html> -EOF diff --git a/docs/hvsupport.py b/docs/hvsupport.py new file mode 100755 index 0000000000..cfbecd82ae --- /dev/null +++ b/docs/hvsupport.py @@ -0,0 +1,479 @@ +#!/usr/bin/env python +# +# Copyright (C) 2011-2019 Red Hat, Inc. +# +# This library is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 2.1 of the License, or (at your option) any later version. +# +# This library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this library. If not, see +# <http://www.gnu.org/licenses/>. + +from __future__ import print_function + +import sys +import os.path +import re + +if len(sys.argv) != 2: + print("syntax: %s SRCDIR\n" % sys.argv[0], file=sys.stderr) + +srcdir = sys.argv[1] + +symslibvirt = os.path.join(srcdir, "libvirt_public.syms") +symsqemu = os.path.join(srcdir, "libvirt_qemu.syms") +symslxc = os.path.join(srcdir, "libvirt_lxc.syms") +drivertablefiles = [ + os.path.join(srcdir, "driver-hypervisor.h"), + os.path.join(srcdir, "driver-interface.h"), + os.path.join(srcdir, "driver-network.h"), + os.path.join(srcdir, "driver-nodedev.h"), + os.path.join(srcdir, "driver-nwfilter.h"), + os.path.join(srcdir, "driver-secret.h"), + os.path.join(srcdir, "driver-state.h"), + os.path.join(srcdir, "driver-storage.h"), + os.path.join(srcdir, "driver-stream.h"), +] + +groupheaders = { + "virHypervisorDriver": "Hypervisor APIs", + "virNetworkDriver": "Virtual Network APIs", + "virInterfaceDriver": "Host Interface APIs", + "virNodeDeviceDriver": "Host Device APIs", + "virStorageDriver": "Storage Pool APIs", + "virSecretDriver": "Secret APIs", + "virNWFilterDriver": "Network Filter APIs", +} + + +srcs = [] +for root, dirs, files in os.walk(srcdir): + for file in files: + if ((file.endswith("driver.c") and + not file.endswith("vbox_driver.c")) or + file.endswith("common.c") or + file.endswith("tmpl.c") or + file.endswith("monitor.c") or + file.endswith("hal.c") or + file.endswith("udev.c")): + srcs.append(os.path.join(root, file)) + + +# Map API functions to the header and documentation files they're in +# so that we can generate proper hyperlinks to their documentation. +# +# The function names are grep'd from the XML output of apibuild.py. +def getAPIFilenames(filename): + files = {} + + with open(filename) as fh: + prog = re.compile(r"\s*<function name='([^']+)' file='([^']+)'.*") + for line in fh: + res = prog.match(line) + if res is not None: + files[res.group(1)] = res.group(2) + + if len(files) == 0: + raise Exception("No functions found in %s. Has the apibuild.py output changed?" % filename) + + return files + + +def parseSymsFile(apisref, prefix, filename, xmlfilename): + vers = None + prevvers = None + + filenames = getAPIFilenames(xmlfilename) + + with open(filename) as fh: + groupstartprog = re.compile(r"^\s*%s_(\d+\.\d+\.\d+)\s*{\s*$" % prefix) + groupendprog1 = re.compile(r"^\s*}\s*;\s*$") + groupendprog2 = re.compile(r"^\s*}\s*%s_(\d+\.\d+\.\d+)\s*;\s*$" % prefix) + symbolprog = re.compile(r"^\s*(\w+)\s*;\s*$") + for line in fh: + line = line.strip() + + if line == "": + continue + if line[0] == '#': + continue + if line.startswith("global:"): + continue + if line.startswith("local:"): + continue + + groupstartmatch = groupstartprog.match(line) + groupendmatch1 = groupendprog1.match(line) + groupendmatch2 = groupendprog2.match(line) + symbolmatch = symbolprog.match(line) + if groupstartmatch is not None: + if vers is not None: + raise Exception("malformed syms file when starting group") + + vers = groupstartmatch.group(1) + elif groupendmatch1 is not None: + if prevvers is not None: + raise Exception("malformed syms file when ending group") + + prevvers = vers + vers = None + elif groupendmatch2 is not None: + if groupendmatch2.group(1) != prevvers: + raise Exception("malformed syms file %s != %s when ending group" % ( + groupendmatch2.group(1), prevvers)) + + prevvers = vers + vers = None + elif symbolmatch is not None: + name = symbolmatch.group(1) + apisref[name] = { + "vers": vers, + "file": filenames.get(name), + } + else: + raise Exception("unexpected data %s" % line) + + +apis = {} +# Get the list of all public APIs and their corresponding version +parseSymsFile(apis, "LIBVIRT", symslibvirt, os.path.join(srcdir, "../docs/libvirt-api.xml")) + +# And the same for the QEMU specific APIs +parseSymsFile(apis, "LIBVIRT_QEMU", symsqemu, os.path.join(srcdir, "../docs/libvirt-qemu-api.xml")) + +# And the same for the LXC specific APIs +parseSymsFile(apis, "LIBVIRT_LXC", symslxc, os.path.join(srcdir, "../docs/libvirt-lxc-api.xml")) + + +# Some special things which aren't public APIs, +# but we want to report +apis["virConnectSupportsFeature"] = { + "vers": "0.3.2" +} +apis["virDomainMigratePrepare"] = { + "vers": "0.3.2" +} +apis["virDomainMigratePerform"] = { + "vers": "0.3.2" +} +apis["virDomainMigrateFinish"] = { + "vers": "0.3.2" +} +apis["virDomainMigratePrepare2"] = { + "vers": "0.5.0" +} +apis["virDomainMigrateFinish2"] = { + "vers": "0.5.0" +} +apis["virDomainMigratePrepareTunnel"] = { + "vers": "0.7.2" +} + +apis["virDomainMigrateBegin3"] = { + "vers": "0.9.2" +} +apis["virDomainMigratePrepare3"] = { + "vers": "0.9.2" +} +apis["virDomainMigratePrepareTunnel3"] = { + "vers": "0.9.2" +} +apis["virDomainMigratePerform3"] = { + "vers": "0.9.2" +} +apis["virDomainMigrateFinish3"] = { + "vers": "0.9.2" +} +apis["virDomainMigrateConfirm3"] = { + "vers": "0.9.2" +} + +apis["virDomainMigrateBegin3Params"] = { + "vers": "1.1.0" +} +apis["virDomainMigratePrepare3Params"] = { + "vers": "1.1.0" +} +apis["virDomainMigratePrepareTunnel3Params"] = { + "vers": "1.1.0" +} +apis["virDomainMigratePerform3Params"] = { + "vers": "1.1.0" +} +apis["virDomainMigrateFinish3Params"] = { + "vers": "1.1.0" +} +apis["virDomainMigrateConfirm3Params"] = { + "vers": "1.1.0" +} + + +# Now we want to get the mapping between public APIs +# and driver struct fields. This lets us later match +# update the driver impls with the public APis. + +# Group name -> hash of APIs { fields -> api name } +groups = {} +ingrp = None +for drivertablefile in drivertablefiles: + with open(drivertablefile) as fh: + starttableprog = re.compile(r"struct _(vir\w*Driver)") + endtableprog = re.compile(r"};") + callbackprog = re.compile(r"^\s*vir(?:Drv)(\w+)\s+(\w+);\s*$") + ignoreapiprog = re.compile(r"\w+(Open|Close|URIProbe)") + for line in fh: + starttablematch = starttableprog.match(line) + if starttablematch is not None: + grp = starttablematch.group(1) + if grp != "virStateDriver" and grp != "virStreamDriver": + ingrp = grp + groups[ingrp] = { + "apis": {}, + "drivers": {} + } + elif ingrp != None: + callbackmatch = callbackprog.match(line) + if callbackmatch is not None: + name = callbackmatch.group(1) + field = callbackmatch.group(2) + + api = "vir" + name + if api in apis: + groups[ingrp]["apis"][field] = api + elif ignoreapiprog.match(api) != None: + continue + else: + raise Exception("driver %s does not have a public API" % name) + elif endtableprog.match(line): + ingrp = None + + +# Finally, we read all the primary driver files and extract +# the driver API tables from each one. + +for src in srcs: + with open(src) as fh: + groupsre = "|".join(groups.keys()) + groupsprog2 = re.compile(r"^\s*(static\s+)?(" + groupsre + r")\s+(\w+)\s*=\s*{") + groupsprog3 = re.compile(r"^\s*(static\s+)?(" + groupsre + r")\s+NAME\(\w+\)\s*=\s*{") + nodedevimplprog = re.compile(r".*/node_device_(\w+)\.c") + miscimplprog = re.compile(r".*/(\w+?)_((\w+)_)?(\w+)\.c") + callbackprog = re.compile(r"\s*\.(\w+)\s*=\s*(\w+)\s*,?\s*(?:/\*\s*(\d+\.\d+\.\d+)\s*(?:-\s*(\d+\.\d+\.\d+))?\s*\*/\s*)?$") + skipapiprog = re.compile(r"\w+(Open|Close|URIProbe)") + + ingrp = None + impl = None + for line in fh: + if ingrp is None: + m = groupsprog2.match(line) + if m is None: + m = groupsprog3.match(line) + if m is not None: + ingrp = m.group(2) + impl = src + + implmatch = nodedevimplprog.match(impl) + if implmatch is None: + implmatch = miscimplprog.match(impl) + if implmatch is None: + raise Exception("Unexpected impl format '%s'" % impl) + impl = implmatch.group(1) + + if impl in groups[ingrp]["drivers"]: + raise Exception("Group %s already contains %s" % (ingrp, impl)) + + groups[ingrp]["drivers"][impl] = {} + else: + callbackmatch = callbackprog.match(line) + if callbackmatch is not None: + api = callbackmatch.group(1) + meth = callbackmatch.group(2) + vers = callbackmatch.group(3) + deleted = callbackmatch.group(4) + + if api == "no" or api == "name": + continue + + if meth == "NULL" and deleted is None: + raise Exception("Method impl for %s is NULL, but no deleted version is provided" % api) + + if meth != "NULL" and deleted is not None: + raise Exception("Method impl for %s is non-NULL, but deleted version is provided" % api) + + if vers is None and api != "connectURIProbe": + raise Exception("Method %s in %s is missing version" % (meth, src)) + + if api not in groups[ingrp]["apis"]: + if skipapiprog.match(api): + continue + + raise Exception("Found unexpected method %s in %s" % (api, ingrp)) + + groups[ingrp]["drivers"][impl][api] = { + "vers": vers, + "deleted": deleted, + } + + if (api == "domainMigratePrepare" or + api == "domainMigratePrepare2" or + api == "domainMigratePrepare3"): + if "domainMigrate" not in groups[ingrp]["drivers"][impl]: + groups[ingrp]["drivers"][impl]["domainMigrate"] = { + "vers": vers, + } + elif line.find("}") != -1: + ingrp = None + + +# The '.open' driver method is used for 3 public APIs, so we +# have a bit of manual fixup todo with the per-driver versioning +# and support matrix + +groups["virHypervisorDriver"]["apis"]["openAuth"] = "virConnectOpenAuth" +groups["virHypervisorDriver"]["apis"]["openReadOnly"] = "virConnectOpenReadOnly" +groups["virHypervisorDriver"]["apis"]["domainMigrate"] = "virDomainMigrate" + +openAuthVers = (0 * 1000 * 1000) + (4 * 1000) + 0 + +for drv in groups["virHypervisorDriver"]["drivers"].keys(): + openVersStr = groups["virHypervisorDriver"]["drivers"][drv]["connectOpen"]["vers"] + openVers = 0 + if openVersStr != "Y": + openVersBits = openVersStr.split(".") + if len(openVersBits) != 3: + raise Exception("Expected 3 digit version for %s" % openVersStr) + openVers = (int(openVersBits[0]) * 1000 * 1000) + (int(openVersBits[1]) * 1000) + int(openVersBits[2]) + + # virConnectOpenReadOnly always matches virConnectOpen version + groups["virHypervisorDriver"]["drivers"][drv]["connectOpenReadOnly"] = \ + groups["virHypervisorDriver"]["drivers"][drv]["connectOpen"] + + # virConnectOpenAuth is always 0.4.0 if the driver existed + # before this time, otherwise it matches the version of + # the driver's virConnectOpen entry + if openVersStr == "Y" or openVers >= openAuthVers: + vers = openVersStr + else: + vers = "0.4.0" + groups["virHypervisorDriver"]["drivers"][drv]["connectOpenAuth"] = { + "vers": vers, + } + + +# Another special case for the virDomainCreateLinux which was replaced +# with virDomainCreateXML +groups["virHypervisorDriver"]["apis"]["domainCreateLinux"] = "virDomainCreateLinux" + +createAPIVers = (0 * 1000 * 1000) + (0 * 1000) + 3 + +for drv in groups["virHypervisorDriver"]["drivers"].keys(): + if "domainCreateXML" not in groups["virHypervisorDriver"]["drivers"][drv]: + continue + createVersStr = groups["virHypervisorDriver"]["drivers"][drv]["domainCreateXML"]["vers"] + createVers = 0 + if createVersStr != "Y": + createVersBits = createVersStr.split(".") + if len(createVersBits) != 3: + raise Exception("Expected 3 digit version for %s" % createVersStr) + createVers = (int(createVersBits[0]) * 1000 * 1000) + (int(createVersBits[1]) * 1000) + int(createVersBits[2]) + + # virCreateLinux is always 0.0.3 if the driver existed + # before this time, otherwise it matches the version of + # the driver's virCreateXML entry + if createVersStr == "Y" or createVers >= createAPIVers: + vers = createVersStr + else: + vers = "0.0.3" + + groups["virHypervisorDriver"]["drivers"][drv]["domainCreateLinux"] = { + "vers": vers, + } + + +# Finally we generate the HTML file with the tables + +print('''<?xml version="1.0" encoding="UTF-8"?> +<!DOCTYPE html> +<html xmlns="http://www.w3.org/1999/xhtml"> +<body class="hvsupport"> +<h1>libvirt API support matrix</h1> + +<ul id="toc"></ul> + +<p> +This page documents which <a href="html/">libvirt calls</a> work on +which libvirt drivers / hypervisors, and which version the API appeared +in. If a hypervisor driver later dropped support for the API, the version +when it was removed is also mentioned (highlighted in +<span class="removedhv">dark red</span>). +</p> +''') + +for grp in sorted(groups.keys()): + print("<h2><a id=\"%s\">%s</a></h2>" % (grp, groupheaders[grp])) + print('''<table class="top_table"> +<thead> +<tr> +<th>API</th> +<th>Version</th>''') + + for drv in sorted(groups[grp]["drivers"].keys()): + print(" <th>%s</th>" % drv) + + + print('''</tr> +</thead> +<tbody>''') + + row = 0 + def sortkey(field): + return groups[grp]["apis"][field] + for field in sorted(groups[grp]["apis"].keys(), key=sortkey): + api = groups[grp]["apis"][field] + vers = apis[api]["vers"] + htmlgrp = apis[api].get("file") + print("<tr>") + + if htmlgrp is not None: + print('''<td>\n<a href=\"html/libvirt-%s.html#%s\">%s</a>\n</td>''' % (htmlgrp, api, api)) + else: + print("<td>\n%s</td>" % api) + + print("<td>%s</td>" % vers) + + for drv in sorted(groups[grp]["drivers"].keys()): + info = "" + if field in groups[grp]["drivers"][drv]: + vers = groups[grp]["drivers"][drv][field]["vers"] + if vers is not None: + info = info + vers + + deleted = groups[grp]["drivers"][drv][field].get("deleted") + if deleted is not None: + info = info + (''' - <span class="removedhv">%s</span>''' % deleted) + + print("<td>%s</td>" % info) + + print("</tr>") + + row = row + 1 + if (row % 15) == 0: + print('''<tr> +<th>API</th> +<th>Version</th>''') + + for drv in sorted(groups[grp]["drivers"].keys()): + print(" <th>%s</th>" % drv) + + print("</tr>") + + print("</tbody>\n</table>") + +print("</body>\n</html>") -- 2.21.0
As part of an goal to eliminate Perl from libvirt build tools, rewrite the check-aclperms.pl tool in Python. This was a straight conversion, manually going line-by-line to change the syntax from Perl to Python. Thus the overall structure of the file and approach is the same. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- src/Makefile.am | 4 +-- src/check-aclperms.pl | 73 ---------------------------------------- src/check-aclperms.py | 77 +++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 79 insertions(+), 75 deletions(-) delete mode 100755 src/check-aclperms.pl create mode 100755 src/check-aclperms.py diff --git a/src/Makefile.am b/src/Makefile.am index f5093b9c90..2eb4387231 100644 --- a/src/Makefile.am +++ b/src/Makefile.am @@ -359,11 +359,11 @@ check-aclrules: $(addprefix $(srcdir)/,$(filter-out /%,$(STATEFUL_DRIVER_SOURCE_FILES))) check-aclperms: - $(AM_V_GEN)$(PERL) $(srcdir)/check-aclperms.pl \ + $(AM_V_GEN)$(PYTHON) $(srcdir)/check-aclperms.py \ $(srcdir)/access/viraccessperm.h \ $(srcdir)/access/viraccessperm.c -EXTRA_DIST += check-driverimpls.pl check-aclrules.pl check-aclperms.pl +EXTRA_DIST += check-driverimpls.pl check-aclrules.pl check-aclperms.py check-local: check-protocol check-symfile check-symsorting \ check-drivername check-driverimpls check-aclrules \ diff --git a/src/check-aclperms.pl b/src/check-aclperms.pl deleted file mode 100755 index 55b6598313..0000000000 --- a/src/check-aclperms.pl +++ /dev/null @@ -1,73 +0,0 @@ -#!/usr/bin/env perl -# -# Copyright (C) 2013 Red Hat, Inc. -# -# This library is free software; you can redistribute it and/or -# modify it under the terms of the GNU Lesser General Public -# License as published by the Free Software Foundation; either -# version 2.1 of the License, or (at your option) any later version. -# -# This library is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# Lesser General Public License for more details. -# -# You should have received a copy of the GNU Lesser General Public -# License along with this library. If not, see -# <http://www.gnu.org/licenses/>. -# -# This script just validates that the stringified version of -# a virAccessPerm enum matches the enum constant name. We do -# a lot of auto-generation of code, so when these don't match -# problems occur, preventing auth from succeeding at all. - -my $hdr = shift; -my $impl = shift; - -my %perms; - -my @perms; - -open HDR, $hdr or die "cannot read $hdr: $!"; - -while (<HDR>) { - if (/^\s+VIR_ACCESS_PERM_([_A-Z]+)(,?|\s|$)/) { - my $perm = $1; - - $perms{$perm} = 1 unless ($perm =~ /_LAST$/); - } -} - -close HDR; - - -open IMPL, $impl or die "cannot read $impl: $!"; - -my $group; -my $warned = 0; - -while (defined (my $line = <IMPL>)) { - if ($line =~ /VIR_ACCESS_PERM_([_A-Z]+)_LAST/) { - $group = $1; - } elsif ($line =~ /"[_a-z]+"/) { - my @bits = split /,/, $line; - foreach my $bit (@bits) { - if ($bit =~ /"([_a-z]+)"/) { - my $perm = uc($group . "_" . $1); - if (!exists $perms{$perm}) { - print STDERR "Unknown perm string $1 for group $group\n"; - $warned = 1; - } - delete $perms{$perm}; - } - } - } -} -close IMPL; - -foreach my $perm (keys %perms) { - print STDERR "Perm $perm had not string form\n"; - $warned = 1; -} - -exit $warned; diff --git a/src/check-aclperms.py b/src/check-aclperms.py new file mode 100755 index 0000000000..ace6ab6757 --- /dev/null +++ b/src/check-aclperms.py @@ -0,0 +1,77 @@ +#!/usr/bin/env python +# +# Copyright (C) 2013-2019 Red Hat, Inc. +# +# This library is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 2.1 of the License, or (at your option) any later version. +# +# This library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this library. If not, see +# <http://www.gnu.org/licenses/>. +# +# This script just validates that the stringified version of +# a virAccessPerm enum matches the enum constant name. We do +# a lot of auto-generation of code, so when these don't match +# problems occur, preventing auth from succeeding at all. + +from __future__ import print_function + +import re +import sys + +if len(sys.argv) != 3: + print("syntax: %s HEADER IMPL" % (sys.argv[0]), file=sys.stderr) + sys.exit(1) + +hdr = sys.argv[1] +impl = sys.argv[2] + +perms = {} + +with open(hdr) as fh: + symprog = re.compile(r"^\s+VIR_ACCESS_PERM_([_A-Z]+)(,?|\s|$).*") + for line in fh: + symmatch = symprog.match(line) + if symmatch is not None: + perm = symmatch.group(1) + + if not perm.endswith("_LAST"): + perms[perm] = 1 + +warned = False + +with open(impl) as fh: + group = None + symlastprog = re.compile(r".*VIR_ACCESS_PERM_([_A-Z]+)_LAST.*") + alnumprog = re.compile(r'''.*"([_a-z]+)".*''') + + for line in fh: + symlastmatch = symlastprog.match(line) + if symlastmatch is not None: + group = symlastmatch.group(1) + elif alnumprog.match(line) is not None: + bits = line.split(",") + for bit in bits: + m = alnumprog.match(bit) + if m is not None: + perm = (group + "_" + m.group(1)).upper() + if perm not in perms: + print("Unknown perm string %s for group %s" % (m.group(1), group), file=sys.stderr) + warned = True + + del perms[perm] + +for perm in perms.keys(): + print("Perm %s had not string form" % perm, file=sys.stderr) + warned = True + +if warned: + sys.exit(1) +sys.exit(0) -- 2.21.0
As part of an goal to eliminate Perl from libvirt build tools, rewrite the check-symsorting.pl tool in Python. This was a straight conversion, manually going line-by-line to change the syntax from Perl to Python. Thus the overall structure of the file and approach is the same. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- src/Makefile.am | 6 +-- src/check-symsorting.pl | 106 ------------------------------------- src/check-symsorting.py | 112 ++++++++++++++++++++++++++++++++++++++++ 3 files changed, 115 insertions(+), 109 deletions(-) delete mode 100755 src/check-symsorting.pl create mode 100755 src/check-symsorting.py diff --git a/src/Makefile.am b/src/Makefile.am index 2eb4387231..f30b085d07 100644 --- a/src/Makefile.am +++ b/src/Makefile.am @@ -285,12 +285,12 @@ check-symfile: check-admin-symfile: endif ! WITH_LINUX check-symsorting: - $(AM_V_GEN)$(PERL) $(srcdir)/check-symsorting.pl \ + $(AM_V_GEN)$(PYTHON) $(srcdir)/check-symsorting.py \ $(srcdir) $(SYM_FILES) check-admin-symsorting: - $(AM_V_GEN)$(PERL) $(srcdir)/check-symsorting.pl \ + $(AM_V_GEN)$(PYTHON) $(srcdir)/check-symsorting.py \ $(srcdir) $(ADMIN_SYM_FILES) -EXTRA_DIST += check-symfile.pl check-symsorting.pl +EXTRA_DIST += check-symfile.pl check-symsorting.py # Keep this list synced with RPC_PROBE_FILES PROTOCOL_STRUCTS = \ diff --git a/src/check-symsorting.pl b/src/check-symsorting.pl deleted file mode 100755 index 51e38bdedb..0000000000 --- a/src/check-symsorting.pl +++ /dev/null @@ -1,106 +0,0 @@ -#!/usr/bin/env perl - -# Copyright (C) 2012-2013 Red Hat, Inc. -# -# This library is free software; you can redistribute it and/or -# modify it under the terms of the GNU Lesser General Public -# License as published by the Free Software Foundation; either -# version 2.1 of the License, or (at your option) any later version. -# -# This library is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# Lesser General Public License for more details. -# -# You should have received a copy of the GNU Lesser General Public -# License along with this library. If not, see -# <http://www.gnu.org/licenses/>. - -use strict; -use warnings; - -die "syntax: $0 SRCDIR SYMFILE..." unless int(@ARGV) >= 2; - -my $ret = 0; -my $srcdir = shift; -my $lastgroup = undef; -foreach my $symfile (@ARGV) { - open SYMFILE, $symfile or die "cannot read $symfile: $!"; - - my $line = 0; - my $groupfile = ""; - my @group; - - while (<SYMFILE>) { - chomp; - - if (/^#\s*((\w+\/)*(\w+\.h))\s*$/) { - $groupfile = $1; - } elsif (/^#/) { - # Ignore comments - } elsif (/^\s*$/) { - if (@group) { - &check_sorting(\@group, $symfile, $line, $groupfile); - } - @group = (); - $line = $.; - } else { - $_ =~ s/;//; - push @group, $_; - } - } - - close SYMFILE; - if (@group) { - &check_sorting(\@group, $symfile, $line, $groupfile); - } - $lastgroup = undef; -} - -sub check_sorting { - my $group = shift; - my $symfile = shift; - my $line = shift; - my $groupfile = shift; - - my @group = @{$group}; - my @sorted = sort { lc $a cmp lc $b } @group; - my $sorted = 1; - my $first; - my $last; - - # Check that groups are in order and groupfile exists - if (defined $lastgroup && lc $lastgroup ge lc $groupfile) { - print "Symbol block at $symfile:$line: block not sorted\n"; - print "Move $groupfile block before $lastgroup block\n"; - print "\n"; - $ret = 1; - } - if (! -e "$srcdir/$groupfile") { - print "Symbol block at $symfile:$line: $groupfile not found\n"; - print "\n"; - $ret = 1; - } - $lastgroup = $groupfile; - - # Check that symbols within a group are in order - for (my $i = 0 ; $i <= $#sorted ; $i++) { - if ($sorted[$i] ne $group[$i]) { - $first = $i unless defined $first; - $last = $i; - $sorted = 0; - } - } - if (!$sorted) { - @group = splice @group, $first, ($last-$first+1); - @sorted = splice @sorted, $first, ($last-$first+1); - print "Symbol block at $symfile:$line: symbols not sorted\n"; - print map { " " . $_ . "\n" } @group; - print "Correct ordering\n"; - print map { " " . $_ . "\n" } @sorted; - print "\n"; - $ret = 1; - } -} - -exit $ret; diff --git a/src/check-symsorting.py b/src/check-symsorting.py new file mode 100755 index 0000000000..a1d16a4c43 --- /dev/null +++ b/src/check-symsorting.py @@ -0,0 +1,112 @@ +#!/usr/bin/env python + +# Copyright (C) 2012-2019 Red Hat, Inc. +# +# This library is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 2.1 of the License, or (at your option) any later version. +# +# This library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this library. If not, see +# <http://www.gnu.org/licenses/>. + +from __future__ import print_function + +import os.path +import re +import sys + +if len(sys.argv) < 3: + print("syntax: %s SRCDIR SYMFILE..." % sys.argv[0], file=sys.stderr) + sys.exit(1) + +def check_sorting(group, symfile, line, groupfile, lastgroup): + sortedgroup = sorted(group, key=str.lower) + issorted = True + first = None + last = None + + err = False + # Check that groups are in order and groupfile exists + if lastgroup is not None and lastgroup.lower() > groupfile.lower(): + print("Symbol block at %s:%s: block not sorted" % (symfile, line), file=sys.stderr) + print("Move %s block before %s block" % (groupfile, lastgroup), file=sys.stderr) + print("", file=sys.stderr) + err = True + + if not os.path.exists(os.path.join(srcdir, groupfile)): + print("Symbol block at %s:%s: %s not found" % (symfile, line, groupfile), file=sys.stderr) + print("", file=sys.stderr) + err = True + + # Check that symbols within a group are in order + for i in range(len(group)): + if sortedgroup[i] != group[i]: + if first is None: + first = i + + last = i + issorted = False + + if not issorted: + actual = group[first:(last-first+1)] + expect = sortedgroup[first:(last-first+1)] + print("Symbol block at %s:%s: symbols not sorted" % (symfile, line), file=sys.stderr) + for g in actual: + print(" %s" % g, file=sys.stderr) + print("Correct ordering", file=sys.stderr) + for g in expect: + print(" %s" % g, file=sys.stderr) + print("", file=sys.stderr) + err = True + + return err + + +ret = 0 +srcdir = sys.argv[1] +lastgroup = None +for symfile in sys.argv[2:]: + with open(symfile, "r") as fh: + lineno = 0 + groupfile = "" + group = [] + thisline = 0 + + filenameprog = re.compile(r'''^#\s*((\w+\/)*(\w+\.h))\s*$''') + + for line in fh: + thisline = thisline + 1 + line = line.strip() + + filenamematch = filenameprog.match(line) + if filenamematch is not None: + groupfile = filenamematch.group(1) + elif line == "": + if len(group) > 0: + if check_sorting(group, symfile, lineno, groupfile, lastgroup): + ret = 1 + + group = [] + lineno = thisline + lastgroup = groupfile + elif line[0] == '#': + # Ignore comments + pass + else: + line = line.strip(";") + group.append(line) + + if len(group) > 0: + if check_sorting(group, symfile, lineno, groupfile, lastgroup): + ret = 1 + + lastgroup = None + +sys.exit(ret) -- 2.21.0
As part of an goal to eliminate Perl from libvirt build tools, rewrite the check-symfile.pl tool in Python. This was a straight conversion, manually going line-by-line to change the syntax from Perl to Python. Thus the overall structure of the file and approach is the same. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- src/Makefile.am | 6 ++-- src/check-symfile.pl | 70 -------------------------------------- src/check-symfile.py | 80 ++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 83 insertions(+), 73 deletions(-) delete mode 100755 src/check-symfile.pl create mode 100755 src/check-symfile.py diff --git a/src/Makefile.am b/src/Makefile.am index f30b085d07..1724b3b809 100644 --- a/src/Makefile.am +++ b/src/Makefile.am @@ -275,10 +275,10 @@ PDWTAGS = \ # rule for libvirt.la. However, checking symbols relies on Linux ELF layout if WITH_LINUX check-symfile: libvirt.syms libvirt.la - $(AM_V_GEN)$(PERL) $(srcdir)/check-symfile.pl libvirt.syms \ + $(AM_V_GEN)$(PYTHON) $(srcdir)/check-symfile.py libvirt.syms \ .libs/libvirt.so check-admin-symfile: libvirt_admin.syms libvirt-admin.la - $(AM_V_GEN)$(PERL) $(srcdir)/check-symfile.pl libvirt_admin.syms \ + $(AM_V_GEN)$(PYTHON) $(srcdir)/check-symfile.py libvirt_admin.syms \ .libs/libvirt-admin.so else ! WITH_LINUX check-symfile: @@ -290,7 +290,7 @@ check-symsorting: check-admin-symsorting: $(AM_V_GEN)$(PYTHON) $(srcdir)/check-symsorting.py \ $(srcdir) $(ADMIN_SYM_FILES) -EXTRA_DIST += check-symfile.pl check-symsorting.py +EXTRA_DIST += check-symfile.py check-symsorting.py # Keep this list synced with RPC_PROBE_FILES PROTOCOL_STRUCTS = \ diff --git a/src/check-symfile.pl b/src/check-symfile.pl deleted file mode 100755 index 4f88300864..0000000000 --- a/src/check-symfile.pl +++ /dev/null @@ -1,70 +0,0 @@ -#!/usr/bin/env perl - -# Copyright (C) 2012-2013 Red Hat, Inc. -# -# This library is free software; you can redistribute it and/or -# modify it under the terms of the GNU Lesser General Public -# License as published by the Free Software Foundation; either -# version 2.1 of the License, or (at your option) any later version. -# -# This library is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# Lesser General Public License for more details. -# -# You should have received a copy of the GNU Lesser General Public -# License along with this library. If not, see -# <http://www.gnu.org/licenses/>. - -die "syntax: $0 SYMFILE ELFLIB(S)" unless int(@ARGV) >= 2; - -my $symfile = shift @ARGV; -my @elflibs = @ARGV; - -my %wantsyms; -my %gotsyms; - -my $ret = 0; - -open SYMFILE, $symfile or die "cannot read $symfile: $!"; - -while (<SYMFILE>) { - next if /{/; - next if /}/; - next if /global:/; - next if /local:/; - next if /^\s*$/; - next if /^\s*#/; - next if /\*/; - - die "malformed line $_" unless /^\s*(\S+);$/; - - if (exists $wantsyms{$1}) { - print STDERR "Symbol $1 is listed twice\n"; - $ret = 1; - } else { - $wantsyms{$1} = 1; - } -} -close SYMFILE; - -foreach my $elflib (@elflibs) { - open NM, "-|", "nm", $elflib or die "cannot run 'nm $elflib': $!"; - - while (<NM>) { - next unless /^\S+\s(?:[TBD])\s(\S+)\s*$/; - - $gotsyms{$1} = 1; - } - - close NM; -} - -foreach my $sym (keys(%wantsyms)) { - next if exists $gotsyms{$sym}; - - print STDERR "Expected symbol $sym is not in ELF library\n"; - $ret = 1; -} - -exit($ret); diff --git a/src/check-symfile.py b/src/check-symfile.py new file mode 100755 index 0000000000..3154619bfe --- /dev/null +++ b/src/check-symfile.py @@ -0,0 +1,80 @@ +#!/usr/bin/env python +# +# Copyright (C) 2012-2019 Red Hat, Inc. +# +# This library is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 2.1 of the License, or (at your option) any later version. +# +# This library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this library. If not, see +# <http://www.gnu.org/licenses/>. + +from __future__ import print_function + +import re +import subprocess +import sys + +if len(sys.argv) < 3: + print("syntax: %s SYMFILE ELFLIB(S)" % sys.argv[0], file=sys.stderr) + +symfile = sys.argv[1] +elflibs = sys.argv[2:] + +wantsyms = {} +gotsyms = {} + +ret = 0 + +with open(symfile, "r") as fh: + for line in fh: + line = line.strip() + if line.find("{") != -1: + continue + if line.find("}") != -1: + continue + if line in ["global:", "local:"]: + continue + if line == "": + continue + if line[0] == '#': + continue + if line.find("*") != -1: + continue + + line = line.strip(";") + + if line in wantsyms: + print("Symbol $1 is listed twice", file=sys.stderr) + ret = 1 + else: + wantsyms[line] = True + +for elflib in elflibs: + nm = subprocess.Popen(["nm", elflib], shell=False, stdout=subprocess.PIPE).stdout + + symprog = re.compile(r'''^\S+\s(?:[TBD])\s(\S+)\s*$''') + for line in nm: + line = line.decode("utf-8") + symmatch = symprog.match(line) + if symmatch is None: + continue + + gotsyms[symmatch.group(1)] = True + + +for sym in wantsyms.keys(): + if sym in gotsyms: + continue + + print("Expected symbol '%s' is not in ELF library" % sym, file=sys.stderr) + ret = 1 + +sys.exit(ret) -- 2.21.0
As part of an goal to eliminate Perl from libvirt build tools, rewrite the augeas-gentest.pl tool in Python. This was a straight conversion, manually going line-by-line to change the syntax from Perl to Python. Thus the overall structure of the file and approach is the same. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- Makefile.am | 2 +- build-aux/augeas-gentest.pl | 60 ------------------------------- build-aux/augeas-gentest.py | 72 +++++++++++++++++++++++++++++++++++++ src/Makefile.am | 2 +- 4 files changed, 74 insertions(+), 62 deletions(-) delete mode 100755 build-aux/augeas-gentest.pl create mode 100755 build-aux/augeas-gentest.py diff --git a/Makefile.am b/Makefile.am index cf9ff94f4f..e909d29d10 100644 --- a/Makefile.am +++ b/Makefile.am @@ -41,7 +41,7 @@ EXTRA_DIST = \ run.in \ README.md \ AUTHORS.in \ - build-aux/augeas-gentest.pl \ + build-aux/augeas-gentest.py \ build-aux/check-spacing.pl \ build-aux/gitlog-to-changelog \ build-aux/header-ifdef.pl \ diff --git a/build-aux/augeas-gentest.pl b/build-aux/augeas-gentest.pl deleted file mode 100755 index 65834b533b..0000000000 --- a/build-aux/augeas-gentest.pl +++ /dev/null @@ -1,60 +0,0 @@ -#!/usr/bin/env perl -# -# augeas-gentest.pl: Generate an augeas test file, from an -# example config file + test file template -# -# This library is free software; you can redistribute it and/or -# modify it under the terms of the GNU Lesser General Public -# License as published by the Free Software Foundation; either -# version 2.1 of the License, or (at your option) any later version. -# -# This library is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# Lesser General Public License for more details. -# -# You should have received a copy of the GNU Lesser General Public -# License along with this library. If not, see -# <http://www.gnu.org/licenses/>. - -use strict; -use warnings; - -die "syntax: $0 CONFIG TEMPLATE\n" unless @ARGV == 2; - -my $config = shift @ARGV; -my $template = shift @ARGV; - -open CONFIG, "<", $config or die "cannot read $config: $!"; -open TEMPLATE, "<", $template or die "cannot read $template: $!"; - -my $group = 0; -while (<TEMPLATE>) { - if (/\@CONFIG\@/) { - my $group = 0; - print " let conf = \""; - while (<CONFIG>) { - if (/^#\w/) { - s/^#//; - s/\"/\\\"/g; - print $_; - $group = /\[\s$/; - } elsif ($group) { - s/\"/\\\"/g; - if (/#\s*\]/) { - $group = 0; - } - if (/^#/) { - s/^#//; - print $_; - } - } - } - print "\"\n"; - } else { - print $_; - } -} - -close TEMPLATE; -close CONFIG; diff --git a/build-aux/augeas-gentest.py b/build-aux/augeas-gentest.py new file mode 100755 index 0000000000..b7f343ef42 --- /dev/null +++ b/build-aux/augeas-gentest.py @@ -0,0 +1,72 @@ +#!/usr/bin/env python +# +# Copyright (C) 2012-2019 Red Hat, Inc. +# +# augeas-gentest.py: Generate an augeas test file, from an +# example config file + test file template +# +# This library is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 2.1 of the License, or (at your option) any later version. +# +# This library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this library. If not, see +# <http://www.gnu.org/licenses/>. + +from __future__ import print_function + +import re +import sys + +if len(sys.argv) != 3: + print("syntax: %s CONFIG TEMPLATE" % sys.argv[0], file=sys.stderr) + sys.exit(1) + +config = sys.argv[1] +template = sys.argv[2] + +def expand_config(config): + with open(config) as fh: + optprog = re.compile(r'''^#\w.*''') + groupstartprog = re.compile(r'''.*\[\s$''') + groupendprog = re.compile(r'''#\s*\].*$''') + + group = False + for line in fh: + if optprog.match(line) is not None: + line = line[1:] + line = line.replace('"', '\\"') + print(line, end='') + if groupstartprog.match(line): + group = True + elif group: + line = line.replace('"', '\\"') + + if groupendprog.match(line): + group = False + + if line[0] == '#': + line = line[1:] + print(line, end='') + + +def expand_template(template, config): + with open(template) as fh: + group = 0 + + markerprog = re.compile(r'''\s*@CONFIG@\s*''') + for line in fh: + if markerprog.match(line) is not None: + print(' let conf = "', end='') + expand_config(config) + print('"') + else: + print(line, end='') + +expand_template(template, config) diff --git a/src/Makefile.am b/src/Makefile.am index 1724b3b809..716c2000e2 100644 --- a/src/Makefile.am +++ b/src/Makefile.am @@ -422,7 +422,7 @@ check-augeas: $(augeas_DATA) $(augeastest_DATA) fi .PHONY: check-augeas -AUG_GENTEST = $(PERL) $(top_srcdir)/build-aux/augeas-gentest.pl +AUG_GENTEST = $(PYTHON) $(top_srcdir)/build-aux/augeas-gentest.py # -- 2.21.0
As part of an goal to eliminate Perl from libvirt build tools, rewrite the minimize-po.pl tool in Python. This was a straight conversion, manually going line-by-line to change the syntax from Perl to Python. Thus the overall structure of the file and approach is the same. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- Makefile.am | 2 +- build-aux/minimize-po.pl | 37 ------------------------- build-aux/minimize-po.py | 60 ++++++++++++++++++++++++++++++++++++++++ po/Makefile.am | 2 +- 4 files changed, 62 insertions(+), 39 deletions(-) delete mode 100755 build-aux/minimize-po.pl create mode 100755 build-aux/minimize-po.py diff --git a/Makefile.am b/Makefile.am index e909d29d10..0a9d867610 100644 --- a/Makefile.am +++ b/Makefile.am @@ -45,7 +45,7 @@ EXTRA_DIST = \ build-aux/check-spacing.pl \ build-aux/gitlog-to-changelog \ build-aux/header-ifdef.pl \ - build-aux/minimize-po.pl \ + build-aux/minimize-po.py \ build-aux/mock-noinline.pl \ build-aux/prohibit-duplicate-header.pl \ build-aux/useless-if-before-free \ diff --git a/build-aux/minimize-po.pl b/build-aux/minimize-po.pl deleted file mode 100755 index 497533a836..0000000000 --- a/build-aux/minimize-po.pl +++ /dev/null @@ -1,37 +0,0 @@ -#!/usr/bin/perl - -my @block; -my $msgstr = 0; -my $empty = 0; -my $unused = 0; -my $fuzzy = 0; -while (<>) { - if (/^$/) { - if (!$empty && !$unused && !$fuzzy) { - print @block; - } - @block = (); - $msgstr = 0; - $fuzzy = 0; - push @block, $_; - } else { - if (/^msgstr/) { - $msgstr = 1; - $empty = 1; - } - if (/^#.*fuzzy/) { - $fuzzy = 1; - } - if (/^#~ msgstr/) { - $unused = 1; - } - if ($msgstr && /".+"/) { - $empty = 0; - } - push @block, $_; - } -} - -if (@block && !$empty && !$unused) { - print @block; -} diff --git a/build-aux/minimize-po.py b/build-aux/minimize-po.py new file mode 100755 index 0000000000..5046bacede --- /dev/null +++ b/build-aux/minimize-po.py @@ -0,0 +1,60 @@ +#!/usr/bin/env python +# +# Copyright (C) 2018-2019 Red Hat, Inc. +# +# This library is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 2.1 of the License, or (at your option) any later version. +# +# This library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this library. If not, see +# <http://www.gnu.org/licenses/>. + +from __future__ import print_function + +import re +import sys + +block = [] +msgstr = False +empty = False +unused = False +fuzzy = False + +strprog = re.compile(r'''.*".+".*''') + +for line in sys.stdin: + line = line.rstrip("\n") + + if line == "": + if not empty and not unused and not fuzzy: + for b in block: + print(b) + + block = [] + msgstr = False + fuzzy = False + block.append(line) + else: + if line.startswith("msgstr"): + msgstr = True + empty = True + + if line[0] == '#' and line.find("fuzzy") != -1: + fuzzy = True + if line.startswith("#~ msgstr"): + unused = True + if msgstr and strprog.match(line): + empty = False + + block.append(line) + +if not empty and not unused and not fuzzy: + for b in block: + print(b) diff --git a/po/Makefile.am b/po/Makefile.am index da117edbd5..b4d2f956ab 100644 --- a/po/Makefile.am +++ b/po/Makefile.am @@ -58,7 +58,7 @@ update-mini-po: $(POTFILE) $(MSGMERGE) --no-location --no-fuzzy-matching --sort-output \ $$lang.po $(POTFILE) | \ $(SED) $(SED_PO_FIXUP_ARGS) | \ - $(PERL) $(top_srcdir)/build-aux/minimize-po.pl > \ + $(PYTHON) $(top_srcdir)/build-aux/minimize-po.py > \ $(srcdir)/$$lang.mini.po ; \ done -- 2.21.0
As part of an goal to eliminate Perl from libvirt build tools, rewrite the prohibit-duplicate-header.pl tool in Python. This was a straight conversion, manually going line-by-line to change the syntax from Perl to Python. Thus the overall structure of the file and approach is the same. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- Makefile.am | 2 +- build-aux/prohibit-duplicate-header.pl | 26 ------------- build-aux/prohibit-duplicate-header.py | 54 ++++++++++++++++++++++++++ cfg.mk | 2 +- 4 files changed, 56 insertions(+), 28 deletions(-) delete mode 100644 build-aux/prohibit-duplicate-header.pl create mode 100644 build-aux/prohibit-duplicate-header.py diff --git a/Makefile.am b/Makefile.am index 0a9d867610..d0077a4f2a 100644 --- a/Makefile.am +++ b/Makefile.am @@ -47,7 +47,7 @@ EXTRA_DIST = \ build-aux/header-ifdef.pl \ build-aux/minimize-po.py \ build-aux/mock-noinline.pl \ - build-aux/prohibit-duplicate-header.pl \ + build-aux/prohibit-duplicate-header.py \ build-aux/useless-if-before-free \ build-aux/vc-list-files \ ci/Makefile \ diff --git a/build-aux/prohibit-duplicate-header.pl b/build-aux/prohibit-duplicate-header.pl deleted file mode 100644 index 4a2ea65665..0000000000 --- a/build-aux/prohibit-duplicate-header.pl +++ /dev/null @@ -1,26 +0,0 @@ -#!/usr/bin/env perl - -use strict; - -my $file = " "; -my $ret = 0; -my %includes = ( ); -my $lineno = 0; - -while (<>) { - if (not $file eq $ARGV) { - %includes = ( ); - $file = $ARGV; - $lineno = 0; - } - $lineno++; - if (/^# *include *[<"]([^>"]*\.h)[">]/) { - $includes{$1}++; - if ($includes{$1} == 2) { - $ret = 1; - print STDERR "$ARGV:$lineno: $_"; - print STDERR "Do not include a header more than once per file\n"; - } - } -} -exit $ret; diff --git a/build-aux/prohibit-duplicate-header.py b/build-aux/prohibit-duplicate-header.py new file mode 100644 index 0000000000..0ef86e8453 --- /dev/null +++ b/build-aux/prohibit-duplicate-header.py @@ -0,0 +1,54 @@ +#!/usr/bin/env python +# +# Copyright (C) 2016-2019 Red Hat, Inc. +# +# This library is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 2.1 of the License, or (at your option) any later version. +# +# This library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this library. If not, see +# <http://www.gnu.org/licenses/>. + +from __future__ import print_function + +import re +import sys + +def check_file(filename): + includes = {} + lineno = 0 + errs = False + with open(filename, "r") as fh: + headerprog = re.compile(r'''^# *include *[<"]([^>"]*\.h)[">]\s*$''') + for line in fh: + lineno = lineno + 1 + + headermatch = headerprog.match(line) + if headermatch is not None: + inc = headermatch.group(1) + + if inc in includes: + print("%s:%d: %s" % (filename, lineno, inc), file=sys.stderr) + errs = True + else: + includes[inc] = True + + return errs + +ret = 0 + +for filename in sys.argv[1:]: + if check_file(filename): + ret = 1 + +if ret == 1: + print("Do not include a header more than once per file", file=sys.stderr) + +sys.exit(ret) diff --git a/cfg.mk b/cfg.mk index d10dfa16a1..da1a1cb3cf 100644 --- a/cfg.mk +++ b/cfg.mk @@ -1148,7 +1148,7 @@ endif # Don't include duplicate header in the source (either *.c or *.h) prohibit-duplicate-header: $(AM_V_GEN)$(VC_LIST_EXCEPT) | $(GREP) '\.[chx]$$' | xargs \ - $(PERL) -W $(top_srcdir)/build-aux/prohibit-duplicate-header.pl + $(PYTHON) $(top_srcdir)/build-aux/prohibit-duplicate-header.py spacing-check: $(AM_V_GEN)$(VC_LIST) | $(GREP) '\.c$$' | xargs \ -- 2.21.0
As part of an goal to eliminate Perl from libvirt build tools, rewrite the check-spacing.pl tool in Python. This was a straight conversion, manually going line-by-line to change the syntax from Perl to Python. Thus the overall structure of the file and approach is the same. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- Makefile.am | 2 +- build-aux/check-spacing.pl | 198 ----------------------------------- build-aux/check-spacing.py | 204 +++++++++++++++++++++++++++++++++++++ cfg.mk | 2 +- 4 files changed, 206 insertions(+), 200 deletions(-) delete mode 100755 build-aux/check-spacing.pl create mode 100755 build-aux/check-spacing.py diff --git a/Makefile.am b/Makefile.am index d0077a4f2a..d28c3ebcea 100644 --- a/Makefile.am +++ b/Makefile.am @@ -42,7 +42,7 @@ EXTRA_DIST = \ README.md \ AUTHORS.in \ build-aux/augeas-gentest.py \ - build-aux/check-spacing.pl \ + build-aux/check-spacing.py \ build-aux/gitlog-to-changelog \ build-aux/header-ifdef.pl \ build-aux/minimize-po.py \ diff --git a/build-aux/check-spacing.pl b/build-aux/check-spacing.pl deleted file mode 100755 index 33377f3dd3..0000000000 --- a/build-aux/check-spacing.pl +++ /dev/null @@ -1,198 +0,0 @@ -#!/usr/bin/env perl -# -# check-spacing.pl: Report any usage of 'function (..args..)' -# Also check for other syntax issues, such as correct use of ';' -# -# This library is free software; you can redistribute it and/or -# modify it under the terms of the GNU Lesser General Public -# License as published by the Free Software Foundation; either -# version 2.1 of the License, or (at your option) any later version. -# -# This library is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# Lesser General Public License for more details. -# -# You should have received a copy of the GNU Lesser General Public -# License along with this library. If not, see -# <http://www.gnu.org/licenses/>. - -use strict; -use warnings; - -my $ret = 0; -my $incomment = 0; - -foreach my $file (@ARGV) { - # Per-file variables for multiline Curly Bracket (cb_) check - my $cb_linenum = 0; - my $cb_code = ""; - my $cb_scolon = 0; - - open FILE, $file; - - while (defined (my $line = <FILE>)) { - my $data = $line; - # For temporary modifications - my $tmpdata; - - # Kill any quoted , ; = or " - $data =~ s/'[";,=]'/'X'/g; - - # Kill any quoted strings - $data =~ s,"(?:[^\\\"]|\\.)*","XXX",g; - - next if $data =~ /^#/; - - # Kill contents of multi-line comments - # and detect end of multi-line comments - if ($incomment) { - if ($data =~ m,\*/,) { - $incomment = 0; - $data =~ s,^.*\*/,*/,; - } else { - $data = ""; - } - } - - # Kill single line comments, and detect - # start of multi-line comments - if ($data =~ m,/\*.*\*/,) { - $data =~ s,/\*.*\*/,/* */,; - } elsif ($data =~ m,/\*,) { - $incomment = 1; - $data =~ s,/\*.*,/*,; - } - - # We need to match things like - # - # int foo (int bar, bool wizz); - # foo (bar, wizz); - # - # but not match things like: - # - # typedef int (*foo)(bar wizz) - # - # we can't do this (efficiently) without - # missing things like - # - # foo (*bar, wizz); - # - # We also don't want to spoil the $data so it can be used - # later on. - $tmpdata = $data; - while ($tmpdata =~ /(\w+)\s\((?!\*)/) { - my $kw = $1; - - # Allow space after keywords only - if ($kw =~ /^(?:if|for|while|switch|return)$/) { - $tmpdata =~ s/(?:$kw\s\()/XXX(/; - } else { - print "Whitespace after non-keyword:\n"; - print "$file:$.: $line"; - $ret = 1; - last; - } - } - - # Require whitespace immediately after keywords - if ($data =~ /\b(?:if|for|while|switch|return)\(/) { - print "No whitespace after keyword:\n"; - print "$file:$.: $line"; - $ret = 1; - } - - # Forbid whitespace between )( of a function typedef - if ($data =~ /\(\*\w+\)\s+\(/) { - print "Whitespace between ')' and '(':\n"; - print "$file:$.: $line"; - $ret = 1; - } - - # Forbid whitespace following ( or prior to ) - # but allow whitespace before ) on a single line - # (optionally followed by a semicolon) - if (($data =~ /\s\)/ && not $data =~ /^\s+\);?$/) || - $data =~ /\((?!$)\s/) { - print "Whitespace after '(' or before ')':\n"; - print "$file:$.: $line"; - $ret = 1; - } - - # Forbid whitespace before ";" or ",". Things like below are allowed: - # - # 1) The expression is empty for "for" loop. E.g. - # for (i = 0; ; i++) - # - # 2) An empty statement. E.g. - # while (write(statuswrite, &status, 1) == -1 && - # errno == EINTR) - # ; - # - if ($data =~ /\s[;,]/) { - unless ($data =~ /\S; ; / || - $data =~ /^\s+;/) { - print "Whitespace before semicolon or comma:\n"; - print "$file:$.: $line"; - $ret = 1; - } - } - - # Require EOL, macro line continuation, or whitespace after ";". - # Allow "for (;;)" as an exception. - if ($data =~ /;[^ \\\n;)]/) { - print "Invalid character after semicolon:\n"; - print "$file:$.: $line"; - $ret = 1; - } - - # Require EOL, space, or enum/struct end after comma. - if ($data =~ /,[^ \\\n)}]/) { - print "Invalid character after comma:\n"; - print "$file:$.: $line"; - $ret = 1; - } - - # Require spaces around assignment '=', compounds and '==' - if ($data =~ /[^ ]\b[!<>&|\-+*\/%\^=]?=/ || - $data =~ /=[^= \\\n]/) { - print "Spacing around '=' or '==':\n"; - print "$file:$.: $line"; - $ret = 1; - } - - # One line conditional statements with one line bodies should - # not use curly brackets. - if ($data =~ /^\s*(if|while|for)\b.*\{$/) { - $cb_linenum = $.; - $cb_code = $line; - $cb_scolon = 0; - } - - # We need to check for exactly one semicolon inside the body, - # because empty statements (e.g. with comment only) are - # allowed - if ($cb_linenum == $. - 1 && $data =~ /^[^;]*;[^;]*$/) { - $cb_code .= $line; - $cb_scolon = 1; - } - - if ($data =~ /^\s*}\s*$/ && - $cb_linenum == $. - 2 && - $cb_scolon) { - - print "Curly brackets around single-line body:\n"; - print "$file:$cb_linenum-$.:\n$cb_code$line"; - $ret = 1; - - # There _should_ be no need to reset the values; but to - # keep my inner peace... - $cb_linenum = 0; - $cb_scolon = 0; - $cb_code = ""; - } - } - close FILE; -} - -exit $ret; diff --git a/build-aux/check-spacing.py b/build-aux/check-spacing.py new file mode 100755 index 0000000000..ec565585ad --- /dev/null +++ b/build-aux/check-spacing.py @@ -0,0 +1,204 @@ +#!/usr/bin/env python +# +# Copyright (C) 2012-2019 Red Hat, Inc. +# +# check-spacing.pl: Report any usage of 'function (..args..)' +# Also check for other syntax issues, such as correct use of ';' +# +# This library is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 2.1 of the License, or (at your option) any later version. +# +# This library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this library. If not, see +# <http://www.gnu.org/licenses/>. + +from __future__ import print_function + +import re +import sys + +def check_whitespace(filename): + errs = False + with open(filename, 'r') as fh: + quotedmetaprog = re.compile(r"""'[";,=]'""") + quotedstringprog = re.compile(r'''"(?:[^\\\"]|\\.)*"''') + commentstartprog = re.compile(r'''^(.*)/\*.*$''') + commentendprog = re.compile(r'''^.*\*/(.*)$''') + commentprog = re.compile(r'''^(.*)/\*.*\*/(.*)''') + funcprog = re.compile(r'''(\w+)\s\((?!\*)''') + keywordprog = re.compile(r'''^.*\b(?:if|for|while|switch|return)\(.*$''') + functypedefprog = re.compile(r'''^.*\(\*\w+\)\s+\(.*$''') + whitespaceprog1 = re.compile(r'''^.*\s\).*$''') + whitespaceprog2 = re.compile(r'''^\s+\);?$''') + whitespaceprog3 = re.compile(r'''^.*\((?!$)\s.*''') + commasemiprog1 = re.compile(r'''.*\s[;,].*''') + commasemiprog2 = re.compile(r'''.*\S; ; .*''') + commasemiprog3 = re.compile(r'''^\s+;''') + semicolonprog = re.compile(r'''.*;[^ \\\n;)].*''') + commaprog = re.compile(r'''.*,[^ \\\n)}].*''') + assignprog1 = re.compile(r'''[^ ]\b[!<>&|\-+*\/%\^=]?=''') + assignprog2 = re.compile(r'''=[^= \\\n]''') + condstartprog = re.compile(r'''^\s*(if|while|for)\b.*\{$''') + statementprog = re.compile(r'''^[^;]*;[^;]*$''') + condendprog = re.compile(r'''^\s*}\s*$''') + + incomment = False + # Per-file variables for multiline Curly Bracket (cb_) check + cb_lineno = 0 + cb_code = "" + cb_scolon = False + + lineno = 0 + for line in fh: + lineno = lineno + 1 + data = line + # For temporary modifications + + # Kill any quoted , ; = or " + data = quotedmetaprog.sub("'X'", data) + + # Kill any quoted strings + data = quotedstringprog.sub('"XXX"', data) + + if data[0] == '#': + continue + + # Kill contents of multi-line comments + # and detect end of multi-line comments + if incomment: + if commentendprog.match(data): + data = commentendprog.sub('*/\2', data) + incomment = False + else: + data = "" + + # Kill single line comments, and detect + # start of multi-line comments + if commentprog.match(data): + data = commentprog.sub(r'''\1/* */\2''', data) + elif commentstartprog.match(data): + data = commentstartprog.sub(r'''\1/*''', data) + incomment = True + + # We need to match things like + # + # int foo (int bar, bool wizz); + # foo (bar, wizz); + # + # but not match things like: + # + # typedef int (*foo)(bar wizz) + # + # we can't do this (efficiently) without + # missing things like + # + # foo (*bar, wizz); + # + for match in funcprog.finditer(data): + kw = match.group(1) + + # Allow space after keywords only + if kw not in ["if", "for", "while", "switch", "return"]: + print("Whitespace after non-keyword:", file=sys.stderr) + print("%s:%d: %s" % (filename, lineno, line), file=sys.stderr) + errs = True + break + + # Require whitespace immediately after keywords + if keywordprog.match(data): + print("No whitespace after keyword:", file=sys.stderr) + print("%s:%d: %s" % (filename, lineno, line), file=sys.stderr) + errs = True + + # Forbid whitespace between )( of a function typedef + if functypedefprog.match(data): + print("Whitespace between ')' and '(':", file=sys.stderr) + print("%s:%d: %s" % (filename, lineno, line), file=sys.stderr) + errs = True + + # Forbid whitespace following ( or prior to ) + # but allow whitespace before ) on a single line + # (optionally followed by a semicolon) + if ((whitespaceprog1.match(data) and + not whitespaceprog2.match(data)) + or whitespaceprog3.match(data)): + print("Whitespace after '(' or before ')':", file=sys.stderr) + print("%s:%d: %s" % (filename, lineno, line), file=sys.stderr) + errs = True + + # Forbid whitespace before ";" or ",". Things like below are allowed: + # + # 1) The expression is empty for "for" loop. E.g. + # for (i = 0; ; i++) + # + # 2) An empty statement. E.g. + # while (write(statuswrite, &status, 1) == -1 && + # errno == EINTR) + # ; + # + if commasemiprog1.match(data) and not ( + commasemiprog2.match(data) or + commasemiprog3.match(data)): + print("Whitespace before semicolon or comma:", file=sys.stderr) + print("%s:%d: %s" % (filename, lineno, line), file=sys.stderr) + errs = True + + # Require EOL, macro line continuation, or whitespace after ";". + # Allow "for (;;)" as an exception. + if semicolonprog.match(data): + print("Invalid character after semicolon:", file=sys.stderr) + print("%s:%d: %s" % (filename, lineno, line), file=sys.stderr) + errs = True + + # Require EOL, space, or enum/struct end after comma. + if commaprog.match(data): + print("Invalid character after comma:", file=sys.stderr) + print("%s:%d: %s" % (filename, lineno, line), file=sys.stderr) + errs = True + + # Require spaces around assignment '=', compounds and '==' + if assignprog1.match(data) or assignprog2.match(data): + print("Spacing around '=' or '==':", file=sys.stderr) + print("%s:%d: %s" % (filename, lineno, line), file=sys.stderr) + errs = True + + # One line conditional statements with one line bodies should + # not use curly brackets. + if condstartprog.match(data): + cb_lineno = lineno + cb_code = line + cb_scolon = False + + # We need to check for exactly one semicolon inside the body, + # because empty statements (e.g. with comment only) are + # allowed + if (cb_lineno == lineno - 1) and statementprog.match(data): + cb_code = cb_code + line + cb_scolon = True + + if condendprog.match(data) and (cb_lineno == lineno - 2) and cb_scolon: + print("Curly brackets around single-line body:", file=sys.stderr) + print("%s:%d:\n%s%s"% (filename, cb_linenum - lineno, cb_code, line), file=sys.stderr) + errs = True + + # There _should_ be no need to reset the values; but to + # keep my inner peace... + cb_linenum = 0 + cb_scolon = False + cb_code = "" + + return errs + +ret = 0 +for filename in sys.argv[1:]: + if check_whitespace(filename): + ret = 1 + +sys.exit(ret) diff --git a/cfg.mk b/cfg.mk index da1a1cb3cf..02bade7f8a 100644 --- a/cfg.mk +++ b/cfg.mk @@ -1152,7 +1152,7 @@ prohibit-duplicate-header: spacing-check: $(AM_V_GEN)$(VC_LIST) | $(GREP) '\.c$$' | xargs \ - $(PERL) $(top_srcdir)/build-aux/check-spacing.pl || \ + $(PYTHON) $(top_srcdir)/build-aux/check-spacing.py || \ { echo '$(ME): incorrect formatting' 1>&2; exit 1; } mock-noinline: -- 2.21.0
As part of an goal to eliminate Perl from libvirt build tools, rewrite the mock-noinline.pl tool in Python. This was a straight conversion, manually going line-by-line to change the syntax from Perl to Python. Thus the overall structure of the file and approach is the same. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- Makefile.am | 2 +- build-aux/mock-noinline.pl | 75 -------------------------------- build-aux/mock-noinline.py | 88 ++++++++++++++++++++++++++++++++++++++ cfg.mk | 2 +- 4 files changed, 90 insertions(+), 77 deletions(-) delete mode 100644 build-aux/mock-noinline.pl create mode 100644 build-aux/mock-noinline.py diff --git a/Makefile.am b/Makefile.am index d28c3ebcea..8bbc6e1747 100644 --- a/Makefile.am +++ b/Makefile.am @@ -46,7 +46,7 @@ EXTRA_DIST = \ build-aux/gitlog-to-changelog \ build-aux/header-ifdef.pl \ build-aux/minimize-po.py \ - build-aux/mock-noinline.pl \ + build-aux/mock-noinline.py \ build-aux/prohibit-duplicate-header.py \ build-aux/useless-if-before-free \ build-aux/vc-list-files \ diff --git a/build-aux/mock-noinline.pl b/build-aux/mock-noinline.pl deleted file mode 100644 index 958e133885..0000000000 --- a/build-aux/mock-noinline.pl +++ /dev/null @@ -1,75 +0,0 @@ -#!/usr/bin/env perl - -my %noninlined; -my %mocked; - -# Functions in public header don't get the noinline annotation -# so whitelist them here -$noninlined{"virEventAddTimeout"} = 1; -# This one confuses the script as its defined in the mock file -# but is actually just a local helper -$noninlined{"virMockStatRedirect"} = 1; - -foreach my $arg (@ARGV) { - if ($arg =~ /\.h$/) { - #print "Scan header $arg\n"; - &scan_annotations($arg); - } elsif ($arg =~ /mock\.c$/) { - #print "Scan mock $arg\n"; - &scan_overrides($arg); - } -} - -my $warned = 0; -foreach my $func (keys %mocked) { - next if exists $noninlined{$func}; - - $warned++; - print STDERR "$func is mocked at $mocked{$func} but missing noinline annotation\n"; -} - -exit $warned ? 1 : 0; - - -sub scan_annotations { - my $file = shift; - - open FH, $file or die "cannot read $file: $!"; - - my $func; - while (<FH>) { - if (/^\s*(\w+)\(/ || /^(?:\w+\*?\s+)+(?:\*\s*)?(\w+)\(/) { - my $name = $1; - if ($name !~ /ATTRIBUTE/) { - $func = $name; - } - } elsif (/^\s*$/) { - $func = undef; - } - if (/ATTRIBUTE_NOINLINE/) { - if (defined $func) { - $noninlined{$func} = 1; - } - } - } - - close FH -} - -sub scan_overrides { - my $file = shift; - - open FH, $file or die "cannot read $file: $!"; - - my $func; - while (<FH>) { - if (/^(\w+)\(/ || /^\w+\s*(?:\*\s*)?(\w+)\(/) { - my $name = $1; - if ($name =~ /^vir/) { - $mocked{$name} = "$file:$."; - } - } - } - - close FH -} diff --git a/build-aux/mock-noinline.py b/build-aux/mock-noinline.py new file mode 100644 index 0000000000..fdc60c5de9 --- /dev/null +++ b/build-aux/mock-noinline.py @@ -0,0 +1,88 @@ +#!/usr/bin/env python +# +# Copyright (C) 2017-2019 Red Hat, Inc. +# +# This library is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 2.1 of the License, or (at your option) any later version. +# +# This library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this library. If not, see +# <http://www.gnu.org/licenses/>. + +from __future__ import print_function + +import re +import sys + +noninlined = {} +mocked = {} + +# Functions in public header don't get the noinline annotation +# so whitelist them here +noninlined["virEventAddTimeout"] = True +# This one confuses the script as its defined in the mock file +# but is actually just a local helper +noninlined["virMockStatRedirect"] = True + +def scan_annotations(filename): + funcprog1 = re.compile(r'''^\s*(\w+)\(.*''') + funcprog2 = re.compile(r'''^(?:\w+\*?\s+)+(?:\*\s*)?(\w+)\(.*''') + with open(filename, "r") as fh: + func = None + for line in fh: + line = line.strip() + m = funcprog1.match(line) + if m is None: + m = funcprog2.match(line) + if m is not None: + name = m.group(1) + if name.find("ATTRIBUTE") == -1: + func = name + elif line == "": + func = None + + if line.find("ATTRIBUTE_NOINLINE") != -1: + if func is not None: + noninlined[func] = True + +def scan_overrides(filename): + funcprog1 = re.compile(r'''^(\w+)\(.*''') + funcprog2 = re.compile(r'''^\w+\s*(?:\*\s*)?(\w+)\(.*''') + with open(filename, "r") as fh: + func = None + lineno = 0 + for line in fh: + lineno = lineno + 1 + #line = line.strip() + + m = funcprog1.match(line) + if m is None: + m = funcprog2.match(line) + if m is not None: + name = m.group(1) + if name.startswith("vir"): + mocked[name] = "%s:%d" % (filename, lineno) + + +for filename in sys.argv[1:]: + if filename.endswith(".h"): + scan_annotations(filename) + elif filename.endswith("mock.c"): + scan_overrides(filename) + +warned = False +for func in mocked.keys(): + if func not in noninlined: + warned = True + print("%s is mocked at %s but missing noinline annotation" % (func, mocked[func]), file=sys.stderr) + +if warned: + sys.exit(1) +sys.exit(0) diff --git a/cfg.mk b/cfg.mk index 02bade7f8a..732ee8084c 100644 --- a/cfg.mk +++ b/cfg.mk @@ -1157,7 +1157,7 @@ spacing-check: mock-noinline: $(AM_V_GEN)$(VC_LIST) | $(GREP) '\.[ch]$$' | xargs \ - $(PERL) $(top_srcdir)/build-aux/mock-noinline.pl + $(PYTHON) $(top_srcdir)/build-aux/mock-noinline.py header-ifdef: $(AM_V_GEN)$(VC_LIST) | $(GREP) '\.[h]$$' | xargs \ -- 2.21.0
As part of an goal to eliminate Perl from libvirt build tools, rewrite the header-ifdef.pl tool in Python. This was a straight conversion, manually going line-by-line to change the syntax from Perl to Python. Thus the overall structure of the file and approach is the same. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- Makefile.am | 2 +- build-aux/header-ifdef.pl | 182 --------------------------------- build-aux/header-ifdef.py | 206 ++++++++++++++++++++++++++++++++++++++ cfg.mk | 2 +- 4 files changed, 208 insertions(+), 184 deletions(-) delete mode 100644 build-aux/header-ifdef.pl create mode 100644 build-aux/header-ifdef.py diff --git a/Makefile.am b/Makefile.am index 8bbc6e1747..22480882fc 100644 --- a/Makefile.am +++ b/Makefile.am @@ -44,7 +44,7 @@ EXTRA_DIST = \ build-aux/augeas-gentest.py \ build-aux/check-spacing.py \ build-aux/gitlog-to-changelog \ - build-aux/header-ifdef.pl \ + build-aux/header-ifdef.py \ build-aux/minimize-po.py \ build-aux/mock-noinline.py \ build-aux/prohibit-duplicate-header.py \ diff --git a/build-aux/header-ifdef.pl b/build-aux/header-ifdef.pl deleted file mode 100644 index dba3dbcbdc..0000000000 --- a/build-aux/header-ifdef.pl +++ /dev/null @@ -1,182 +0,0 @@ -#!/usr/bin/perl -# -# Validate that header files follow a standard layout: -# -# /* -# ...copyright header... -# */ -# <one blank line> -# #pragma once -# ....content.... -# -#--- -# -# For any file ending priv.h, before the #pragma once -# We will have a further section -# -# #ifndef SYMBOL_ALLOW -# # error .... -# #endif /* SYMBOL_ALLOW */ -# <one blank line> -# -#--- -# -# For public headers (files in include/), use the standard header guard instead of #pragma once: -# #ifndef SYMBOL -# # define SYMBOL -# ....content.... -# #endif /* SYMBOL */ - -use strict; -use warnings; - -my $STATE_COPYRIGHT_COMMENT = 0; -my $STATE_COPYRIGHT_BLANK = 1; -my $STATE_PRIV_START = 2; -my $STATE_PRIV_ERROR = 3; -my $STATE_PRIV_END = 4; -my $STATE_PRIV_BLANK = 5; -my $STATE_GUARD_START = 6; -my $STATE_GUARD_DEFINE = 7; -my $STATE_GUARD_END = 8; -my $STATE_EOF = 9; -my $STATE_PRAGMA = 10; - -my $file = " "; -my $ret = 0; -my $ifdef = ""; -my $ifdefpriv = ""; -my $publicheader = 0; - -my $state = $STATE_EOF; -my $mistake = 0; - -sub mistake { - my $msg = shift; - warn $msg; - $mistake = 1; - $ret = 1; -} - -while (<>) { - if (not $file eq $ARGV) { - if ($state == $STATE_COPYRIGHT_COMMENT) { - &mistake("$file: missing copyright comment"); - } elsif ($state == $STATE_COPYRIGHT_BLANK) { - &mistake("$file: missing blank line after copyright header"); - } elsif ($state == $STATE_PRIV_START) { - &mistake("$file: missing '#ifndef $ifdefpriv'"); - } elsif ($state == $STATE_PRIV_ERROR) { - &mistake("$file: missing '# error ...priv allow...'"); - } elsif ($state == $STATE_PRIV_END) { - &mistake("$file: missing '#endif /* $ifdefpriv */'"); - } elsif ($state == $STATE_PRIV_BLANK) { - &mistake("$file: missing blank line after priv header check"); - } elsif ($state == $STATE_GUARD_START) { - if ($publicheader) { - &mistake("$file: missing '#ifndef $ifdef'"); - } else { - &mistake("$file: missing '#pragma once' header guard"); - } - } elsif ($state == $STATE_GUARD_DEFINE) { - &mistake("$file: missing '# define $ifdef'"); - } elsif ($state == $STATE_GUARD_END) { - &mistake("$file: missing '#endif /* $ifdef */'"); - } - - $ifdef = uc $ARGV; - $ifdef =~ s,.*/,,; - $ifdef =~ s,[^A-Z0-9],_,g; - $ifdef =~ s,__+,_,g; - unless ($ifdef =~ /^LIBVIRT_/ && $ARGV !~ /libvirt_internal.h/) { - $ifdef = "LIBVIRT_" . $ifdef; - } - $ifdefpriv = $ifdef . "_ALLOW"; - - $file = $ARGV; - $state = $STATE_COPYRIGHT_COMMENT; - $mistake = 0; - $publicheader = ($ARGV =~ /include\//); - } - - if ($mistake || - $ARGV =~ /config-post\.h$/ || - $ARGV =~ /vbox_(CAPI|XPCOM)/) { - $state = $STATE_EOF; - next; - } - - if ($state == $STATE_COPYRIGHT_COMMENT) { - if (m,\*/,) { - $state = $STATE_COPYRIGHT_BLANK; - } - } elsif ($state == $STATE_COPYRIGHT_BLANK) { - if (! /^$/) { - &mistake("$file: missing blank line after copyright header"); - } - if ($ARGV =~ /priv\.h$/) { - $state = $STATE_PRIV_START; - } else { - $state = $STATE_GUARD_START; - } - } elsif ($state == $STATE_PRIV_START) { - if (/^$/) { - &mistake("$file: too many blank lines after copyright header"); - } elsif (/#ifndef $ifdefpriv$/) { - $state = $STATE_PRIV_ERROR; - } else { - &mistake("$file: missing '#ifndef $ifdefpriv'"); - } - } elsif ($state == $STATE_PRIV_ERROR) { - if (/# error ".*"$/) { - $state = $STATE_PRIV_END; - } else { - &mistake("$file: missing '# error ...priv allow...'"); - } - } elsif ($state == $STATE_PRIV_END) { - if (m,#endif /\* $ifdefpriv \*/,) { - $state = $STATE_PRIV_BLANK; - } else { - &mistake("$file: missing '#endif /* $ifdefpriv */'"); - } - } elsif ($state == $STATE_PRIV_BLANK) { - if (! /^$/) { - &mistake("$file: missing blank line after priv guard"); - } - $state = $STATE_GUARD_START; - } elsif ($state == $STATE_GUARD_START) { - if (/^$/) { - &mistake("$file: too many blank lines after copyright header"); - } - if ($publicheader) { - if (/#ifndef $ifdef$/) { - $state = $STATE_GUARD_DEFINE; - } else { - &mistake("$file: missing '#ifndef $ifdef'"); - } - } else { - if (/#pragma once/) { - $state = $STATE_PRAGMA; - } else { - &mistake("$file: missing '#pragma once' header guard"); - } - } - } elsif ($state == $STATE_GUARD_DEFINE) { - if (/# define $ifdef$/) { - $state = $STATE_GUARD_END; - } else { - &mistake("$file: missing '# define $ifdef'"); - } - } elsif ($state == $STATE_GUARD_END) { - if (m,#endif /\* $ifdef \*/$,) { - $state = $STATE_EOF; - } - } elsif ($state == $STATE_PRAGMA) { - next; - } elsif ($state == $STATE_EOF) { - die "$file: unexpected content after '#endif /* $ifdef */'"; - } else { - die "$file: unexpected state $state"; - } -} -exit $ret; diff --git a/build-aux/header-ifdef.py b/build-aux/header-ifdef.py new file mode 100644 index 0000000000..86c4781a7e --- /dev/null +++ b/build-aux/header-ifdef.py @@ -0,0 +1,206 @@ +#!/usr/bin/env python +# +# Copyright (C) 2018-2019 Red Hat, Inc. +# +# This library is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 2.1 of the License, or (at your option) any later version. +# +# This library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this library. If not, see +# <http://www.gnu.org/licenses/>. +# +# Validate that header files follow a standard layout: +# +# /* +# ...copyright header... +# */ +# <one blank line> +# #pragma once +# ....content.... +# +#--- +# +# For any file ending priv.h, before the #pragma once +# We will have a further section +# +# #ifndef SYMBOL_ALLOW +# # error .... +# #endif /* SYMBOL_ALLOW */ +# <one blank line> +# +#--- +# +# For public headers (files in include/), use the standard header guard instead of #pragma once: +# #ifndef SYMBOL +# # define SYMBOL +# ....content.... +# #endif /* SYMBOL */ + +from __future__ import print_function + +import os.path +import re +import sys + +STATE_COPYRIGHT_COMMENT = 0 +STATE_COPYRIGHT_BLANK = 1 +STATE_PRIV_START = 2 +STATE_PRIV_ERROR = 3 +STATE_PRIV_END = 4 +STATE_PRIV_BLANK = 5 +STATE_GUARD_START = 6 +STATE_GUARD_DEFINE = 7 +STATE_GUARD_END = 8 +STATE_EOF = 9 +STATE_PRAGMA = 10 + +def check_header(filename): + ifdef = "" + ifdefpriv = "" + + state = STATE_EOF + + ifdef = os.path.basename(filename).upper() + ifdef = re.sub(r"""[^A-Z0-9]""", "_", ifdef) + ifdef = re.sub(r"""__+""", "_", ifdef) + + if not ifdef.startswith("LIBVIRT_") or filename.find("libvirt_internal.h") != -1: + ifdef = "LIBVIRT_" + ifdef + + ifdefpriv = ifdef + "_ALLOW" + + state = STATE_COPYRIGHT_COMMENT + publicheader = False + if filename.find("include/") != -1: + publicheader = True + + with open(filename, "r") as fh: + for line in fh: + line = line.rstrip("\n") + if state == STATE_COPYRIGHT_COMMENT: + if line.find("*/") != -1: + state = STATE_COPYRIGHT_BLANK + elif state == STATE_COPYRIGHT_BLANK: + if line != "": + print("%s: missing blank line after copyright header" % filename, file=sys.stderr) + return True + + if filename.endswith("priv.h"): + state = STATE_PRIV_START + else: + state = STATE_GUARD_START + elif state == STATE_PRIV_START: + if line == "": + print("%s: too many blank lines after copyright header" % filename, file=sys.stderr) + return True + elif re.match(r"""#ifndef %s$""" % ifdefpriv, line): + state = STATE_PRIV_ERROR + else: + print("%s: missing '#ifndef %s'" % (filename, ifdefpriv), file=sys.stderr) + return True + elif state == STATE_PRIV_ERROR: + if re.match(r"""# error ".*"$""", line): + state = STATE_PRIV_END + else: + print("%s: missing '# error ...priv allow...'" % filename, file=sys.stderr) + return True + elif state == STATE_PRIV_END: + if re.match(r"""#endif /\* %s \*/""" % ifdefpriv, line): + state = STATE_PRIV_BLANK + else: + print("%s: missing '#endif /* %s */'" % (filename, ifdefpriv), file=sys.stderr) + return True + elif state == STATE_PRIV_BLANK: + if line != "": + print("%s: missing blank line after priv guard" % filename, file=sys.stderr) + return True + state = STATE_GUARD_START + elif state == STATE_GUARD_START: + if line == "": + print("%s: too many blank lines after copyright header" % filename, file=sys.stderr) + return True + if publicheader: + if re.match(r"""#ifndef %s$""" % ifdef, line): + state = STATE_GUARD_DEFINE + else: + print("%s: missing '#ifndef %s'" % (filename, ifdef), file=sys.stderr) + return True + else: + if re.match(r"""#pragma once""", line): + state = STATE_PRAGMA + else: + print("%s: missing '#pragma once' header guard" % filename, file=sys.stderr) + return True + elif state == STATE_GUARD_DEFINE: + if re.match(r"""# define %s$""" % ifdef, line): + state = STATE_GUARD_END + else: + print("%s: missing '# define %s'" % (filename, ifdef), file=sys.stderr) + return True + elif state == STATE_GUARD_END: + if re.match(r"""#endif /\* %s \*/$""" % ifdef, line): + state = STATE_EOF + elif state == STATE_PRAGMA: + next + elif state == STATE_EOF: + print("%s: unexpected content after '#endif /* %s */'" % (filename, ifdef), file=sys.stderr) + return True + else: + print("%s: unexpected state $state" % filename, file=sys.stderr) + return True + + + if state == STATE_COPYRIGHT_COMMENT: + print("%s: missing copyright comment" % filename, file=sys.stderr) + return True + elif state == STATE_COPYRIGHT_BLANK: + print("%s: missing blank line after copyright header" % filename, file=sys.stderr) + return True + elif state == STATE_PRIV_START: + print("%s: missing '#ifndef %s'" % (filename, ifdefpriv), file=sys.stderr) + return True + elif state == STATE_PRIV_ERROR: + print("%s: missing '# error ...priv allow...'" % filename, file=sys.stderr) + return True + elif state == STATE_PRIV_END: + print("%s: missing '#endif /* %s */'" % (filename, ifdefpriv), file=sys.stderr) + return True + elif state == STATE_PRIV_BLANK: + print("%s: missing blank line after priv header check" % filename, file=sys.stderr) + return True + elif state == STATE_GUARD_START: + if publicheader: + print("%s: missing '#ifndef %s'" % (filename, ifdef), file=sys.stderr) + return True + else: + print("%s: missing '#pragma once' header guard" % filename, file=sys.stderr) + return True + elif state == STATE_GUARD_DEFINE: + print("%s: missing '# define %s'" % (filename, ifdef), file=sys.stderr) + return True + elif state == STATE_GUARD_END: + print("%s: missing '#endif /* %s */'" % (filename, ifdef), file=sys.stderr) + return True + + return False + +ret = 0 + +for filename in sys.argv[1:]: + if filename.find("config-post.h") != -1: + continue + if filename.find("vbox_CAPI") != -1: + continue + if filename.find("vbox_XPCOM") != -1: + continue + if check_header(filename): + ret = 1 + +sys.exit(ret) diff --git a/cfg.mk b/cfg.mk index 732ee8084c..c398cf9d9f 100644 --- a/cfg.mk +++ b/cfg.mk @@ -1161,7 +1161,7 @@ mock-noinline: header-ifdef: $(AM_V_GEN)$(VC_LIST) | $(GREP) '\.[h]$$' | xargs \ - $(PERL) $(top_srcdir)/build-aux/header-ifdef.pl + $(PYTHON) $(top_srcdir)/build-aux/header-ifdef.py test-wrap-argv: $(AM_V_GEN)$(VC_LIST) | $(GREP) -E '\.(ldargs|args)' | xargs \ -- 2.21.0
On Thu, Sep 05, 2019 at 11:21:55AM +0100, Daniel P. Berrangé wrote: [snipping the zoo]
Daniel P. Berrangé (11): cfg.mk: fix comment detection for python semicolon check docs: rewrite hvsupport.html page generator in python docs: rewrite ACL permissions checker in Python docs: rewrite symfile sorting checker in Python docs: rewrite symfile library checker in Python docs: rewrite augest test generator in Python docs: rewrite po file minimizer in Python docs: rewrite duplicate header checker in Python docs: rewrite whitespace checker in Python docs: rewrite mock inline checker in Python docs: rewrite header ifdef checker in Python
As I said in RE: Language Consolidation, this could use a preceding HACKING change with justification. Also, rewrite to a new language does mean throwing away debugged and optimized code - but if the end goal is to drop automake and configure, overall the build will be much quicker anyway. Also, this does not work with Python 3.6 on my Gentoo: Traceback (most recent call last): File "./build-aux/prohibit-duplicate-header.py", line 48, in <module> if check_file(filename): File "./build-aux/prohibit-duplicate-header.py", line 30, in check_file for line in fh: File "/usr/lib/python-exec/python3.6/../../../lib64/python3.6/encodings/ascii.py", line 26, in decode return codecs.ascii_decode(input, self.errors)[0] UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 548: ordinal not in range(128) makefile_conditionals po_check preprocessor_indentation prohibit_HAVE_MBRTOWC make: *** [cfg.mk:1150: prohibit-duplicate-header] Error 123 make: *** Waiting for unfinished jobs.... prohibit_PATH_MAX Traceback (most recent call last): File "./build-aux/header-ifdef.py", line 203, in <module> if check_header(filename): File "./build-aux/header-ifdef.py", line 85, in check_header for line in fh: File "/usr/lib/python-exec/python3.6/../../../lib64/python3.6/encodings/ascii.py", line 26, in decode return codecs.ascii_decode(input, self.errors)[0] UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 100: ordinal not in range(128) make: *** [cfg.mk:1163: header-ifdef] Error 123 Traceback (most recent call last): File "./build-aux/mock-noinline.py", line 76, in <module> scan_annotations(filename) File "./build-aux/mock-noinline.py", line 39, in scan_annotations for line in fh: File "/usr/lib/python-exec/python3.6/../../../lib64/python3.6/encodings/ascii.py", line 26, in decode return codecs.ascii_decode(input, self.errors)[0] UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 100: ordinal not in range(128) make: *** [cfg.mk:1159: mock-noinline] Error 123 Traceback (most recent call last): File "./build-aux/check-spacing.py", line 201, in <module> if check_whitespace(filename): File "./build-aux/check-spacing.py", line 59, in check_whitespace for line in fh: File "/usr/lib/python-exec/python3.6/../../../lib64/python3.6/encodings/ascii.py", line 26, in decode return codecs.ascii_decode(input, self.errors)[0] UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 548: ordinal not in range(128) maint.mk: incorrect formatting make: *** [cfg.mk:1154: spacing-check] Error 1 Jano
On Thu, Sep 05, 2019 at 01:43:32PM +0200, Ján Tomko wrote:
On Thu, Sep 05, 2019 at 11:21:55AM +0100, Daniel P. Berrangé wrote: [snipping the zoo]
Daniel P. Berrangé (11): cfg.mk: fix comment detection for python semicolon check docs: rewrite hvsupport.html page generator in python docs: rewrite ACL permissions checker in Python docs: rewrite symfile sorting checker in Python docs: rewrite symfile library checker in Python docs: rewrite augest test generator in Python docs: rewrite po file minimizer in Python docs: rewrite duplicate header checker in Python docs: rewrite whitespace checker in Python docs: rewrite mock inline checker in Python docs: rewrite header ifdef checker in Python
As I said in RE: Language Consolidation, this could use a preceding HACKING change with justification.
Also, rewrite to a new language does mean throwing away debugged and optimized code - but if the end goal is to drop automake and configure, overall the build will be much quicker anyway.
Debugged yes, but I don't think I'd call this optimized code - it is pretty much all a quick hack.
Also, this does not work with Python 3.6 on my Gentoo: Traceback (most recent call last): File "./build-aux/prohibit-duplicate-header.py", line 48, in <module> if check_file(filename): File "./build-aux/prohibit-duplicate-header.py", line 30, in check_file for line in fh: File "/usr/lib/python-exec/python3.6/../../../lib64/python3.6/encodings/ascii.py", line 26, in decode return codecs.ascii_decode(input, self.errors)[0] UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 548: ordinal not in range(128)
Is your locale ($LANG / $LC_*) not using a UTF-8 character set perhaps ? Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Thu, Sep 05, 2019 at 12:48:34PM +0100, Daniel P. Berrangé wrote:
On Thu, Sep 05, 2019 at 01:43:32PM +0200, Ján Tomko wrote:
On Thu, Sep 05, 2019 at 11:21:55AM +0100, Daniel P. Berrangé wrote: [snipping the zoo]
Daniel P. Berrangé (11): cfg.mk: fix comment detection for python semicolon check docs: rewrite hvsupport.html page generator in python docs: rewrite ACL permissions checker in Python docs: rewrite symfile sorting checker in Python docs: rewrite symfile library checker in Python docs: rewrite augest test generator in Python docs: rewrite po file minimizer in Python docs: rewrite duplicate header checker in Python docs: rewrite whitespace checker in Python docs: rewrite mock inline checker in Python docs: rewrite header ifdef checker in Python
As I said in RE: Language Consolidation, this could use a preceding HACKING change with justification.
Also, rewrite to a new language does mean throwing away debugged and optimized code - but if the end goal is to drop automake and configure, overall the build will be much quicker anyway.
Debugged yes, but I don't think I'd call this optimized code - it is pretty much all a quick hack.
I stand by that. I actually ran a profiler on some of these back in 2016 :)
Also, this does not work with Python 3.6 on my Gentoo: Traceback (most recent call last): File "./build-aux/prohibit-duplicate-header.py", line 48, in <module> if check_file(filename): File "./build-aux/prohibit-duplicate-header.py", line 30, in check_file for line in fh: File "/usr/lib/python-exec/python3.6/../../../lib64/python3.6/encodings/ascii.py", line 26, in decode return codecs.ascii_decode(input, self.errors)[0] UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 548: ordinal not in range(128)
Is your locale ($LANG / $LC_*) not using a UTF-8 character set perhaps ?
Running 'print (locale.getlocale())' in a manually invoked shell has it: ('en_US', 'UTF-8') when invoked via 'make prohibit-duplicate-header' the locale is not set: (None, None) That seems to be gnulib's fault: top/maint.mk:156:export LC_ALL = C Jano
Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
-- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
On Thu, Sep 05, 2019 at 02:20:55PM +0200, Ján Tomko wrote:
On Thu, Sep 05, 2019 at 12:48:34PM +0100, Daniel P. Berrangé wrote:
On Thu, Sep 05, 2019 at 01:43:32PM +0200, Ján Tomko wrote:
On Thu, Sep 05, 2019 at 11:21:55AM +0100, Daniel P. Berrangé wrote: [snipping the zoo]
Daniel P. Berrangé (11): cfg.mk: fix comment detection for python semicolon check docs: rewrite hvsupport.html page generator in python docs: rewrite ACL permissions checker in Python docs: rewrite symfile sorting checker in Python docs: rewrite symfile library checker in Python docs: rewrite augest test generator in Python docs: rewrite po file minimizer in Python docs: rewrite duplicate header checker in Python docs: rewrite whitespace checker in Python docs: rewrite mock inline checker in Python docs: rewrite header ifdef checker in Python
As I said in RE: Language Consolidation, this could use a preceding HACKING change with justification.
Also, rewrite to a new language does mean throwing away debugged and optimized code - but if the end goal is to drop automake and configure, overall the build will be much quicker anyway.
Debugged yes, but I don't think I'd call this optimized code - it is pretty much all a quick hack.
I stand by that. I actually ran a profiler on some of these back in 2016 :)
Also, this does not work with Python 3.6 on my Gentoo: Traceback (most recent call last): File "./build-aux/prohibit-duplicate-header.py", line 48, in <module> if check_file(filename): File "./build-aux/prohibit-duplicate-header.py", line 30, in check_file for line in fh: File "/usr/lib/python-exec/python3.6/../../../lib64/python3.6/encodings/ascii.py", line 26, in decode return codecs.ascii_decode(input, self.errors)[0] UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 548: ordinal not in range(128)
Is your locale ($LANG / $LC_*) not using a UTF-8 character set perhaps ?
Running 'print (locale.getlocale())' in a manually invoked shell has it: ('en_US', 'UTF-8') when invoked via 'make prohibit-duplicate-header' the locale is not set: (None, None)
That seems to be gnulib's fault: top/maint.mk:156:export LC_ALL = C
Interesting, I wonder why that gnulib thing didn't affect me too. Anyway, it should be easy enough to fix this to guarantee UTF8 for python Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
participants (3)
-
Daniel P. Berrangé -
Eric Blake -
Ján Tomko