[libvirt] [PATCH v2 00/24] scripts: convert most perl scripts to python

This series is an effort to reduce the number of different languages we use by eliminating most use of perl in favour of python. This aligns with fact that the likely future build system we'll use (meson) is written in python, and that python is much more commonly used/understood by developers these days than perl. With this applied we use perl in a handful of places only: - src/rpc/gendispatch.pl - this is a horrendously large script and very hard to understand/follow. A straight syntax conversion to Python would still leave a hgue and hard to understand/follow script. It really needs a clean room rewrite from scratch, with better structure. - src/rpc/genprotocol.pl - fairly easy to convert, but might be obsolete depending on approach for rewriting gendispatch.pl, so ignored for now - tests/oomtrace.pl - will be purge by the patches that drop OOM handling anyway - tools/wireshark/util/genxdrstub.pl - a very large script, which I haven't got the courage to tackle yet. - cfg.mk/maint.mk - many syntax rules involve regexes which are fed to perl. Decision on what to do with syntax-check rules punted to another time. - build-aux/gitlog-to-changelog - build-aux/useless-if-before-free - Both pulled in from gnulib. Could be rewritten quite easily if desired, but given that we aren't maintainers of them right now, they're ignored as they don't really impact our developers. In v2: - Pulled in patch to hacking file - Converted many more scripts - Forced UTF-8 character set to avoid ascii codec on py3 < 3.7 Daniel P. Berrangé (24): docs: document that C & Python are the preferred languages cfg.mk: fix comment detection for python semicolon check build: force a UTF-8 locale for python build-aux: rewrite augest test generator in Python build-aux: rewrite po file minimizer in Python build-aux: rewrite duplicate header checker in Python build-aux: rewrite whitespace checker in Python build-aux: rewrite mock inline checker in Python build-aux: rewrite header ifdef checker in Python src: rewrite ACL permissions checker in Python src: rewrite symfile sorting checker in Python src: rewrite symfile library checker in Python src: rewrite systemtap probe generator in Python src: rewrite systemtap function generator in Python src: rewrite driver name checker in Python src: rewrite driver impl checker in Python src: rewrite ACL rule checker in Python src: rewrite polkit ACL generator in Python src: rewrite remote protocol checker in Python tests: rewrite test argv line wrapper in Python tests: rewrite qemu capability grouper in Python tests: rewrite file access checker in Python docs: rewrite hvsupport.html page generator in python docs: rewrite polkit docs generator in Python Makefile.am | 14 +- build-aux/augeas-gentest.pl | 60 ---- build-aux/augeas-gentest.py | 72 ++++ build-aux/check-spacing.pl | 198 ---------- build-aux/check-spacing.py | 204 +++++++++++ build-aux/header-ifdef.pl | 182 ---------- build-aux/header-ifdef.py | 206 +++++++++++ build-aux/minimize-po.pl | 37 -- build-aux/minimize-po.py | 60 ++++ build-aux/mock-noinline.pl | 75 ---- build-aux/mock-noinline.py | 88 +++++ build-aux/prohibit-duplicate-header.pl | 26 -- build-aux/prohibit-duplicate-header.py | 54 +++ cfg.mk | 32 +- configure.ac | 8 + docs/Makefile.am | 17 +- docs/genaclperms.pl | 125 ------- docs/genaclperms.py | 122 +++++++ docs/hacking.html.in | 30 ++ docs/hvsupport.pl | 458 ----------------------- docs/hvsupport.py | 479 +++++++++++++++++++++++++ po/Makefile.am | 2 +- src/Makefile.am | 156 +++----- src/access/Makefile.inc.am | 6 +- src/access/genpolkit.pl | 119 ------ src/access/genpolkit.py | 119 ++++++ src/bhyve/Makefile.inc.am | 4 +- src/check-aclperms.pl | 73 ---- src/check-aclperms.py | 77 ++++ src/check-aclrules.pl | 252 ------------- src/check-aclrules.py | 244 +++++++++++++ src/check-driverimpls.pl | 80 ----- src/check-driverimpls.py | 96 +++++ src/check-drivername.pl | 83 ----- src/check-drivername.py | 112 ++++++ src/check-remote-protocol.py | 131 +++++++ src/check-symfile.pl | 70 ---- src/check-symfile.py | 80 +++++ src/check-symsorting.pl | 106 ------ src/check-symsorting.py | 112 ++++++ src/dtrace2systemtap.pl | 130 ------- src/dtrace2systemtap.py | 140 ++++++++ src/esx/Makefile.inc.am | 2 +- src/hyperv/Makefile.inc.am | 2 +- src/interface/Makefile.inc.am | 2 +- src/libxl/Makefile.inc.am | 4 +- src/locking/Makefile.inc.am | 6 +- src/logging/Makefile.inc.am | 2 +- src/lxc/Makefile.inc.am | 4 +- src/network/Makefile.inc.am | 2 +- src/node_device/Makefile.inc.am | 2 +- src/nwfilter/Makefile.inc.am | 2 +- src/qemu/Makefile.inc.am | 4 +- src/remote/Makefile.inc.am | 4 +- src/rpc/Makefile.inc.am | 2 +- src/rpc/gensystemtap.pl | 193 ---------- src/rpc/gensystemtap.py | 174 +++++++++ src/secret/Makefile.inc.am | 2 +- src/storage/Makefile.inc.am | 2 +- src/util/Makefile.inc.am | 8 +- src/vbox/Makefile.inc.am | 2 +- src/vz/Makefile.inc.am | 2 +- tests/Makefile.am | 4 +- tests/check-file-access.pl | 126 ------- tests/check-file-access.py | 121 +++++++ tests/file_access_whitelist.txt | 2 +- tests/group-qemu-caps.pl | 124 ------- tests/group-qemu-caps.py | 115 ++++++ tests/test-wrap-argv.pl | 174 --------- tests/test-wrap-argv.py | 162 +++++++++ tests/testutils.c | 16 +- 71 files changed, 3132 insertions(+), 2872 deletions(-) delete mode 100755 build-aux/augeas-gentest.pl create mode 100755 build-aux/augeas-gentest.py delete mode 100755 build-aux/check-spacing.pl create mode 100755 build-aux/check-spacing.py delete mode 100644 build-aux/header-ifdef.pl create mode 100644 build-aux/header-ifdef.py delete mode 100755 build-aux/minimize-po.pl create mode 100755 build-aux/minimize-po.py delete mode 100644 build-aux/mock-noinline.pl create mode 100644 build-aux/mock-noinline.py delete mode 100644 build-aux/prohibit-duplicate-header.pl create mode 100644 build-aux/prohibit-duplicate-header.py delete mode 100755 docs/genaclperms.pl create mode 100755 docs/genaclperms.py delete mode 100755 docs/hvsupport.pl create mode 100755 docs/hvsupport.py delete mode 100755 src/access/genpolkit.pl create mode 100755 src/access/genpolkit.py delete mode 100755 src/check-aclperms.pl create mode 100755 src/check-aclperms.py delete mode 100755 src/check-aclrules.pl create mode 100755 src/check-aclrules.py delete mode 100755 src/check-driverimpls.pl create mode 100755 src/check-driverimpls.py delete mode 100755 src/check-drivername.pl create mode 100644 src/check-drivername.py create mode 100644 src/check-remote-protocol.py delete mode 100755 src/check-symfile.pl create mode 100755 src/check-symfile.py delete mode 100755 src/check-symsorting.pl create mode 100755 src/check-symsorting.py delete mode 100755 src/dtrace2systemtap.pl create mode 100755 src/dtrace2systemtap.py delete mode 100755 src/rpc/gensystemtap.pl create mode 100755 src/rpc/gensystemtap.py delete mode 100755 tests/check-file-access.pl create mode 100755 tests/check-file-access.py delete mode 100755 tests/group-qemu-caps.pl create mode 100755 tests/group-qemu-caps.py delete mode 100755 tests/test-wrap-argv.pl create mode 100755 tests/test-wrap-argv.py -- 2.21.0

Blacklist Perl and Shell code in favour of Python for sake of readability and portability. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- docs/hacking.html.in | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/docs/hacking.html.in b/docs/hacking.html.in index 90376968c3..edf2f54ce3 100644 --- a/docs/hacking.html.in +++ b/docs/hacking.html.in @@ -343,6 +343,36 @@ Richard Jones' guide to working with open source projects</a>. </p> + <h2><a id="lang">Language Usage</a></h2> + + <p> + The libvirt repository makes use of a large number of programming + languages. It is anticipated that in the future libvirt will adopt + use of other new languages. To reduce the overall burden on developers, + there is thus a general desire to phase out usage of some of the + existing languages. + </p> + + <p> + The preferred languages at this time are: + </p> + + <ul> + <li>C - for the main libvirt codebase. Dialect supported by + GCC/CLang only.</li> + <li>Python - for supporting build scripts / tools. Code must + run with both version 2.7 and 3.x at this time.</li> + </ul> + + <p> + Languages that should not be used for any new contributions: + </p> + + <ul> + <li>Perl - build scripts must be written in Python instead.</li> + <li>Shell - build scripts must be written in Python instead.</li> + </ul> + <h2><a id="tooling">Tooling</a></h2> <p> -- 2.21.0

The pattern ^[^#].*\;$$ Was attempting to detect any trailing ';' in python code which was not in a comment. This does not allow for the comment '#' character to be indented with whitespace. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- cfg.mk | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cfg.mk b/cfg.mk index 1f29729949..d09ab507e0 100644 --- a/cfg.mk +++ b/cfg.mk @@ -815,7 +815,7 @@ sc_require_enum_last_marker: # In Python files we don't want to end lines with a semicolon like in C sc_prohibit_semicolon_at_eol_in_python: - @prohibit='^[^#].*\;$$' \ + @prohibit='^[ \t]*[^# \t].*\;$$' \ in_vc_files='\.py$$' \ halt='python does not require to end lines with a semicolon' \ $(_sc_search_regexp) -- 2.21.0

On 9/11/19 11:23 AM, Daniel P. Berrangé wrote:
The pattern
^[^#].*\;$$
Was attempting to detect any trailing ';' in python code which was not in a comment. This does not allow for the comment '#' character to be indented with whitespace.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- cfg.mk | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Reviewed-by: Eric Blake <eblake@redhat.com>
diff --git a/cfg.mk b/cfg.mk index 1f29729949..d09ab507e0 100644 --- a/cfg.mk +++ b/cfg.mk @@ -815,7 +815,7 @@ sc_require_enum_last_marker:
# In Python files we don't want to end lines with a semicolon like in C sc_prohibit_semicolon_at_eol_in_python: - @prohibit='^[^#].*\;$$' \ + @prohibit='^[ \t]*[^# \t].*\;$$' \ in_vc_files='\.py$$' \ halt='python does not require to end lines with a semicolon' \ $(_sc_search_regexp)
-- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org

Python3 versions less than 3.7 have very unhelpful handling of the C locale where they assume data is 7-bit only. This violates POSIX which requires the C locale to be 8-bit clean. Python3 >= 3.7 now assumes that the C locale is always UTF-8. Set env variables to force LC_CTYPE to en_US.UTF-8 so that we get UTF-8 handling on all python versions. Note we do not use C.UTF-8 since not all C libraries support that. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- Makefile.am | 2 +- configure.ac | 8 ++++++++ docs/Makefile.am | 3 ++- src/esx/Makefile.inc.am | 2 +- src/hyperv/Makefile.inc.am | 2 +- src/util/Makefile.inc.am | 8 ++++---- 6 files changed, 17 insertions(+), 8 deletions(-) diff --git a/Makefile.am b/Makefile.am index cf9ff94f4f..711f365504 100644 --- a/Makefile.am +++ b/Makefile.am @@ -69,7 +69,7 @@ NEWS: \ $(srcdir)/docs/news.xml \ >$@-tmp \ || { rm -f $@-tmp; exit 1; }; \ - $(PYTHON) $(srcdir)/docs/reformat-news.py $@-tmp >$@ \ + $(RUNUTF8) $(PYTHON) $(srcdir)/docs/reformat-news.py $@-tmp >$@ \ || { rm -f $@-tmp; exit 1; }; \ rm -f $@-tmp; \ fi diff --git a/configure.ac b/configure.ac index 7c76a9c9ec..31810b9010 100644 --- a/configure.ac +++ b/configure.ac @@ -706,6 +706,14 @@ AC_PATH_PROGS([PYTHON], [python3 python2 python]) if test -z "$PYTHON"; then AC_MSG_ERROR(['python3', 'python2' or 'python' binary is required to build libvirt]) fi + +dnl Python3 < 3.7 treats the C locale as 7-bit only. +dnl We must force env vars so it treats it as UTF-8 +dnl regardless of the user's locale. +RUNUTF8="LC_ALL= LANG=C LC_CTYPE=en_US.UTF-8" +AC_SUBST(RUNUTF8) + + AC_PATH_PROG([PERL], [perl]) if test -z "$PERL"; then AC_MSG_ERROR(['perl' binary is required to build libvirt]) diff --git a/docs/Makefile.am b/docs/Makefile.am index 1cdb584b0b..14f3faffd4 100644 --- a/docs/Makefile.am +++ b/docs/Makefile.am @@ -348,7 +348,8 @@ $(APIBUILD_STAMP): $(srcdir)/apibuild.py \ $(top_srcdir)/src/util/virerror.c \ $(top_srcdir)/src/util/virevent.c \ $(top_srcdir)/src/util/virtypedparam.c - $(AM_V_GEN)srcdir=$(srcdir) builddir=$(builddir) $(PYTHON) $(APIBUILD) + $(AM_V_GEN)srcdir=$(srcdir) builddir=$(builddir) \ + $(RUNUTF8) $(PYTHON) $(APIBUILD) touch $@ diff --git a/src/esx/Makefile.inc.am b/src/esx/Makefile.inc.am index 2d55c6573c..70a13ea829 100644 --- a/src/esx/Makefile.inc.am +++ b/src/esx/Makefile.inc.am @@ -63,7 +63,7 @@ $(ESX_DRIVER_GENERATED): $(ESX_GENERATED_STAMP) $(ESX_GENERATED_STAMP): $(srcdir)/esx/esx_vi_generator.input \ $(srcdir)/esx/esx_vi_generator.py - $(AM_V_GEN)srcdir=$(srcdir) $(PYTHON) $(srcdir)/esx/esx_vi_generator.py \ + $(AM_V_GEN)srcdir=$(srcdir) $(RUNUTF8) $(PYTHON) $(srcdir)/esx/esx_vi_generator.py \ && touch $@ MAINTAINERCLEANFILES += $(ESX_DRIVER_GENERATED) $(ESX_GENERATED_STAMP) diff --git a/src/hyperv/Makefile.inc.am b/src/hyperv/Makefile.inc.am index 51be8bcacc..6728b39c90 100644 --- a/src/hyperv/Makefile.inc.am +++ b/src/hyperv/Makefile.inc.am @@ -41,7 +41,7 @@ $(HYPERV_DRIVER_GENERATED): $(HYPERV_GENERATED_STAMP) $(HYPERV_GENERATED_STAMP): $(srcdir)/hyperv/hyperv_wmi_generator.input \ $(srcdir)/hyperv/hyperv_wmi_generator.py - $(AM_V_GEN)srcdir=$(srcdir) $(PYTHON) \ + $(AM_V_GEN)srcdir=$(srcdir) $(RUNUTF8) $(PYTHON) \ $(srcdir)/hyperv/hyperv_wmi_generator.py \ && touch $@ diff --git a/src/util/Makefile.inc.am b/src/util/Makefile.inc.am index 46866cf213..adb5e6d727 100644 --- a/src/util/Makefile.inc.am +++ b/src/util/Makefile.inc.am @@ -300,7 +300,7 @@ util/virkeycodetable_%.h: $(srcdir)/keycodemapdb/data/keymaps.csv \ $(AM_V_GEN)export NAME=`echo $@ | sed -e 's,util/virkeycodetable_,,' \ -e 's,\.h,,'` && \ $(MKDIR_P) util/ && \ - $(PYTHON) $(srcdir)/keycodemapdb/tools/keymap-gen \ + $(RUNUTF8) $(PYTHON) $(srcdir)/keycodemapdb/tools/keymap-gen \ --lang stdc --varname virKeyCodeTable_$$NAME code-table \ $(srcdir)/keycodemapdb/data/keymaps.csv $$NAME > $@-tmp && \ mv $@-tmp $@ || rm -f $@-tmp @@ -310,7 +310,7 @@ util/virkeynametable_%.h: $(srcdir)/keycodemapdb/data/keymaps.csv \ $(AM_V_GEN)export NAME=`echo $@ | sed -e 's,util/virkeynametable_,,' \ -e 's,\.h,,'` && \ $(MKDIR_P) util/ && \ - $(PYTHON) $(srcdir)/keycodemapdb/tools/keymap-gen \ + $(RUNUTF8) $(PYTHON) $(srcdir)/keycodemapdb/tools/keymap-gen \ --lang stdc --varname virKeyNameTable_$$NAME name-table \ $(srcdir)/keycodemapdb/data/keymaps.csv $$NAME > $@-tmp && \ mv $@-tmp $@ || rm -f $@-tmp @@ -320,7 +320,7 @@ util/virkeycode-%.pod: $(srcdir)/keycodemapdb/data/keymaps.csv \ $(AM_V_GEN)export NAME=`echo $@ | sed -e 's,util/virkeycode-,,' \ -e 's,\.pod,,'` && \ $(MKDIR_P) util/ && \ - $(PYTHON) $(srcdir)/keycodemapdb/tools/keymap-gen \ + $(RUNUTF8) $(PYTHON) $(srcdir)/keycodemapdb/tools/keymap-gen \ --lang pod \ --varname "virkeycode-$$NAME - Key code values for $$NAME" \ code-docs \ @@ -332,7 +332,7 @@ util/virkeyname-%.pod: $(srcdir)/keycodemapdb/data/keymaps.csv \ $(AM_V_GEN)export NAME=`echo $@ | sed -e 's,util/virkeyname-,,' \ -e 's,\.pod,,'` && \ $(MKDIR_P) util/ && \ - $(PYTHON) $(srcdir)/keycodemapdb/tools/keymap-gen \ + $(RUNUTF8) $(PYTHON) $(srcdir)/keycodemapdb/tools/keymap-gen \ --lang pod \ --varname "virkeyname-$$NAME - Key name values for $$NAME" \ name-docs \ -- 2.21.0

On 9/11/19 11:23 AM, Daniel P. Berrangé wrote:
Python3 versions less than 3.7 have very unhelpful handling of the C locale where they assume data is 7-bit only. This violates POSIX which requires the C locale to be 8-bit clean. Python3 >= 3.7 now assumes that the C locale is always UTF-8.
Being UTF-8 vs. being 8-bit clean are not necessarily synonymous, but the difference shouldn't matter when we only use UTF-8 encoding in our script source code and inputs.
Set env variables to force LC_CTYPE to en_US.UTF-8 so that we get UTF-8 handling on all python versions. Note we do not use C.UTF-8 since not all C libraries support that.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- Makefile.am | 2 +- configure.ac | 8 ++++++++ docs/Makefile.am | 3 ++- src/esx/Makefile.inc.am | 2 +- src/hyperv/Makefile.inc.am | 2 +- src/util/Makefile.inc.am | 8 ++++---- 6 files changed, 17 insertions(+), 8 deletions(-)
Reviewed-by: Eric Blake <eblake@redhat.com>
+++ b/src/esx/Makefile.inc.am @@ -63,7 +63,7 @@ $(ESX_DRIVER_GENERATED): $(ESX_GENERATED_STAMP)
$(ESX_GENERATED_STAMP): $(srcdir)/esx/esx_vi_generator.input \ $(srcdir)/esx/esx_vi_generator.py - $(AM_V_GEN)srcdir=$(srcdir) $(PYTHON) $(srcdir)/esx/esx_vi_generator.py \ + $(AM_V_GEN)srcdir=$(srcdir) $(RUNUTF8) $(PYTHON) $(srcdir)/esx/esx_vi_generator.py \ && touch $@
Worth rewrapping long lines any differently, since your addition pushes this beyond 80 columns? -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org

On Wed, Sep 11, 2019 at 12:27:45PM -0500, Eric Blake wrote:
On 9/11/19 11:23 AM, Daniel P. Berrangé wrote:
Python3 versions less than 3.7 have very unhelpful handling of the C locale where they assume data is 7-bit only. This violates POSIX which requires the C locale to be 8-bit clean. Python3 >= 3.7 now assumes that the C locale is always UTF-8.
Being UTF-8 vs. being 8-bit clean are not necessarily synonymous, but the difference shouldn't matter when we only use UTF-8 encoding in our script source code and inputs.
Set env variables to force LC_CTYPE to en_US.UTF-8 so that we get UTF-8 handling on all python versions. Note we do not use C.UTF-8 since not all C libraries support that.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- Makefile.am | 2 +- configure.ac | 8 ++++++++ docs/Makefile.am | 3 ++- src/esx/Makefile.inc.am | 2 +- src/hyperv/Makefile.inc.am | 2 +- src/util/Makefile.inc.am | 8 ++++---- 6 files changed, 17 insertions(+), 8 deletions(-)
Reviewed-by: Eric Blake <eblake@redhat.com>
+++ b/src/esx/Makefile.inc.am @@ -63,7 +63,7 @@ $(ESX_DRIVER_GENERATED): $(ESX_GENERATED_STAMP)
$(ESX_GENERATED_STAMP): $(srcdir)/esx/esx_vi_generator.input \ $(srcdir)/esx/esx_vi_generator.py - $(AM_V_GEN)srcdir=$(srcdir) $(PYTHON) $(srcdir)/esx/esx_vi_generator.py \ + $(AM_V_GEN)srcdir=$(srcdir) $(RUNUTF8) $(PYTHON) $(srcdir)/esx/esx_vi_generator.py \ && touch $@
Worth rewrapping long lines any differently, since your addition pushes this beyond 80 columns?
Ohh, bug-tastic. make syntax-check validates long lines, but we filter for Makefile.am, so we've been silently missing out Makefile.in.am files. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|

As part of an goal to eliminate Perl from libvirt build tools, rewrite the augeas-gentest.pl tool in Python. This was a straight conversion, manually going line-by-line to change the syntax from Perl to Python. Thus the overall structure of the file and approach is the same. The use of $(AUG_GENTEST) as a dependancy in the makefiles needed to be fixed, because this was assumed to be the filename of the script, but is in fact a full shell command line. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- Makefile.am | 2 +- build-aux/augeas-gentest.pl | 60 --------------------------- build-aux/augeas-gentest.py | 72 +++++++++++++++++++++++++++++++++ src/Makefile.am | 3 +- src/bhyve/Makefile.inc.am | 4 +- src/interface/Makefile.inc.am | 2 +- src/libxl/Makefile.inc.am | 4 +- src/locking/Makefile.inc.am | 6 +-- src/logging/Makefile.inc.am | 2 +- src/lxc/Makefile.inc.am | 4 +- src/network/Makefile.inc.am | 2 +- src/node_device/Makefile.inc.am | 2 +- src/nwfilter/Makefile.inc.am | 2 +- src/qemu/Makefile.inc.am | 4 +- src/remote/Makefile.inc.am | 4 +- src/secret/Makefile.inc.am | 2 +- src/storage/Makefile.inc.am | 2 +- src/vbox/Makefile.inc.am | 2 +- src/vz/Makefile.inc.am | 2 +- 19 files changed, 97 insertions(+), 84 deletions(-) delete mode 100755 build-aux/augeas-gentest.pl create mode 100755 build-aux/augeas-gentest.py diff --git a/Makefile.am b/Makefile.am index 711f365504..17448a914e 100644 --- a/Makefile.am +++ b/Makefile.am @@ -41,7 +41,7 @@ EXTRA_DIST = \ run.in \ README.md \ AUTHORS.in \ - build-aux/augeas-gentest.pl \ + build-aux/augeas-gentest.py \ build-aux/check-spacing.pl \ build-aux/gitlog-to-changelog \ build-aux/header-ifdef.pl \ diff --git a/build-aux/augeas-gentest.pl b/build-aux/augeas-gentest.pl deleted file mode 100755 index 65834b533b..0000000000 --- a/build-aux/augeas-gentest.pl +++ /dev/null @@ -1,60 +0,0 @@ -#!/usr/bin/env perl -# -# augeas-gentest.pl: Generate an augeas test file, from an -# example config file + test file template -# -# This library is free software; you can redistribute it and/or -# modify it under the terms of the GNU Lesser General Public -# License as published by the Free Software Foundation; either -# version 2.1 of the License, or (at your option) any later version. -# -# This library is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# Lesser General Public License for more details. -# -# You should have received a copy of the GNU Lesser General Public -# License along with this library. If not, see -# <http://www.gnu.org/licenses/>. - -use strict; -use warnings; - -die "syntax: $0 CONFIG TEMPLATE\n" unless @ARGV == 2; - -my $config = shift @ARGV; -my $template = shift @ARGV; - -open CONFIG, "<", $config or die "cannot read $config: $!"; -open TEMPLATE, "<", $template or die "cannot read $template: $!"; - -my $group = 0; -while (<TEMPLATE>) { - if (/\@CONFIG\@/) { - my $group = 0; - print " let conf = \""; - while (<CONFIG>) { - if (/^#\w/) { - s/^#//; - s/\"/\\\"/g; - print $_; - $group = /\[\s$/; - } elsif ($group) { - s/\"/\\\"/g; - if (/#\s*\]/) { - $group = 0; - } - if (/^#/) { - s/^#//; - print $_; - } - } - } - print "\"\n"; - } else { - print $_; - } -} - -close TEMPLATE; -close CONFIG; diff --git a/build-aux/augeas-gentest.py b/build-aux/augeas-gentest.py new file mode 100755 index 0000000000..b7f343ef42 --- /dev/null +++ b/build-aux/augeas-gentest.py @@ -0,0 +1,72 @@ +#!/usr/bin/env python +# +# Copyright (C) 2012-2019 Red Hat, Inc. +# +# augeas-gentest.py: Generate an augeas test file, from an +# example config file + test file template +# +# This library is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 2.1 of the License, or (at your option) any later version. +# +# This library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this library. If not, see +# <http://www.gnu.org/licenses/>. + +from __future__ import print_function + +import re +import sys + +if len(sys.argv) != 3: + print("syntax: %s CONFIG TEMPLATE" % sys.argv[0], file=sys.stderr) + sys.exit(1) + +config = sys.argv[1] +template = sys.argv[2] + +def expand_config(config): + with open(config) as fh: + optprog = re.compile(r'''^#\w.*''') + groupstartprog = re.compile(r'''.*\[\s$''') + groupendprog = re.compile(r'''#\s*\].*$''') + + group = False + for line in fh: + if optprog.match(line) is not None: + line = line[1:] + line = line.replace('"', '\\"') + print(line, end='') + if groupstartprog.match(line): + group = True + elif group: + line = line.replace('"', '\\"') + + if groupendprog.match(line): + group = False + + if line[0] == '#': + line = line[1:] + print(line, end='') + + +def expand_template(template, config): + with open(template) as fh: + group = 0 + + markerprog = re.compile(r'''\s*@CONFIG@\s*''') + for line in fh: + if markerprog.match(line) is not None: + print(' let conf = "', end='') + expand_config(config) + print('"') + else: + print(line, end='') + +expand_template(template, config) diff --git a/src/Makefile.am b/src/Makefile.am index f5093b9c90..7a4de2aa53 100644 --- a/src/Makefile.am +++ b/src/Makefile.am @@ -422,7 +422,8 @@ check-augeas: $(augeas_DATA) $(augeastest_DATA) fi .PHONY: check-augeas -AUG_GENTEST = $(PERL) $(top_srcdir)/build-aux/augeas-gentest.pl +AUG_GENTEST_SCRIPT = $(top_srcdir)/build-aux/augeas-gentest.py +AUG_GENTEST = $(RUNUTF8) $(PYTHON) $(AUG_GENTEST_SCRIPT) # diff --git a/src/bhyve/Makefile.inc.am b/src/bhyve/Makefile.inc.am index 195069872a..7e49a8bb5c 100644 --- a/src/bhyve/Makefile.inc.am +++ b/src/bhyve/Makefile.inc.am @@ -77,7 +77,7 @@ bhyve/virtbhyved.aug: remote/libvirtd.aug.in $< > $@ bhyve/test_virtbhyved.aug: remote/test_libvirtd.aug.in \ - bhyve/virtbhyved.conf $(AUG_GENTEST) + bhyve/virtbhyved.conf $(AUG_GENTEST_SCRIPT) $(AM_V_GEN)$(AUG_GENTEST) bhyve/virtbhyved.conf \ $(srcdir)/remote/test_libvirtd.aug.in | \ $(SED) \ @@ -91,7 +91,7 @@ augeas_DATA += bhyve/libvirtd_bhyve.aug augeastest_DATA += bhyve/test_libvirtd_bhyve.aug bhyve/test_libvirtd_bhyve.aug: bhyve/test_libvirtd_bhyve.aug.in \ - $(srcdir)/bhyve/bhyve.conf $(AUG_GENTEST) + $(srcdir)/bhyve/bhyve.conf $(AUG_GENTEST_SCRIPT) $(AM_V_GEN)$(AUG_GENTEST) $(srcdir)/bhyve/bhyve.conf $< > $@ endif WITH_BHYVE diff --git a/src/interface/Makefile.inc.am b/src/interface/Makefile.inc.am index a88df0bd11..b042da4c57 100644 --- a/src/interface/Makefile.inc.am +++ b/src/interface/Makefile.inc.am @@ -95,7 +95,7 @@ interface/virtinterfaced.aug: remote/libvirtd.aug.in $< > $@ interface/test_virtinterfaced.aug: remote/test_libvirtd.aug.in \ - interface/virtinterfaced.conf $(AUG_GENTEST) + interface/virtinterfaced.conf $(AUG_GENTEST_SCRIPT) $(AM_V_GEN)$(AUG_GENTEST) interface/virtinterfaced.conf \ $(srcdir)/remote/test_libvirtd.aug.in | \ $(SED) \ diff --git a/src/libxl/Makefile.inc.am b/src/libxl/Makefile.inc.am index a2151162e6..617b2a9019 100644 --- a/src/libxl/Makefile.inc.am +++ b/src/libxl/Makefile.inc.am @@ -108,7 +108,7 @@ libxl/virtxend.aug: remote/libvirtd.aug.in $< > $@ libxl/test_virtxend.aug: remote/test_libvirtd.aug.in \ - libxl/virtxend.conf $(AUG_GENTEST) + libxl/virtxend.conf $(AUG_GENTEST_SCRIPT) $(AM_V_GEN)$(AUG_GENTEST) libxl/virtxend.conf \ $(srcdir)/remote/test_libvirtd.aug.in | \ $(SED) \ @@ -122,7 +122,7 @@ augeas_DATA += libxl/libvirtd_libxl.aug augeastest_DATA += libxl/test_libvirtd_libxl.aug libxl/test_libvirtd_libxl.aug: libxl/test_libvirtd_libxl.aug.in \ - $(srcdir)/libxl/libxl.conf $(AUG_GENTEST) + $(srcdir)/libxl/libxl.conf $(AUG_GENTEST_SCRIPT) $(AM_V_GEN)$(AUG_GENTEST) $(srcdir)/libxl/libxl.conf $< > $@ INSTALL_DATA_DIRS += libxl diff --git a/src/locking/Makefile.inc.am b/src/locking/Makefile.inc.am index fae92a6e45..df0c51aaf6 100644 --- a/src/locking/Makefile.inc.am +++ b/src/locking/Makefile.inc.am @@ -219,7 +219,7 @@ endif WITH_SANLOCK if WITH_SANLOCK if WITH_QEMU locking/test_libvirt_sanlock.aug: locking/test_libvirt_sanlock.aug.in \ - locking/qemu-sanlock.conf $(AUG_GENTEST) + locking/qemu-sanlock.conf $(AUG_GENTEST_SCRIPT) $(AM_V_GEN)$(AUG_GENTEST) locking/qemu-sanlock.conf $< > $@ endif WITH_QEMU @@ -227,12 +227,12 @@ endif WITH_SANLOCK if WITH_QEMU locking/test_libvirt_lockd.aug: locking/test_libvirt_lockd.aug.in \ - locking/qemu-lockd.conf $(AUG_GENTEST) + locking/qemu-lockd.conf $(AUG_GENTEST_SCRIPT) $(AM_V_GEN)$(AUG_GENTEST) locking/qemu-lockd.conf $< > $@ endif WITH_QEMU locking/test_virtlockd.aug: locking/test_virtlockd.aug.in \ - locking/virtlockd.conf $(AUG_GENTEST) + locking/virtlockd.conf $(AUG_GENTEST_SCRIPT) $(AM_V_GEN)$(AUG_GENTEST) $(srcdir)/locking/virtlockd.conf $< > $@ endif WITH_LIBVIRTD diff --git a/src/logging/Makefile.inc.am b/src/logging/Makefile.inc.am index 7e441dbffb..ce777e278a 100644 --- a/src/logging/Makefile.inc.am +++ b/src/logging/Makefile.inc.am @@ -101,7 +101,7 @@ augeas_DATA += logging/virtlogd.aug augeastest_DATA += logging/test_virtlogd.aug logging/test_virtlogd.aug: logging/test_virtlogd.aug.in \ - logging/virtlogd.conf $(AUG_GENTEST) + logging/virtlogd.conf $(AUG_GENTEST_SCRIPT) $(AM_V_GEN)$(AUG_GENTEST) $(srcdir)/logging/virtlogd.conf $< > $@ endif WITH_LIBVIRTD diff --git a/src/lxc/Makefile.inc.am b/src/lxc/Makefile.inc.am index b4d560702c..0e12800886 100644 --- a/src/lxc/Makefile.inc.am +++ b/src/lxc/Makefile.inc.am @@ -164,7 +164,7 @@ lxc/virtlxcd.aug: remote/libvirtd.aug.in $< > $@ lxc/test_virtlxcd.aug: remote/test_libvirtd.aug.in \ - lxc/virtlxcd.conf $(AUG_GENTEST) + lxc/virtlxcd.conf $(AUG_GENTEST_SCRIPT) $(AM_V_GEN)$(AUG_GENTEST) lxc/virtlxcd.conf \ $(srcdir)/remote/test_libvirtd.aug.in | \ $(SED) \ @@ -222,7 +222,7 @@ augeas_DATA += lxc/libvirtd_lxc.aug augeastest_DATA += lxc/test_libvirtd_lxc.aug lxc/test_libvirtd_lxc.aug: lxc/test_libvirtd_lxc.aug.in \ - $(srcdir)/lxc/lxc.conf $(AUG_GENTEST) + $(srcdir)/lxc/lxc.conf $(AUG_GENTEST_SCRIPT) $(AM_V_GEN)$(AUG_GENTEST) $(srcdir)/lxc/lxc.conf $< > $@ INSTALL_DATA_DIRS += lxc diff --git a/src/network/Makefile.inc.am b/src/network/Makefile.inc.am index 9f20dad7b3..31e4b1fbe4 100644 --- a/src/network/Makefile.inc.am +++ b/src/network/Makefile.inc.am @@ -102,7 +102,7 @@ network/virtnetworkd.aug: remote/libvirtd.aug.in $< > $@ network/test_virtnetworkd.aug: remote/test_libvirtd.aug.in \ - network/virtnetworkd.conf $(AUG_GENTEST) + network/virtnetworkd.conf $(AUG_GENTEST_SCRIPT) $(AM_V_GEN)$(AUG_GENTEST) network/virtnetworkd.conf \ $(srcdir)/remote/test_libvirtd.aug.in | \ $(SED) \ diff --git a/src/node_device/Makefile.inc.am b/src/node_device/Makefile.inc.am index 5a6525d843..dfe16d62d6 100644 --- a/src/node_device/Makefile.inc.am +++ b/src/node_device/Makefile.inc.am @@ -118,7 +118,7 @@ node_device/virtnodedevd.aug: remote/libvirtd.aug.in $< > $@ node_device/test_virtnodedevd.aug: remote/test_libvirtd.aug.in \ - node_device/virtnodedevd.conf $(AUG_GENTEST) + node_device/virtnodedevd.conf $(AUG_GENTEST_SCRIPT) $(AM_V_GEN)$(AUG_GENTEST) node_device/virtnodedevd.conf \ $(srcdir)/remote/test_libvirtd.aug.in | \ $(SED) \ diff --git a/src/nwfilter/Makefile.inc.am b/src/nwfilter/Makefile.inc.am index 7693634e29..476768ef11 100644 --- a/src/nwfilter/Makefile.inc.am +++ b/src/nwfilter/Makefile.inc.am @@ -103,7 +103,7 @@ nwfilter/virtnwfilterd.aug: remote/libvirtd.aug.in $< > $@ nwfilter/test_virtnwfilterd.aug: remote/test_libvirtd.aug.in \ - nwfilter/virtnwfilterd.conf $(AUG_GENTEST) + nwfilter/virtnwfilterd.conf $(AUG_GENTEST_SCRIPT) $(AM_V_GEN)$(AUG_GENTEST) nwfilter/virtnwfilterd.conf \ $(srcdir)/remote/test_libvirtd.aug.in | \ $(SED) \ diff --git a/src/qemu/Makefile.inc.am b/src/qemu/Makefile.inc.am index d16b315ebc..caa79cae6a 100644 --- a/src/qemu/Makefile.inc.am +++ b/src/qemu/Makefile.inc.am @@ -165,7 +165,7 @@ qemu/virtqemud.aug: remote/libvirtd.aug.in $< > $@ qemu/test_virtqemud.aug: remote/test_libvirtd.aug.in \ - qemu/virtqemud.conf $(AUG_GENTEST) + qemu/virtqemud.conf $(AUG_GENTEST_SCRIPT) $(AM_V_GEN)$(AUG_GENTEST) qemu/virtqemud.conf \ $(srcdir)/remote/test_libvirtd.aug.in | \ $(SED) \ @@ -180,7 +180,7 @@ augeas_DATA += qemu/libvirtd_qemu.aug augeastest_DATA += qemu/test_libvirtd_qemu.aug qemu/test_libvirtd_qemu.aug: qemu/test_libvirtd_qemu.aug.in \ - $(srcdir)/qemu/qemu.conf $(AUG_GENTEST) + $(srcdir)/qemu/qemu.conf $(AUG_GENTEST_SCRIPT) $(AM_V_GEN)$(AUG_GENTEST) $(srcdir)/qemu/qemu.conf $< > $@ INSTALL_DATA_DIRS += qemu diff --git a/src/remote/Makefile.inc.am b/src/remote/Makefile.inc.am index abf04d998a..0c5e2fd376 100644 --- a/src/remote/Makefile.inc.am +++ b/src/remote/Makefile.inc.am @@ -285,7 +285,7 @@ remote/virtproxyd.aug: remote/libvirtd.aug.in $< > $@ remote/test_libvirtd.aug: remote/test_libvirtd.aug.in \ - remote/libvirtd.conf $(AUG_GENTEST) + remote/libvirtd.conf $(AUG_GENTEST_SCRIPT) $(AM_V_GEN)$(AUG_GENTEST) remote/libvirtd.conf \ $(srcdir)/remote/test_libvirtd.aug.in | \ $(SED) \ @@ -298,7 +298,7 @@ remote/test_libvirtd.aug: remote/test_libvirtd.aug.in \ > $@ || rm -f $@ remote/test_virtproxyd.aug: remote/test_libvirtd.aug.in \ - remote/virtproxyd.conf $(AUG_GENTEST) + remote/virtproxyd.conf $(AUG_GENTEST_SCRIPT) $(AM_V_GEN)$(AUG_GENTEST) remote/virtproxyd.conf \ $(srcdir)/remote/test_libvirtd.aug.in | \ $(SED) \ diff --git a/src/secret/Makefile.inc.am b/src/secret/Makefile.inc.am index 37f816406e..1e25993225 100644 --- a/src/secret/Makefile.inc.am +++ b/src/secret/Makefile.inc.am @@ -91,7 +91,7 @@ secret/virtsecretd.aug: remote/libvirtd.aug.in $< > $@ secret/test_virtsecretd.aug: remote/test_libvirtd.aug.in \ - secret/virtsecretd.conf $(AUG_GENTEST) + secret/virtsecretd.conf $(AUG_GENTEST_SCRIPT) $(AM_V_GEN)$(AUG_GENTEST) secret/virtsecretd.conf \ $(srcdir)/remote/test_libvirtd.aug.in | \ $(SED) \ diff --git a/src/storage/Makefile.inc.am b/src/storage/Makefile.inc.am index fd0d57d2b6..3542ee52e8 100644 --- a/src/storage/Makefile.inc.am +++ b/src/storage/Makefile.inc.am @@ -195,7 +195,7 @@ storage/virtstoraged.aug: remote/libvirtd.aug.in $< > $@ storage/test_virtstoraged.aug: remote/test_libvirtd.aug.in \ - storage/virtstoraged.conf $(AUG_GENTEST) + storage/virtstoraged.conf $(AUG_GENTEST_SCRIPT) $(AM_V_GEN)$(AUG_GENTEST) storage/virtstoraged.conf \ $(srcdir)/remote/test_libvirtd.aug.in | \ $(SED) \ diff --git a/src/vbox/Makefile.inc.am b/src/vbox/Makefile.inc.am index 178c360b99..1f2bcb5488 100644 --- a/src/vbox/Makefile.inc.am +++ b/src/vbox/Makefile.inc.am @@ -117,7 +117,7 @@ vbox/virtvboxd.aug: remote/libvirtd.aug.in $< > $@ vbox/test_virtvboxd.aug: remote/test_libvirtd.aug.in \ - vbox/virtvboxd.conf $(AUG_GENTEST) + vbox/virtvboxd.conf $(AUG_GENTEST_SCRIPT) $(AM_V_GEN)$(AUG_GENTEST) vbox/virtvboxd.conf \ $(srcdir)/remote/test_libvirtd.aug.in | \ $(SED) \ diff --git a/src/vz/Makefile.inc.am b/src/vz/Makefile.inc.am index f56fceb8f7..fa7b8b79dd 100644 --- a/src/vz/Makefile.inc.am +++ b/src/vz/Makefile.inc.am @@ -91,7 +91,7 @@ vz/virtvzd.aug: remote/libvirtd.aug.in $< > $@ vz/test_virtvzd.aug: remote/test_libvirtd.aug.in \ - vz/virtvzd.conf $(AUG_GENTEST) + vz/virtvzd.conf $(AUG_GENTEST_SCRIPT) $(AM_V_GEN)$(AUG_GENTEST) vz/virtvzd.conf \ $(srcdir)/remote/test_libvirtd.aug.in | \ $(SED) \ -- 2.21.0

As part of an goal to eliminate Perl from libvirt build tools, rewrite the minimize-po.pl tool in Python. This was a straight conversion, manually going line-by-line to change the syntax from Perl to Python. Thus the overall structure of the file and approach is the same. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- Makefile.am | 2 +- build-aux/minimize-po.pl | 37 ------------------------- build-aux/minimize-po.py | 60 ++++++++++++++++++++++++++++++++++++++++ po/Makefile.am | 2 +- 4 files changed, 62 insertions(+), 39 deletions(-) delete mode 100755 build-aux/minimize-po.pl create mode 100755 build-aux/minimize-po.py diff --git a/Makefile.am b/Makefile.am index 17448a914e..8f688d40d0 100644 --- a/Makefile.am +++ b/Makefile.am @@ -45,7 +45,7 @@ EXTRA_DIST = \ build-aux/check-spacing.pl \ build-aux/gitlog-to-changelog \ build-aux/header-ifdef.pl \ - build-aux/minimize-po.pl \ + build-aux/minimize-po.py \ build-aux/mock-noinline.pl \ build-aux/prohibit-duplicate-header.pl \ build-aux/useless-if-before-free \ diff --git a/build-aux/minimize-po.pl b/build-aux/minimize-po.pl deleted file mode 100755 index 497533a836..0000000000 --- a/build-aux/minimize-po.pl +++ /dev/null @@ -1,37 +0,0 @@ -#!/usr/bin/perl - -my @block; -my $msgstr = 0; -my $empty = 0; -my $unused = 0; -my $fuzzy = 0; -while (<>) { - if (/^$/) { - if (!$empty && !$unused && !$fuzzy) { - print @block; - } - @block = (); - $msgstr = 0; - $fuzzy = 0; - push @block, $_; - } else { - if (/^msgstr/) { - $msgstr = 1; - $empty = 1; - } - if (/^#.*fuzzy/) { - $fuzzy = 1; - } - if (/^#~ msgstr/) { - $unused = 1; - } - if ($msgstr && /".+"/) { - $empty = 0; - } - push @block, $_; - } -} - -if (@block && !$empty && !$unused) { - print @block; -} diff --git a/build-aux/minimize-po.py b/build-aux/minimize-po.py new file mode 100755 index 0000000000..5046bacede --- /dev/null +++ b/build-aux/minimize-po.py @@ -0,0 +1,60 @@ +#!/usr/bin/env python +# +# Copyright (C) 2018-2019 Red Hat, Inc. +# +# This library is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 2.1 of the License, or (at your option) any later version. +# +# This library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this library. If not, see +# <http://www.gnu.org/licenses/>. + +from __future__ import print_function + +import re +import sys + +block = [] +msgstr = False +empty = False +unused = False +fuzzy = False + +strprog = re.compile(r'''.*".+".*''') + +for line in sys.stdin: + line = line.rstrip("\n") + + if line == "": + if not empty and not unused and not fuzzy: + for b in block: + print(b) + + block = [] + msgstr = False + fuzzy = False + block.append(line) + else: + if line.startswith("msgstr"): + msgstr = True + empty = True + + if line[0] == '#' and line.find("fuzzy") != -1: + fuzzy = True + if line.startswith("#~ msgstr"): + unused = True + if msgstr and strprog.match(line): + empty = False + + block.append(line) + +if not empty and not unused and not fuzzy: + for b in block: + print(b) diff --git a/po/Makefile.am b/po/Makefile.am index da117edbd5..90025b15dd 100644 --- a/po/Makefile.am +++ b/po/Makefile.am @@ -58,7 +58,7 @@ update-mini-po: $(POTFILE) $(MSGMERGE) --no-location --no-fuzzy-matching --sort-output \ $$lang.po $(POTFILE) | \ $(SED) $(SED_PO_FIXUP_ARGS) | \ - $(PERL) $(top_srcdir)/build-aux/minimize-po.pl > \ + $(RUNUTF8) $(PYTHON) $(top_srcdir)/build-aux/minimize-po.py > \ $(srcdir)/$$lang.mini.po ; \ done -- 2.21.0

As part of an goal to eliminate Perl from libvirt build tools, rewrite the prohibit-duplicate-header.pl tool in Python. This was a straight conversion, manually going line-by-line to change the syntax from Perl to Python. Thus the overall structure of the file and approach is the same. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- Makefile.am | 2 +- build-aux/prohibit-duplicate-header.pl | 26 ------------- build-aux/prohibit-duplicate-header.py | 54 ++++++++++++++++++++++++++ cfg.mk | 4 +- 4 files changed, 57 insertions(+), 29 deletions(-) delete mode 100644 build-aux/prohibit-duplicate-header.pl create mode 100644 build-aux/prohibit-duplicate-header.py diff --git a/Makefile.am b/Makefile.am index 8f688d40d0..a7d8da7146 100644 --- a/Makefile.am +++ b/Makefile.am @@ -47,7 +47,7 @@ EXTRA_DIST = \ build-aux/header-ifdef.pl \ build-aux/minimize-po.py \ build-aux/mock-noinline.pl \ - build-aux/prohibit-duplicate-header.pl \ + build-aux/prohibit-duplicate-header.py \ build-aux/useless-if-before-free \ build-aux/vc-list-files \ ci/Makefile \ diff --git a/build-aux/prohibit-duplicate-header.pl b/build-aux/prohibit-duplicate-header.pl deleted file mode 100644 index 4a2ea65665..0000000000 --- a/build-aux/prohibit-duplicate-header.pl +++ /dev/null @@ -1,26 +0,0 @@ -#!/usr/bin/env perl - -use strict; - -my $file = " "; -my $ret = 0; -my %includes = ( ); -my $lineno = 0; - -while (<>) { - if (not $file eq $ARGV) { - %includes = ( ); - $file = $ARGV; - $lineno = 0; - } - $lineno++; - if (/^# *include *[<"]([^>"]*\.h)[">]/) { - $includes{$1}++; - if ($includes{$1} == 2) { - $ret = 1; - print STDERR "$ARGV:$lineno: $_"; - print STDERR "Do not include a header more than once per file\n"; - } - } -} -exit $ret; diff --git a/build-aux/prohibit-duplicate-header.py b/build-aux/prohibit-duplicate-header.py new file mode 100644 index 0000000000..0ef86e8453 --- /dev/null +++ b/build-aux/prohibit-duplicate-header.py @@ -0,0 +1,54 @@ +#!/usr/bin/env python +# +# Copyright (C) 2016-2019 Red Hat, Inc. +# +# This library is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 2.1 of the License, or (at your option) any later version. +# +# This library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this library. If not, see +# <http://www.gnu.org/licenses/>. + +from __future__ import print_function + +import re +import sys + +def check_file(filename): + includes = {} + lineno = 0 + errs = False + with open(filename, "r") as fh: + headerprog = re.compile(r'''^# *include *[<"]([^>"]*\.h)[">]\s*$''') + for line in fh: + lineno = lineno + 1 + + headermatch = headerprog.match(line) + if headermatch is not None: + inc = headermatch.group(1) + + if inc in includes: + print("%s:%d: %s" % (filename, lineno, inc), file=sys.stderr) + errs = True + else: + includes[inc] = True + + return errs + +ret = 0 + +for filename in sys.argv[1:]: + if check_file(filename): + ret = 1 + +if ret == 1: + print("Do not include a header more than once per file", file=sys.stderr) + +sys.exit(ret) diff --git a/cfg.mk b/cfg.mk index d09ab507e0..933857bc2f 100644 --- a/cfg.mk +++ b/cfg.mk @@ -1147,8 +1147,8 @@ endif # Don't include duplicate header in the source (either *.c or *.h) prohibit-duplicate-header: - $(AM_V_GEN)$(VC_LIST_EXCEPT) | $(GREP) '\.[chx]$$' | xargs \ - $(PERL) -W $(top_srcdir)/build-aux/prohibit-duplicate-header.pl + $(AM_V_GEN)$(VC_LIST_EXCEPT) | $(GREP) '\.[chx]$$' | $(RUNUTF8) xargs \ + $(PYTHON) $(top_srcdir)/build-aux/prohibit-duplicate-header.py spacing-check: $(AM_V_GEN)$(VC_LIST) | $(GREP) '\.c$$' | xargs \ -- 2.21.0

As part of an goal to eliminate Perl from libvirt build tools, rewrite the check-spacing.pl tool in Python. This was a straight conversion, manually going line-by-line to change the syntax from Perl to Python. Thus the overall structure of the file and approach is the same. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- Makefile.am | 2 +- build-aux/check-spacing.pl | 198 ----------------------------------- build-aux/check-spacing.py | 204 +++++++++++++++++++++++++++++++++++++ cfg.mk | 4 +- 4 files changed, 207 insertions(+), 201 deletions(-) delete mode 100755 build-aux/check-spacing.pl create mode 100755 build-aux/check-spacing.py diff --git a/Makefile.am b/Makefile.am index a7d8da7146..03bf1beb78 100644 --- a/Makefile.am +++ b/Makefile.am @@ -42,7 +42,7 @@ EXTRA_DIST = \ README.md \ AUTHORS.in \ build-aux/augeas-gentest.py \ - build-aux/check-spacing.pl \ + build-aux/check-spacing.py \ build-aux/gitlog-to-changelog \ build-aux/header-ifdef.pl \ build-aux/minimize-po.py \ diff --git a/build-aux/check-spacing.pl b/build-aux/check-spacing.pl deleted file mode 100755 index 33377f3dd3..0000000000 --- a/build-aux/check-spacing.pl +++ /dev/null @@ -1,198 +0,0 @@ -#!/usr/bin/env perl -# -# check-spacing.pl: Report any usage of 'function (..args..)' -# Also check for other syntax issues, such as correct use of ';' -# -# This library is free software; you can redistribute it and/or -# modify it under the terms of the GNU Lesser General Public -# License as published by the Free Software Foundation; either -# version 2.1 of the License, or (at your option) any later version. -# -# This library is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# Lesser General Public License for more details. -# -# You should have received a copy of the GNU Lesser General Public -# License along with this library. If not, see -# <http://www.gnu.org/licenses/>. - -use strict; -use warnings; - -my $ret = 0; -my $incomment = 0; - -foreach my $file (@ARGV) { - # Per-file variables for multiline Curly Bracket (cb_) check - my $cb_linenum = 0; - my $cb_code = ""; - my $cb_scolon = 0; - - open FILE, $file; - - while (defined (my $line = <FILE>)) { - my $data = $line; - # For temporary modifications - my $tmpdata; - - # Kill any quoted , ; = or " - $data =~ s/'[";,=]'/'X'/g; - - # Kill any quoted strings - $data =~ s,"(?:[^\\\"]|\\.)*","XXX",g; - - next if $data =~ /^#/; - - # Kill contents of multi-line comments - # and detect end of multi-line comments - if ($incomment) { - if ($data =~ m,\*/,) { - $incomment = 0; - $data =~ s,^.*\*/,*/,; - } else { - $data = ""; - } - } - - # Kill single line comments, and detect - # start of multi-line comments - if ($data =~ m,/\*.*\*/,) { - $data =~ s,/\*.*\*/,/* */,; - } elsif ($data =~ m,/\*,) { - $incomment = 1; - $data =~ s,/\*.*,/*,; - } - - # We need to match things like - # - # int foo (int bar, bool wizz); - # foo (bar, wizz); - # - # but not match things like: - # - # typedef int (*foo)(bar wizz) - # - # we can't do this (efficiently) without - # missing things like - # - # foo (*bar, wizz); - # - # We also don't want to spoil the $data so it can be used - # later on. - $tmpdata = $data; - while ($tmpdata =~ /(\w+)\s\((?!\*)/) { - my $kw = $1; - - # Allow space after keywords only - if ($kw =~ /^(?:if|for|while|switch|return)$/) { - $tmpdata =~ s/(?:$kw\s\()/XXX(/; - } else { - print "Whitespace after non-keyword:\n"; - print "$file:$.: $line"; - $ret = 1; - last; - } - } - - # Require whitespace immediately after keywords - if ($data =~ /\b(?:if|for|while|switch|return)\(/) { - print "No whitespace after keyword:\n"; - print "$file:$.: $line"; - $ret = 1; - } - - # Forbid whitespace between )( of a function typedef - if ($data =~ /\(\*\w+\)\s+\(/) { - print "Whitespace between ')' and '(':\n"; - print "$file:$.: $line"; - $ret = 1; - } - - # Forbid whitespace following ( or prior to ) - # but allow whitespace before ) on a single line - # (optionally followed by a semicolon) - if (($data =~ /\s\)/ && not $data =~ /^\s+\);?$/) || - $data =~ /\((?!$)\s/) { - print "Whitespace after '(' or before ')':\n"; - print "$file:$.: $line"; - $ret = 1; - } - - # Forbid whitespace before ";" or ",". Things like below are allowed: - # - # 1) The expression is empty for "for" loop. E.g. - # for (i = 0; ; i++) - # - # 2) An empty statement. E.g. - # while (write(statuswrite, &status, 1) == -1 && - # errno == EINTR) - # ; - # - if ($data =~ /\s[;,]/) { - unless ($data =~ /\S; ; / || - $data =~ /^\s+;/) { - print "Whitespace before semicolon or comma:\n"; - print "$file:$.: $line"; - $ret = 1; - } - } - - # Require EOL, macro line continuation, or whitespace after ";". - # Allow "for (;;)" as an exception. - if ($data =~ /;[^ \\\n;)]/) { - print "Invalid character after semicolon:\n"; - print "$file:$.: $line"; - $ret = 1; - } - - # Require EOL, space, or enum/struct end after comma. - if ($data =~ /,[^ \\\n)}]/) { - print "Invalid character after comma:\n"; - print "$file:$.: $line"; - $ret = 1; - } - - # Require spaces around assignment '=', compounds and '==' - if ($data =~ /[^ ]\b[!<>&|\-+*\/%\^=]?=/ || - $data =~ /=[^= \\\n]/) { - print "Spacing around '=' or '==':\n"; - print "$file:$.: $line"; - $ret = 1; - } - - # One line conditional statements with one line bodies should - # not use curly brackets. - if ($data =~ /^\s*(if|while|for)\b.*\{$/) { - $cb_linenum = $.; - $cb_code = $line; - $cb_scolon = 0; - } - - # We need to check for exactly one semicolon inside the body, - # because empty statements (e.g. with comment only) are - # allowed - if ($cb_linenum == $. - 1 && $data =~ /^[^;]*;[^;]*$/) { - $cb_code .= $line; - $cb_scolon = 1; - } - - if ($data =~ /^\s*}\s*$/ && - $cb_linenum == $. - 2 && - $cb_scolon) { - - print "Curly brackets around single-line body:\n"; - print "$file:$cb_linenum-$.:\n$cb_code$line"; - $ret = 1; - - # There _should_ be no need to reset the values; but to - # keep my inner peace... - $cb_linenum = 0; - $cb_scolon = 0; - $cb_code = ""; - } - } - close FILE; -} - -exit $ret; diff --git a/build-aux/check-spacing.py b/build-aux/check-spacing.py new file mode 100755 index 0000000000..ec565585ad --- /dev/null +++ b/build-aux/check-spacing.py @@ -0,0 +1,204 @@ +#!/usr/bin/env python +# +# Copyright (C) 2012-2019 Red Hat, Inc. +# +# check-spacing.pl: Report any usage of 'function (..args..)' +# Also check for other syntax issues, such as correct use of ';' +# +# This library is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 2.1 of the License, or (at your option) any later version. +# +# This library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this library. If not, see +# <http://www.gnu.org/licenses/>. + +from __future__ import print_function + +import re +import sys + +def check_whitespace(filename): + errs = False + with open(filename, 'r') as fh: + quotedmetaprog = re.compile(r"""'[";,=]'""") + quotedstringprog = re.compile(r'''"(?:[^\\\"]|\\.)*"''') + commentstartprog = re.compile(r'''^(.*)/\*.*$''') + commentendprog = re.compile(r'''^.*\*/(.*)$''') + commentprog = re.compile(r'''^(.*)/\*.*\*/(.*)''') + funcprog = re.compile(r'''(\w+)\s\((?!\*)''') + keywordprog = re.compile(r'''^.*\b(?:if|for|while|switch|return)\(.*$''') + functypedefprog = re.compile(r'''^.*\(\*\w+\)\s+\(.*$''') + whitespaceprog1 = re.compile(r'''^.*\s\).*$''') + whitespaceprog2 = re.compile(r'''^\s+\);?$''') + whitespaceprog3 = re.compile(r'''^.*\((?!$)\s.*''') + commasemiprog1 = re.compile(r'''.*\s[;,].*''') + commasemiprog2 = re.compile(r'''.*\S; ; .*''') + commasemiprog3 = re.compile(r'''^\s+;''') + semicolonprog = re.compile(r'''.*;[^ \\\n;)].*''') + commaprog = re.compile(r'''.*,[^ \\\n)}].*''') + assignprog1 = re.compile(r'''[^ ]\b[!<>&|\-+*\/%\^=]?=''') + assignprog2 = re.compile(r'''=[^= \\\n]''') + condstartprog = re.compile(r'''^\s*(if|while|for)\b.*\{$''') + statementprog = re.compile(r'''^[^;]*;[^;]*$''') + condendprog = re.compile(r'''^\s*}\s*$''') + + incomment = False + # Per-file variables for multiline Curly Bracket (cb_) check + cb_lineno = 0 + cb_code = "" + cb_scolon = False + + lineno = 0 + for line in fh: + lineno = lineno + 1 + data = line + # For temporary modifications + + # Kill any quoted , ; = or " + data = quotedmetaprog.sub("'X'", data) + + # Kill any quoted strings + data = quotedstringprog.sub('"XXX"', data) + + if data[0] == '#': + continue + + # Kill contents of multi-line comments + # and detect end of multi-line comments + if incomment: + if commentendprog.match(data): + data = commentendprog.sub('*/\2', data) + incomment = False + else: + data = "" + + # Kill single line comments, and detect + # start of multi-line comments + if commentprog.match(data): + data = commentprog.sub(r'''\1/* */\2''', data) + elif commentstartprog.match(data): + data = commentstartprog.sub(r'''\1/*''', data) + incomment = True + + # We need to match things like + # + # int foo (int bar, bool wizz); + # foo (bar, wizz); + # + # but not match things like: + # + # typedef int (*foo)(bar wizz) + # + # we can't do this (efficiently) without + # missing things like + # + # foo (*bar, wizz); + # + for match in funcprog.finditer(data): + kw = match.group(1) + + # Allow space after keywords only + if kw not in ["if", "for", "while", "switch", "return"]: + print("Whitespace after non-keyword:", file=sys.stderr) + print("%s:%d: %s" % (filename, lineno, line), file=sys.stderr) + errs = True + break + + # Require whitespace immediately after keywords + if keywordprog.match(data): + print("No whitespace after keyword:", file=sys.stderr) + print("%s:%d: %s" % (filename, lineno, line), file=sys.stderr) + errs = True + + # Forbid whitespace between )( of a function typedef + if functypedefprog.match(data): + print("Whitespace between ')' and '(':", file=sys.stderr) + print("%s:%d: %s" % (filename, lineno, line), file=sys.stderr) + errs = True + + # Forbid whitespace following ( or prior to ) + # but allow whitespace before ) on a single line + # (optionally followed by a semicolon) + if ((whitespaceprog1.match(data) and + not whitespaceprog2.match(data)) + or whitespaceprog3.match(data)): + print("Whitespace after '(' or before ')':", file=sys.stderr) + print("%s:%d: %s" % (filename, lineno, line), file=sys.stderr) + errs = True + + # Forbid whitespace before ";" or ",". Things like below are allowed: + # + # 1) The expression is empty for "for" loop. E.g. + # for (i = 0; ; i++) + # + # 2) An empty statement. E.g. + # while (write(statuswrite, &status, 1) == -1 && + # errno == EINTR) + # ; + # + if commasemiprog1.match(data) and not ( + commasemiprog2.match(data) or + commasemiprog3.match(data)): + print("Whitespace before semicolon or comma:", file=sys.stderr) + print("%s:%d: %s" % (filename, lineno, line), file=sys.stderr) + errs = True + + # Require EOL, macro line continuation, or whitespace after ";". + # Allow "for (;;)" as an exception. + if semicolonprog.match(data): + print("Invalid character after semicolon:", file=sys.stderr) + print("%s:%d: %s" % (filename, lineno, line), file=sys.stderr) + errs = True + + # Require EOL, space, or enum/struct end after comma. + if commaprog.match(data): + print("Invalid character after comma:", file=sys.stderr) + print("%s:%d: %s" % (filename, lineno, line), file=sys.stderr) + errs = True + + # Require spaces around assignment '=', compounds and '==' + if assignprog1.match(data) or assignprog2.match(data): + print("Spacing around '=' or '==':", file=sys.stderr) + print("%s:%d: %s" % (filename, lineno, line), file=sys.stderr) + errs = True + + # One line conditional statements with one line bodies should + # not use curly brackets. + if condstartprog.match(data): + cb_lineno = lineno + cb_code = line + cb_scolon = False + + # We need to check for exactly one semicolon inside the body, + # because empty statements (e.g. with comment only) are + # allowed + if (cb_lineno == lineno - 1) and statementprog.match(data): + cb_code = cb_code + line + cb_scolon = True + + if condendprog.match(data) and (cb_lineno == lineno - 2) and cb_scolon: + print("Curly brackets around single-line body:", file=sys.stderr) + print("%s:%d:\n%s%s"% (filename, cb_linenum - lineno, cb_code, line), file=sys.stderr) + errs = True + + # There _should_ be no need to reset the values; but to + # keep my inner peace... + cb_linenum = 0 + cb_scolon = False + cb_code = "" + + return errs + +ret = 0 +for filename in sys.argv[1:]: + if check_whitespace(filename): + ret = 1 + +sys.exit(ret) diff --git a/cfg.mk b/cfg.mk index 933857bc2f..4da3554b30 100644 --- a/cfg.mk +++ b/cfg.mk @@ -1151,8 +1151,8 @@ prohibit-duplicate-header: $(PYTHON) $(top_srcdir)/build-aux/prohibit-duplicate-header.py spacing-check: - $(AM_V_GEN)$(VC_LIST) | $(GREP) '\.c$$' | xargs \ - $(PERL) $(top_srcdir)/build-aux/check-spacing.pl || \ + $(AM_V_GEN)$(VC_LIST) | $(GREP) '\.c$$' | $(RUNUTF8) xargs \ + $(PYTHON) $(top_srcdir)/build-aux/check-spacing.py || \ { echo '$(ME): incorrect formatting' 1>&2; exit 1; } mock-noinline: -- 2.21.0

As part of an goal to eliminate Perl from libvirt build tools, rewrite the mock-noinline.pl tool in Python. This was a straight conversion, manually going line-by-line to change the syntax from Perl to Python. Thus the overall structure of the file and approach is the same. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- Makefile.am | 2 +- build-aux/mock-noinline.pl | 75 -------------------------------- build-aux/mock-noinline.py | 88 ++++++++++++++++++++++++++++++++++++++ cfg.mk | 4 +- 4 files changed, 91 insertions(+), 78 deletions(-) delete mode 100644 build-aux/mock-noinline.pl create mode 100644 build-aux/mock-noinline.py diff --git a/Makefile.am b/Makefile.am index 03bf1beb78..db77da890c 100644 --- a/Makefile.am +++ b/Makefile.am @@ -46,7 +46,7 @@ EXTRA_DIST = \ build-aux/gitlog-to-changelog \ build-aux/header-ifdef.pl \ build-aux/minimize-po.py \ - build-aux/mock-noinline.pl \ + build-aux/mock-noinline.py \ build-aux/prohibit-duplicate-header.py \ build-aux/useless-if-before-free \ build-aux/vc-list-files \ diff --git a/build-aux/mock-noinline.pl b/build-aux/mock-noinline.pl deleted file mode 100644 index 958e133885..0000000000 --- a/build-aux/mock-noinline.pl +++ /dev/null @@ -1,75 +0,0 @@ -#!/usr/bin/env perl - -my %noninlined; -my %mocked; - -# Functions in public header don't get the noinline annotation -# so whitelist them here -$noninlined{"virEventAddTimeout"} = 1; -# This one confuses the script as its defined in the mock file -# but is actually just a local helper -$noninlined{"virMockStatRedirect"} = 1; - -foreach my $arg (@ARGV) { - if ($arg =~ /\.h$/) { - #print "Scan header $arg\n"; - &scan_annotations($arg); - } elsif ($arg =~ /mock\.c$/) { - #print "Scan mock $arg\n"; - &scan_overrides($arg); - } -} - -my $warned = 0; -foreach my $func (keys %mocked) { - next if exists $noninlined{$func}; - - $warned++; - print STDERR "$func is mocked at $mocked{$func} but missing noinline annotation\n"; -} - -exit $warned ? 1 : 0; - - -sub scan_annotations { - my $file = shift; - - open FH, $file or die "cannot read $file: $!"; - - my $func; - while (<FH>) { - if (/^\s*(\w+)\(/ || /^(?:\w+\*?\s+)+(?:\*\s*)?(\w+)\(/) { - my $name = $1; - if ($name !~ /ATTRIBUTE/) { - $func = $name; - } - } elsif (/^\s*$/) { - $func = undef; - } - if (/ATTRIBUTE_NOINLINE/) { - if (defined $func) { - $noninlined{$func} = 1; - } - } - } - - close FH -} - -sub scan_overrides { - my $file = shift; - - open FH, $file or die "cannot read $file: $!"; - - my $func; - while (<FH>) { - if (/^(\w+)\(/ || /^\w+\s*(?:\*\s*)?(\w+)\(/) { - my $name = $1; - if ($name =~ /^vir/) { - $mocked{$name} = "$file:$."; - } - } - } - - close FH -} diff --git a/build-aux/mock-noinline.py b/build-aux/mock-noinline.py new file mode 100644 index 0000000000..fdc60c5de9 --- /dev/null +++ b/build-aux/mock-noinline.py @@ -0,0 +1,88 @@ +#!/usr/bin/env python +# +# Copyright (C) 2017-2019 Red Hat, Inc. +# +# This library is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 2.1 of the License, or (at your option) any later version. +# +# This library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this library. If not, see +# <http://www.gnu.org/licenses/>. + +from __future__ import print_function + +import re +import sys + +noninlined = {} +mocked = {} + +# Functions in public header don't get the noinline annotation +# so whitelist them here +noninlined["virEventAddTimeout"] = True +# This one confuses the script as its defined in the mock file +# but is actually just a local helper +noninlined["virMockStatRedirect"] = True + +def scan_annotations(filename): + funcprog1 = re.compile(r'''^\s*(\w+)\(.*''') + funcprog2 = re.compile(r'''^(?:\w+\*?\s+)+(?:\*\s*)?(\w+)\(.*''') + with open(filename, "r") as fh: + func = None + for line in fh: + line = line.strip() + m = funcprog1.match(line) + if m is None: + m = funcprog2.match(line) + if m is not None: + name = m.group(1) + if name.find("ATTRIBUTE") == -1: + func = name + elif line == "": + func = None + + if line.find("ATTRIBUTE_NOINLINE") != -1: + if func is not None: + noninlined[func] = True + +def scan_overrides(filename): + funcprog1 = re.compile(r'''^(\w+)\(.*''') + funcprog2 = re.compile(r'''^\w+\s*(?:\*\s*)?(\w+)\(.*''') + with open(filename, "r") as fh: + func = None + lineno = 0 + for line in fh: + lineno = lineno + 1 + #line = line.strip() + + m = funcprog1.match(line) + if m is None: + m = funcprog2.match(line) + if m is not None: + name = m.group(1) + if name.startswith("vir"): + mocked[name] = "%s:%d" % (filename, lineno) + + +for filename in sys.argv[1:]: + if filename.endswith(".h"): + scan_annotations(filename) + elif filename.endswith("mock.c"): + scan_overrides(filename) + +warned = False +for func in mocked.keys(): + if func not in noninlined: + warned = True + print("%s is mocked at %s but missing noinline annotation" % (func, mocked[func]), file=sys.stderr) + +if warned: + sys.exit(1) +sys.exit(0) diff --git a/cfg.mk b/cfg.mk index 4da3554b30..490e6b7f14 100644 --- a/cfg.mk +++ b/cfg.mk @@ -1156,8 +1156,8 @@ spacing-check: { echo '$(ME): incorrect formatting' 1>&2; exit 1; } mock-noinline: - $(AM_V_GEN)$(VC_LIST) | $(GREP) '\.[ch]$$' | xargs \ - $(PERL) $(top_srcdir)/build-aux/mock-noinline.pl + $(AM_V_GEN)$(VC_LIST) | $(GREP) '\.[ch]$$' | $(RUNUTF8) xargs \ + $(PYTHON) $(top_srcdir)/build-aux/mock-noinline.py header-ifdef: $(AM_V_GEN)$(VC_LIST) | $(GREP) '\.[h]$$' | xargs \ -- 2.21.0

As part of an goal to eliminate Perl from libvirt build tools, rewrite the header-ifdef.pl tool in Python. This was a straight conversion, manually going line-by-line to change the syntax from Perl to Python. Thus the overall structure of the file and approach is the same. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- Makefile.am | 2 +- build-aux/header-ifdef.pl | 182 --------------------------------- build-aux/header-ifdef.py | 206 ++++++++++++++++++++++++++++++++++++++ cfg.mk | 4 +- 4 files changed, 209 insertions(+), 185 deletions(-) delete mode 100644 build-aux/header-ifdef.pl create mode 100644 build-aux/header-ifdef.py diff --git a/Makefile.am b/Makefile.am index db77da890c..4ba729d3f7 100644 --- a/Makefile.am +++ b/Makefile.am @@ -44,7 +44,7 @@ EXTRA_DIST = \ build-aux/augeas-gentest.py \ build-aux/check-spacing.py \ build-aux/gitlog-to-changelog \ - build-aux/header-ifdef.pl \ + build-aux/header-ifdef.py \ build-aux/minimize-po.py \ build-aux/mock-noinline.py \ build-aux/prohibit-duplicate-header.py \ diff --git a/build-aux/header-ifdef.pl b/build-aux/header-ifdef.pl deleted file mode 100644 index dba3dbcbdc..0000000000 --- a/build-aux/header-ifdef.pl +++ /dev/null @@ -1,182 +0,0 @@ -#!/usr/bin/perl -# -# Validate that header files follow a standard layout: -# -# /* -# ...copyright header... -# */ -# <one blank line> -# #pragma once -# ....content.... -# -#--- -# -# For any file ending priv.h, before the #pragma once -# We will have a further section -# -# #ifndef SYMBOL_ALLOW -# # error .... -# #endif /* SYMBOL_ALLOW */ -# <one blank line> -# -#--- -# -# For public headers (files in include/), use the standard header guard instead of #pragma once: -# #ifndef SYMBOL -# # define SYMBOL -# ....content.... -# #endif /* SYMBOL */ - -use strict; -use warnings; - -my $STATE_COPYRIGHT_COMMENT = 0; -my $STATE_COPYRIGHT_BLANK = 1; -my $STATE_PRIV_START = 2; -my $STATE_PRIV_ERROR = 3; -my $STATE_PRIV_END = 4; -my $STATE_PRIV_BLANK = 5; -my $STATE_GUARD_START = 6; -my $STATE_GUARD_DEFINE = 7; -my $STATE_GUARD_END = 8; -my $STATE_EOF = 9; -my $STATE_PRAGMA = 10; - -my $file = " "; -my $ret = 0; -my $ifdef = ""; -my $ifdefpriv = ""; -my $publicheader = 0; - -my $state = $STATE_EOF; -my $mistake = 0; - -sub mistake { - my $msg = shift; - warn $msg; - $mistake = 1; - $ret = 1; -} - -while (<>) { - if (not $file eq $ARGV) { - if ($state == $STATE_COPYRIGHT_COMMENT) { - &mistake("$file: missing copyright comment"); - } elsif ($state == $STATE_COPYRIGHT_BLANK) { - &mistake("$file: missing blank line after copyright header"); - } elsif ($state == $STATE_PRIV_START) { - &mistake("$file: missing '#ifndef $ifdefpriv'"); - } elsif ($state == $STATE_PRIV_ERROR) { - &mistake("$file: missing '# error ...priv allow...'"); - } elsif ($state == $STATE_PRIV_END) { - &mistake("$file: missing '#endif /* $ifdefpriv */'"); - } elsif ($state == $STATE_PRIV_BLANK) { - &mistake("$file: missing blank line after priv header check"); - } elsif ($state == $STATE_GUARD_START) { - if ($publicheader) { - &mistake("$file: missing '#ifndef $ifdef'"); - } else { - &mistake("$file: missing '#pragma once' header guard"); - } - } elsif ($state == $STATE_GUARD_DEFINE) { - &mistake("$file: missing '# define $ifdef'"); - } elsif ($state == $STATE_GUARD_END) { - &mistake("$file: missing '#endif /* $ifdef */'"); - } - - $ifdef = uc $ARGV; - $ifdef =~ s,.*/,,; - $ifdef =~ s,[^A-Z0-9],_,g; - $ifdef =~ s,__+,_,g; - unless ($ifdef =~ /^LIBVIRT_/ && $ARGV !~ /libvirt_internal.h/) { - $ifdef = "LIBVIRT_" . $ifdef; - } - $ifdefpriv = $ifdef . "_ALLOW"; - - $file = $ARGV; - $state = $STATE_COPYRIGHT_COMMENT; - $mistake = 0; - $publicheader = ($ARGV =~ /include\//); - } - - if ($mistake || - $ARGV =~ /config-post\.h$/ || - $ARGV =~ /vbox_(CAPI|XPCOM)/) { - $state = $STATE_EOF; - next; - } - - if ($state == $STATE_COPYRIGHT_COMMENT) { - if (m,\*/,) { - $state = $STATE_COPYRIGHT_BLANK; - } - } elsif ($state == $STATE_COPYRIGHT_BLANK) { - if (! /^$/) { - &mistake("$file: missing blank line after copyright header"); - } - if ($ARGV =~ /priv\.h$/) { - $state = $STATE_PRIV_START; - } else { - $state = $STATE_GUARD_START; - } - } elsif ($state == $STATE_PRIV_START) { - if (/^$/) { - &mistake("$file: too many blank lines after copyright header"); - } elsif (/#ifndef $ifdefpriv$/) { - $state = $STATE_PRIV_ERROR; - } else { - &mistake("$file: missing '#ifndef $ifdefpriv'"); - } - } elsif ($state == $STATE_PRIV_ERROR) { - if (/# error ".*"$/) { - $state = $STATE_PRIV_END; - } else { - &mistake("$file: missing '# error ...priv allow...'"); - } - } elsif ($state == $STATE_PRIV_END) { - if (m,#endif /\* $ifdefpriv \*/,) { - $state = $STATE_PRIV_BLANK; - } else { - &mistake("$file: missing '#endif /* $ifdefpriv */'"); - } - } elsif ($state == $STATE_PRIV_BLANK) { - if (! /^$/) { - &mistake("$file: missing blank line after priv guard"); - } - $state = $STATE_GUARD_START; - } elsif ($state == $STATE_GUARD_START) { - if (/^$/) { - &mistake("$file: too many blank lines after copyright header"); - } - if ($publicheader) { - if (/#ifndef $ifdef$/) { - $state = $STATE_GUARD_DEFINE; - } else { - &mistake("$file: missing '#ifndef $ifdef'"); - } - } else { - if (/#pragma once/) { - $state = $STATE_PRAGMA; - } else { - &mistake("$file: missing '#pragma once' header guard"); - } - } - } elsif ($state == $STATE_GUARD_DEFINE) { - if (/# define $ifdef$/) { - $state = $STATE_GUARD_END; - } else { - &mistake("$file: missing '# define $ifdef'"); - } - } elsif ($state == $STATE_GUARD_END) { - if (m,#endif /\* $ifdef \*/$,) { - $state = $STATE_EOF; - } - } elsif ($state == $STATE_PRAGMA) { - next; - } elsif ($state == $STATE_EOF) { - die "$file: unexpected content after '#endif /* $ifdef */'"; - } else { - die "$file: unexpected state $state"; - } -} -exit $ret; diff --git a/build-aux/header-ifdef.py b/build-aux/header-ifdef.py new file mode 100644 index 0000000000..86c4781a7e --- /dev/null +++ b/build-aux/header-ifdef.py @@ -0,0 +1,206 @@ +#!/usr/bin/env python +# +# Copyright (C) 2018-2019 Red Hat, Inc. +# +# This library is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 2.1 of the License, or (at your option) any later version. +# +# This library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this library. If not, see +# <http://www.gnu.org/licenses/>. +# +# Validate that header files follow a standard layout: +# +# /* +# ...copyright header... +# */ +# <one blank line> +# #pragma once +# ....content.... +# +#--- +# +# For any file ending priv.h, before the #pragma once +# We will have a further section +# +# #ifndef SYMBOL_ALLOW +# # error .... +# #endif /* SYMBOL_ALLOW */ +# <one blank line> +# +#--- +# +# For public headers (files in include/), use the standard header guard instead of #pragma once: +# #ifndef SYMBOL +# # define SYMBOL +# ....content.... +# #endif /* SYMBOL */ + +from __future__ import print_function + +import os.path +import re +import sys + +STATE_COPYRIGHT_COMMENT = 0 +STATE_COPYRIGHT_BLANK = 1 +STATE_PRIV_START = 2 +STATE_PRIV_ERROR = 3 +STATE_PRIV_END = 4 +STATE_PRIV_BLANK = 5 +STATE_GUARD_START = 6 +STATE_GUARD_DEFINE = 7 +STATE_GUARD_END = 8 +STATE_EOF = 9 +STATE_PRAGMA = 10 + +def check_header(filename): + ifdef = "" + ifdefpriv = "" + + state = STATE_EOF + + ifdef = os.path.basename(filename).upper() + ifdef = re.sub(r"""[^A-Z0-9]""", "_", ifdef) + ifdef = re.sub(r"""__+""", "_", ifdef) + + if not ifdef.startswith("LIBVIRT_") or filename.find("libvirt_internal.h") != -1: + ifdef = "LIBVIRT_" + ifdef + + ifdefpriv = ifdef + "_ALLOW" + + state = STATE_COPYRIGHT_COMMENT + publicheader = False + if filename.find("include/") != -1: + publicheader = True + + with open(filename, "r") as fh: + for line in fh: + line = line.rstrip("\n") + if state == STATE_COPYRIGHT_COMMENT: + if line.find("*/") != -1: + state = STATE_COPYRIGHT_BLANK + elif state == STATE_COPYRIGHT_BLANK: + if line != "": + print("%s: missing blank line after copyright header" % filename, file=sys.stderr) + return True + + if filename.endswith("priv.h"): + state = STATE_PRIV_START + else: + state = STATE_GUARD_START + elif state == STATE_PRIV_START: + if line == "": + print("%s: too many blank lines after copyright header" % filename, file=sys.stderr) + return True + elif re.match(r"""#ifndef %s$""" % ifdefpriv, line): + state = STATE_PRIV_ERROR + else: + print("%s: missing '#ifndef %s'" % (filename, ifdefpriv), file=sys.stderr) + return True + elif state == STATE_PRIV_ERROR: + if re.match(r"""# error ".*"$""", line): + state = STATE_PRIV_END + else: + print("%s: missing '# error ...priv allow...'" % filename, file=sys.stderr) + return True + elif state == STATE_PRIV_END: + if re.match(r"""#endif /\* %s \*/""" % ifdefpriv, line): + state = STATE_PRIV_BLANK + else: + print("%s: missing '#endif /* %s */'" % (filename, ifdefpriv), file=sys.stderr) + return True + elif state == STATE_PRIV_BLANK: + if line != "": + print("%s: missing blank line after priv guard" % filename, file=sys.stderr) + return True + state = STATE_GUARD_START + elif state == STATE_GUARD_START: + if line == "": + print("%s: too many blank lines after copyright header" % filename, file=sys.stderr) + return True + if publicheader: + if re.match(r"""#ifndef %s$""" % ifdef, line): + state = STATE_GUARD_DEFINE + else: + print("%s: missing '#ifndef %s'" % (filename, ifdef), file=sys.stderr) + return True + else: + if re.match(r"""#pragma once""", line): + state = STATE_PRAGMA + else: + print("%s: missing '#pragma once' header guard" % filename, file=sys.stderr) + return True + elif state == STATE_GUARD_DEFINE: + if re.match(r"""# define %s$""" % ifdef, line): + state = STATE_GUARD_END + else: + print("%s: missing '# define %s'" % (filename, ifdef), file=sys.stderr) + return True + elif state == STATE_GUARD_END: + if re.match(r"""#endif /\* %s \*/$""" % ifdef, line): + state = STATE_EOF + elif state == STATE_PRAGMA: + next + elif state == STATE_EOF: + print("%s: unexpected content after '#endif /* %s */'" % (filename, ifdef), file=sys.stderr) + return True + else: + print("%s: unexpected state $state" % filename, file=sys.stderr) + return True + + + if state == STATE_COPYRIGHT_COMMENT: + print("%s: missing copyright comment" % filename, file=sys.stderr) + return True + elif state == STATE_COPYRIGHT_BLANK: + print("%s: missing blank line after copyright header" % filename, file=sys.stderr) + return True + elif state == STATE_PRIV_START: + print("%s: missing '#ifndef %s'" % (filename, ifdefpriv), file=sys.stderr) + return True + elif state == STATE_PRIV_ERROR: + print("%s: missing '# error ...priv allow...'" % filename, file=sys.stderr) + return True + elif state == STATE_PRIV_END: + print("%s: missing '#endif /* %s */'" % (filename, ifdefpriv), file=sys.stderr) + return True + elif state == STATE_PRIV_BLANK: + print("%s: missing blank line after priv header check" % filename, file=sys.stderr) + return True + elif state == STATE_GUARD_START: + if publicheader: + print("%s: missing '#ifndef %s'" % (filename, ifdef), file=sys.stderr) + return True + else: + print("%s: missing '#pragma once' header guard" % filename, file=sys.stderr) + return True + elif state == STATE_GUARD_DEFINE: + print("%s: missing '# define %s'" % (filename, ifdef), file=sys.stderr) + return True + elif state == STATE_GUARD_END: + print("%s: missing '#endif /* %s */'" % (filename, ifdef), file=sys.stderr) + return True + + return False + +ret = 0 + +for filename in sys.argv[1:]: + if filename.find("config-post.h") != -1: + continue + if filename.find("vbox_CAPI") != -1: + continue + if filename.find("vbox_XPCOM") != -1: + continue + if check_header(filename): + ret = 1 + +sys.exit(ret) diff --git a/cfg.mk b/cfg.mk index 490e6b7f14..3e63fee9ff 100644 --- a/cfg.mk +++ b/cfg.mk @@ -1160,8 +1160,8 @@ mock-noinline: $(PYTHON) $(top_srcdir)/build-aux/mock-noinline.py header-ifdef: - $(AM_V_GEN)$(VC_LIST) | $(GREP) '\.[h]$$' | xargs \ - $(PERL) $(top_srcdir)/build-aux/header-ifdef.pl + $(AM_V_GEN)$(VC_LIST) | $(GREP) '\.[h]$$' | $(RUNUTF8) xargs \ + $(PYTHON) $(top_srcdir)/build-aux/header-ifdef.py test-wrap-argv: $(AM_V_GEN)$(VC_LIST) | $(GREP) -E '\.(ldargs|args)' | xargs \ -- 2.21.0

As part of an goal to eliminate Perl from libvirt build tools, rewrite the check-aclperms.pl tool in Python. This was a straight conversion, manually going line-by-line to change the syntax from Perl to Python. Thus the overall structure of the file and approach is the same. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- src/Makefile.am | 4 +-- src/check-aclperms.pl | 73 ---------------------------------------- src/check-aclperms.py | 77 +++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 79 insertions(+), 75 deletions(-) delete mode 100755 src/check-aclperms.pl create mode 100755 src/check-aclperms.py diff --git a/src/Makefile.am b/src/Makefile.am index 7a4de2aa53..c441525b7d 100644 --- a/src/Makefile.am +++ b/src/Makefile.am @@ -359,11 +359,11 @@ check-aclrules: $(addprefix $(srcdir)/,$(filter-out /%,$(STATEFUL_DRIVER_SOURCE_FILES))) check-aclperms: - $(AM_V_GEN)$(PERL) $(srcdir)/check-aclperms.pl \ + $(AM_V_GEN)$(RUNUTF8) $(PYTHON) $(srcdir)/check-aclperms.py \ $(srcdir)/access/viraccessperm.h \ $(srcdir)/access/viraccessperm.c -EXTRA_DIST += check-driverimpls.pl check-aclrules.pl check-aclperms.pl +EXTRA_DIST += check-driverimpls.pl check-aclrules.pl check-aclperms.py check-local: check-protocol check-symfile check-symsorting \ check-drivername check-driverimpls check-aclrules \ diff --git a/src/check-aclperms.pl b/src/check-aclperms.pl deleted file mode 100755 index 55b6598313..0000000000 --- a/src/check-aclperms.pl +++ /dev/null @@ -1,73 +0,0 @@ -#!/usr/bin/env perl -# -# Copyright (C) 2013 Red Hat, Inc. -# -# This library is free software; you can redistribute it and/or -# modify it under the terms of the GNU Lesser General Public -# License as published by the Free Software Foundation; either -# version 2.1 of the License, or (at your option) any later version. -# -# This library is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# Lesser General Public License for more details. -# -# You should have received a copy of the GNU Lesser General Public -# License along with this library. If not, see -# <http://www.gnu.org/licenses/>. -# -# This script just validates that the stringified version of -# a virAccessPerm enum matches the enum constant name. We do -# a lot of auto-generation of code, so when these don't match -# problems occur, preventing auth from succeeding at all. - -my $hdr = shift; -my $impl = shift; - -my %perms; - -my @perms; - -open HDR, $hdr or die "cannot read $hdr: $!"; - -while (<HDR>) { - if (/^\s+VIR_ACCESS_PERM_([_A-Z]+)(,?|\s|$)/) { - my $perm = $1; - - $perms{$perm} = 1 unless ($perm =~ /_LAST$/); - } -} - -close HDR; - - -open IMPL, $impl or die "cannot read $impl: $!"; - -my $group; -my $warned = 0; - -while (defined (my $line = <IMPL>)) { - if ($line =~ /VIR_ACCESS_PERM_([_A-Z]+)_LAST/) { - $group = $1; - } elsif ($line =~ /"[_a-z]+"/) { - my @bits = split /,/, $line; - foreach my $bit (@bits) { - if ($bit =~ /"([_a-z]+)"/) { - my $perm = uc($group . "_" . $1); - if (!exists $perms{$perm}) { - print STDERR "Unknown perm string $1 for group $group\n"; - $warned = 1; - } - delete $perms{$perm}; - } - } - } -} -close IMPL; - -foreach my $perm (keys %perms) { - print STDERR "Perm $perm had not string form\n"; - $warned = 1; -} - -exit $warned; diff --git a/src/check-aclperms.py b/src/check-aclperms.py new file mode 100755 index 0000000000..ace6ab6757 --- /dev/null +++ b/src/check-aclperms.py @@ -0,0 +1,77 @@ +#!/usr/bin/env python +# +# Copyright (C) 2013-2019 Red Hat, Inc. +# +# This library is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 2.1 of the License, or (at your option) any later version. +# +# This library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this library. If not, see +# <http://www.gnu.org/licenses/>. +# +# This script just validates that the stringified version of +# a virAccessPerm enum matches the enum constant name. We do +# a lot of auto-generation of code, so when these don't match +# problems occur, preventing auth from succeeding at all. + +from __future__ import print_function + +import re +import sys + +if len(sys.argv) != 3: + print("syntax: %s HEADER IMPL" % (sys.argv[0]), file=sys.stderr) + sys.exit(1) + +hdr = sys.argv[1] +impl = sys.argv[2] + +perms = {} + +with open(hdr) as fh: + symprog = re.compile(r"^\s+VIR_ACCESS_PERM_([_A-Z]+)(,?|\s|$).*") + for line in fh: + symmatch = symprog.match(line) + if symmatch is not None: + perm = symmatch.group(1) + + if not perm.endswith("_LAST"): + perms[perm] = 1 + +warned = False + +with open(impl) as fh: + group = None + symlastprog = re.compile(r".*VIR_ACCESS_PERM_([_A-Z]+)_LAST.*") + alnumprog = re.compile(r'''.*"([_a-z]+)".*''') + + for line in fh: + symlastmatch = symlastprog.match(line) + if symlastmatch is not None: + group = symlastmatch.group(1) + elif alnumprog.match(line) is not None: + bits = line.split(",") + for bit in bits: + m = alnumprog.match(bit) + if m is not None: + perm = (group + "_" + m.group(1)).upper() + if perm not in perms: + print("Unknown perm string %s for group %s" % (m.group(1), group), file=sys.stderr) + warned = True + + del perms[perm] + +for perm in perms.keys(): + print("Perm %s had not string form" % perm, file=sys.stderr) + warned = True + +if warned: + sys.exit(1) +sys.exit(0) -- 2.21.0

As part of an goal to eliminate Perl from libvirt build tools, rewrite the check-symsorting.pl tool in Python. This was a straight conversion, manually going line-by-line to change the syntax from Perl to Python. Thus the overall structure of the file and approach is the same. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- src/Makefile.am | 6 +-- src/check-symsorting.pl | 106 ------------------------------------- src/check-symsorting.py | 112 ++++++++++++++++++++++++++++++++++++++++ 3 files changed, 115 insertions(+), 109 deletions(-) delete mode 100755 src/check-symsorting.pl create mode 100755 src/check-symsorting.py diff --git a/src/Makefile.am b/src/Makefile.am index c441525b7d..4bdd16af22 100644 --- a/src/Makefile.am +++ b/src/Makefile.am @@ -285,12 +285,12 @@ check-symfile: check-admin-symfile: endif ! WITH_LINUX check-symsorting: - $(AM_V_GEN)$(PERL) $(srcdir)/check-symsorting.pl \ + $(AM_V_GEN)$(RUNUTF8) $(PYTHON) $(srcdir)/check-symsorting.py \ $(srcdir) $(SYM_FILES) check-admin-symsorting: - $(AM_V_GEN)$(PERL) $(srcdir)/check-symsorting.pl \ + $(AM_V_GEN)$(RUNUTF8) $(PYTHON) $(srcdir)/check-symsorting.py \ $(srcdir) $(ADMIN_SYM_FILES) -EXTRA_DIST += check-symfile.pl check-symsorting.pl +EXTRA_DIST += check-symfile.pl check-symsorting.py # Keep this list synced with RPC_PROBE_FILES PROTOCOL_STRUCTS = \ diff --git a/src/check-symsorting.pl b/src/check-symsorting.pl deleted file mode 100755 index 51e38bdedb..0000000000 --- a/src/check-symsorting.pl +++ /dev/null @@ -1,106 +0,0 @@ -#!/usr/bin/env perl - -# Copyright (C) 2012-2013 Red Hat, Inc. -# -# This library is free software; you can redistribute it and/or -# modify it under the terms of the GNU Lesser General Public -# License as published by the Free Software Foundation; either -# version 2.1 of the License, or (at your option) any later version. -# -# This library is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# Lesser General Public License for more details. -# -# You should have received a copy of the GNU Lesser General Public -# License along with this library. If not, see -# <http://www.gnu.org/licenses/>. - -use strict; -use warnings; - -die "syntax: $0 SRCDIR SYMFILE..." unless int(@ARGV) >= 2; - -my $ret = 0; -my $srcdir = shift; -my $lastgroup = undef; -foreach my $symfile (@ARGV) { - open SYMFILE, $symfile or die "cannot read $symfile: $!"; - - my $line = 0; - my $groupfile = ""; - my @group; - - while (<SYMFILE>) { - chomp; - - if (/^#\s*((\w+\/)*(\w+\.h))\s*$/) { - $groupfile = $1; - } elsif (/^#/) { - # Ignore comments - } elsif (/^\s*$/) { - if (@group) { - &check_sorting(\@group, $symfile, $line, $groupfile); - } - @group = (); - $line = $.; - } else { - $_ =~ s/;//; - push @group, $_; - } - } - - close SYMFILE; - if (@group) { - &check_sorting(\@group, $symfile, $line, $groupfile); - } - $lastgroup = undef; -} - -sub check_sorting { - my $group = shift; - my $symfile = shift; - my $line = shift; - my $groupfile = shift; - - my @group = @{$group}; - my @sorted = sort { lc $a cmp lc $b } @group; - my $sorted = 1; - my $first; - my $last; - - # Check that groups are in order and groupfile exists - if (defined $lastgroup && lc $lastgroup ge lc $groupfile) { - print "Symbol block at $symfile:$line: block not sorted\n"; - print "Move $groupfile block before $lastgroup block\n"; - print "\n"; - $ret = 1; - } - if (! -e "$srcdir/$groupfile") { - print "Symbol block at $symfile:$line: $groupfile not found\n"; - print "\n"; - $ret = 1; - } - $lastgroup = $groupfile; - - # Check that symbols within a group are in order - for (my $i = 0 ; $i <= $#sorted ; $i++) { - if ($sorted[$i] ne $group[$i]) { - $first = $i unless defined $first; - $last = $i; - $sorted = 0; - } - } - if (!$sorted) { - @group = splice @group, $first, ($last-$first+1); - @sorted = splice @sorted, $first, ($last-$first+1); - print "Symbol block at $symfile:$line: symbols not sorted\n"; - print map { " " . $_ . "\n" } @group; - print "Correct ordering\n"; - print map { " " . $_ . "\n" } @sorted; - print "\n"; - $ret = 1; - } -} - -exit $ret; diff --git a/src/check-symsorting.py b/src/check-symsorting.py new file mode 100755 index 0000000000..a1d16a4c43 --- /dev/null +++ b/src/check-symsorting.py @@ -0,0 +1,112 @@ +#!/usr/bin/env python + +# Copyright (C) 2012-2019 Red Hat, Inc. +# +# This library is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 2.1 of the License, or (at your option) any later version. +# +# This library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this library. If not, see +# <http://www.gnu.org/licenses/>. + +from __future__ import print_function + +import os.path +import re +import sys + +if len(sys.argv) < 3: + print("syntax: %s SRCDIR SYMFILE..." % sys.argv[0], file=sys.stderr) + sys.exit(1) + +def check_sorting(group, symfile, line, groupfile, lastgroup): + sortedgroup = sorted(group, key=str.lower) + issorted = True + first = None + last = None + + err = False + # Check that groups are in order and groupfile exists + if lastgroup is not None and lastgroup.lower() > groupfile.lower(): + print("Symbol block at %s:%s: block not sorted" % (symfile, line), file=sys.stderr) + print("Move %s block before %s block" % (groupfile, lastgroup), file=sys.stderr) + print("", file=sys.stderr) + err = True + + if not os.path.exists(os.path.join(srcdir, groupfile)): + print("Symbol block at %s:%s: %s not found" % (symfile, line, groupfile), file=sys.stderr) + print("", file=sys.stderr) + err = True + + # Check that symbols within a group are in order + for i in range(len(group)): + if sortedgroup[i] != group[i]: + if first is None: + first = i + + last = i + issorted = False + + if not issorted: + actual = group[first:(last-first+1)] + expect = sortedgroup[first:(last-first+1)] + print("Symbol block at %s:%s: symbols not sorted" % (symfile, line), file=sys.stderr) + for g in actual: + print(" %s" % g, file=sys.stderr) + print("Correct ordering", file=sys.stderr) + for g in expect: + print(" %s" % g, file=sys.stderr) + print("", file=sys.stderr) + err = True + + return err + + +ret = 0 +srcdir = sys.argv[1] +lastgroup = None +for symfile in sys.argv[2:]: + with open(symfile, "r") as fh: + lineno = 0 + groupfile = "" + group = [] + thisline = 0 + + filenameprog = re.compile(r'''^#\s*((\w+\/)*(\w+\.h))\s*$''') + + for line in fh: + thisline = thisline + 1 + line = line.strip() + + filenamematch = filenameprog.match(line) + if filenamematch is not None: + groupfile = filenamematch.group(1) + elif line == "": + if len(group) > 0: + if check_sorting(group, symfile, lineno, groupfile, lastgroup): + ret = 1 + + group = [] + lineno = thisline + lastgroup = groupfile + elif line[0] == '#': + # Ignore comments + pass + else: + line = line.strip(";") + group.append(line) + + if len(group) > 0: + if check_sorting(group, symfile, lineno, groupfile, lastgroup): + ret = 1 + + lastgroup = None + +sys.exit(ret) -- 2.21.0

As part of an goal to eliminate Perl from libvirt build tools, rewrite the check-symfile.pl tool in Python. This was a straight conversion, manually going line-by-line to change the syntax from Perl to Python. Thus the overall structure of the file and approach is the same. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- src/Makefile.am | 10 +++--- src/check-symfile.pl | 70 -------------------------------------- src/check-symfile.py | 80 ++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 85 insertions(+), 75 deletions(-) delete mode 100755 src/check-symfile.pl create mode 100755 src/check-symfile.py diff --git a/src/Makefile.am b/src/Makefile.am index 4bdd16af22..a145f69140 100644 --- a/src/Makefile.am +++ b/src/Makefile.am @@ -275,11 +275,11 @@ PDWTAGS = \ # rule for libvirt.la. However, checking symbols relies on Linux ELF layout if WITH_LINUX check-symfile: libvirt.syms libvirt.la - $(AM_V_GEN)$(PERL) $(srcdir)/check-symfile.pl libvirt.syms \ - .libs/libvirt.so + $(AM_V_GEN)$(RUNUTF8) $(PYTHON) $(srcdir)/check-symfile.py \ + libvirt.syms .libs/libvirt.so check-admin-symfile: libvirt_admin.syms libvirt-admin.la - $(AM_V_GEN)$(PERL) $(srcdir)/check-symfile.pl libvirt_admin.syms \ - .libs/libvirt-admin.so + $(AM_V_GEN)$(RUNUTF8) $(PYTHON) $(srcdir)/check-symfile.py \ + libvirt_admin.syms .libs/libvirt-admin.so else ! WITH_LINUX check-symfile: check-admin-symfile: @@ -290,7 +290,7 @@ check-symsorting: check-admin-symsorting: $(AM_V_GEN)$(RUNUTF8) $(PYTHON) $(srcdir)/check-symsorting.py \ $(srcdir) $(ADMIN_SYM_FILES) -EXTRA_DIST += check-symfile.pl check-symsorting.py +EXTRA_DIST += check-symfile.py check-symsorting.py # Keep this list synced with RPC_PROBE_FILES PROTOCOL_STRUCTS = \ diff --git a/src/check-symfile.pl b/src/check-symfile.pl deleted file mode 100755 index 4f88300864..0000000000 --- a/src/check-symfile.pl +++ /dev/null @@ -1,70 +0,0 @@ -#!/usr/bin/env perl - -# Copyright (C) 2012-2013 Red Hat, Inc. -# -# This library is free software; you can redistribute it and/or -# modify it under the terms of the GNU Lesser General Public -# License as published by the Free Software Foundation; either -# version 2.1 of the License, or (at your option) any later version. -# -# This library is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# Lesser General Public License for more details. -# -# You should have received a copy of the GNU Lesser General Public -# License along with this library. If not, see -# <http://www.gnu.org/licenses/>. - -die "syntax: $0 SYMFILE ELFLIB(S)" unless int(@ARGV) >= 2; - -my $symfile = shift @ARGV; -my @elflibs = @ARGV; - -my %wantsyms; -my %gotsyms; - -my $ret = 0; - -open SYMFILE, $symfile or die "cannot read $symfile: $!"; - -while (<SYMFILE>) { - next if /{/; - next if /}/; - next if /global:/; - next if /local:/; - next if /^\s*$/; - next if /^\s*#/; - next if /\*/; - - die "malformed line $_" unless /^\s*(\S+);$/; - - if (exists $wantsyms{$1}) { - print STDERR "Symbol $1 is listed twice\n"; - $ret = 1; - } else { - $wantsyms{$1} = 1; - } -} -close SYMFILE; - -foreach my $elflib (@elflibs) { - open NM, "-|", "nm", $elflib or die "cannot run 'nm $elflib': $!"; - - while (<NM>) { - next unless /^\S+\s(?:[TBD])\s(\S+)\s*$/; - - $gotsyms{$1} = 1; - } - - close NM; -} - -foreach my $sym (keys(%wantsyms)) { - next if exists $gotsyms{$sym}; - - print STDERR "Expected symbol $sym is not in ELF library\n"; - $ret = 1; -} - -exit($ret); diff --git a/src/check-symfile.py b/src/check-symfile.py new file mode 100755 index 0000000000..3154619bfe --- /dev/null +++ b/src/check-symfile.py @@ -0,0 +1,80 @@ +#!/usr/bin/env python +# +# Copyright (C) 2012-2019 Red Hat, Inc. +# +# This library is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 2.1 of the License, or (at your option) any later version. +# +# This library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this library. If not, see +# <http://www.gnu.org/licenses/>. + +from __future__ import print_function + +import re +import subprocess +import sys + +if len(sys.argv) < 3: + print("syntax: %s SYMFILE ELFLIB(S)" % sys.argv[0], file=sys.stderr) + +symfile = sys.argv[1] +elflibs = sys.argv[2:] + +wantsyms = {} +gotsyms = {} + +ret = 0 + +with open(symfile, "r") as fh: + for line in fh: + line = line.strip() + if line.find("{") != -1: + continue + if line.find("}") != -1: + continue + if line in ["global:", "local:"]: + continue + if line == "": + continue + if line[0] == '#': + continue + if line.find("*") != -1: + continue + + line = line.strip(";") + + if line in wantsyms: + print("Symbol $1 is listed twice", file=sys.stderr) + ret = 1 + else: + wantsyms[line] = True + +for elflib in elflibs: + nm = subprocess.Popen(["nm", elflib], shell=False, stdout=subprocess.PIPE).stdout + + symprog = re.compile(r'''^\S+\s(?:[TBD])\s(\S+)\s*$''') + for line in nm: + line = line.decode("utf-8") + symmatch = symprog.match(line) + if symmatch is None: + continue + + gotsyms[symmatch.group(1)] = True + + +for sym in wantsyms.keys(): + if sym in gotsyms: + continue + + print("Expected symbol '%s' is not in ELF library" % sym, file=sys.stderr) + ret = 1 + +sys.exit(ret) -- 2.21.0

As part of an goal to eliminate Perl from libvirt build tools, rewrite the dtrace2systemtap.pl tool in Python. This was a straight conversion, manually going line-by-line to change the syntax from Perl to Python. Thus the overall structure of the file and approach is the same. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- src/Makefile.am | 9 ++- src/dtrace2systemtap.pl | 130 ------------------------------------- src/dtrace2systemtap.py | 140 ++++++++++++++++++++++++++++++++++++++++ 3 files changed, 144 insertions(+), 135 deletions(-) delete mode 100755 src/dtrace2systemtap.pl create mode 100755 src/dtrace2systemtap.py diff --git a/src/Makefile.am b/src/Makefile.am index a145f69140..8b046cafe4 100644 --- a/src/Makefile.am +++ b/src/Makefile.am @@ -605,7 +605,6 @@ nodist_libvirt_la_SOURCES = libvirt_probes.h if WITH_REMOTE nodist_libvirt_driver_remote_la_SOURCES = libvirt_probes.h endif WITH_REMOTE -DTRACE2SYSTEMTAP_FLAGS = --with-modules BUILT_SOURCES += libvirt_probes.h libvirt_probes.stp libvirt_functions.stp @@ -640,10 +639,10 @@ RPC_PROBE_FILES += $(srcdir)/rpc/virnetprotocol.x \ libvirt_functions.stp: $(RPC_PROBE_FILES) $(srcdir)/rpc/gensystemtap.pl $(AM_V_GEN)$(PERL) -w $(srcdir)/rpc/gensystemtap.pl $(RPC_PROBE_FILES) > $@ -%_probes.stp: %_probes.d $(srcdir)/dtrace2systemtap.pl \ +%_probes.stp: %_probes.d $(srcdir)/dtrace2systemtap.py \ $(top_builddir)/config.status - $(AM_V_GEN)$(PERL) -w $(srcdir)/dtrace2systemtap.pl \ - $(DTRACE2SYSTEMTAP_FLAGS) $(bindir) $(sbindir) $(libdir) $< > $@ + $(AM_V_GEN)$(RUNUTF8) $(PYTHON) $(srcdir)/dtrace2systemtap.py \ + $(bindir) $(sbindir) $(libdir) $< > $@ CLEANFILES += libvirt_probes.h libvirt_probes.o libvirt_probes.lo \ libvirt_functions.stp libvirt_probes.stp @@ -764,7 +763,7 @@ endif LIBVIRT_INIT_SCRIPT_SYSTEMD endif WITH_LIBVIRTD -EXTRA_DIST += dtrace2systemtap.pl +EXTRA_DIST += dtrace2systemtap.py if WITH_LIBVIRTD diff --git a/src/dtrace2systemtap.pl b/src/dtrace2systemtap.pl deleted file mode 100755 index c5fce248b4..0000000000 --- a/src/dtrace2systemtap.pl +++ /dev/null @@ -1,130 +0,0 @@ -#!/usr/bin/env perl -# -# Copyright (C) 2011-2012 Red Hat, Inc. -# -# This library is free software; you can redistribute it and/or -# modify it under the terms of the GNU Lesser General Public -# License as published by the Free Software Foundation; either -# version 2.1 of the License, or (at your option) any later version. -# -# This library is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# Lesser General Public License for more details. -# -# You should have received a copy of the GNU Lesser General Public -# License along with this library. If not, see -# <http://www.gnu.org/licenses/>. -# -# -# Generate a set of systemtap probe definitions corresponding to -# DTrace probe markers in libvirt.so -# -# perl dtrace2systemtap.pl probes.d > libvirt_probes.stp -# - -use strict; -use warnings; - -my $file; -my @files; -my %files; - -my $with_modules = 0; -if ($ARGV[0] eq "--with-modules") { - # set if we want to honor the "module" setting in the .d file - $with_modules = 1; - shift @ARGV; -} - -my $bindir = shift @ARGV; -my $sbindir = shift @ARGV; -my $libdir = shift @ARGV; - -my $probe; -my $args; - -# Read the DTraceprobes definition -while (<>) { - next if m,^\s*$,; - - next if /^\s*provider\s+\w+\s*{\s*$/; - next if /^\s*};\s*$/; - - if (m,^\s*\#,) { - if (m,^\s*\#\s*file:\s*(\S+)\s*$,) { - $file = $1; - push @files, $file; - $files{$file} = { prefix => undef, probes => [] }; - } elsif (m,^\s*\#\s*prefix:\s*(\S+)\s*$,) { - $files{$file}->{prefix} = $1; - } elsif (m,^\s*\#\s*binary:\s*(\S+)\s*$,) { - $files{$file}->{binary} = $1; - } elsif (m,^\s*\#\s*module:\s*(\S+)\s*$,) { - $files{$file}->{module} = $1; - } else { - # ignore unknown comments - } - } else { - if (m,\s*probe\s+([a-zA-Z0-9_]+)\((.*?)(\);)?$,) { - $probe = $1; - $args = $2; - if ($3) { - push @{$files{$file}->{probes}}, [$probe, $args]; - $probe = $args = undef; - } - } elsif ($probe) { - if (m,^(.*?)(\);)?$,) { - $args .= $1; - if ($2) { - push @{$files{$file}->{probes}}, [$probe, $args]; - $probe = $args = undef; - } - } else { - die "unexpected data $_ on line $."; - } - } else { - die "unexpected data $_ on line $."; - } - } -} - -# Write out the SystemTap probes -foreach my $file (@files) { - my $prefix = $files{$file}->{prefix}; - my @probes = @{$files{$file}->{probes}}; - - print "# $file\n\n"; - foreach my $probe (@probes) { - my $name = $probe->[0]; - my $args = $probe->[1]; - - my $pname = $name; - $pname =~ s/${prefix}_/libvirt.$prefix./; - - my $binary = "$libdir/libvirt.so"; - if (exists $files{$file}->{binary}) { - $binary = $sbindir . "/" . $files{$file}->{binary}; - } - if ($with_modules && exists $files{$file}->{module}) { - $binary = $libdir . "/" . $files{$file}->{module}; - } - - print "probe $pname = process(\"$binary\").mark(\"$name\") {\n"; - - my @args = split /,/, $args; - for (my $i = 0 ; $i <= $#args ; $i++) { - my $arg = $args[$i]; - my $isstr = $arg =~ /char\s+\*/; - $arg =~ s/^.*\s\*?(\S+)$/$1/; - - if ($isstr) { - print " $arg = user_string(\$arg", $i + 1, ");\n"; - } else { - print " $arg = \$arg", $i + 1, ";\n"; - } - } - print "}\n\n"; - } - print "\n"; -} diff --git a/src/dtrace2systemtap.py b/src/dtrace2systemtap.py new file mode 100755 index 0000000000..9dc9aaf9e5 --- /dev/null +++ b/src/dtrace2systemtap.py @@ -0,0 +1,140 @@ +#!/usr/bin/env python +# +# Copyright (C) 2011-2019 Red Hat, Inc. +# +# This library is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 2.1 of the License, or (at your option) any later version. +# +# This library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this library. If not, see +# <http://www.gnu.org/licenses/>. +# +# +# Generate a set of systemtap probe definitions corresponding to +# DTrace probe markers in libvirt.so +# +# python dtrace2systemtap.py probes.d > libvirt_probes.stp +# + +from __future__ import print_function + +import re +import sys + +file = None +filelist = [] +files = {} + +bindir = sys.argv[1] +sbindir = sys.argv[2] +libdir = sys.argv[3] +dtrace = sys.argv[4] + +probe = None +args = None + +# Read the DTraceprobes definition + +with open(dtrace, "r") as fh: + lineno = 0 + for line in fh: + lineno = lineno + 1 + line = line.strip() + if line == "": + continue + if line.find("provider ") != -1 and line.find("{") != -1: + continue + if line.find("};") != -1: + continue + + if line.startswith("#"): + m = re.match(r'''^\#\s*file:\s*(\S+)$''', line) + if m is not None: + file = m.group(1) + filelist.append(file) + files[file] = { "prefix": None, "probes": [] } + continue + + m = re.match(r'''^\#\s*prefix:\s*(\S+)$''', line) + if m is not None: + files[file]["prefix"] = m.group(1) + continue + + m = re.match(r'''^\#\s*binary:\s*(\S+)$''', line) + if m is not None: + files[file]["binary"] = m.group(1) + continue + + m = re.match(r'''^\#\s*module:\s*(\S+)$''', line) + if m is not None: + files[file]["module"] = m.group(1) + + # ignore unknown comments + else: + m = re.match(r'''probe\s+([a-zA-Z0-9_]+)\((.*?)(\);)?$''', line) + if m is not None: + probe = m.group(1) + args = m.group(2) + if m.group(3) is not None: + files[file]["probes"].append([probe, args]) + probe = None + args = None + elif probe is not None: + m = re.match(r'''^(.*?)(\);)?$''', line) + if m is not None: + args = args + m.group(1) + if m.group(2) is not None: + files[file]["probes"].append([probe, args]) + probe = None + args = None + else: + raise Exception("unexpected data %s on line %d" % (line, lineno)) + else: + raise Exception("unexpected data %s on line %d" % (line, lineno)) + +# Write out the SystemTap probes +for file in filelist: + prefix = files[file]["prefix"] + probes = files[file]["probes"] + + print("# %s\n" % file) + for probe in probes: + name = probe[0] + args = probe[1] + + pname = name.replace(prefix + "_", "libvirt." + prefix + ".") + + binary = libdir + "/libvirt.so" + if "binary" in files[file]: + binary = sbindir + "/" + files[file]["binary"] + if "module" in files[file]: + binary = libdir + "/" + files[file]["module"] + + print("probe %s = process(\"%s\").mark(\"%s\") {" % (pname, binary, name)) + + argbits = args.split(",") + for idx in range(len(argbits)): + arg = argbits[idx] + isstr = False + if arg.find("char *") != -1: + isstr = True + + m = re.match(r'''^.*\s\*?(\S+)$''', arg) + if m is not None: + arg = m.group(1) + else: + raise Exception("Malformed arg %s" % arg) + + if isstr: + print(" %s = user_string($arg%d);" % (arg, idx + 1)) + else: + print(" %s = $arg%d;" % (arg, idx + 1)) + print("}\n") + print("") -- 2.21.0

As part of an goal to eliminate Perl from libvirt build tools, rewrite the gensystemtap.pl tool in Python. This was a straight conversion, manually going line-by-line to change the syntax from Perl to Python. Thus the overall structure of the file and approach is the same. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- cfg.mk | 4 +- src/Makefile.am | 5 +- src/rpc/Makefile.inc.am | 2 +- src/rpc/gensystemtap.pl | 193 ---------------------------------------- src/rpc/gensystemtap.py | 174 ++++++++++++++++++++++++++++++++++++ 5 files changed, 179 insertions(+), 199 deletions(-) delete mode 100755 src/rpc/gensystemtap.pl create mode 100755 src/rpc/gensystemtap.py diff --git a/cfg.mk b/cfg.mk index 3e63fee9ff..45b16ee8c3 100644 --- a/cfg.mk +++ b/cfg.mk @@ -407,6 +407,7 @@ sc_prohibit_risky_id_promotion: # since gnulib has more guarantees for snprintf portability sc_prohibit_sprintf: @prohibit='\<[s]printf\>' \ + in_vc_files='\.[ch]$$' \ halt='use snprintf, not sprintf' \ $(_sc_search_regexp) @@ -1267,9 +1268,6 @@ exclude_file_name_regexp--sc_prohibit_readlink = \ exclude_file_name_regexp--sc_prohibit_setuid = ^src/util/virutil\.c|tools/virt-login-shell\.c$$ -exclude_file_name_regexp--sc_prohibit_sprintf = \ - ^(cfg\.mk|docs/hacking\.html\.in|.*\.stp|.*\.pl)$$ - exclude_file_name_regexp--sc_prohibit_strncpy = ^src/util/virstring\.c$$ exclude_file_name_regexp--sc_prohibit_strtol = ^examples/.*$$ diff --git a/src/Makefile.am b/src/Makefile.am index 8b046cafe4..539d8b0f58 100644 --- a/src/Makefile.am +++ b/src/Makefile.am @@ -636,8 +636,9 @@ RPC_PROBE_FILES += $(srcdir)/rpc/virnetprotocol.x \ $(srcdir)/remote/qemu_protocol.x \ $(srcdir)/admin/admin_protocol.x -libvirt_functions.stp: $(RPC_PROBE_FILES) $(srcdir)/rpc/gensystemtap.pl - $(AM_V_GEN)$(PERL) -w $(srcdir)/rpc/gensystemtap.pl $(RPC_PROBE_FILES) > $@ +libvirt_functions.stp: $(RPC_PROBE_FILES) $(srcdir)/rpc/gensystemtap.py + $(AM_V_GEN)$(RUNUTF8) $(PYTHON) $(srcdir)/rpc/gensystemtap.py \ + $(RPC_PROBE_FILES) > $@ %_probes.stp: %_probes.d $(srcdir)/dtrace2systemtap.py \ $(top_builddir)/config.status diff --git a/src/rpc/Makefile.inc.am b/src/rpc/Makefile.inc.am index b8ca53c69a..b43ee4ab7a 100644 --- a/src/rpc/Makefile.inc.am +++ b/src/rpc/Makefile.inc.am @@ -3,7 +3,7 @@ EXTRA_DIST += \ rpc/gendispatch.pl \ rpc/genprotocol.pl \ - rpc/gensystemtap.pl \ + rpc/gensystemtap.py \ rpc/virnetprotocol.x \ rpc/virkeepaliveprotocol.x \ $(NULL) diff --git a/src/rpc/gensystemtap.pl b/src/rpc/gensystemtap.pl deleted file mode 100755 index 6693d4d6f5..0000000000 --- a/src/rpc/gensystemtap.pl +++ /dev/null @@ -1,193 +0,0 @@ -#!/usr/bin/env perl -# -# Copyright (C) 2011-2012 Red Hat, Inc. -# -# This library is free software; you can redistribute it and/or -# modify it under the terms of the GNU Lesser General Public -# License as published by the Free Software Foundation; either -# version 2.1 of the License, or (at your option) any later version. -# -# This library is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# Lesser General Public License for more details. -# -# You should have received a copy of the GNU Lesser General Public -# License along with this library. If not, see -# <http://www.gnu.org/licenses/>. -# -# Generate a set of systemtap functions for translating various -# RPC enum values into strings -# -# perl gensystemtap.pl */*.x > libvirt_functions.stp -# - -use strict; - -my %funcs; - -my %type; -my %status; -my %auth; - -my $instatus = 0; -my $intype = 0; -my $inauth = 0; -while (<>) { - if (/enum\s+virNetMessageType/) { - $intype = 1; - } elsif (/enum\s+virNetMessageStatus/) { - $instatus = 1; - } elsif (/enum remote_auth_type/) { - $inauth = 1; - } elsif (/}/) { - $instatus = $intype = $inauth = 0; - } elsif ($instatus) { - if (/^\s+VIR_NET_(\w+)\s*=\s*(\d+),?$/) { - $status{$2} = lc $1; - } - } elsif ($intype) { - if (/^\s+VIR_NET_(\w+)\s*=\s*(\d+),?$/) { - $type{$2} = lc $1; - } - } elsif ($inauth) { - if (/^\s+REMOTE_AUTH_(\w+)\s*=\s*(\d+),?$/) { - $auth{$2} = lc $1; - } - } else { - if (/(?:VIR_)?(\w+?)(?:_PROTOCOL)?_PROGRAM\s*=\s*0x([a-fA-F0-9]+)\s*;/) { - $funcs{lc $1} = { id => hex($2), version => undef, progs => [] }; - } elsif (/(?:VIR_)?(\w+?)(?:_PROTOCOL)?_(?:PROGRAM|PROTOCOL)_VERSION\s*=\s*(\d+)\s*;/) { - $funcs{lc $1}->{version} = $2; - } elsif (/(?:VIR_)?(\w+?)(?:_PROTOCOL)?_PROC_(.*?)\s+=\s+(\d+)/) { - $funcs{lc $1}->{progs}->[$3] = lc $2; - } - } -} - -print <<EOF; -function libvirt_rpc_auth_name(type, verbose) -{ -EOF -my $first = 1; -foreach my $type (sort(keys %auth)) { - my $cond = $first ? "if" : "} else if"; - $first = 0; - print " $cond (type == ", $type, ") {\n"; - print " typestr = \"", $auth{$type}, "\"\n"; -} -print <<EOF; - } else { - typestr = "unknown"; - verbose = 1; - } - if (verbose) { - typestr = typestr . sprintf(":%d", type) - } - return typestr; -} -EOF - -print <<EOF; -function libvirt_rpc_type_name(type, verbose) -{ -EOF -$first = 1; -foreach my $type (sort(keys %type)) { - my $cond = $first ? "if" : "} else if"; - $first = 0; - print " $cond (type == ", $type, ") {\n"; - print " typestr = \"", $type{$type}, "\"\n"; -} -print <<EOF; - } else { - typestr = "unknown"; - verbose = 1; - } - if (verbose) { - typestr = typestr . sprintf(":%d", type) - } - return typestr; -} -EOF - -print <<EOF; -function libvirt_rpc_status_name(status, verbose) -{ -EOF -$first = 1; -foreach my $status (sort(keys %status)) { - my $cond = $first ? "if" : "} else if"; - $first = 0; - print " $cond (status == ", $status, ") {\n"; - print " statusstr = \"", $status{$status}, "\"\n"; -} -print <<EOF; - } else { - statusstr = "unknown"; - verbose = 1; - } - if (verbose) { - statusstr = statusstr . sprintf(":%d", status) - } - return statusstr; -} -EOF - -print <<EOF; -function libvirt_rpc_program_name(program, verbose) -{ -EOF -$first = 1; -foreach my $prog (sort(keys %funcs)) { - my $cond = $first ? "if" : "} else if"; - $first = 0; - print " $cond (program == ", $funcs{$prog}->{id}, ") {\n"; - print " programstr = \"", $prog, "\"\n"; -} -print <<EOF; - } else { - programstr = "unknown"; - verbose = 1; - } - if (verbose) { - programstr = programstr . sprintf(":%d", program) - } - return programstr; -} -EOF - - -print <<EOF; -function libvirt_rpc_procedure_name(program, version, proc, verbose) -{ -EOF -$first = 1; -foreach my $prog (sort(keys %funcs)) { - my $cond = $first ? "if" : "} else if"; - $first = 0; - print " $cond (program == ", $funcs{$prog}->{id}, " && version == ", $funcs{$prog}->{version}, ") {\n"; - - my $pfirst = 1; - for (my $id = 1 ; $id <= $#{$funcs{$prog}->{progs}} ; $id++) { - my $cond = $pfirst ? "if" : "} else if"; - $pfirst = 0; - print " $cond (proc == $id) {\n"; - print " procstr = \"", $funcs{$prog}->{progs}->[$id], "\";\n"; - } - print " } else {\n"; - print " procstr = \"unknown\";\n"; - print " verbose = 1;\n"; - print " }\n"; -} -print <<EOF; - } else { - procstr = "unknown"; - verbose = 1; - } - if (verbose) { - procstr = procstr . sprintf(":%d", proc) - } - return procstr; -} -EOF diff --git a/src/rpc/gensystemtap.py b/src/rpc/gensystemtap.py new file mode 100755 index 0000000000..4013c0969f --- /dev/null +++ b/src/rpc/gensystemtap.py @@ -0,0 +1,174 @@ +#!/usr/bin/env python +# +# Copyright (C) 2011-2019 Red Hat, Inc. +# +# This library is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 2.1 of the License, or (at your option) any later version. +# +# This library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this library. If not, see +# <http://www.gnu.org/licenses/>. +# +# Generate a set of systemtap functions for translating various +# RPC enum values into strings +# +# python gensystemtap.py */*.x > libvirt_functions.stp +# + +from __future__ import print_function + +import re +import sys + +funcs = {} + +types = {} +status = {} +auth = {} + +def load_file(fh): + instatus = False + intype = False + inauth = False + + for line in fh: + if re.match(r'''\s*enum\s+virNetMessageType\s*''', line): + intype = True + elif re.match(r'''\s*enum\s+virNetMessageStatus\s*''', line): + instatus = True + elif re.match(r'''\s*enum remote_auth_type.*''', line): + inauth = True + elif line.find("}") != -1: + intype = False + instatus = False + inauth = False + elif instatus: + m = re.match(r'''^\s+VIR_NET_(\w+)\s*=\s*(\d+),?$''', line) + if m is not None: + status[m.group(2)] = m.group(1).lower() + elif intype: + m = re.match(r'''^\s+VIR_NET_(\w+)\s*=\s*(\d+),?$''', line) + if m is not None: + types[m.group(2)] = m.group(1).lower() + elif inauth: + m = re.match(r'''^\s+REMOTE_AUTH_(\w+)\s*=\s*(\d+),?$''', line) + if m is not None: + auth[m.group(2)] = m.group(1).lower() + else: + m = re.match(r'''^.*?(?:VIR_)?(\w+?)(?:_PROTOCOL)?_PROGRAM\s*=\s*0x([a-fA-F0-9]+)\s*;''', line) + if m is not None: + funcs[m.group(1).lower()] = { + "id": int(m.group(2), 16), + "version": None, + "progs": [] + } + continue + + m = re.match(r'''^.*?(?:VIR_)?(\w+?)(?:_PROTOCOL)?_(?:PROGRAM|PROTOCOL)_VERSION\s*=\s*(\d+)\s*;''', line) + if m is not None: + funcs[m.group(1).lower()]["version"] = m.group(2) + continue + + m = re.match(r'''^.*?(?:VIR_)?(\w+?)(?:_PROTOCOL)?_PROC_(.*?)\s+=\s+(\d+)''', line) + if m is not None: + funcs[m.group(1).lower()]["progs"].insert(int(m.group(3)), m.group(2).lower()) + +for file in sys.argv[1:]: + with open(file, "r") as fh: + load_file(fh) + + +def genfunc(name, varname, types): + print("function %s(%s, verbose)" % (name, varname)) + print("{") + + first = True + for typename in sorted(types.keys()): + cond = "} else if" + if first: + cond = "if" + first = False + + print(" %s (%s == %s) {" % (cond, varname, typename)) + print(" %sstr = \"%s\"" % (varname, types[typename])) + + print(" } else {") + print(" %sstr = \"unknown\";" % varname) + print(" verbose = 1;") + print(" }") + print(" if (verbose) {") + print(" %sstr = %sstr . sprintf(\":%%d\", %s)" % (varname, varname, varname)) + print(" }") + print(" return %sstr;" % varname) + print("}") + +genfunc("libvirt_rpc_auth_name", "type", auth) +genfunc("libvirt_rpc_type_name", "type", types) +genfunc("libvirt_rpc_status_name", "status", status) + +print("function libvirt_rpc_program_name(program, verbose)") +print("{") + +first = True +for funcname in sorted(funcs.keys()): + cond = "} else if" + if first: + cond = "if" + first = False + + print(" %s (program == %s) {" % (cond, funcs[funcname]["id"])) + print(" programstr = \"%s\"" % funcname) + +print(" } else {") +print(" programstr = \"unknown\";") +print(" verbose = 1;") +print(" }") +print(" if (verbose) {") +print(" programstr = programstr . sprintf(\":%d\", program)") +print(" }") +print(" return programstr;") +print("}") + +print("function libvirt_rpc_procedure_name(program, version, proc, verbose)") +print("{") + +first = True +for prog in sorted(funcs.keys()): + cond = "} else if" + if first: + cond = "if" + first = False + + print(" %s (program == %s && version == %s) {" % (cond, funcs[prog]["id"], funcs[prog]["version"])) + + pfirst = True + for id in range(len(funcs[prog]["progs"])): + pcond = "} else if" + if pfirst: + pcond = "if" + pfirst = False + + print(" %s (proc == %s) {" % (pcond, id + 1)) + print(" procstr = \"%s\";" % funcs[prog]["progs"][id]) + + print(" } else {") + print(" procstr = \"unknown\";") + print(" verbose = 1;") + print(" }") + +print(" } else {") +print(" procstr = \"unknown\";") +print(" verbose = 1;") +print(" }") +print(" if (verbose) {") +print(" procstr = procstr . sprintf(\":%d\", proc)") +print(" }") +print(" return procstr;") +print("}") -- 2.21.0

As part of an goal to eliminate Perl from libvirt build tools, rewrite the check-drivername.pl tool in Python. This was mostly a straight conversion, manually going line-by-line to change the syntax from Perl to Python. Thus the overall structure of the file and approach is the same. In testing though it was discovered the existing code was broken since it hadn't been updated after driver.h was split into many files. Since the old code is being thrown away, the fix was done as part of the rewrite rather than split into a separate commit. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- src/Makefile.am | 18 +++++-- src/check-drivername.pl | 83 ----------------------------- src/check-drivername.py | 112 ++++++++++++++++++++++++++++++++++++++++ 3 files changed, 127 insertions(+), 86 deletions(-) delete mode 100755 src/check-drivername.pl create mode 100644 src/check-drivername.py diff --git a/src/Makefile.am b/src/Makefile.am index 539d8b0f58..58d228eb4f 100644 --- a/src/Makefile.am +++ b/src/Makefile.am @@ -336,15 +336,27 @@ check-protocol: endif !WITH_REMOTE EXTRA_DIST += $(PROTOCOL_STRUCTS) +DRIVERS = \ + $(srcdir)/driver-hypervisor.h \ + $(srcdir)/driver-interface.h \ + $(srcdir)/driver-network.h \ + $(srcdir)/driver-nodedev.h \ + $(srcdir)/driver-nwfilter.h \ + $(srcdir)/driver-secret.h \ + $(srcdir)/driver-state.h \ + $(srcdir)/driver-storage.h \ + $(srcdir)/driver-stream.h \ + $(NULL) + check-drivername: - $(AM_V_GEN)$(PERL) $(srcdir)/check-drivername.pl \ - $(srcdir)/driver.h \ + $(AM_V_GEN)$(RUNUTF8) $(PYTHON) $(srcdir)/check-drivername.py \ + $(DRIVERS) \ $(srcdir)/libvirt_public.syms \ $(srcdir)/libvirt_admin_public.syms \ $(srcdir)/libvirt_qemu.syms \ $(srcdir)/libvirt_lxc.syms -EXTRA_DIST += check-drivername.pl +EXTRA_DIST += check-drivername.py check-driverimpls: $(AM_V_GEN)$(PERL) $(srcdir)/check-driverimpls.pl \ diff --git a/src/check-drivername.pl b/src/check-drivername.pl deleted file mode 100755 index 3a62193e33..0000000000 --- a/src/check-drivername.pl +++ /dev/null @@ -1,83 +0,0 @@ -#!/usr/bin/env perl -# -# Copyright (C) 2013 Red Hat, Inc. -# -# This library is free software; you can redistribute it and/or -# modify it under the terms of the GNU Lesser General Public -# License as published by the Free Software Foundation; either -# version 2.1 of the License, or (at your option) any later version. -# -# This library is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# Lesser General Public License for more details. -# -# You should have received a copy of the GNU Lesser General Public -# License along with this library. If not, see -# <http://www.gnu.org/licenses/>. -# - -use strict; -use warnings; - -my $drvfile = shift; -my @symfiles = @ARGV; - -my %symbols; - -foreach my $symfile (@symfiles) { - open SYMFILE, "<", $symfile - or die "cannot read $symfile: $!"; - while (<SYMFILE>) { - if (/^\s*(vir\w+)\s*;\s*$/) { - $symbols{$1} = 1; - } - } - - close SYMFILE; -} - -open DRVFILE, "<", $drvfile - or die "cannot read $drvfile: $!"; - -my $status = 0; - -while (<DRVFILE>) { - next if /virDrvConnectSupportsFeature/; - if (/\*(virDrv\w+)\s*\)/) { - - my $drv = $1; - - next if $drv =~ /virDrvState/; - next if $drv =~ /virDrvDomainMigrate(Prepare|Perform|Confirm|Begin|Finish)/; - - my $sym = $drv; - $sym =~ s/virDrv/vir/; - - unless (exists $symbols{$sym}) { - print "Driver method name $drv doesn't match public API name\n"; - $status = 1; - } - } elsif (/^\*(vir\w+)\s*\)/) { - my $name = $1; - print "Bogus name $1\n"; - $status = 1; - } elsif (/^\s*(virDrv\w+)\s+(\w+);\s*/) { - my $drv = $1; - my $field = $2; - - my $tmp = $drv; - $tmp =~ s/virDrv//; - $tmp =~ s/^NWFilter/nwfilter/; - $tmp =~ s/^(\w)/lc $1/e; - - unless ($tmp eq $field) { - print "Driver struct field $field should be named $tmp\n"; - $status = 1; - } - } -} - -close DRVFILE; - -exit $status; diff --git a/src/check-drivername.py b/src/check-drivername.py new file mode 100644 index 0000000000..e87202be6d --- /dev/null +++ b/src/check-drivername.py @@ -0,0 +1,112 @@ +#!/usr/bin/env python +# +# Copyright (C) 2013-2019 Red Hat, Inc. +# +# This library is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 2.1 of the License, or (at your option) any later version. +# +# This library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this library. If not, see +# <http://www.gnu.org/licenses/>. +# + +from __future__ import print_function + +import re +import sys + +drvfiles = [] +symfiles = [] +for arg in sys.argv: + if arg.endswith(".h"): + drvfiles.append(arg) + else: + symfiles.append(arg) + +symbols = {} + +for symfile in symfiles: + with open(symfile, "r") as fh: + for line in fh: + m = re.match(r'''^\s*(vir\w+)\s*;\s*$''', line) + if m is not None: + symbols[m.group(1)] = True + +status = 0 +for drvfile in drvfiles: + with open(drvfile, "r") as fh: + for line in fh: + if line.find("virDrvConnectSupportsFeature") != -1: + continue + + m = re.match(r'''.*\*(virDrv\w+)\s*\).*''', line) + if m is not None: + drv = m.group(1) + + skip = [ + "virDrvStateInitialize", + "virDrvStateCleanup", + "virDrvStateReload", + "virDrvStateStop", + "virDrvConnectURIProbe", + "virDrvDomainMigratePrepare", + "virDrvDomainMigratePrepare2", + "virDrvDomainMigratePrepare3", + "virDrvDomainMigratePrepare3Params", + "virDrvDomainMigratePrepareTunnel", + "virDrvDomainMigratePrepareTunnelParams", + "virDrvDomainMigratePrepareTunnel3", + "virDrvDomainMigratePrepareTunnel3Params", + "virDrvDomainMigratePerform", + "virDrvDomainMigratePerform3", + "virDrvDomainMigratePerform3Params", + "virDrvDomainMigrateConfirm", + "virDrvDomainMigrateConfirm3", + "virDrvDomainMigrateConfirm3Params", + "virDrvDomainMigrateBegin", + "virDrvDomainMigrateBegin3", + "virDrvDomainMigrateBegin3Params", + "virDrvDomainMigrateFinish", + "virDrvDomainMigrateFinish2", + "virDrvDomainMigrateFinish3", + "virDrvDomainMigrateFinish3Params", + "virDrvStreamInData", + ] + if drv in skip: + continue + + sym = drv.replace("virDrv", "vir") + + if sym not in symbols: + print("Driver method name %s doesn't match public API name" % drv) + continue + + m = re.match(r'''^\*(vir\w+)\s*\)''', line) + if m is not None: + name = m.group(1) + print("Bogus name %s" % name) + status = 1 + continue + + m = re.match(r'''^\s*(virDrv\w+)\s+(\w+);\s*''', line) + if m is not None: + drv = m.group(1) + field = m.group(2) + + tmp = drv.replace("virDrv", "") + if tmp.startswith("NWFilter"): + tmp = "nwfilter" + tmp[8:] + tmp = tmp[0:1].lower() + tmp[1:] + + if tmp != field: + print("Driver struct field %s should be named %s" % (field, tmp)) + status = 1 + +sys.exit(status) -- 2.21.0

As part of an goal to eliminate Perl from libvirt build tools, rewrite the check-driverimpls.pl tool in Python. This was a straight conversion, manually going line-by-line to change the syntax from Perl to Python. Thus the overall structure of the file and approach is the same. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- src/Makefile.am | 4 +- src/check-driverimpls.pl | 80 --------------------------------- src/check-driverimpls.py | 96 ++++++++++++++++++++++++++++++++++++++++ 3 files changed, 98 insertions(+), 82 deletions(-) delete mode 100755 src/check-driverimpls.pl create mode 100755 src/check-driverimpls.py diff --git a/src/Makefile.am b/src/Makefile.am index 58d228eb4f..17d8885fe8 100644 --- a/src/Makefile.am +++ b/src/Makefile.am @@ -359,7 +359,7 @@ check-drivername: EXTRA_DIST += check-drivername.py check-driverimpls: - $(AM_V_GEN)$(PERL) $(srcdir)/check-driverimpls.pl \ + $(AM_V_GEN)$(RUNUTF8) $(PYTHON) $(srcdir)/check-driverimpls.py \ $(filter /%,$(DRIVER_SOURCE_FILES)) \ $(filter $(srcdir)/%,$(DRIVER_SOURCE_FILES)) \ $(addprefix $(srcdir)/,$(filter-out $(srcdir)/%, \ @@ -375,7 +375,7 @@ check-aclperms: $(srcdir)/access/viraccessperm.h \ $(srcdir)/access/viraccessperm.c -EXTRA_DIST += check-driverimpls.pl check-aclrules.pl check-aclperms.py +EXTRA_DIST += check-driverimpls.py check-aclrules.pl check-aclperms.py check-local: check-protocol check-symfile check-symsorting \ check-drivername check-driverimpls check-aclrules \ diff --git a/src/check-driverimpls.pl b/src/check-driverimpls.pl deleted file mode 100755 index 3c0d54724c..0000000000 --- a/src/check-driverimpls.pl +++ /dev/null @@ -1,80 +0,0 @@ -#!/usr/bin/env perl -# -# Copyright (C) 2013 Red Hat, Inc. -# -# This library is free software; you can redistribute it and/or -# modify it under the terms of the GNU Lesser General Public -# License as published by the Free Software Foundation; either -# version 2.1 of the License, or (at your option) any later version. -# -# This library is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# Lesser General Public License for more details. -# -# You should have received a copy of the GNU Lesser General Public -# License along with this library. If not, see -# <http://www.gnu.org/licenses/>. -# - -use strict; -use warnings; - -my $intable = 0; -my $table; -my $mainprefix; - -my $status = 0; -while (<>) { - if ($intable) { - if (/}/) { - $intable = 0; - $table = undef; - $mainprefix = undef; - } elsif (/\.(\w+)\s*=\s*(\w+),?/) { - my $api = $1; - my $impl = $2; - - next if $api eq "no"; - next if $api eq "name"; - next if $impl eq "NULL"; - - my $suffix = $impl; - my $prefix = $impl; - $prefix =~ s/^([a-z]+)(.*?)$/$1/; - - if (defined $mainprefix) { - if ($mainprefix ne $prefix) { - print "$ARGV:$. Bad prefix '$prefix' for API '$api', expecting '$mainprefix'\n"; - $status = 1; - } - } else { - $mainprefix = $prefix; - } - - if ($api !~ /^$mainprefix/) { - $suffix =~ s/^[a-z]+//; - $suffix =~ s/^([A-Z]+)/lc $1/e; - } - - if ($api ne $suffix) { - my $want = $api; - $want =~ s/^nwf/NWF/; - if ($api !~ /^$mainprefix/) { - $want =~ s/^([a-z])/uc $1/e; - $want = $mainprefix . $want; - } - print "$ARGV:$. Bad impl name '$impl' for API '$api', expecting '$want'\n"; - $status = 1; - } - } - } elsif (/^(?:static\s+)?(vir(?:\w+)?Driver)\s+(?!.*;)/) { - next if $1 eq "virNWFilterCallbackDriver" || - $1 eq "virNWFilterTechDriver" || - $1 eq "virConnectDriver"; - $intable = 1; - $table = $1; - } -} - -exit $status; diff --git a/src/check-driverimpls.py b/src/check-driverimpls.py new file mode 100755 index 0000000000..b02398276f --- /dev/null +++ b/src/check-driverimpls.py @@ -0,0 +1,96 @@ +#!/usr/bin/env python +# +# Copyright (C) 2013-2019 Red Hat, Inc. +# +# This library is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 2.1 of the License, or (at your option) any later version. +# +# This library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this library. If not, see +# <http://www.gnu.org/licenses/>. +# + +from __future__ import print_function + +import re +import sys + +def checkdriverimpls(filename): + intable = False + table = None + mainprefix = None + + errs = False + with open(filename, "r") as fh: + lineno = 0 + for line in fh: + lineno = lineno + 1 + if intable: + if line.find("}") != -1: + intable = False + table = None + mainprefix = None + continue + + m = re.match(r'''\s*\.(\w+)\s*=\s*(\w+),?\s*/''', line) + if m is not None: + api = m.group(1) + impl = m.group(2) + + if api in ["no", "name"]: + continue + if impl in ["NULL"]: + continue + + suffix = impl + prefix = re.sub(r'''^([a-z]+)(.*?)$''', r'''\1''', impl) + + if mainprefix is not None: + if mainprefix != prefix: + print("%s:%d Bad prefix '%s' for API '%s', expecting '%s'" % ( + filename, lineno, prefix, api, mainprefix), file=sys.stderr) + errs = True + else: + mainprefix = prefix + + if not api.startswith(mainprefix): + suffix = re.sub(r'''^[a-z]+''', "", suffix) + suffix = re.sub(r'''^([A-Z]+)''', lambda m: m.group(1).lower(), suffix) + + if api != suffix: + want = api + if want.startswith("nwf"): + want = "NWF" + want[3:] + + if not api.startswith(mainprefix): + want = re.sub(r'''^([a-z])''', lambda m: m.group(1).upper(), want) + want = mainprefix + want + + print ("%s:%d Bad impl name '%s' for API '%s', expecting '%s'" % ( + filename, lineno, impl, api, want), file=sys.stderr) + errs = True + else: + m = re.match(r'''^(?:static\s+)?(vir(?:\w+)?Driver)\s+(?!.*;)''', line) + if m is not None: + drv = m.group(1) + if drv in ["virNWFilterCallbackDriver", + "virNWFilterTechDriver", + "virConnectDriver"]: + continue + intable = True + table = drv + + return errs + +status = 0 +for filename in sys.argv[1:]: + if checkdriverimpls(filename): + status = 1 +sys.exit(status) -- 2.21.0

As part of an goal to eliminate Perl from libvirt build tools, rewrite the check-aclrules.pl tool in Python. This was a straight conversion, manually going line-by-line to change the syntax from Perl to Python. Thus the overall structure of the file and approach is the same. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- src/Makefile.am | 4 +- src/check-aclrules.pl | 252 ------------------------------------------ src/check-aclrules.py | 244 ++++++++++++++++++++++++++++++++++++++++ 3 files changed, 246 insertions(+), 254 deletions(-) delete mode 100755 src/check-aclrules.pl create mode 100755 src/check-aclrules.py diff --git a/src/Makefile.am b/src/Makefile.am index 17d8885fe8..7319aee3b9 100644 --- a/src/Makefile.am +++ b/src/Makefile.am @@ -366,7 +366,7 @@ check-driverimpls: $(filter-out /%,$(DRIVER_SOURCE_FILES)))) check-aclrules: - $(AM_V_GEN)$(PERL) $(srcdir)/check-aclrules.pl \ + $(AM_V_GEN)$(RUNUTF8) $(PYTHON) $(srcdir)/check-aclrules.py \ $(REMOTE_PROTOCOL) \ $(addprefix $(srcdir)/,$(filter-out /%,$(STATEFUL_DRIVER_SOURCE_FILES))) @@ -375,7 +375,7 @@ check-aclperms: $(srcdir)/access/viraccessperm.h \ $(srcdir)/access/viraccessperm.c -EXTRA_DIST += check-driverimpls.py check-aclrules.pl check-aclperms.py +EXTRA_DIST += check-driverimpls.py check-aclrules.py check-aclperms.py check-local: check-protocol check-symfile check-symsorting \ check-drivername check-driverimpls check-aclrules \ diff --git a/src/check-aclrules.pl b/src/check-aclrules.pl deleted file mode 100755 index 0d4cac17ca..0000000000 --- a/src/check-aclrules.pl +++ /dev/null @@ -1,252 +0,0 @@ -#!/usr/bin/env perl -# -# Copyright (C) 2013-2014 Red Hat, Inc. -# -# This library is free software; you can redistribute it and/or -# modify it under the terms of the GNU Lesser General Public -# License as published by the Free Software Foundation; either -# version 2.1 of the License, or (at your option) any later version. -# -# This library is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# Lesser General Public License for more details. -# -# You should have received a copy of the GNU Lesser General Public -# License along with this library. If not, see -# <http://www.gnu.org/licenses/>. -# -# This script validates that the driver implementation of any -# public APIs contain ACL checks. -# -# As the script reads each source file, it attempts to identify -# top level function names. -# -# When reading the body of the functions, it looks for anything -# that looks like an API called named XXXEnsureACL. It will -# validate that the XXX prefix matches the name of the function -# it occurs in. -# -# When it later finds the virDriverPtr table, for each entry -# point listed, it will validate if there was a previously -# detected EnsureACL call recorded. -# -use strict; -use warnings; - -my $status = 0; - -my $brace = 0; -my $maybefunc; -my $intable = 0; -my $table; - -my %acls; -my %aclfilters; - -my %whitelist = ( - "connectClose" => 1, - "connectIsEncrypted" => 1, - "connectIsSecure" => 1, - "connectIsAlive" => 1, - "networkOpen" => 1, - "networkClose" => 1, - "nwfilterOpen" => 1, - "nwfilterClose" => 1, - "secretOpen" => 1, - "secretClose" => 1, - "storageOpen" => 1, - "storageClose" => 1, - "interfaceOpen" => 1, - "interfaceClose" => 1, - "connectURIProbe" => 1, - "localOnly" => 1, - "domainQemuAttach" => 1, - ); - -# XXX this vzDomainMigrateConfirm3Params looks -# bogus - determine why it doesn't have a valid -# ACL check. -my %implwhitelist = ( - "vzDomainMigrateConfirm3Params" => 1, - ); - -my $lastfile; - -sub fixup_name { - my $name = shift; - - $name =~ s/Nwfilter/NWFilter/; - $name =~ s/Xml$/XML/; - $name =~ s/Uri$/URI/; - $name =~ s/Uuid$/UUID/; - $name =~ s/Id$/ID/; - $name =~ s/Mac$/MAC/; - $name =~ s/Cpu$/CPU/; - $name =~ s/Os$/OS/; - $name =~ s/Nmi$/NMI/; - $name =~ s/Pm/PM/; - $name =~ s/Fstrim$/FSTrim/; - $name =~ s/Scsi/SCSI/; - $name =~ s/Wwn$/WWN/; - - return $name; -} - -sub name_to_ProcName { - my $name = shift; - - my @elems; - if ($name =~ /_/ || (lc $name) eq "open" || (lc $name) eq "close") { - @elems = split /_/, $name; - @elems = map lc, @elems; - @elems = map ucfirst, @elems; - } else { - @elems = $name; - } - @elems = map { fixup_name($_) } @elems; - my $procname = join "", @elems; - - $procname =~ s/^([A-Z])/lc $1/e; - - return $procname; -} - - -my $proto = shift @ARGV; - -open PROTO, "<$proto" or die "cannot read $proto"; - -my %filtered; -my $incomment = 0; -my $filtered = 0; -while (<PROTO>) { - if (m,/\*\*,) { - $incomment = 1; - $filtered = 0; - } elsif ($incomment) { - if (m,\*\s\@aclfilter,) { - $filtered = 1; - } elsif ($filtered && - m,REMOTE_PROC_(.*)\s+=\s*\d+,) { - my $api = name_to_ProcName($1); - # Event filtering is handled in daemon/remote.c instead of drivers - if (! m,_EVENT_REGISTER,) { - $filtered{$api} = 1; - } - $incomment = 0; - } - } -} - -close PROTO; - -while (<>) { - if (!defined $lastfile || - $lastfile ne $ARGV) { - %acls = (); - $brace = 0; - $maybefunc = undef; - $lastfile = $ARGV; - } - if ($brace == 0) { - # Looks for anything which appears to be a function - # body name. Doesn't matter if we pick up bogus stuff - # here, as long as we don't miss valid stuff - if (m,\b(\w+)\(,) { - $maybefunc = $1; - } - } elsif ($brace > 0) { - if (m,(\w+)EnsureACL,) { - # Record the fact that maybefunc contains an - # ACL call, and make sure it is the right call! - my $func = $1; - $func =~ s/^vir//; - if (!defined $maybefunc) { - print "$ARGV:$. Unexpected check '$func' outside function\n"; - $status = 1; - } else { - unless ($maybefunc =~ /$func$/i) { - print "$ARGV:$. Mismatch check 'vir${func}EnsureACL' for function '$maybefunc'\n"; - $status = 1; - } - } - $acls{$maybefunc} = 1; - } elsif (m,(\w+)CheckACL,) { - # Record the fact that maybefunc contains an - # ACL filter call, and make sure it is the right call! - my $func = $1; - $func =~ s/^vir//; - if (!defined $maybefunc) { - print "$ARGV:$. Unexpected check '$func' outside function\n"; - $status = 1; - } else { - unless ($maybefunc =~ /$func$/i) { - print "$ARGV:$. Mismatch check 'vir${func}CheckACL' for function '$maybefunc'\n"; - $status = 1; - } - } - $aclfilters{$maybefunc} = 1; - } elsif (m,\b(\w+)\(,) { - # Handles case where we replaced an API with a new - # one which adds new parameters, and we're left with - # a simple stub calling the new API. - my $callfunc = $1; - if (exists $acls{$callfunc}) { - $acls{$maybefunc} = 1; - } - if (exists $aclfilters{$callfunc}) { - $aclfilters{$maybefunc} = 1; - } - } - } - - # Pass the vir*DriverPtr tables and make sure that - # every func listed there, has an impl which calls - # an ACL function - if ($intable) { - if (/\}/) { - $intable = 0; - $table = undef; - } elsif (/\.(\w+)\s*=\s*(\w+),?/) { - my $api = $1; - my $impl = $2; - - next if $impl eq "NULL"; - - if ($api ne "no" && - $api ne "name" && - $table ne "virStateDriver" && - !exists $acls{$impl} && - !exists $whitelist{$api} && - !exists $implwhitelist{$impl}) { - print "$ARGV:$. Missing ACL check in function '$impl' for '$api'\n"; - $status = 1; - } - - if (exists $filtered{$api} && - !exists $aclfilters{$impl}) { - print "$ARGV:$. Missing ACL filter in function '$impl' for '$api'\n"; - $status = 1; - } - } - } elsif (/^(?:static\s+)?(vir(?:\w+)?Driver)\s+/) { - if ($1 ne "virNWFilterCallbackDriver" && - $1 ne "virNWFilterTechDriver" && - $1 ne "virDomainConfNWFilterDriver") { - $intable = 1; - $table = $1; - } - } - - - my $count; - $count = s/{//g; - $brace += $count; - $count = s/}//g; - $brace -= $count; -} continue { - close ARGV if eof; -} - -exit $status; diff --git a/src/check-aclrules.py b/src/check-aclrules.py new file mode 100755 index 0000000000..0f4c210851 --- /dev/null +++ b/src/check-aclrules.py @@ -0,0 +1,244 @@ +#!/usr/bin/env python +# +# Copyright (C) 2013-2019 Red Hat, Inc. +# +# This library is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 2.1 of the License, or (at your option) any later version. +# +# This library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this library. If not, see +# <http://www.gnu.org/licenses/>. +# +# This script validates that the driver implementation of any +# public APIs contain ACL checks. +# +# As the script reads each source file, it attempts to identify +# top level function names. +# +# When reading the body of the functions, it looks for anything +# that looks like an API called named XXXEnsureACL. It will +# validate that the XXX prefix matches the name of the function +# it occurs in. +# +# When it later finds the virDriverPtr table, for each entry +# point listed, it will validate if there was a previously +# detected EnsureACL call recorded. +# + +from __future__ import print_function + +import re +import sys + +whitelist = { + "connectClose": True, + "connectIsEncrypted": True, + "connectIsSecure": True, + "connectIsAlive": True, + "networkOpen": True, + "networkClose": True, + "nwfilterOpen": True, + "nwfilterClose": True, + "secretOpen": True, + "secretClose": True, + "storageOpen": True, + "storageClose": True, + "interfaceOpen": True, + "interfaceClose": True, + "connectURIProbe": True, + "localOnly": True, + "domainQemuAttach": True, +} + +# XXX this vzDomainMigrateConfirm3Params looks +# bogus - determine why it doesn't have a valid +# ACL check. +implwhitelist = { + "vzDomainMigrateConfirm3Params": True, +} + +lastfile = None + +def fixup_name(name): + name.replace("Nwfilter", "NWFilter") + name.replace("Pm", "PM") + name.replace("Scsi", "SCSI") + if name.endswith("Xml"): + name = name[:-3] + "XML" + elif name.endswith("Uri"): + name = name[:-3] + "URI" + elif name.endswith("Uuid"): + name = name[:-4] + "UUID" + elif name.endswith("Id"): + name = name[:-2] + "ID" + elif name.endswith("Mac"): + name = name[:-3] + "MAC" + elif name.endswith("Cpu"): + name = name[:-3] + "MAC" + elif name.endswith("Os"): + name = name[:-2] + "OS" + elif name.endswith("Nmi"): + name = name[:-3] + "NMI" + elif name.endswith("Fstrim"): + name = name[:-6] + "FSTrim" + elif name.endswith("Wwn"): + name = name[:-3] + "WWN" + + return name + + +def name_to_ProcName(name): + elems = [] + if name.find("_") != -1 or name.lower() in ["open", "close"]: + elems = [n.lower().capitalize() for n in name.split("_")] + else: + elems = [name] + + elems = [fixup_name(n) for n in elems] + procname = "".join(elems) + + return procname[0:1].lower() + procname[1:] + + +proto = sys.argv[1] + +filteredmap = {} +with open(proto, "r") as fh: + incomment = False + filtered = False + + for line in fh: + if line.find("/**") != -1: + incomment = True + filtered = False + elif incomment: + if line.find("* @aclfilter") != -1: + filtered = True + elif filtered: + m = re.match(r'''.*REMOTE_PROC_(.*)\s+=\s*\d+.*''', line) + if m is not None: + api = name_to_ProcName(m.group(1)) + # Event filtering is handled in daemon/remote.c instead of drivers + if line.find("_EVENT_REGISTER") == -1: + filteredmap[api] = True + incomment = False + +def process_file(filename): + brace = 0 + maybefunc = None + intable = False + table = None + + acls = {} + aclfilters = {} + errs = False + with open(filename, "r") as fh: + lineno = 0 + for line in fh: + lineno = lineno + 1 + if brace == 0: + # Looks for anything which appears to be a function + # body name. Doesn't matter if we pick up bogus stuff + # here, as long as we don't miss valid stuff + m = re.match(r'''.*\b(\w+)\(.*''', line) + if m is not None: + maybefunc = m.group(1) + elif brace > 0: + ensureacl = re.match(r'''.*?(\w+)EnsureACL.*''', line) + checkacl = re.match(r'''.*?(\w+)CheckACL.*''', line) + stub = re.match(r'''.*?\b(\w+)\(.*''', line) + if ensureacl is not None: + # Record the fact that maybefunc contains an + # ACL call, and make sure it is the right call! + func = ensureacl.group(1) + if func.startswith("vir"): + func = func[3:] + + if maybefunc is None: + print("%s:%d Unexpected check '%s' outside function" % (filename, lineno, func), file=sys.stderr) + errs = True + else: + if not maybefunc.lower().endswith(func.lower()): + print("%s:%d Mismatch check 'vir%sEnsureACL' for function '%s'" % (filename, lineno, func, maybefunc), file=sys.stderr) + errs = True + acls[maybefunc] = True + elif checkacl: + # Record the fact that maybefunc contains an + # ACL filter call, and make sure it is the right call! + func = checkacl.group(1) + if func.startswith("vir"): + func = func[3:] + + if maybefunc is None: + print("%s:%d Unexpected check '%s' outside function" % (filename, lineno, func), file=sys.stderr) + errs = True + else: + if not maybefunc.lower().endswith(func.lower()): + print("%s:%d Mismatch check 'vir%sEnsureACL' for function '%s'" % (filename, lineno, func, maybefunc), file=sys.stderr) + errs = True + aclfilters[maybefunc] = True + elif stub: + # Handles case where we replaced an API with a new + # one which adds new parameters, and we're left with + # a simple stub calling the new API. + callfunc = stub.group(1) + if callfunc in acls: + acls[maybefunc] = True + + if callfunc in aclfilters: + aclfilters[maybefunc] = True + + # Pass the vir*DriverPtr tables and make sure that + # every func listed there, has an impl which calls + # an ACL function + if intable: + assign = re.match(r'''.*?\.(\w+)\s*=\s*(\w+),?.*''', line) + if line.find("}") != -1: + intable = False + table = None + elif assign is not None: + api = assign.group(1) + impl = assign.group(2) + + if (impl != "NULL" and + api not in ["no", "name"] and + table != "virStateDriver"): + if (impl not in acls and + api not in whitelist and + impl not in implwhitelist): + print("%s:%d Missing ACL check in function '%s' for '%s'" % (filename, lineno, impl, api), file=sys.stderr) + errs = True + + if api in filteredmap and impl not in aclfilters: + print("%s:%d Missing ACL filter in function '%s' for '%s'" % (filename, lineno, impl, api), file=sys.stderr) + errs = True + else: + m = re.match(r'''^(?:static\s+)?(vir(?:\w+)?Driver)\s+''', line) + if m is not None: + name = m.group(1) + if name not in ["virNWFilterCallbackDriver", + "virNWFilterTechDriver", + "virDomainConfNWFilterDriver"]: + intable = True + table = name + + if line.find("{") != -1: + brace = brace + 1 + if line.find("}") != -1: + brace = brace - 1 + + return errs + +status = 0 +for filename in sys.argv[2:]: + if process_file(filename): + status = 1 + +sys.exit(status) -- 2.21.0

As part of an goal to eliminate Perl from libvirt build tools, rewrite the genpolkit.pl tool in Python. This was a straight conversion, manually going line-by-line to change the syntax from Perl to Python. Thus the overall structure of the file and approach is the same. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- src/access/Makefile.inc.am | 6 +- src/access/genpolkit.pl | 119 ------------------------------------- src/access/genpolkit.py | 119 +++++++++++++++++++++++++++++++++++++ 3 files changed, 122 insertions(+), 122 deletions(-) delete mode 100755 src/access/genpolkit.pl create mode 100755 src/access/genpolkit.py diff --git a/src/access/Makefile.inc.am b/src/access/Makefile.inc.am index 4dc742f4e5..5f3648aa6b 100644 --- a/src/access/Makefile.inc.am +++ b/src/access/Makefile.inc.am @@ -44,7 +44,7 @@ GENERATED_SYM_FILES += $(ACCESS_DRIVER_SYM_FILES) EXTRA_DIST += \ $(ACCESS_DRIVER_POLKIT_POLICY) \ - access/genpolkit.pl \ + access/genpolkit.py \ $(NULL) @@ -63,8 +63,8 @@ libvirt_driver_access_la_LIBADD = $(ACCESS_DRIVER_POLKIT_POLICY): $(srcdir)/access/viraccessperm.h \ - $(srcdir)/access/genpolkit.pl Makefile.am - $(AM_V_GEN)$(PERL) $(srcdir)/access/genpolkit.pl < $< > $@ || rm -f $@ + $(srcdir)/access/genpolkit.py Makefile.am + $(AM_V_GEN)$(RUNUTF8) $(PYTHON) $(srcdir)/access/genpolkit.py $< > $@ || rm -f $@ if WITH_POLKIT libvirt_driver_access_la_SOURCES += $(ACCESS_DRIVER_POLKIT_SOURCES) diff --git a/src/access/genpolkit.pl b/src/access/genpolkit.pl deleted file mode 100755 index f8f20caf65..0000000000 --- a/src/access/genpolkit.pl +++ /dev/null @@ -1,119 +0,0 @@ -#!/usr/bin/env perl -# -# Copyright (C) 2012-2013 Red Hat, Inc. -# -# This library is free software; you can redistribute it and/or -# modify it under the terms of the GNU Lesser General Public -# License as published by the Free Software Foundation; either -# version 2.1 of the License, or (at your option) any later version. -# -# This library is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# Lesser General Public License for more details. -# -# You should have received a copy of the GNU Lesser General Public -# License along with this library. If not, see -# <http://www.gnu.org/licenses/>. -# - -use strict; -use warnings; - -my @objects = ( - "CONNECT", "DOMAIN", "INTERFACE", "NETWORK_PORT", - "NETWORK","NODE_DEVICE", "NWFILTER_BINDING", "NWFILTER", - "SECRET", "STORAGE_POOL", "STORAGE_VOL", - ); - -my $objects = join ("|", @objects); - -# Data we're going to be generating looks like this -# -# <policyconfig> -# <action id="org.libvirt.unix.monitor"> -# <description>Monitor local virtualized systems</description> -# <message>System policy prevents monitoring of local virtualized systems</message> -# <defaults> -# <allow_any>yes</allow_any> -# <allow_inactive>yes</allow_inactive> -# <allow_active>yes</allow_active> -# </defaults> -# </action> -# ...more <action> rules... -# </policyconfig> - -my %opts; -my $in_opts = 0; - -my %perms; - -while (<>) { - if ($in_opts) { - if (m,\*/,) { - $in_opts = 0; - } elsif (/\*\s*\@(\w+):\s*(.*?)\s*$/) { - $opts{$1} = $2; - } - } elsif (m,/\*\*,) { - $in_opts = 1; - } elsif (/VIR_ACCESS_PERM_($objects)_((?:\w|_)+),/) { - my $object = lc $1; - my $perm = lc $2; - next if $perm eq "last"; - - $object =~ s/_/-/g; - $perm =~ s/_/-/g; - - $perms{$object} = {} unless exists $perms{$object}; - $perms{$object}->{$perm} = { - desc => $opts{desc}, - message => $opts{message}, - anonymous => $opts{anonymous} - }; - %opts = (); - } -} - -print <<EOF; -<?xml version="1.0" encoding="UTF-8"?> -<!DOCTYPE policyconfig PUBLIC "-//freedesktop//DTD polkit Policy Configuration 1.0//EN" - "http://www.freedesktop.org/software/polkit/policyconfig-1.dtd"> -<policyconfig> - <vendor>Libvirt Project</vendor> - <vendor_url>https://libvirt.org</vendor_url> -EOF - -foreach my $object (sort { $a cmp $b } keys %perms) { - foreach my $perm (sort { $a cmp $b } keys %{$perms{$object}}) { - my $description = $perms{$object}->{$perm}->{desc}; - my $message = $perms{$object}->{$perm}->{message}; - my $anonymous = $perms{$object}->{$perm}->{anonymous}; - - die "missing description for $object.$perm" unless - defined $description; - die "missing message for $object.$perm" unless - defined $message; - - my $allow_any = $anonymous ? "yes" : "no"; - my $allow_inactive = $allow_any; - my $allow_active = $allow_any; - - print <<EOF; - <action id="org.libvirt.api.$object.$perm"> - <description>$description</description> - <message>$message</message> - <defaults> - <allow_any>$allow_any</allow_any> - <allow_inactive>$allow_inactive</allow_inactive> - <allow_active>$allow_active</allow_active> - </defaults> - </action> -EOF - - } -} - -print <<EOF; -</policyconfig> -EOF diff --git a/src/access/genpolkit.py b/src/access/genpolkit.py new file mode 100755 index 0000000000..5a70200f87 --- /dev/null +++ b/src/access/genpolkit.py @@ -0,0 +1,119 @@ +#!/usr/bin/env python +# +# Copyright (C) 2012-2019 Red Hat, Inc. +# +# This library is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 2.1 of the License, or (at your option) any later version. +# +# This library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this library. If not, see +# <http://www.gnu.org/licenses/>. +# + +from __future__ import print_function + +import re +import sys + +objects = [ + "CONNECT", "DOMAIN", "INTERFACE", "NETWORK_PORT", + "NETWORK","NODE_DEVICE", "NWFILTER_BINDING", "NWFILTER", + "SECRET", "STORAGE_POOL", "STORAGE_VOL", +] + +objectstr = "|".join(objects) + +# Data we're going to be generating looks like this +# +# <policyconfig> +# <action id="org.libvirt.unix.monitor"> +# <description>Monitor local virtualized systems</description> +# <message>System policy prevents monitoring of local virtualized systems</message> +# <defaults> +# <allow_any>yes</allow_any> +# <allow_inactive>yes</allow_inactive> +# <allow_active>yes</allow_active> +# </defaults> +# </action> +# ...more <action> rules... +# </policyconfig> + +opts = {} +in_opts = False + +perms = {} + +aclfile = sys.argv[1] +with open(aclfile, "r") as fh: + for line in fh: + if in_opts: + if line.find("*/") != -1: + in_opts = False + else: + m = re.match(r'''.*\*\s*\@(\w+):\s*(.*?)\s*$''', line) + if m is not None: + opts[m.group(1)] = m.group(2) + elif line.find("**") != -1: + in_opts = True + else: + m = re.match(r'''.*VIR_ACCESS_PERM_(%s)_((?:\w|_)+),''' % objectstr, line) + if m is not None: + obj = m.group(1).lower() + perm = m.group(2).lower() + if perm == "last": + continue + + obj = obj.replace("_", "-") + perm = perm.replace("_", "-") + + if obj not in perms: + perms[obj] = {} + perms[obj][perm] = { + "desc": opts.get("desc", None), + "message": opts.get("message", None), + "anonymous": opts.get("anonymous", None), + } + opts = {} + +print('<?xml version="1.0" encoding="UTF-8"?>') +print('<!DOCTYPE policyconfig PUBLIC "-//freedesktop//DTD polkit Policy Configuration 1.0//EN"') +print(' "http://www.freedesktop.org/software/polkit/policyconfig-1.dtd">') +print('<policyconfig>') +print(' <vendor>Libvirt Project</vendor>') +print(' <vendor_url>https://libvirt.org</vendor_url>') + +for obj in sorted(perms.keys()): + for perm in sorted(perms[obj].keys()): + description = perms[obj][perm]["desc"] + message = perms[obj][perm]["message"] + anonymous = perms[obj][perm]["anonymous"] + + if description is None: + raise Exception("missing description for %s.%s" % (obj, perm)) + if message is None: + raise Exception("missing message for %s.%s" % (obj, perm)) + + allow_any = "no" + if anonymous: + allow_any = "yes" + allow_inactive = allow_any + allow_active = allow_any + + print(' <action id="org.libvirt.api.%s.%s">' % (obj, perm)) + print(' <description>%s</description>' % description) + print(' <message>%s</message>' % message) + print(' <defaults>') + print(' <allow_any>%s</allow_any>' % allow_any) + print(' <allow_inactive>%s</allow_inactive>' % allow_inactive) + print(' <allow_active>%s</allow_active>' % allow_active) + print(' </defaults>') + print(' </action>') + +print('</policyconfig>') -- 2.21.0

As part of an goal to eliminate Perl from libvirt build tools, rewrite the pdwtags processing script in Python. The original inline shell and perl code was completely unintelligible. The new python code is a manual conversion that attempts todo basically the same thing. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- cfg.mk | 3 +- src/Makefile.am | 99 ++++---------------------- src/check-remote-protocol.py | 131 +++++++++++++++++++++++++++++++++++ 3 files changed, 148 insertions(+), 85 deletions(-) create mode 100644 src/check-remote-protocol.py diff --git a/cfg.mk b/cfg.mk index 45b16ee8c3..0f5e5b09e7 100644 --- a/cfg.mk +++ b/cfg.mk @@ -353,6 +353,7 @@ sc_prohibit_mkstemp: # access with F_OK or R_OK is okay, though. sc_prohibit_access_xok: @prohibit='access(at)? *\(.*X_OK' \ + in_vc_files='\.[ch]$$' \ halt='use virFileIsExecutable instead of access(,X_OK)' \ $(_sc_search_regexp) @@ -1222,7 +1223,7 @@ exclude_file_name_regexp--sc_prohibit_PATH_MAX = \ ^cfg\.mk$$ exclude_file_name_regexp--sc_prohibit_access_xok = \ - ^(cfg\.mk|src/util/virutil\.c)$$ + ^(src/util/virutil\.c)$$ exclude_file_name_regexp--sc_prohibit_asprintf = \ ^(cfg\.mk|bootstrap.conf$$|examples/|src/util/virstring\.[ch]$$|tests/vircgroupmock\.c|tools/virt-login-shell\.c|tools/nss/libvirt_nss\.c$$) diff --git a/src/Makefile.am b/src/Makefile.am index 7319aee3b9..7c98da34dd 100644 --- a/src/Makefile.am +++ b/src/Makefile.am @@ -193,84 +193,6 @@ DRIVER_SOURCES += \ -# Ensure that we don't change the struct or member names or member ordering -# in remote_protocol.x The embedded perl below needs a few comments, and -# presumes you know what pdwtags output looks like: -# * use -0777 -n to slurp the entire file into $_. -# * the "split" splits on the /* DD */ comments, so that $p iterates -# through the struct definitions. -# * process only "struct remote_..." entries -# * remove comments and preceding TAB throughout -# * remove empty lines throughout -# * remove white space at end of buffer - -# With pdwtags 1.8, --verbose output includes separators like these: -# /* 93 */ -# /* <0> (null):0 */ -# with the second line omitted for intrinsic types. -# Whereas with pdwtags 1.3, they look like this: -# /* <2d2> /usr/include/libio.h:180 */ -# The alternation of the following regexps matches both cases. -r1 = /\* \d+ \*/ -r2 = /\* <[[:xdigit:]]+> \S+:\d+ \*/ -libs_prefix = remote_|qemu_|lxc_|admin_ -other_prefix = keepalive|vir(Net|LockSpace|LXCMonitor) -struct_prefix = ($(libs_prefix)|$(other_prefix)) - -# Depending on configure options, libtool creates one or both of -# remote/{,.libs/}libvirt_driver_remote_la-remote_protocol.o. We want -# the newest of the two, in case configure options changed and a stale -# file is left around from an earlier build. -# The pdwtags output is completely different when building with clang -# which causes the comparison against expected output to fail, so skip -# if using clang as CC. -PDWTAGS = \ - $(AM_V_GEN)if $(CC) -v 2>&1 | grep -q clang; then \ - echo 'WARNING: skipping pdwtags test with Clang' >&2; \ - exit 0; \ - fi; \ - if (pdwtags --help) > /dev/null 2>&1; then \ - o=`ls -t $(<:.lo=.$(OBJEXT)) \ - $(subst /,/.libs/,$(<:.lo=.$(OBJEXT))) \ - 2>/dev/null | sed -n 1p`; \ - test -f "$$o" || { echo ".o for $< not found" >&2; exit 1; }; \ - pdwtags --verbose $$o > $(@F)-t1 2> $(@F)-t2; \ - if test ! -s $(@F)-t1 && test -s $(@F)-t2; then \ - rm -rf $(@F)-t?; \ - echo 'WARNING: pdwtags appears broken; skipping the $@ test' >&2;\ - else \ - $(PERL) -0777 -n \ - -e 'foreach my $$p (split m!\n*(?:$(r1)|$(r2))\n!) {' \ - -e ' if ($$p =~ /^(struct|enum) $(struct_prefix)/ ||' \ - -e ' $$p =~ /^enum \{/) {' \ - -e ' $$p =~ s!\t*/\*.*?\*/!!sg;' \ - -e ' $$p =~ s!\s+\n!\n!sg;' \ - -e ' $$p =~ s!\s+$$!!;' \ - -e ' $$p =~ s!\t! !g;' \ - -e ' print "$$p\n";' \ - -e ' $$n++;' \ - -e ' }' \ - -e '}' \ - -e 'BEGIN {' \ - -e ' print "/* -*- c -*- */\n";' \ - -e '}' \ - -e 'END {' \ - -e ' if ($$n < 1) {' \ - -e ' warn "WARNING: your pdwtags program is too old\n";' \ - -e ' warn "WARNING: skipping the $@ test\n";' \ - -e ' warn "WARNING: install dwarves-1.3 or newer\n";' \ - -e ' exit 8;' \ - -e ' }' \ - -e '}' \ - < $(@F)-t1 > $(@F)-t3; \ - case $$? in 8) rm -f $(@F)-t?; exit 0;; 0) ;; *) exit 1;; esac;\ - diff -u $(@)s $(@F)-t3; st=$$?; rm -f $(@F)-t?; exit $$st; \ - fi; \ - else \ - echo 'WARNING: you lack pdwtags; skipping the $@ test' >&2; \ - echo 'WARNING: install the dwarves package to get pdwtags' >&2; \ - fi - # .libs/libvirt.so is built by libtool as a side-effect of the Makefile # rule for libvirt.la. However, checking symbols relies on Linux ELF layout if WITH_LINUX @@ -307,34 +229,43 @@ PROTOCOL_STRUCTS = \ if WITH_REMOTE check-protocol: $(PROTOCOL_STRUCTS) $(PROTOCOL_STRUCTS:structs=struct) +# Ensure that we don't change the struct or member names or member ordering +# in remote_protocol.x The process-pdwtags.py post-processes output to +# extract the bits we want. + # The .o file that pdwtags parses is created as a side effect of running # libtool; but from make's perspective we depend on the .lo file. $(srcdir)/remote_protocol-struct \ $(srcdir)/qemu_protocol-struct \ $(srcdir)/lxc_protocol-struct: \ $(srcdir)/%-struct: remote/libvirt_driver_remote_la-%.lo - $(PDWTAGS) + $(AM_V_GEN)$(RUNUTF8) $(PYTHON) $(srcdir)/check-remote-protocol.py \ + "$(CC)" "$(OBJEXT)" $< $(@)s $(srcdir)/virnetprotocol-struct $(srcdir)/virkeepaliveprotocol-struct: \ $(srcdir)/%-struct: rpc/libvirt_net_rpc_la-%.lo - $(PDWTAGS) + $(AM_V_GEN)$(RUNUTF8) $(PYTHON) $(srcdir)/check-remote-protocol.py \ + "$(CC)" "$(OBJEXT)" $< $(@)s if WITH_LXC $(srcdir)/lxc_monitor_protocol-struct: \ $(srcdir)/%-struct: lxc/libvirt_driver_lxc_impl_la-%.lo - $(PDWTAGS) + $(AM_V_GEN)$(RUNUTF8) $(PYTHON) $(srcdir)/check-remote-protocol.py \ + "$(CC)" "$(OBJEXT)" $< $(@)s endif WITH_LXC $(srcdir)/lock_protocol-struct: \ $(srcdir)/%-struct: locking/lockd_la-%.lo - $(PDWTAGS) + $(AM_V_GEN)$(RUNUTF8) $(PYTHON) $(srcdir)/check-remote-protocol.py \ + "$(CC)" "$(OBJEXT)" $< $(@)s $(srcdir)/admin_protocol-struct: \ $(srcdir)/%-struct: admin/libvirt_admin_la-%.lo - $(PDWTAGS) + $(AM_V_GEN)$(RUNUTF8) $(PYTHON) $(srcdir)/check-remote-protocol.py \ + "$(CC)" "$(OBJEXT)" $< $(@)s else !WITH_REMOTE # The $(PROTOCOL_STRUCTS) files must live in git, because they cannot be # re-generated when configured --without-remote. check-protocol: endif !WITH_REMOTE -EXTRA_DIST += $(PROTOCOL_STRUCTS) +EXTRA_DIST += $(PROTOCOL_STRUCTS) check-remote-protocol.py DRIVERS = \ $(srcdir)/driver-hypervisor.h \ diff --git a/src/check-remote-protocol.py b/src/check-remote-protocol.py new file mode 100644 index 0000000000..68b0af7c04 --- /dev/null +++ b/src/check-remote-protocol.py @@ -0,0 +1,131 @@ +#!/usr/bin/env python +# +# Copyright (C) 2019 Red Hat, Inc. +# +# This library is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 2.1 of the License, or (at your option) any later version. +# +# This library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this library. If not, see +# <http://www.gnu.org/licenses/>. +# +# This uses pdwtags to check remote protocol defs +# +# * the "split" splits on the /* DD */ comments, so that $p iterates +# through the struct definitions. +# * process only "struct remote_..." entries +# * remove comments and preceding TAB throughout +# * remove empty lines throughout +# * remove white space at end of buffer + +from __future__ import print_function + +import os.path +import re +import subprocess +import sys + +cc = sys.argv[1] +objext = sys.argv[2] +proto_lo = sys.argv[3] +expected = sys.argv[4] + +proto_lo = proto_lo.replace("/", "/.libs/") + +ccargv = cc.split(" ") +ccargv.append("-v") +ccproc = subprocess.Popen(ccargv, stdout=subprocess.PIPE, stderr=subprocess.STDOUT) +out, err = ccproc.communicate() +out = out.decode("utf-8") +if out.find("clang") != -1: + print("WARNING: skipping pdwtags test with Clang", file=sys.stderr) + sys.exit(0) + +def which(program): + import os + def is_exe(fpath): + return os.path.isfile(fpath) and os.access(fpath, os.X_OK) + + fpath, fname = os.path.split(program) + if fpath: + if is_exe(program): + return program + else: + for path in os.environ["PATH"].split(os.pathsep): + exe_file = os.path.join(path, program) + if is_exe(exe_file): + return exe_file + + return None + +pdwtags = which("pdwtags") +if pdwtags is None: + print("WARNING: you lack pdwtags; skipping the protocol test", file=sys.stderr) + print("WARNING: install the dwarves package to get pdwtags", file=sys.stderr) + sys.exit(0) + +proto_o = proto_lo.replace(".lo", ".o") + +if not os.path.exists(proto_o): + raise Exception("Missing %s", proto_o) + +pdwtagsproc = subprocess.Popen(["pdwtags", "--verbose", proto_o], + stdout=subprocess.PIPE, stderr=subprocess.PIPE) +out, err = pdwtagsproc.communicate() +out = out.decode("utf-8") +err = err.decode("utf-8") + +if out == "" and err != "": + print("WARNING: no output, pdwtags appears broken:", file=sys.stderr) + for l in err.strip().split("\n"): + print("WARNING: %s" % l, file=sys.stderr) + print("WARNING: skipping the remote protocol test", file=sys.stderr) + sys.exit(0) + +# With pdwtags 1.8, --verbose output includes separators like these: +# /* 93 */ +# /* <0> (null):0 */ +# with the second line omitted for intrinsic types. +# Whereas with pdwtags 1.3, they look like this: +# /* <2d2> /usr/include/libio.h:180 */ +# The alternation of the following regexps matches both cases. +r1 = r'''/\* \d+ \*/''' +r2 = r'''/\* <[0-9a-fA-F]+> \S+:\d+ \*/''' + +libs_prefix = "remote_|qemu_|lxc_|admin_" +other_prefix = "keepalive|vir(Net|LockSpace|LXCMonitor)" +struct_prefix = "(" + libs_prefix + "|" + other_prefix + ")" + +n = 0 +bits = re.split(r'''\n*(?:%s|%s)\n''' % (r1, r2), out) +actual = ["/* -*- c -*- */"] + +for bit in bits: + if (re.match(r'''^(struct|enum)\s+''' + struct_prefix, bit) or + bit.startswith("enum {")): + bit = re.sub(r'''\t*/\*.*?\*/''', "", bit) + bit = re.sub(r'''\s+\n''', '''\n''', bit) + bit = re.sub(r'''\s+$''', "", bit) + bit = re.sub(r'''\t''', " ", bit) + actual.append(bit) + n = n + 1 + +if n < 1: + print("WARNING: No structs/enums matched. Your", file=sys.stderr) + print("WARNING: pdwtags program is probably too old", file=sys.stderr) + print("WARNING: skipping the remote protocol test", file=sys.stderr) + print("WARNING: install dwarves-1.3 or newer", file=sys.stderr) + sys.exit(8) + +diff = subprocess.Popen(["diff", "-u", expected, "-"], stdin=subprocess.PIPE) +actualstr = "\n".join(actual) + "\n" +diff.communicate(input=actualstr.encode("utf-8")) + +sys.exit(diff.returncode) -- 2.21.0

As part of an goal to eliminate Perl from libvirt build tools, rewrite the test-wrap-argv.pl tool in Python. This was a straight conversion, manually going line-by-line to change the syntax from Perl to Python. Thus the overall structure of the file and approach is the same. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- cfg.mk | 4 +- tests/test-wrap-argv.pl | 174 ---------------------------------------- tests/test-wrap-argv.py | 162 +++++++++++++++++++++++++++++++++++++ tests/testutils.c | 16 ++-- 4 files changed, 172 insertions(+), 184 deletions(-) delete mode 100755 tests/test-wrap-argv.pl create mode 100755 tests/test-wrap-argv.py diff --git a/cfg.mk b/cfg.mk index 0f5e5b09e7..a0fc7a4708 100644 --- a/cfg.mk +++ b/cfg.mk @@ -1166,8 +1166,8 @@ header-ifdef: $(PYTHON) $(top_srcdir)/build-aux/header-ifdef.py test-wrap-argv: - $(AM_V_GEN)$(VC_LIST) | $(GREP) -E '\.(ldargs|args)' | xargs \ - $(PERL) $(top_srcdir)/tests/test-wrap-argv.pl --check + $(AM_V_GEN)$(VC_LIST) | $(GREP) -E '\.(ldargs|args)' | $(RUNUTF8) xargs \ + $(PYTHON) $(top_srcdir)/tests/test-wrap-argv.py --check group-qemu-caps: $(AM_V_GEN)$(PERL) $(top_srcdir)/tests/group-qemu-caps.pl --check $(top_srcdir)/ diff --git a/tests/test-wrap-argv.pl b/tests/test-wrap-argv.pl deleted file mode 100755 index 7867e9d719..0000000000 --- a/tests/test-wrap-argv.pl +++ /dev/null @@ -1,174 +0,0 @@ -#!/usr/bin/env perl -# -# Copyright (C) 2015 Red Hat, Inc. -# -# This library is free software; you can redistribute it and/or -# modify it under the terms of the GNU Lesser General Public -# License as published by the Free Software Foundation; either -# version 2.1 of the License, or (at your option) any later version. -# -# This library is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# Lesser General Public License for more details. -# -# You should have received a copy of the GNU Lesser General Public -# License along with this library. If not, see -# <http://www.gnu.org/licenses/>. -# -# This script is intended to be passed a list of .args files, used -# to store command line ARGV for the test suites. It will reformat -# them such that there is at most one '-param value' on each line -# of the file. Parameter values that are longer than 80 chars will -# also be split. -# -# If --in-place is supplied as the first parameter of this script, -# the files will be changed in place. -# If --check is the first parameter, the script will return -# a non-zero value if a file is not wrapped correctly. -# Otherwise the rewrapped files are printed to the standard output. - -$in_place = 0; -$check = 0; - -if (@ARGV[0] eq "--in-place" or @ARGV[0] eq "-i") { - $in_place = 1; - shift @ARGV; -} elsif (@ARGV[0] eq "--check") { - $check = 1; - shift @ARGV; -} - -$ret = 0; -foreach my $file (@ARGV) { - if (&rewrap($file) < 0) { - $ret = 1; - } -} - -exit $ret; - -sub rewrap { - my $file = shift; - - # Read the original file - open FILE, "<", $file or die "cannot read $file: $!"; - my @orig_lines = <FILE>; - close FILE; - my @lines = @orig_lines; - foreach (@lines) { - # If there is a trailing '\' then kill the new line - if (/\\$/) { - chomp; - $_ =~ s/\\$//; - } - } - - # Skip empty files - return unless @lines; - - # Kill the last new line in the file - chomp @lines[$#lines]; - - # Reconstruct the master data by joining all lines - # and then split again based on the real desired - # newlines - @lines = split /\n/, join('', @lines); - - # Now each @lines represents a single command, we - # can process them - @lines = map { &rewrap_line($_) } @lines; - - if ($in_place) { - open FILE, ">", $file or die "cannot write $file: $!"; - foreach my $line (@lines) { - print FILE $line; - } - close FILE; - } elsif ($check) { - my $nl = join('', @lines); - my $ol = join('', @orig_lines); - unless ($nl eq $ol) { - open DIFF, "| diff -u $file -" or die "cannot run diff: $!"; - print DIFF $nl; - close DIFF; - - print STDERR "Incorrect line wrapping in $file\n"; - print STDERR "Use test-wrap-argv.pl to wrap test data files\n"; - return -1; - } - } else { - foreach my $line (@lines) { - print $line; - } - } - return 0; -} - -sub rewrap_line { - my $line = shift; - my @bits = split / /, join('', $line); - - # @bits contains env vars, then the command line - # and then the arguments - my @env; - my $cmd; - my @args; - - if ($bits[0] !~ /=/) { - $cmd = shift @bits; - } - - foreach my $bit (@bits) { - # If no command is defined yet, we must still - # have env vars - if (!defined $cmd) { - # Look for leading / to indicate command name - if ($bit =~ m,^/,) { - $cmd = $bit; - } else { - push @env, $bit; - } - } else { - # If there's a leading '-' then this is a new - # parameter, otherwise its a value for the prev - # parameter. - if ($bit =~ m,^-,) { - push @args, $bit; - } else { - $args[$#args] .= " " . $bit; - } - } - } - - # We might have to split line argument values... - @args = map { &rewrap_arg($_) } @args; - # Print env + command first - return join(" \\\n", @env, $cmd, @args), "\n"; -} - -sub rewrap_arg { - my $arg = shift; - my @ret; - my $max_len = 78; - - while (length($arg) > $max_len) { - my $split = rindex $arg, ",", $max_len; - if ($split == -1) { - $split = rindex $arg, ":", $max_len; - } - if ($split == -1) { - $split = rindex $arg, " ", $max_len; - } - if ($split == -1) { - warn "cannot find nice place to split '$arg' below 80 chars\n"; - $split = $max_len - 1; - } - $split++; - - push @ret, substr $arg, 0, $split; - $arg = substr $arg, $split; - } - push @ret, $arg; - return join("\\\n", @ret); -} diff --git a/tests/test-wrap-argv.py b/tests/test-wrap-argv.py new file mode 100755 index 0000000000..7221f0c5d5 --- /dev/null +++ b/tests/test-wrap-argv.py @@ -0,0 +1,162 @@ +#!/usr/bin/env python +# +# Copyright (C) 2019 Red Hat, Inc. +# +# This library is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 2.1 of the License, or (at your option) any later version. +# +# This library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this library. If not, see +# <http://www.gnu.org/licenses/>. +# +# This script is intended to be passed a list of .args files, used +# to store command line ARGV for the test suites. It will reformat +# them such that there is at most one '-param value' on each line +# of the file. Parameter values that are longer than 80 chars will +# also be split. +# +# If --in-place is supplied as the first parameter of this script, +# the files will be changed in place. +# If --check is the first parameter, the script will return +# a non-zero value if a file is not wrapped correctly. +# Otherwise the rewrapped files are printed to the standard output. + +from __future__ import print_function + +import argparse +import subprocess +import sys + + +def rewrap_line(line): + bits = line.split(" ") + + # bits contains env vars, then the command line + # and then the arguments + env = [] + cmd = None + args = [] + + if bits[0].find("=") == -1: + cmd = bits[0] + bits = bits[1:] + + for bit in bits: + # If no command is defined yet, we must still + # have env vars + if cmd is None: + # Look for leading / to indicate command name + if bit.startswith("/"): + cmd = bit + else: + env.append(bit) + else: + # If there's a leading '-' then this is a new + # parameter, otherwise its a value for the prev + # parameter. + if bit.startswith("-"): + args.append(bit) + else: + args[-1] = args[-1] + " " + bit + + # We might have to split line argument values... + args = [rewrap_arg(arg) for arg in args] + + # Print env + command first + return " \\\n".join(env + [cmd] + args) + "\n" + + +def rewrap_arg(arg): + ret = [] + max_len = 78 + + while len(arg) > max_len: + split = arg.rfind(",", 0, max_len + 1) + if split == -1: + split = arg.rfind(":", 0, max_len + 1) + if split == -1: + split = arg.rfind(" ", 0, max_len + 1) + if split == -1: + print("cannot find nice place to split '%s' below 80 chars" % arg, file=sys.stderr) + split = max_len - 1 + + split = split + 1 + + ret.append(arg[0:split]) + arg = arg[split:] + + ret.append(arg) + return "\\\n".join(ret) + + +def rewrap(filename, in_place, check): + # Read the original file + with open(filename, 'r') as fh: + orig_lines = [] + for line in fh: + orig_lines.append(line) + + if len(orig_lines) == 0: + return + + lines = [] + for line in orig_lines: + if line.endswith("\\\n"): + line = line[:-2] + lines.append(line) + + # Kill the last new line in the file + lines[-1] = lines[-1].rstrip("\n") + + # Reconstruct the master data by joining all lines + # and then split again based on the real desired + # newlines + lines = "".join(lines).split("\n") + + # Now each 'lines' entry represents a single command, we + # can process them + new_lines = [] + for line in lines: + new_lines.append(rewrap_line(line)) + + if in_place: + with open(filename, "w") as fh: + for line in new_lines: + print(line, file=fh) + elif check: + orig = "".join(orig_lines) + new = "".join(new_lines) + if new != orig: + diff = subprocess.Popen(["diff", "-u", filename, "-"], stdin=subprocess.PIPE) + diff.communicate(input=new.encode('utf-8')) + + print("Incorrect line wrapping in $file", file=sys.stderr) + print("Use test-wrap-argv.py to wrap test data files", file=sys.stderr) + return False + else: + for line in new_lines: + print(line) + + return True + +parser = argparse.ArgumentParser(description='Test arg line wrapper') +parser.add_argument('--in-place', '-i', action="store_true", help='modify files in-place') +parser.add_argument('--check', action="store_true", help='check existing files only') +parser.add_argument('files', nargs="+", help="filenames to check") +args = parser.parse_args() + +errs = False +for filename in args.files: + if not rewrap(filename, args.in_place, args.check): + errs = True + +if errs: + sys.exit(1) +sys.exit(0) diff --git a/tests/testutils.c b/tests/testutils.c index 8b2c51044e..0ed58f3931 100644 --- a/tests/testutils.c +++ b/tests/testutils.c @@ -77,7 +77,7 @@ static size_t testCounter; static virBitmapPtr testBitmap; char *progname; -static char *perl; +static char *python; bool virTestOOMActive(void) { @@ -528,15 +528,15 @@ virTestRewrapFile(const char *filename) virStringHasSuffix(filename, ".ldargs"))) return 0; - if (!perl) { - fprintf(stderr, "cannot rewrap %s: unable to find perl in path", filename); + if (!python) { + fprintf(stderr, "cannot rewrap %s: unable to find python in path", filename); return -1; } - if (virAsprintf(&script, "%s/test-wrap-argv.pl", abs_srcdir) < 0) + if (virAsprintf(&script, "%s/test-wrap-argv.py", abs_srcdir) < 0) goto cleanup; - cmd = virCommandNewArgList(perl, script, "--in-place", filename, NULL); + cmd = virCommandNewArgList(python, script, "--in-place", filename, NULL); if (virCommandRun(cmd, NULL) < 0) goto cleanup; @@ -1088,8 +1088,8 @@ int virTestMain(int argc, } #endif /* TEST_OOM */ - /* Find perl early because some tests override PATH */ - perl = virFindFileInPath("perl"); + /* Find python early because some tests override PATH */ + python = virFindFileInPath("python"); ret = (func)(); @@ -1100,7 +1100,7 @@ int virTestMain(int argc, fprintf(stderr, " %-3zu %s\n", testCounter, ret == 0 ? "OK" : "FAIL"); } virLogReset(); - VIR_FREE(perl); + VIR_FREE(python); return ret; } -- 2.21.0

As part of an goal to eliminate Perl from libvirt build tools, rewrite the group-qemu-caps.pl tool in Python. This was a straight conversion, manually going line-by-line to change the syntax from Perl to Python. Thus the overall structure of the file and approach is the same. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- cfg.mk | 3 +- tests/group-qemu-caps.pl | 124 --------------------------------------- tests/group-qemu-caps.py | 115 ++++++++++++++++++++++++++++++++++++ 3 files changed, 117 insertions(+), 125 deletions(-) delete mode 100755 tests/group-qemu-caps.pl create mode 100755 tests/group-qemu-caps.py diff --git a/cfg.mk b/cfg.mk index a0fc7a4708..241fb4b9d7 100644 --- a/cfg.mk +++ b/cfg.mk @@ -1170,7 +1170,8 @@ test-wrap-argv: $(PYTHON) $(top_srcdir)/tests/test-wrap-argv.py --check group-qemu-caps: - $(AM_V_GEN)$(PERL) $(top_srcdir)/tests/group-qemu-caps.pl --check $(top_srcdir)/ + $(AM_V_GEN)$(RUNUTF8) $(PYTHON) $(top_srcdir)/tests/group-qemu-caps.py \ + --check --prefix $(top_srcdir)/ # sc_po_check can fail if generated files are not built first sc_po_check: \ diff --git a/tests/group-qemu-caps.pl b/tests/group-qemu-caps.pl deleted file mode 100755 index 829e63a562..0000000000 --- a/tests/group-qemu-caps.pl +++ /dev/null @@ -1,124 +0,0 @@ -#!/usr/bin/env perl -# -# This library is free software; you can redistribute it and/or -# modify it under the terms of the GNU Lesser General Public -# License as published by the Free Software Foundation; either -# version 2.1 of the License, or (at your option) any later version. -# -# This library is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# Lesser General Public License for more details. -# -# You should have received a copy of the GNU Lesser General Public -# License along with this library. If not, see -# <http://www.gnu.org/licenses/>. -# -# -# Regroup array values into smaller groups separated by numbered comments. -# -# If --check is the first parameter, the script will return -# a non-zero value if a file is not grouped correctly. -# Otherwise the files are regrouped in place. - -use strict; -use warnings; - -my $check = 0; - -if (defined $ARGV[0] && $ARGV[0] eq "--check") { - $check = 1; - shift @ARGV; -} - -my $prefix = ''; -if (defined $ARGV[0]) { - $prefix = $ARGV[0]; - shift @ARGV; -} - -my $ret = 0; -if (®roup_caps($prefix . 'src/qemu/qemu_capabilities.c', - 'virQEMUCaps grouping marker', - '\);', - 0, - " ") < 0) { - $ret = 1; -} -if (®roup_caps($prefix . 'src/qemu/qemu_capabilities.h', - 'virQEMUCapsFlags grouping marker', - 'QEMU_CAPS_LAST \/\* this must', - 1, - " ") < 0) { - $ret = 1; -} - -exit $ret; - -sub regroup_caps { - my $filename = shift; - my $start_regex = shift; - my $end_regex = shift; - my $trailing_newline = shift; - my $counter_prefix = shift; - my $step = 5; - - open FILE, '<', $filename or die "cannot open $filename: $!"; - my @original = <FILE>; - close FILE; - - my @fixed; - my $game_on = 0; - my $counter = 0; - foreach (@original) { - if ($game_on) { - next if ($_ =~ '/\* [0-9]+ \*/'); - next if (/^\s+$/); - if ($counter % $step == 0) { - if ($counter != 0) { - push @fixed, "\n"; - } - push @fixed, "$counter_prefix/* $counter */\n"; - } - if (!($_ =~ '/\*' && !($_ =~ '\*/'))) { - # count two-line comments as one line - $counter++; - } - } - if (/$start_regex/) { - $game_on = 1; - } elsif ($game_on && $_ =~ /$end_regex/) { - if (($counter -1) % $step == 0) { - pop @fixed; # /* $counter */ - if ($counter != 1) { - pop @fixed; # \n - } - } - if ($trailing_newline) { - push @fixed, "\n"; - } - $game_on = 0; - } - push @fixed, $_; - } - - if ($check) { - my $nl = join('', @fixed); - my $ol = join('', @original); - unless ($nl eq $ol) { - open DIFF, "| diff -u $filename -" or die "cannot run diff: $!"; - print DIFF $nl; - close DIFF; - - print STDERR "Incorrect array grouping in $filename\n"; - print STDERR "Use group-qemu-caps.pl to group long array members\n"; - return -1; - } - } else { - open FILE, '>', $filename or die "cannot open $filename: $!"; - foreach my $line (@fixed) { - print FILE $line; - } - close FILE; - } -} diff --git a/tests/group-qemu-caps.py b/tests/group-qemu-caps.py new file mode 100755 index 0000000000..35f317587c --- /dev/null +++ b/tests/group-qemu-caps.py @@ -0,0 +1,115 @@ +#!/usr/bin/env python +# +# This library is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 2.1 of the License, or (at your option) any later version. +# +# This library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this library. If not, see +# <http://www.gnu.org/licenses/>. +# +# +# Regroup array values into smaller groups separated by numbered comments. +# +# If --check is the first parameter, the script will return +# a non-zero value if a file is not grouped correctly. +# Otherwise the files are regrouped in place. + +from __future__ import print_function + +import argparse +import re +import subprocess +import sys + +def regroup_caps(check, filename, start_regex, end_regex, trailing_newline, counter_prefix): + step = 5 + + original = [] + with open(filename, "r") as fh: + for line in fh: + original.append(line) + + fixed = [] + game_on = False + counter = 0 + for line in original: + line = line.rstrip("\n") + if game_on: + if re.match(r'''.*/\* [0-9]+ \*/.*''', line): + continue + if re.match(r'''^\s*$''', line): + continue + if counter % step == 0: + if counter != 0: + fixed.append("\n") + fixed.append("%s/* %d */\n" % (counter_prefix, counter)) + + if not (line.find("/*") != -1 and line.find("*/") == -1): + # count two-line comments as one line + counter = counter + 1 + + if re.match(start_regex, line): + game_on = True + elif game_on and re.match(end_regex, line): + if (counter - 1) % step == 0: + fixed = fixed[:-1] # /* $counter */ + if counter != 1: + fixed = fixed[:-1] # \n + + if trailing_newline: + fixed.append("\n") + + game_on = False + + fixed.append(line + "\n") + + if check: + orig = "".join(original) + new = "".join(fixed) + if new != orig: + diff = subprocess.Popen(["diff", "-u", filename, "-"], stdin=subprocess.PIPE) + diff.communicate(input=new.encode('utf-8')) + + print("Incorrect line wrapping in $file", file=sys.stderr) + print("Use test-wrap-argv.py to wrap test data files", file=sys.stderr) + return False + else: + with open(filename, "w") as fh: + for line in fixed: + print(line, file=fh, end='') + + return True + +parser = argparse.ArgumentParser(description='Test arg line wrapper') +parser.add_argument('--check', action="store_true", help='check existing files only') +parser.add_argument('--prefix', default='', help='source code tree prefix') +args = parser.parse_args() + +errs = False + +if not regroup_caps(args.check, + args.prefix + 'src/qemu/qemu_capabilities.c', + r'.*virQEMUCaps grouping marker.*', + r'.*\);.*', + 0, + " "): + errs = True + +if not regroup_caps(args.check, + args.prefix + 'src/qemu/qemu_capabilities.h', + r'.*virQEMUCapsFlags grouping marker.*', + r'.*QEMU_CAPS_LAST \/\* this must.*', + 1, + " "): + errs = True + +if errs: + sys.exit(1) +sys.exit(0) -- 2.21.0

As part of an goal to eliminate Perl from libvirt build tools, rewrite the check-file-access.pl tool in Python. This was a straight conversion, manually going line-by-line to change the syntax from Perl to Python. Thus the overall structure of the file and approach is the same. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- tests/Makefile.am | 4 +- tests/check-file-access.pl | 126 -------------------------------- tests/check-file-access.py | 121 ++++++++++++++++++++++++++++++ tests/file_access_whitelist.txt | 2 +- 4 files changed, 124 insertions(+), 129 deletions(-) delete mode 100755 tests/check-file-access.pl create mode 100755 tests/check-file-access.py diff --git a/tests/Makefile.am b/tests/Makefile.am index f92710db43..48d1924707 100644 --- a/tests/Makefile.am +++ b/tests/Makefile.am @@ -450,14 +450,14 @@ EXTRA_DIST += $(test_scripts) if WITH_LINUX check-access: file-access-clean VIR_TEST_FILE_ACCESS=1 $(MAKE) $(AM_MAKEFLAGS) check - $(PERL) check-file-access.pl | sort -u + $(RUNUTF8) $(PYTHON) check-file-access.py | sort -u file-access-clean: > test_file_access.txt endif WITH_LINUX EXTRA_DIST += \ - check-file-access.pl \ + check-file-access.py \ file_access_whitelist.txt if WITH_TESTS diff --git a/tests/check-file-access.pl b/tests/check-file-access.pl deleted file mode 100755 index ea0b7a18a2..0000000000 --- a/tests/check-file-access.pl +++ /dev/null @@ -1,126 +0,0 @@ -#!/usr/bin/env perl -# -# Copyright (C) 2016 Red Hat, Inc. -# -# This library is free software; you can redistribute it and/or -# modify it under the terms of the GNU Lesser General Public -# License as published by the Free Software Foundation; either -# version 2.1 of the License, or (at your option) any later version. -# -# This library is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# Lesser General Public License for more details. -# -# You should have received a copy of the GNU Lesser General Public -# License along with this library. If not, see -# <http://www.gnu.org/licenses/>. -# -# This script is supposed to check test_file_access.txt file and -# warn about file accesses outside our working tree. -# -# - -use strict; -use warnings; - -my $access_file = "test_file_access.txt"; -my $whitelist_file = "file_access_whitelist.txt"; - -my @known_actions = ("open", "fopen", "access", "stat", "lstat", "connect"); - -my @files; -my @whitelist; - -open FILE, "<", $access_file or die "Unable to open $access_file: $!"; -while (<FILE>) { - chomp; - if (/^(\S*):\s*(\S*):\s*(\S*)(\s*:\s*(.*))?$/) { - my %rec; - ${rec}{path} = $1; - ${rec}{action} = $2; - ${rec}{progname} = $3; - if (defined $5) { - ${rec}{testname} = $5; - } - push (@files, \%rec); - } else { - die "Malformed line $_"; - } -} -close FILE; - -open FILE, "<", $whitelist_file or die "Unable to open $whitelist_file: $!"; -while (<FILE>) { - chomp; - if (/^\s*#.*$/) { - # comment - } elsif (/^(\S*):\s*(\S*)(:\s*(\S*)(\s*:\s*(.*))?)?$/ and - grep /^$2$/, @known_actions) { - # $path: $action: $progname: $testname - my %rec; - ${rec}{path} = $1; - ${rec}{action} = $3; - if (defined $4) { - ${rec}{progname} = $4; - } - if (defined $6) { - ${rec}{testname} = $6; - } - push (@whitelist, \%rec); - } elsif (/^(\S*)(:\s*(\S*)(\s*:\s*(.*))?)?$/) { - # $path: $progname: $testname - my %rec; - ${rec}{path} = $1; - if (defined $3) { - ${rec}{progname} = $3; - } - if (defined $5) { - ${rec}{testname} = $5; - } - push (@whitelist, \%rec); - } else { - die "Malformed line $_"; - } -} -close FILE; - -# Now we should check if %traces is included in $whitelist. For -# now checking just keys is sufficient -my $error = 0; -for my $file (@files) { - my $match = 0; - - for my $rule (@whitelist) { - if (not %${file}{path} =~ m/^$rule->{path}$/) { - next; - } - - if (defined %${rule}{action} and - not %${file}{action} =~ m/^$rule->{action}$/) { - next; - } - - if (defined %${rule}{progname} and - not %${file}{progname} =~ m/^$rule->{progname}$/) { - next; - } - - if (defined %${rule}{testname} and - defined %${file}{testname} and - not %${file}{testname} =~ m/^$rule->{testname}$/) { - next; - } - - $match = 1; - } - - if (not $match) { - $error = 1; - print "$file->{path}: $file->{action}: $file->{progname}"; - print ": $file->{testname}" if defined %${file}{testname}; - print "\n"; - } -} - -exit $error; diff --git a/tests/check-file-access.py b/tests/check-file-access.py new file mode 100755 index 0000000000..33583a5b52 --- /dev/null +++ b/tests/check-file-access.py @@ -0,0 +1,121 @@ +#!/usr/bin/env python +# +# Copyright (C) 2016-2019 Red Hat, Inc. +# +# This library is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 2.1 of the License, or (at your option) any later version. +# +# This library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this library. If not, see +# <http://www.gnu.org/licenses/>. +# +# This script is supposed to check test_file_access.txt file and +# warn about file accesses outside our working tree. +# +# + +from __future__ import print_function + +import re +import sys + +access_file = "test_file_access.txt" +whitelist_file = "file_access_whitelist.txt" + +known_actions = ["open", "fopen", "access", "stat", "lstat", "connect"] + +files = [] +whitelist = [] + +with open(access_file, "r") as fh: + for line in fh: + line = line.rstrip("\n") + + m = re.match(r'''^(\S*):\s*(\S*):\s*(\S*)(\s*:\s*(.*))?$''', line) + if m is not None: + rec = { + "path": m.group(1), + "action": m.group(2), + "progname": m.group(3), + "testname": m.group(5), + } + files.append(rec) + else: + raise Exception("Malformed line %s" % line) + +with open(whitelist_file, "r") as fh: + for line in fh: + line = line.rstrip("\n") + + if re.match(r'''^\s*#.*$''', line): + continue # comment + if line == "": + continue + + m = re.match(r'''^(\S*):\s*(\S*)(:\s*(\S*)(\s*:\s*(.*))?)?$''', line) + if m is not None and m.group(2) in known_actions: + # $path: $action: $progname: $testname + rec = { + "path": m.group(1), + "action": m.group(3), + "progname": m.group(4), + "testname": m.group(6), + } + whitelist.append(rec) + else: + m = re.match(r'''^(\S*)(:\s*(\S*)(\s*:\s*(.*))?)?$''', line) + if m is not None: + # $path: $progname: $testname + rec = { + "path": m.group(1), + "action": None, + "progname": m.group(3), + "testname": m.group(5), + } + whitelist.append(rec) + else: + raise Exception("Malformed line %s" % line) + + +# Now we should check if %traces is included in $whitelist. For +# now checking just keys is sufficient +err = False +for file in files: + match = False + + for rule in whitelist: + if not re.match("^" + rule["path"], file["path"]): + continue + + if (rule["action"] is not None and + not re.match("^" + rule["action"], file["action"])): + continue + + if (rule["progname"] is not None and + not re.match("^" + rule["progname"], file["progname"])): + continue + + if (rule["testname"] is not None and + file["testname"] is not None and + not re.match("^" + rule["testname"], file["testname"])): + continue + + match = True + + if not match: + err = True + print("%s: %s: %s" % (file["path"], file["action"], file["progname"]), file=sys.stderr, end="") + if file["testname"] is not None: + print(": %s" % file["testname"], file=sys.stderr, end="") + print("", file=sys.stderr) + +if err: + sys.exit(1) +sys.exit(0) diff --git a/tests/file_access_whitelist.txt b/tests/file_access_whitelist.txt index 3fb318cbab..5ec7ee63bb 100644 --- a/tests/file_access_whitelist.txt +++ b/tests/file_access_whitelist.txt @@ -5,7 +5,7 @@ # $path: $progname: $testname # $path: $action: $progname: $testname # -# All these variables are evaluated as perl RE. So to allow +# All these variables are evaluated as python RE. So to allow # /dev/sda and /dev/sdb, you can just '/dev/sd[a-b]', or to allow # /proc/$pid/status you can '/proc/\d+/status' and so on. # Moreover, $action, $progname and $testname can be empty, in which -- 2.21.0

As part of an goal to eliminate Perl from libvirt build tools, rewrite the hvsupport.pl tool in Python. This was a straight conversion, manually going line-by-line to change the syntax from Perl to Python. Thus the overall structure of the file and approach is the same. The new impl generates byte-for-byte identical output to the old impl. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- docs/Makefile.am | 8 +- docs/hvsupport.pl | 458 -------------------------------------------- docs/hvsupport.py | 479 ++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 483 insertions(+), 462 deletions(-) delete mode 100755 docs/hvsupport.pl create mode 100755 docs/hvsupport.py diff --git a/docs/Makefile.am b/docs/Makefile.am index 14f3faffd4..e4aa8b4ad9 100644 --- a/docs/Makefile.am +++ b/docs/Makefile.am @@ -194,7 +194,7 @@ EXTRA_DIST= \ $(internals_html_in) $(internals_html) $(fonts) \ $(kbase_html_in) $(kbase_html) \ aclperms.htmlinc \ - hvsupport.pl \ + hvsupport.py \ $(schema_DATA) acl_generated = aclperms.htmlinc @@ -230,12 +230,12 @@ web: $(dot_html) $(internals_html) $(kbase_html) \ hvsupport.html: $(srcdir)/hvsupport.html.in -$(srcdir)/hvsupport.html.in: $(srcdir)/hvsupport.pl $(api_DATA) \ +$(srcdir)/hvsupport.html.in: $(srcdir)/hvsupport.py $(api_DATA) \ $(top_srcdir)/src/libvirt_public.syms \ $(top_srcdir)/src/libvirt_qemu.syms $(top_srcdir)/src/libvirt_lxc.syms \ $(top_srcdir)/src/driver.h - $(AM_V_GEN)$(PERL) $(srcdir)/hvsupport.pl $(top_srcdir)/src > $@ \ - || { rm $@ && exit 1; } + $(AM_V_GEN)$(RUNUTF8) $(PYTHON) $(srcdir)/hvsupport.py \ + $(top_srcdir)/src > $@ || { rm $@ && exit 1; } news.html.in: \ $(srcdir)/news.xml \ diff --git a/docs/hvsupport.pl b/docs/hvsupport.pl deleted file mode 100755 index 494b8a27ec..0000000000 --- a/docs/hvsupport.pl +++ /dev/null @@ -1,458 +0,0 @@ -#!/usr/bin/env perl - -use strict; -use warnings; - -use File::Find; - -die "syntax: $0 SRCDIR\n" unless int(@ARGV) == 1; - -my $srcdir = shift @ARGV; - -my $symslibvirt = "$srcdir/libvirt_public.syms"; -my $symsqemu = "$srcdir/libvirt_qemu.syms"; -my $symslxc = "$srcdir/libvirt_lxc.syms"; -my @drivertable = ( - "$srcdir/driver-hypervisor.h", - "$srcdir/driver-interface.h", - "$srcdir/driver-network.h", - "$srcdir/driver-nodedev.h", - "$srcdir/driver-nwfilter.h", - "$srcdir/driver-secret.h", - "$srcdir/driver-state.h", - "$srcdir/driver-storage.h", - "$srcdir/driver-stream.h", - ); - -my %groupheaders = ( - "virHypervisorDriver" => "Hypervisor APIs", - "virNetworkDriver" => "Virtual Network APIs", - "virInterfaceDriver" => "Host Interface APIs", - "virNodeDeviceDriver" => "Host Device APIs", - "virStorageDriver" => "Storage Pool APIs", - "virSecretDriver" => "Secret APIs", - "virNWFilterDriver" => "Network Filter APIs", - ); - - -my @srcs; -find({ - wanted => sub { - if (m!$srcdir/.*/\w+_(driver|common|tmpl|monitor|hal|udev)\.c$!) { - push @srcs, $_ if $_ !~ /vbox_driver\.c/; - } - }, no_chdir => 1}, $srcdir); - -# Map API functions to the header and documentation files they're in -# so that we can generate proper hyperlinks to their documentation. -# -# The function names are grep'd from the XML output of apibuild.py. -sub getAPIFilenames { - my $filename = shift; - - my %files; - my $line; - - open FILE, "<", $filename or die "cannot read $filename: $!"; - - while (defined($line = <FILE>)) { - if ($line =~ /function name='([^']+)' file='([^']+)'/) { - $files{$1} = $2; - } - } - - close FILE; - - if (keys %files == 0) { - die "No functions found in $filename. Has the apibuild.py output changed?"; - } - return \%files; -} - -sub parseSymsFile { - my $apisref = shift; - my $prefix = shift; - my $filename = shift; - my $xmlfilename = shift; - - my $line; - my $vers; - my $prevvers; - - my $filenames = getAPIFilenames($xmlfilename); - - open FILE, "<$filename" - or die "cannot read $filename: $!"; - - while (defined($line = <FILE>)) { - chomp $line; - next if $line =~ /^\s*#/; - next if $line =~ /^\s*$/; - next if $line =~ /^\s*(global|local):/; - if ($line =~ /^\s*${prefix}_(\d+\.\d+\.\d+)\s*{\s*$/) { - if (defined $vers) { - die "malformed syms file"; - } - $vers = $1; - } elsif ($line =~ /\s*}\s*;\s*$/) { - if (defined $prevvers) { - die "malformed syms file"; - } - $prevvers = $vers; - $vers = undef; - } elsif ($line =~ /\s*}\s*${prefix}_(\d+\.\d+\.\d+)\s*;\s*$/) { - if ($1 ne $prevvers) { - die "malformed syms file $1 != $vers"; - } - $prevvers = $vers; - $vers = undef; - } elsif ($line =~ /\s*(\w+)\s*;\s*$/) { - $$apisref{$1} = {}; - $$apisref{$1}->{vers} = $vers; - $$apisref{$1}->{file} = $$filenames{$1}; - } else { - die "unexpected data $line\n"; - } - } - - close FILE; -} - -my %apis; -# Get the list of all public APIs and their corresponding version -parseSymsFile(\%apis, "LIBVIRT", $symslibvirt, "$srcdir/../docs/libvirt-api.xml"); - -# And the same for the QEMU specific APIs -parseSymsFile(\%apis, "LIBVIRT_QEMU", $symsqemu, "$srcdir/../docs/libvirt-qemu-api.xml"); - -# And the same for the LXC specific APIs -parseSymsFile(\%apis, "LIBVIRT_LXC", $symslxc, "$srcdir/../docs/libvirt-lxc-api.xml"); - - -# Some special things which aren't public APIs, -# but we want to report -$apis{virConnectSupportsFeature}->{vers} = "0.3.2"; -$apis{virDomainMigratePrepare}->{vers} = "0.3.2"; -$apis{virDomainMigratePerform}->{vers} = "0.3.2"; -$apis{virDomainMigrateFinish}->{vers} = "0.3.2"; -$apis{virDomainMigratePrepare2}->{vers} = "0.5.0"; -$apis{virDomainMigrateFinish2}->{vers} = "0.5.0"; -$apis{virDomainMigratePrepareTunnel}->{vers} = "0.7.2"; - -$apis{virDomainMigrateBegin3}->{vers} = "0.9.2"; -$apis{virDomainMigratePrepare3}->{vers} = "0.9.2"; -$apis{virDomainMigratePrepareTunnel3}->{vers} = "0.9.2"; -$apis{virDomainMigratePerform3}->{vers} = "0.9.2"; -$apis{virDomainMigrateFinish3}->{vers} = "0.9.2"; -$apis{virDomainMigrateConfirm3}->{vers} = "0.9.2"; - -$apis{virDomainMigrateBegin3Params}->{vers} = "1.1.0"; -$apis{virDomainMigratePrepare3Params}->{vers} = "1.1.0"; -$apis{virDomainMigratePrepareTunnel3Params}->{vers} = "1.1.0"; -$apis{virDomainMigratePerform3Params}->{vers} = "1.1.0"; -$apis{virDomainMigrateFinish3Params}->{vers} = "1.1.0"; -$apis{virDomainMigrateConfirm3Params}->{vers} = "1.1.0"; - - - -# Now we want to get the mapping between public APIs -# and driver struct fields. This lets us later match -# update the driver impls with the public APis. - -my $line; - -# Group name -> hash of APIs { fields -> api name } -my %groups; -my $ingrp; -foreach my $drivertable (@drivertable) { - open FILE, "<$drivertable" - or die "cannot read $drivertable: $!"; - - while (defined($line = <FILE>)) { - if ($line =~ /struct _(vir\w*Driver)/) { - my $grp = $1; - if ($grp ne "virStateDriver" && - $grp ne "virStreamDriver") { - $ingrp = $grp; - $groups{$ingrp} = { apis => {}, drivers => {} }; - } - } elsif ($ingrp) { - if ($line =~ /^\s*vir(?:Drv)(\w+)\s+(\w+);\s*$/) { - my $field = $2; - my $name = $1; - - my $api; - if (exists $apis{"vir$name"}) { - $api = "vir$name"; - } elsif ($name =~ /\w+(Open|Close|URIProbe)/) { - next; - } else { - die "driver $name does not have a public API"; - } - $groups{$ingrp}->{apis}->{$field} = $api; - } elsif ($line =~ /};/) { - $ingrp = undef; - } - } - } - - close FILE; -} - - -# Finally, we read all the primary driver files and extract -# the driver API tables from each one. - -foreach my $src (@srcs) { - open FILE, "<$src" or - die "cannot read $src: $!"; - - my $groups_regex = join("|", keys %groups); - $ingrp = undef; - my $impl; - while (defined($line = <FILE>)) { - if (!$ingrp) { - # skip non-matching lines early to save time - next if not $line =~ /$groups_regex/; - - if ($line =~ /^\s*(?:static\s+)?($groups_regex)\s+(\w+)\s*=\s*{/ || - $line =~ /^\s*(?:static\s+)?($groups_regex)\s+NAME\(\w+\)\s*=\s*{/) { - $ingrp = $1; - $impl = $src; - - if ($impl =~ m,.*/node_device_(\w+)\.c,) { - $impl = $1; - } else { - $impl =~ s,.*/(\w+?)_((\w+)_)?(\w+)\.c,$1,; - } - - if ($groups{$ingrp}->{drivers}->{$impl}) { - die "Group $ingrp already contains $impl"; - } - - $groups{$ingrp}->{drivers}->{$impl} = {}; - } - - } else { - if ($line =~ m!\s*\.(\w+)\s*=\s*(\w+)\s*,?\s*(?:/\*\s*(\d+\.\d+\.\d+)\s*(?:-\s*(\d+\.\d+\.\d+))?\s*\*/\s*)?$!) { - my $api = $1; - my $meth = $2; - my $vers = $3; - my $deleted = $4; - - next if $api eq "no" || $api eq "name"; - - if ($meth eq "NULL" && !defined $deleted) { - die "Method impl for $api is NULL, but no deleted version is provided"; - } - if ($meth ne "NULL" && defined $deleted) { - die "Method impl for $api is non-NULL, but deleted version is provided"; - } - - die "Method $meth in $src is missing version" unless defined $vers || $api eq "connectURIProbe"; - - if (!exists($groups{$ingrp}->{apis}->{$api})) { - next if $api =~ /\w(Open|Close|URIProbe)/; - - die "Found unexpected method $api in $ingrp\n"; - } - - $groups{$ingrp}->{drivers}->{$impl}->{$api} = {}; - $groups{$ingrp}->{drivers}->{$impl}->{$api}->{vers} = $vers; - $groups{$ingrp}->{drivers}->{$impl}->{$api}->{deleted} = $deleted; - if ($api eq "domainMigratePrepare" || - $api eq "domainMigratePrepare2" || - $api eq "domainMigratePrepare3") { - if (!$groups{$ingrp}->{drivers}->{$impl}->{"domainMigrate"}) { - $groups{$ingrp}->{drivers}->{$impl}->{"domainMigrate"} = {}; - $groups{$ingrp}->{drivers}->{$impl}->{"domainMigrate"}->{vers} = $vers; - } - } - - } elsif ($line =~ /}/) { - $ingrp = undef; - } - } - } - - close FILE; -} - - -# The '.open' driver method is used for 3 public APIs, so we -# have a bit of manual fixup todo with the per-driver versioning -# and support matrix - -$groups{virHypervisorDriver}->{apis}->{"openAuth"} = "virConnectOpenAuth"; -$groups{virHypervisorDriver}->{apis}->{"openReadOnly"} = "virConnectOpenReadOnly"; -$groups{virHypervisorDriver}->{apis}->{"domainMigrate"} = "virDomainMigrate"; - -my $openAuthVers = (0 * 1000 * 1000) + (4 * 1000) + 0; - -foreach my $drv (keys %{$groups{"virHypervisorDriver"}->{drivers}}) { - my $openVersStr = $groups{"virHypervisorDriver"}->{drivers}->{$drv}->{"connectOpen"}->{vers}; - my $openVers; - if ($openVersStr =~ /(\d+)\.(\d+)\.(\d+)/) { - $openVers = ($1 * 1000 * 1000) + ($2 * 1000) + $3; - } - - # virConnectOpenReadOnly always matches virConnectOpen version - $groups{"virHypervisorDriver"}->{drivers}->{$drv}->{"connectOpenReadOnly"} = - $groups{"virHypervisorDriver"}->{drivers}->{$drv}->{"connectOpen"}; - - $groups{"virHypervisorDriver"}->{drivers}->{$drv}->{"connectOpenAuth"} = {}; - - # virConnectOpenAuth is always 0.4.0 if the driver existed - # before this time, otherwise it matches the version of - # the driver's virConnectOpen entry - if ($openVersStr eq "Y" || - $openVers >= $openAuthVers) { - $groups{"virHypervisorDriver"}->{drivers}->{$drv}->{"connectOpenAuth"}->{vers} = $openVersStr; - } else { - $groups{"virHypervisorDriver"}->{drivers}->{$drv}->{"connectOpenAuth"}->{vers} = "0.4.0"; - } -} - - -# Another special case for the virDomainCreateLinux which was replaced -# with virDomainCreateXML -$groups{virHypervisorDriver}->{apis}->{"domainCreateLinux"} = "virDomainCreateLinux"; - -my $createAPIVers = (0 * 1000 * 1000) + (0 * 1000) + 3; - -foreach my $drv (keys %{$groups{"virHypervisorDriver"}->{drivers}}) { - my $createVersStr = $groups{"virHypervisorDriver"}->{drivers}->{$drv}->{"domainCreateXML"}->{vers}; - next unless defined $createVersStr; - my $createVers; - if ($createVersStr =~ /(\d+)\.(\d+)\.(\d+)/) { - $createVers = ($1 * 1000 * 1000) + ($2 * 1000) + $3; - } - - $groups{"virHypervisorDriver"}->{drivers}->{$drv}->{"domainCreateLinux"} = {}; - - # virCreateLinux is always 0.0.3 if the driver existed - # before this time, otherwise it matches the version of - # the driver's virCreateXML entry - if ($createVersStr eq "Y" || - $createVers >= $createAPIVers) { - $groups{"virHypervisorDriver"}->{drivers}->{$drv}->{"domainCreateLinux"}->{vers} = $createVersStr; - } else { - $groups{"virHypervisorDriver"}->{drivers}->{$drv}->{"domainCreateLinux"}->{vers} = "0.0.3"; - } -} - - -# Finally we generate the HTML file with the tables - -print <<EOF; -<?xml version="1.0" encoding="UTF-8"?> -<!DOCTYPE html> -<html xmlns="http://www.w3.org/1999/xhtml"> -<body class="hvsupport"> -<h1>libvirt API support matrix</h1> - -<ul id="toc"></ul> - -<p> -This page documents which <a href="html/">libvirt calls</a> work on -which libvirt drivers / hypervisors, and which version the API appeared -in. If a hypervisor driver later dropped support for the API, the version -when it was removed is also mentioned (highlighted in -<span class="removedhv">dark red</span>). -</p> - -EOF - - foreach my $grp (sort { $a cmp $b } keys %groups) { - print "<h2><a id=\"$grp\">", $groupheaders{$grp}, "</a></h2>\n"; - print <<EOF; -<table class="top_table"> -<thead> -<tr> -<th>API</th> -<th>Version</th> -EOF - - foreach my $drv (sort { $a cmp $b } keys %{$groups{$grp}->{drivers}}) { - print " <th>$drv</th>\n"; - } - - print <<EOF; -</tr> -</thead> -<tbody> -EOF - - my $row = 0; - foreach my $field (sort { - $groups{$grp}->{apis}->{$a} - cmp - $groups{$grp}->{apis}->{$b} - } keys %{$groups{$grp}->{apis}}) { - my $api = $groups{$grp}->{apis}->{$field}; - my $vers = $apis{$api}->{vers}; - my $htmlgrp = $apis{$api}->{file}; - print <<EOF; -<tr> -<td> -EOF - - if (defined $htmlgrp) { - print <<EOF; -<a href=\"html/libvirt-$htmlgrp.html#$api\">$api</a> -EOF - - } else { - print $api; - } - print <<EOF; -</td> -<td>$vers</td> -EOF - - foreach my $drv (sort {$a cmp $b } keys %{$groups{$grp}->{drivers}}) { - print "<td>"; - if (exists $groups{$grp}->{drivers}->{$drv}->{$field}) { - if ($groups{$grp}->{drivers}->{$drv}->{$field}->{vers}) { - print $groups{$grp}->{drivers}->{$drv}->{$field}->{vers}; - } - if ($groups{$grp}->{drivers}->{$drv}->{$field}->{deleted}) { - print " - <span class=\"removedhv\">", $groups{$grp}->{drivers}->{$drv}->{$field}->{deleted}, "</span>"; - } - } - print "</td>\n"; - } - - print <<EOF; -</tr> -EOF - - $row++; - if (($row % 15) == 0) { - print <<EOF; -<tr> -<th>API</th> -<th>Version</th> -EOF - - foreach my $drv (sort { $a cmp $b } keys %{$groups{$grp}->{drivers}}) { - print " <th>$drv</th>\n"; - } - - print <<EOF; -</tr> -EOF - } - - } - - print <<EOF; -</tbody> -</table> -EOF -} - -print <<EOF; -</body> -</html> -EOF diff --git a/docs/hvsupport.py b/docs/hvsupport.py new file mode 100755 index 0000000000..cfbecd82ae --- /dev/null +++ b/docs/hvsupport.py @@ -0,0 +1,479 @@ +#!/usr/bin/env python +# +# Copyright (C) 2011-2019 Red Hat, Inc. +# +# This library is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 2.1 of the License, or (at your option) any later version. +# +# This library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this library. If not, see +# <http://www.gnu.org/licenses/>. + +from __future__ import print_function + +import sys +import os.path +import re + +if len(sys.argv) != 2: + print("syntax: %s SRCDIR\n" % sys.argv[0], file=sys.stderr) + +srcdir = sys.argv[1] + +symslibvirt = os.path.join(srcdir, "libvirt_public.syms") +symsqemu = os.path.join(srcdir, "libvirt_qemu.syms") +symslxc = os.path.join(srcdir, "libvirt_lxc.syms") +drivertablefiles = [ + os.path.join(srcdir, "driver-hypervisor.h"), + os.path.join(srcdir, "driver-interface.h"), + os.path.join(srcdir, "driver-network.h"), + os.path.join(srcdir, "driver-nodedev.h"), + os.path.join(srcdir, "driver-nwfilter.h"), + os.path.join(srcdir, "driver-secret.h"), + os.path.join(srcdir, "driver-state.h"), + os.path.join(srcdir, "driver-storage.h"), + os.path.join(srcdir, "driver-stream.h"), +] + +groupheaders = { + "virHypervisorDriver": "Hypervisor APIs", + "virNetworkDriver": "Virtual Network APIs", + "virInterfaceDriver": "Host Interface APIs", + "virNodeDeviceDriver": "Host Device APIs", + "virStorageDriver": "Storage Pool APIs", + "virSecretDriver": "Secret APIs", + "virNWFilterDriver": "Network Filter APIs", +} + + +srcs = [] +for root, dirs, files in os.walk(srcdir): + for file in files: + if ((file.endswith("driver.c") and + not file.endswith("vbox_driver.c")) or + file.endswith("common.c") or + file.endswith("tmpl.c") or + file.endswith("monitor.c") or + file.endswith("hal.c") or + file.endswith("udev.c")): + srcs.append(os.path.join(root, file)) + + +# Map API functions to the header and documentation files they're in +# so that we can generate proper hyperlinks to their documentation. +# +# The function names are grep'd from the XML output of apibuild.py. +def getAPIFilenames(filename): + files = {} + + with open(filename) as fh: + prog = re.compile(r"\s*<function name='([^']+)' file='([^']+)'.*") + for line in fh: + res = prog.match(line) + if res is not None: + files[res.group(1)] = res.group(2) + + if len(files) == 0: + raise Exception("No functions found in %s. Has the apibuild.py output changed?" % filename) + + return files + + +def parseSymsFile(apisref, prefix, filename, xmlfilename): + vers = None + prevvers = None + + filenames = getAPIFilenames(xmlfilename) + + with open(filename) as fh: + groupstartprog = re.compile(r"^\s*%s_(\d+\.\d+\.\d+)\s*{\s*$" % prefix) + groupendprog1 = re.compile(r"^\s*}\s*;\s*$") + groupendprog2 = re.compile(r"^\s*}\s*%s_(\d+\.\d+\.\d+)\s*;\s*$" % prefix) + symbolprog = re.compile(r"^\s*(\w+)\s*;\s*$") + for line in fh: + line = line.strip() + + if line == "": + continue + if line[0] == '#': + continue + if line.startswith("global:"): + continue + if line.startswith("local:"): + continue + + groupstartmatch = groupstartprog.match(line) + groupendmatch1 = groupendprog1.match(line) + groupendmatch2 = groupendprog2.match(line) + symbolmatch = symbolprog.match(line) + if groupstartmatch is not None: + if vers is not None: + raise Exception("malformed syms file when starting group") + + vers = groupstartmatch.group(1) + elif groupendmatch1 is not None: + if prevvers is not None: + raise Exception("malformed syms file when ending group") + + prevvers = vers + vers = None + elif groupendmatch2 is not None: + if groupendmatch2.group(1) != prevvers: + raise Exception("malformed syms file %s != %s when ending group" % ( + groupendmatch2.group(1), prevvers)) + + prevvers = vers + vers = None + elif symbolmatch is not None: + name = symbolmatch.group(1) + apisref[name] = { + "vers": vers, + "file": filenames.get(name), + } + else: + raise Exception("unexpected data %s" % line) + + +apis = {} +# Get the list of all public APIs and their corresponding version +parseSymsFile(apis, "LIBVIRT", symslibvirt, os.path.join(srcdir, "../docs/libvirt-api.xml")) + +# And the same for the QEMU specific APIs +parseSymsFile(apis, "LIBVIRT_QEMU", symsqemu, os.path.join(srcdir, "../docs/libvirt-qemu-api.xml")) + +# And the same for the LXC specific APIs +parseSymsFile(apis, "LIBVIRT_LXC", symslxc, os.path.join(srcdir, "../docs/libvirt-lxc-api.xml")) + + +# Some special things which aren't public APIs, +# but we want to report +apis["virConnectSupportsFeature"] = { + "vers": "0.3.2" +} +apis["virDomainMigratePrepare"] = { + "vers": "0.3.2" +} +apis["virDomainMigratePerform"] = { + "vers": "0.3.2" +} +apis["virDomainMigrateFinish"] = { + "vers": "0.3.2" +} +apis["virDomainMigratePrepare2"] = { + "vers": "0.5.0" +} +apis["virDomainMigrateFinish2"] = { + "vers": "0.5.0" +} +apis["virDomainMigratePrepareTunnel"] = { + "vers": "0.7.2" +} + +apis["virDomainMigrateBegin3"] = { + "vers": "0.9.2" +} +apis["virDomainMigratePrepare3"] = { + "vers": "0.9.2" +} +apis["virDomainMigratePrepareTunnel3"] = { + "vers": "0.9.2" +} +apis["virDomainMigratePerform3"] = { + "vers": "0.9.2" +} +apis["virDomainMigrateFinish3"] = { + "vers": "0.9.2" +} +apis["virDomainMigrateConfirm3"] = { + "vers": "0.9.2" +} + +apis["virDomainMigrateBegin3Params"] = { + "vers": "1.1.0" +} +apis["virDomainMigratePrepare3Params"] = { + "vers": "1.1.0" +} +apis["virDomainMigratePrepareTunnel3Params"] = { + "vers": "1.1.0" +} +apis["virDomainMigratePerform3Params"] = { + "vers": "1.1.0" +} +apis["virDomainMigrateFinish3Params"] = { + "vers": "1.1.0" +} +apis["virDomainMigrateConfirm3Params"] = { + "vers": "1.1.0" +} + + +# Now we want to get the mapping between public APIs +# and driver struct fields. This lets us later match +# update the driver impls with the public APis. + +# Group name -> hash of APIs { fields -> api name } +groups = {} +ingrp = None +for drivertablefile in drivertablefiles: + with open(drivertablefile) as fh: + starttableprog = re.compile(r"struct _(vir\w*Driver)") + endtableprog = re.compile(r"};") + callbackprog = re.compile(r"^\s*vir(?:Drv)(\w+)\s+(\w+);\s*$") + ignoreapiprog = re.compile(r"\w+(Open|Close|URIProbe)") + for line in fh: + starttablematch = starttableprog.match(line) + if starttablematch is not None: + grp = starttablematch.group(1) + if grp != "virStateDriver" and grp != "virStreamDriver": + ingrp = grp + groups[ingrp] = { + "apis": {}, + "drivers": {} + } + elif ingrp != None: + callbackmatch = callbackprog.match(line) + if callbackmatch is not None: + name = callbackmatch.group(1) + field = callbackmatch.group(2) + + api = "vir" + name + if api in apis: + groups[ingrp]["apis"][field] = api + elif ignoreapiprog.match(api) != None: + continue + else: + raise Exception("driver %s does not have a public API" % name) + elif endtableprog.match(line): + ingrp = None + + +# Finally, we read all the primary driver files and extract +# the driver API tables from each one. + +for src in srcs: + with open(src) as fh: + groupsre = "|".join(groups.keys()) + groupsprog2 = re.compile(r"^\s*(static\s+)?(" + groupsre + r")\s+(\w+)\s*=\s*{") + groupsprog3 = re.compile(r"^\s*(static\s+)?(" + groupsre + r")\s+NAME\(\w+\)\s*=\s*{") + nodedevimplprog = re.compile(r".*/node_device_(\w+)\.c") + miscimplprog = re.compile(r".*/(\w+?)_((\w+)_)?(\w+)\.c") + callbackprog = re.compile(r"\s*\.(\w+)\s*=\s*(\w+)\s*,?\s*(?:/\*\s*(\d+\.\d+\.\d+)\s*(?:-\s*(\d+\.\d+\.\d+))?\s*\*/\s*)?$") + skipapiprog = re.compile(r"\w+(Open|Close|URIProbe)") + + ingrp = None + impl = None + for line in fh: + if ingrp is None: + m = groupsprog2.match(line) + if m is None: + m = groupsprog3.match(line) + if m is not None: + ingrp = m.group(2) + impl = src + + implmatch = nodedevimplprog.match(impl) + if implmatch is None: + implmatch = miscimplprog.match(impl) + if implmatch is None: + raise Exception("Unexpected impl format '%s'" % impl) + impl = implmatch.group(1) + + if impl in groups[ingrp]["drivers"]: + raise Exception("Group %s already contains %s" % (ingrp, impl)) + + groups[ingrp]["drivers"][impl] = {} + else: + callbackmatch = callbackprog.match(line) + if callbackmatch is not None: + api = callbackmatch.group(1) + meth = callbackmatch.group(2) + vers = callbackmatch.group(3) + deleted = callbackmatch.group(4) + + if api == "no" or api == "name": + continue + + if meth == "NULL" and deleted is None: + raise Exception("Method impl for %s is NULL, but no deleted version is provided" % api) + + if meth != "NULL" and deleted is not None: + raise Exception("Method impl for %s is non-NULL, but deleted version is provided" % api) + + if vers is None and api != "connectURIProbe": + raise Exception("Method %s in %s is missing version" % (meth, src)) + + if api not in groups[ingrp]["apis"]: + if skipapiprog.match(api): + continue + + raise Exception("Found unexpected method %s in %s" % (api, ingrp)) + + groups[ingrp]["drivers"][impl][api] = { + "vers": vers, + "deleted": deleted, + } + + if (api == "domainMigratePrepare" or + api == "domainMigratePrepare2" or + api == "domainMigratePrepare3"): + if "domainMigrate" not in groups[ingrp]["drivers"][impl]: + groups[ingrp]["drivers"][impl]["domainMigrate"] = { + "vers": vers, + } + elif line.find("}") != -1: + ingrp = None + + +# The '.open' driver method is used for 3 public APIs, so we +# have a bit of manual fixup todo with the per-driver versioning +# and support matrix + +groups["virHypervisorDriver"]["apis"]["openAuth"] = "virConnectOpenAuth" +groups["virHypervisorDriver"]["apis"]["openReadOnly"] = "virConnectOpenReadOnly" +groups["virHypervisorDriver"]["apis"]["domainMigrate"] = "virDomainMigrate" + +openAuthVers = (0 * 1000 * 1000) + (4 * 1000) + 0 + +for drv in groups["virHypervisorDriver"]["drivers"].keys(): + openVersStr = groups["virHypervisorDriver"]["drivers"][drv]["connectOpen"]["vers"] + openVers = 0 + if openVersStr != "Y": + openVersBits = openVersStr.split(".") + if len(openVersBits) != 3: + raise Exception("Expected 3 digit version for %s" % openVersStr) + openVers = (int(openVersBits[0]) * 1000 * 1000) + (int(openVersBits[1]) * 1000) + int(openVersBits[2]) + + # virConnectOpenReadOnly always matches virConnectOpen version + groups["virHypervisorDriver"]["drivers"][drv]["connectOpenReadOnly"] = \ + groups["virHypervisorDriver"]["drivers"][drv]["connectOpen"] + + # virConnectOpenAuth is always 0.4.0 if the driver existed + # before this time, otherwise it matches the version of + # the driver's virConnectOpen entry + if openVersStr == "Y" or openVers >= openAuthVers: + vers = openVersStr + else: + vers = "0.4.0" + groups["virHypervisorDriver"]["drivers"][drv]["connectOpenAuth"] = { + "vers": vers, + } + + +# Another special case for the virDomainCreateLinux which was replaced +# with virDomainCreateXML +groups["virHypervisorDriver"]["apis"]["domainCreateLinux"] = "virDomainCreateLinux" + +createAPIVers = (0 * 1000 * 1000) + (0 * 1000) + 3 + +for drv in groups["virHypervisorDriver"]["drivers"].keys(): + if "domainCreateXML" not in groups["virHypervisorDriver"]["drivers"][drv]: + continue + createVersStr = groups["virHypervisorDriver"]["drivers"][drv]["domainCreateXML"]["vers"] + createVers = 0 + if createVersStr != "Y": + createVersBits = createVersStr.split(".") + if len(createVersBits) != 3: + raise Exception("Expected 3 digit version for %s" % createVersStr) + createVers = (int(createVersBits[0]) * 1000 * 1000) + (int(createVersBits[1]) * 1000) + int(createVersBits[2]) + + # virCreateLinux is always 0.0.3 if the driver existed + # before this time, otherwise it matches the version of + # the driver's virCreateXML entry + if createVersStr == "Y" or createVers >= createAPIVers: + vers = createVersStr + else: + vers = "0.0.3" + + groups["virHypervisorDriver"]["drivers"][drv]["domainCreateLinux"] = { + "vers": vers, + } + + +# Finally we generate the HTML file with the tables + +print('''<?xml version="1.0" encoding="UTF-8"?> +<!DOCTYPE html> +<html xmlns="http://www.w3.org/1999/xhtml"> +<body class="hvsupport"> +<h1>libvirt API support matrix</h1> + +<ul id="toc"></ul> + +<p> +This page documents which <a href="html/">libvirt calls</a> work on +which libvirt drivers / hypervisors, and which version the API appeared +in. If a hypervisor driver later dropped support for the API, the version +when it was removed is also mentioned (highlighted in +<span class="removedhv">dark red</span>). +</p> +''') + +for grp in sorted(groups.keys()): + print("<h2><a id=\"%s\">%s</a></h2>" % (grp, groupheaders[grp])) + print('''<table class="top_table"> +<thead> +<tr> +<th>API</th> +<th>Version</th>''') + + for drv in sorted(groups[grp]["drivers"].keys()): + print(" <th>%s</th>" % drv) + + + print('''</tr> +</thead> +<tbody>''') + + row = 0 + def sortkey(field): + return groups[grp]["apis"][field] + for field in sorted(groups[grp]["apis"].keys(), key=sortkey): + api = groups[grp]["apis"][field] + vers = apis[api]["vers"] + htmlgrp = apis[api].get("file") + print("<tr>") + + if htmlgrp is not None: + print('''<td>\n<a href=\"html/libvirt-%s.html#%s\">%s</a>\n</td>''' % (htmlgrp, api, api)) + else: + print("<td>\n%s</td>" % api) + + print("<td>%s</td>" % vers) + + for drv in sorted(groups[grp]["drivers"].keys()): + info = "" + if field in groups[grp]["drivers"][drv]: + vers = groups[grp]["drivers"][drv][field]["vers"] + if vers is not None: + info = info + vers + + deleted = groups[grp]["drivers"][drv][field].get("deleted") + if deleted is not None: + info = info + (''' - <span class="removedhv">%s</span>''' % deleted) + + print("<td>%s</td>" % info) + + print("</tr>") + + row = row + 1 + if (row % 15) == 0: + print('''<tr> +<th>API</th> +<th>Version</th>''') + + for drv in sorted(groups[grp]["drivers"].keys()): + print(" <th>%s</th>" % drv) + + print("</tr>") + + print("</tbody>\n</table>") + +print("</body>\n</html>") -- 2.21.0

As part of an goal to eliminate Perl from libvirt build tools, rewrite the genaclperms.pl tool in Python. This was a straight conversion, manually going line-by-line to change the syntax from Perl to Python. Thus the overall structure of the file and approach is the same. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- docs/Makefile.am | 6 +-- docs/genaclperms.pl | 125 -------------------------------------------- docs/genaclperms.py | 122 ++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 125 insertions(+), 128 deletions(-) delete mode 100755 docs/genaclperms.pl create mode 100755 docs/genaclperms.py diff --git a/docs/Makefile.am b/docs/Makefile.am index e4aa8b4ad9..2f702821ba 100644 --- a/docs/Makefile.am +++ b/docs/Makefile.am @@ -184,7 +184,7 @@ schemadir = $(pkgdatadir)/schemas schema_DATA = $(wildcard $(srcdir)/schemas/*.rng) EXTRA_DIST= \ - apibuild.py genaclperms.pl \ + apibuild.py genaclperms.py \ site.xsl subsite.xsl newapi.xsl page.xsl \ wrapstring.xsl \ $(dot_html) $(dot_html_in) $(gif) $(apihtml) $(apipng) \ @@ -200,8 +200,8 @@ EXTRA_DIST= \ acl_generated = aclperms.htmlinc $(srcdir)/aclperms.htmlinc: $(top_srcdir)/src/access/viraccessperm.h \ - $(srcdir)/genaclperms.pl Makefile.am - $(AM_V_GEN)$(PERL) $(srcdir)/genaclperms.pl $< > $@ + $(srcdir)/genaclperms.py Makefile.am + $(AM_V_GEN)$(RUNUTF8) $(PYTHON) $(srcdir)/genaclperms.py $< > $@ MAINTAINERCLEANFILES = \ $(addprefix $(srcdir)/,$(dot_html)) \ diff --git a/docs/genaclperms.pl b/docs/genaclperms.pl deleted file mode 100755 index e20b4c11c3..0000000000 --- a/docs/genaclperms.pl +++ /dev/null @@ -1,125 +0,0 @@ -#!/usr/bin/env perl -# -# Copyright (C) 2013 Red Hat, Inc. -# -# This library is free software; you can redistribute it and/or -# modify it under the terms of the GNU Lesser General Public -# License as published by the Free Software Foundation; either -# version 2.1 of the License, or (at your option) any later version. -# -# This library is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# Lesser General Public License for more details. -# -# You should have received a copy of the GNU Lesser General Public -# License along with this library. If not, see -# <http://www.gnu.org/licenses/>. -# - -use strict; -use warnings; - -my @objects = ( - "CONNECT", "DOMAIN", "INTERFACE", - "NETWORK_PORT", "NETWORK", "NODE_DEVICE", - "NWFILTER_BINDING", "NWFILTER", - "SECRET", "STORAGE_POOL", "STORAGE_VOL", - ); - -my %class; - -foreach my $object (@objects) { - my $class = lc $object; - - $class =~ s/(^\w|_\w)/uc $1/eg; - $class =~ s/_//g; - $class =~ s/Nwfilter/NWFilter/; - $class = "vir" . $class . "Ptr"; - - $class{$object} = $class; -} - -my $objects = join ("|", @objects); - -my %opts; -my $in_opts = 0; - -my %perms; - -while (<>) { - if ($in_opts) { - if (m,\*/,) { - $in_opts = 0; - } elsif (/\*\s*\@(\w+):\s*(.*?)\s*$/) { - $opts{$1} = $2; - } - } elsif (m,/\*\*,) { - $in_opts = 1; - } elsif (/VIR_ACCESS_PERM_($objects)_((?:\w|_)+),/) { - my $object = $1; - my $perm = lc $2; - next if $perm eq "last"; - - $perm =~ s/_/-/g; - - $perms{$object} = {} unless exists $perms{$object}; - $perms{$object}->{$perm} = { - desc => $opts{desc}, - message => $opts{message}, - anonymous => $opts{anonymous} - }; - %opts = (); - } -} - -print <<EOF; -<?xml version="1.0" encoding="UTF-8"?> -<!DOCTYPE html> -<html xmlns="http://www.w3.org/1999/xhtml"> - <body> -EOF - -foreach my $object (sort { $a cmp $b } keys %perms) { - my $class = $class{$object}; - my $olink = lc "object_" . $object; - print <<EOF; -<h3><a id="$olink">$class</a></h3> -<table class="acl"> - <thead> - <tr> - <th>Permission</th> - <th>Description</th> - </tr> - </thead> - <tbody> -EOF - - foreach my $perm (sort { $a cmp $b } keys %{$perms{$object}}) { - my $description = $perms{$object}->{$perm}->{desc}; - - die "missing description for $object.$perm" unless - defined $description; - - my $plink = lc "perm_" . $object . "_" . $perm; - $plink =~ s/-/_/g; - - print <<EOF; - <tr> - <td><a id="$plink">$perm</a></td> - <td>$description</td> - </tr> -EOF - - } - - print <<EOF; - </tbody> -</table> -EOF -} - -print <<EOF; - </body> -</html> -EOF diff --git a/docs/genaclperms.py b/docs/genaclperms.py new file mode 100755 index 0000000000..0cb5856716 --- /dev/null +++ b/docs/genaclperms.py @@ -0,0 +1,122 @@ +#!/usr/bin/env python +# +# Copyright (C) 2013-2019 Red Hat, Inc. +# +# This library is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 2.1 of the License, or (at your option) any later version. +# +# This library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this library. If not, see +# <http://www.gnu.org/licenses/>. +# + +from __future__ import print_function + +import re +import sys + +objects = [ + "CONNECT", "DOMAIN", "INTERFACE", + "NETWORK_PORT", "NETWORK", "NODE_DEVICE", + "NWFILTER_BINDING", "NWFILTER", + "SECRET", "STORAGE_POOL", "STORAGE_VOL", +] + + +classes = {} + +for obj in objects: + klass = obj.lower() + + klass = re.sub(r'''(^\w|_\w)''', lambda a: a.group(1).upper(), klass) + klass = klass.replace("_", "") + klass = klass.replace("Nwfilter", "NWFilter") + klass = "vir" + klass + "Ptr" + + classes[obj] = klass + + +objectstr = "|".join(objects) + +opts = {} +in_opts = {} + +perms = {} + +aclfile = sys.argv[1] +with open(aclfile, "r") as fh: + for line in fh: + if in_opts: + if line.find("*/") != -1: + in_opts = False + else: + m = re.match(r'''.*\*\s*\@(\w+):\s*(.*?)\s*$''', line) + if m is not None: + opts[m.group(1)] = m.group(2) + elif line.find("**") != -1: + in_opts = True + else: + m = re.match(r'''.*VIR_ACCESS_PERM_(%s)_((?:\w|_)+),''' % objectstr, line) + if m is not None: + obj = m.group(1) + perm = m.group(2).lower() + if perm == "last": + continue + + perm = perm.replace("_", "-") + + if obj not in perms: + perms[obj] = {} + perms[obj][perm] = { + "desc": opts.get("desc", None), + "message": opts.get("message", None), + "anonymous": opts.get("anonymous", None), + } + opts = {} + +print('<?xml version="1.0" encoding="UTF-8"?>') +print('<!DOCTYPE html>') +print('<html xmlns="http://www.w3.org/1999/xhtml">') +print(' <body>') + +for obj in sorted(perms.keys()): + klass = classes[obj] + + olink = "object_" + obj.lower() + + print(' <h3><a id="%s">%s</a></h3>' % (olink, klass)) + print(' <table class="acl">') + print(' <thead>') + print(' <tr>') + print(' <th>Permission</th>') + print(' <th>Description</th>') + print(' </tr>') + print(' </thead>') + print(' <tbody>') + + for perm in sorted(perms[obj].keys()): + description = perms[obj][perm]["desc"] + + if description is None: + raise Exception("missing description for %s.%s" % (obj, perm)) + + plink = "perm_" + obj.lower() + "_" + perm.lower() + plink = plink.replace("-", "_") + + print(' <tr>') + print(' <td><a id="%s">%s</a></td>' % (plink, perm)) + print(' <td>%s</td>' % description) + print(' </tr>') + + print(' </tbody>') + print(' </table>') + +print(' </body>') +print('</html>') -- 2.21.0

On Wed, 2019-09-11 at 17:23 +0100, Daniel P. Berrangé wrote:
This series is an effort to reduce the number of different languages we use by eliminating most use of perl in favour of python.
Just today I found out about https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/htm... which means that if we interpret "supporting RHEL 7" as "supporting the most recent RHEL 7 point release", which I believe we do, then that's one less platform where we are forced to use Python 2! \o/ It might even be the last one, but I'm not entirely sure what the situation is like for SLES and OpenSUSE... Jim, does SLES 12 have Python 3? And, as a side note: do you think you could find the time to add OpenSUSE support to the libvirt-jenkins-ci project? That'd be very useful, because it makes grepping for this kind of information trivial, and also would open the door to running actual CI jobs on the OS :)
This aligns with fact that the likely future build system we'll use (meson) is written in python, and that python is much more commonly used/understood by developers these days than perl.
I believe Meson is itself implemented in Python 3, so platform availability will have to be taken into consideration. My hope is that we can finally ditch Python 2 for good. The https://pythonclock.org/ keeps ticking... -- Andrea Bolognani / Red Hat / Virtualization

On Fri, Sep 13, 2019 at 10:56:45AM +0200, Andrea Bolognani wrote:
On Wed, 2019-09-11 at 17:23 +0100, Daniel P. Berrangé wrote:
This series is an effort to reduce the number of different languages we use by eliminating most use of perl in favour of python.
Just today I found out about
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/htm...
which means that if we interpret "supporting RHEL 7" as "supporting the most recent RHEL 7 point release", which I believe we do, then that's one less platform where we are forced to use Python 2! \o/
I should have said that I did in fact test this series with Python 2.7, Python 3.5 and Python 3.7 - all using the Fedora 31 RPMS of these respective versions.
This aligns with fact that the likely future build system we'll use (meson) is written in python, and that python is much more commonly used/understood by developers these days than perl.
I believe Meson is itself implemented in Python 3, so platform availability will have to be taken into consideration.
Yes, Meson needs py 3.4 I believe
My hope is that we can finally ditch Python 2 for good. The https://pythonclock.org/ keeps ticking...
My intention was to submit patches to purge Py2 support in December, so that we have it gone for the Jan 15 2020 release which matches the pythonclock timeout nicely. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|

On Fri, 2019-09-13 at 09:59 +0100, Daniel P. Berrangé wrote:
On Fri, Sep 13, 2019 at 10:56:45AM +0200, Andrea Bolognani wrote:
On Wed, 2019-09-11 at 17:23 +0100, Daniel P. Berrangé wrote:
This aligns with fact that the likely future build system we'll use (meson) is written in python, and that python is much more commonly used/understood by developers these days than perl.
I believe Meson is itself implemented in Python 3, so platform availability will have to be taken into consideration.
Yes, Meson needs py 3.4 I believe
If I understand the SLES / OpenSUSE situation correctly, then SLES 12.3 should be comparable to OpenSUSE 42.3, which according to Repology has Python 3.4.6. All other platforms have newer Python 3 versions, so we should be good.
My hope is that we can finally ditch Python 2 for good. The https://pythonclock.org/ keeps ticking...
My intention was to submit patches to purge Py2 support in December, so that we have it gone for the Jan 15 2020 release which matches the pythonclock timeout nicely.
The reason I brought it up is because I don't think the situation is going to change significantly in the next three months, so it would probably make sense for these replacement Python scripts to be Python 3 only from day one. On the other hand, you've already gone through the trouble of making them Python 2.7 compatible... -- Andrea Bolognani / Red Hat / Virtualization

On Fri, Sep 13, 2019 at 11:09:16AM +0200, Andrea Bolognani wrote:
On Fri, 2019-09-13 at 09:59 +0100, Daniel P. Berrangé wrote:
On Fri, Sep 13, 2019 at 10:56:45AM +0200, Andrea Bolognani wrote:
On Wed, 2019-09-11 at 17:23 +0100, Daniel P. Berrangé wrote:
This aligns with fact that the likely future build system we'll use (meson) is written in python, and that python is much more commonly used/understood by developers these days than perl.
I believe Meson is itself implemented in Python 3, so platform availability will have to be taken into consideration.
Yes, Meson needs py 3.4 I believe
If I understand the SLES / OpenSUSE situation correctly, then SLES 12.3 should be comparable to OpenSUSE 42.3, which according to Repology has Python 3.4.6. All other platforms have newer Python 3 versions, so we should be good.
My hope is that we can finally ditch Python 2 for good. The https://pythonclock.org/ keeps ticking...
My intention was to submit patches to purge Py2 support in December, so that we have it gone for the Jan 15 2020 release which matches the pythonclock timeout nicely.
The reason I brought it up is because I don't think the situation is going to change significantly in the next three months, so it would probably make sense for these replacement Python scripts to be Python 3 only from day one.
On the other hand, you've already gone through the trouble of making them Python 2.7 compatible...
Honestly the only "trouble" I took was adding from __future__ import print_function everything else I just wrote in the normal way & I got lucky that nothing I did was py3 specific. Most of these scripts are using very boring python code, so its not that surprising. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|

On Fri, 2019-09-13 at 10:12 +0100, Daniel P. Berrangé wrote:
On Fri, Sep 13, 2019 at 11:09:16AM +0200, Andrea Bolognani wrote:
On Fri, 2019-09-13 at 09:59 +0100, Daniel P. Berrangé wrote:
On Fri, Sep 13, 2019 at 10:56:45AM +0200, Andrea Bolognani wrote:
My hope is that we can finally ditch Python 2 for good. The https://pythonclock.org/ keeps ticking...
My intention was to submit patches to purge Py2 support in December, so that we have it gone for the Jan 15 2020 release which matches the pythonclock timeout nicely.
The reason I brought it up is because I don't think the situation is going to change significantly in the next three months, so it would probably make sense for these replacement Python scripts to be Python 3 only from day one.
On the other hand, you've already gone through the trouble of making them Python 2.7 compatible...
Honestly the only "trouble" I took was adding
from __future__ import print_function
everything else I just wrote in the normal way & I got lucky that nothing I did was py3 specific. Most of these scripts are using very boring python code, so its not that surprising.
Good to know! Still, no need to wait until December before dropping Python 2 support in my opinion: we should either be able to do it right now, or we're going to find ourselves in an interesting position three and a half short months from now... Let's see what Jim has to say about SUSE :) -- Andrea Bolognani / Red Hat / Virtualization

On Fri, Sep 13, 2019 at 11:31:34AM +0200, Andrea Bolognani wrote:
On Fri, 2019-09-13 at 10:12 +0100, Daniel P. Berrangé wrote:
On Fri, Sep 13, 2019 at 11:09:16AM +0200, Andrea Bolognani wrote:
On Fri, 2019-09-13 at 09:59 +0100, Daniel P. Berrangé wrote:
On Fri, Sep 13, 2019 at 10:56:45AM +0200, Andrea Bolognani wrote:
My hope is that we can finally ditch Python 2 for good. The https://pythonclock.org/ keeps ticking...
My intention was to submit patches to purge Py2 support in December, so that we have it gone for the Jan 15 2020 release which matches the pythonclock timeout nicely.
The reason I brought it up is because I don't think the situation is going to change significantly in the next three months, so it would probably make sense for these replacement Python scripts to be Python 3 only from day one.
On the other hand, you've already gone through the trouble of making them Python 2.7 compatible...
Honestly the only "trouble" I took was adding
from __future__ import print_function
everything else I just wrote in the normal way & I got lucky that nothing I did was py3 specific. Most of these scripts are using very boring python code, so its not that surprising.
Good to know! Still, no need to wait until December before dropping Python 2 support in my opinion: we should either be able to do it right now, or we're going to find ourselves in an interesting position three and a half short months from now... Let's see what Jim has to say about SUSE :)
I don't see a need to rush into it right now. Synchronizing with end of life of python upstream, makes sense as a time point and matches what QEMU will doo too, with its first release of 2020 dropping py2. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|

On Fri, 2019-09-13 at 10:40 +0100, Daniel P. Berrangé wrote:
On Fri, Sep 13, 2019 at 11:31:34AM +0200, Andrea Bolognani wrote:
Good to know! Still, no need to wait until December before dropping Python 2 support in my opinion: we should either be able to do it right now, or we're going to find ourselves in an interesting position three and a half short months from now... Let's see what Jim has to say about SUSE :)
I don't see a need to rush into it right now. Synchronizing with end of life of python upstream, makes sense as a time point and matches what QEMU will doo too, with its first release of 2020 dropping py2.
Sure, that sounds fine as long as we've made sure we're ready for the switch when the time comes. -- Andrea Bolognani / Red Hat / Virtualization

On 9/13/19 3:09 AM, Andrea Bolognani wrote:
If I understand the SLES / OpenSUSE situation correctly, then SLES 12.3 should be comparable to OpenSUSE 42.3,
Nice job! Even SUSE folks are known to get confused by the numbering :-). openSUSE 42.3 ~= SLES12 SP3 SLES12 SP4 and SP5 have no openSUSE equivalent openSUSE 15 ~= SLES15 openSUSE 15.1 ~= SLES15 SP1 openSUSE 15.2 ~= SLES15 SP2 ... openSUSE 16 ~= SLES16 (assumed, since they don't exist) Regards, Jim

On 9/13/19 2:56 AM, Andrea Bolognani wrote:
On Wed, 2019-09-11 at 17:23 +0100, Daniel P. Berrangé wrote:
This series is an effort to reduce the number of different languages we use by eliminating most use of perl in favour of python.
Just today I found out about
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/htm...
which means that if we interpret "supporting RHEL 7" as "supporting the most recent RHEL 7 point release", which I believe we do, then that's one less platform where we are forced to use Python 2! \o/
It might even be the last one, but I'm not entirely sure what the situation is like for SLES and OpenSUSE... Jim, does SLES 12 have Python 3?
Yes, python 3.4.6. And python 2.7.13.
And, as a side note: do you think you could find the time to add OpenSUSE support to the libvirt-jenkins-ci project? That'd be very useful, because it makes grepping for this kind of information trivial, and also would open the door to running actual CI jobs on the OS :)
I have internal jobs but agreed it would be nice to have openSUSE included in upstream CI on vanilla upstream :-). Any pointers on how to do that? Regards, Jim

On Fri, 2019-09-13 at 13:58 +0000, Jim Fehlig wrote:
On 9/13/19 2:56 AM, Andrea Bolognani wrote:
Jim, does SLES 12 have Python 3?
Yes, python 3.4.6. And python 2.7.13.
That's *amazing* news! \o/
And, as a side note: do you think you could find the time to add OpenSUSE support to the libvirt-jenkins-ci project? That'd be very useful, because it makes grepping for this kind of information trivial, and also would open the door to running actual CI jobs on the OS :)
I have internal jobs but agreed it would be nice to have openSUSE included in upstream CI on vanilla upstream :-). Any pointers on how to do that?
Good to hear you're willing to help with this effort! I've been wanting to introduce OpenSUSE support myself for a very long time, but unfortunately I've never quite managed to scrap together the necessary time. The repository is https://libvirt.org/git/?p=libvirt-jenkins-ci.git and most of the stuff you'd have to touch is inside the guests/ subdirectory; more specifically, you're definitely going to have to: * add OpenSUSE 12 and 15 to inventory and host_vars/; * write an AutoYaST configuration file that can be used to install a minimal OpenSUSE guest without user interaction and add it to the configs/ directory along with the existing preseed and kickstart configurations; * add mappings from abstract package names, such as 'dtrace', to the corresponding OpenSUSE concrete package names, such as 'systemtap-sdt-devel', to vars/mappings.yml; * test installation and see what breaks, fix it, rinse and repeat :D There are more steps necessary to reach full integration after the above, such as ensuring Dockerfiles can be generated, figuring out which projects can be build on the new target platforms and tweaking the Jenkins / Ansible configuration, but the above is a good start and also the part that I'm less comfortable doing myself since I'm not at all familiar with SUSE, so if you could take care of the above that'd be a big help indeed! The guests/README.md file provides some more information, and the lcitool script is hopefully not too difficult to understand. Last but not least, if you have any questions you already know how to get in touch with me ;) -- Andrea Bolognani / Red Hat / Virtualization

On 9/13/19 9:28 AM, Andrea Bolognani wrote:
On Fri, 2019-09-13 at 13:58 +0000, Jim Fehlig wrote:
On 9/13/19 2:56 AM, Andrea Bolognani wrote:
Jim, does SLES 12 have Python 3?
Yes, python 3.4.6. And python 2.7.13.
That's *amazing* news! \o/
And, as a side note: do you think you could find the time to add OpenSUSE support to the libvirt-jenkins-ci project? That'd be very useful, because it makes grepping for this kind of information trivial, and also would open the door to running actual CI jobs on the OS :)
I have internal jobs but agreed it would be nice to have openSUSE included in upstream CI on vanilla upstream :-). Any pointers on how to do that?
Good to hear you're willing to help with this effort! I've been wanting to introduce OpenSUSE support myself for a very long time, but unfortunately I've never quite managed to scrap together the necessary time.
Thanks for the detailed info! I have a busy month ahead, including traveling the next two weeks, so I likely wont get to it until Oct. But I will get to it :-). Regards, Jim
The repository is
https://libvirt.org/git/?p=libvirt-jenkins-ci.git
and most of the stuff you'd have to touch is inside the guests/ subdirectory; more specifically, you're definitely going to have to:
* add OpenSUSE 12 and 15 to inventory and host_vars/;
* write an AutoYaST configuration file that can be used to install a minimal OpenSUSE guest without user interaction and add it to the configs/ directory along with the existing preseed and kickstart configurations;
* add mappings from abstract package names, such as 'dtrace', to the corresponding OpenSUSE concrete package names, such as 'systemtap-sdt-devel', to vars/mappings.yml;
* test installation and see what breaks, fix it, rinse and repeat :D
There are more steps necessary to reach full integration after the above, such as ensuring Dockerfiles can be generated, figuring out which projects can be build on the new target platforms and tweaking the Jenkins / Ansible configuration, but the above is a good start and also the part that I'm less comfortable doing myself since I'm not at all familiar with SUSE, so if you could take care of the above that'd be a big help indeed!
The guests/README.md file provides some more information, and the lcitool script is hopefully not too difficult to understand. Last but not least, if you have any questions you already know how to get in touch with me ;)

On Fri, 2019-09-13 at 18:24 +0000, Jim Fehlig wrote:
On 9/13/19 9:28 AM, Andrea Bolognani wrote:
Good to hear you're willing to help with this effort! I've been wanting to introduce OpenSUSE support myself for a very long time, but unfortunately I've never quite managed to scrap together the necessary time.
Thanks for the detailed info! I have a busy month ahead, including traveling the next two weeks, so I likely wont get to it until Oct. But I will get to it :-).
No need to rush! :) -- Andrea Bolognani / Red Hat / Virtualization

On 9/13/19 9:28 AM, Andrea Bolognani wrote:
On Fri, 2019-09-13 at 13:58 +0000, Jim Fehlig wrote:
On 9/13/19 2:56 AM, Andrea Bolognani wrote:
Jim, does SLES 12 have Python 3?
Yes, python 3.4.6. And python 2.7.13.
That's *amazing* news! \o/
And, as a side note: do you think you could find the time to add OpenSUSE support to the libvirt-jenkins-ci project? That'd be very useful, because it makes grepping for this kind of information trivial, and also would open the door to running actual CI jobs on the OS :)
I have internal jobs but agreed it would be nice to have openSUSE included in upstream CI on vanilla upstream :-). Any pointers on how to do that?
Sorry for the delay. Thanks again for the below info!
Good to hear you're willing to help with this effort! I've been wanting to introduce OpenSUSE support myself for a very long time, but unfortunately I've never quite managed to scrap together the necessary time.
The repository is
https://libvirt.org/git/?p=libvirt-jenkins-ci.git
and most of the stuff you'd have to touch is inside the guests/ subdirectory; more specifically, you're definitely going to have to:
* add OpenSUSE 12 and 15 to inventory and host_vars/;
* write an AutoYaST configuration file that can be used to install a minimal OpenSUSE guest without user interaction and add it to the configs/ directory along with the existing preseed and kickstart configurations;
* add mappings from abstract package names, such as 'dtrace', to the corresponding OpenSUSE concrete package names, such as 'systemtap-sdt-devel', to vars/mappings.yml;
I've completed these steps.
* test installation and see what breaks, fix it, rinse and repeat :D
But I'm still in the wash, rinse, repeat cycle :-). I'm able to 'lcitool install' an openSUSE Leap 15.1 guest, still working on 'lcitool update'. Nonetheless I thought I'd send out my WIP before calling it a week. Any comments on what I have thus far is appreciated. Regards, Jim

On Sat, 2019-11-16 at 00:35 +0000, Jim Fehlig wrote:
On 9/13/19 9:28 AM, Andrea Bolognani wrote:
[...] you're definitely going to have to:
* add OpenSUSE 12 and 15 to inventory and host_vars/;
* write an AutoYaST configuration file that can be used to install a minimal OpenSUSE guest without user interaction and add it to the configs/ directory along with the existing preseed and kickstart configurations;
* add mappings from abstract package names, such as 'dtrace', to the corresponding OpenSUSE concrete package names, such as 'systemtap-sdt-devel', to vars/mappings.yml;
I've completed these steps.
* test installation and see what breaks, fix it, rinse and repeat :D
But I'm still in the wash, rinse, repeat cycle :-). I'm able to 'lcitool install' an openSUSE Leap 15.1 guest, still working on 'lcitool update'.
Nonetheless I thought I'd send out my WIP before calling it a week. Any comments on what I have thus far is appreciated.
Do you mind sendind the patch out as an RFC against the libvirt-jenkins-ci project? That way I can conveniently add comments there instead of hijacking this unrelated thread further :) -- Andrea Bolognani / Red Hat / Virtualization

On Wed, Sep 11, 2019 at 05:23:09PM +0100, Daniel P. Berrangé wrote:
This series is an effort to reduce the number of different languages we use by eliminating most use of perl in favour of python.
This aligns with fact that the likely future build system we'll use (meson) is written in python, and that python is much more commonly used/understood by developers these days than perl.
With this applied we use perl in a handful of places only:
BTW, after the first 3, these patches are all independant so can be applied in any order at any time, so no need to review it all for a big bang merge. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
participants (4)
-
Andrea Bolognani
-
Daniel P. Berrangé
-
Eric Blake
-
Jim Fehlig