[libvirt] [PATCH v6 0/4] scripts: convert most perl scripts to python
This series is an effort to reduce the number of different languages we use by eliminating most use of perl in favour of python. This aligns with fact that the likely future build system we'll use (meson) is written in python, and that python is much more commonly used/understood by developers these days than perl. With this applied we use perl in a handful of places only: - src/rpc/gendispatch.pl - this is a horrendously large script and very hard to understand/follow. A straight syntax conversion to Python would still leave a hgue and hard to understand/follow script. It really needs a clean room rewrite from scratch, with better structure. - src/rpc/genprotocol.pl - fairly easy to convert, but might be obsolete depending on approach for rewriting gendispatch.pl, so ignored for now - tests/oomtrace.pl - will be purge by the patches that drop OOM handling anyway - tools/wireshark/util/genxdrstub.pl - a very large script, which I haven't got the courage to tackle yet. - cfg.mk/maint.mk - many syntax rules involve regexes which are fed to perl. Decision on what to do with syntax-check rules punted to another time. - build-aux/gitlog-to-changelog - build-aux/useless-if-before-free - Both pulled in from gnulib. Could be rewritten quite easily if desired, but given that we aren't maintainers of them right now, they're ignored as they don't really impact our developers. Note that the check-spacing.py script is significantly slower in Python than in Perl. After researching this it appears there is nothing that can be done. The Perl regex engine is simply much better optimized than the Python one. As previously discussed we need to loook at uncrustify or clang-format or some other tool to validate whitespace formatting. This is ongoing. We can either accept the slow down in the short term or keep the Perl version in the short term. In v6: - Fix various bugs in check-file-access.py so it now reports same output as original perl script. - Merged the already-ACKd scripts In v5: - Rebased to cope with changes to require VPATH build - Merged the already-ACKd scripts In v4: - Moved all scripts into the scripts/ directory instead of having them scattered around source tree - Use re.search instead of re.match - Don't bother re-compiling regexes In v3: - All scripts comply with all flake8 style rules with exception of E129 visually indented line with same indent as next logical line In v2: - Pulled in patch to hacking file - Converted many more scripts - Forced UTF-8 character set to avoid ascii codec on py3 < 3.7 Daniel P. Berrangé (4): tests: rewrite qemu capability grouper in Python tests: rewrite file access checker in Python docs: rewrite hvsupport.html page generator in python docs: rewrite polkit docs generator in Python Makefile.am | 4 + build-aux/syntax-check.mk | 3 +- docs/Makefile.am | 12 +- docs/genaclperms.pl | 125 -------- docs/hvsupport.pl | 459 ----------------------------- scripts/check-file-access.py | 125 ++++++++ scripts/genaclperms.py | 121 ++++++++ scripts/group-qemu-caps.py | 121 ++++++++ scripts/hvsupport.py | 502 ++++++++++++++++++++++++++++++++ tests/Makefile.am | 3 +- tests/check-file-access.pl | 130 --------- tests/file_access_whitelist.txt | 2 +- tests/group-qemu-caps.pl | 124 -------- 13 files changed, 882 insertions(+), 849 deletions(-) delete mode 100755 docs/genaclperms.pl delete mode 100755 docs/hvsupport.pl create mode 100755 scripts/check-file-access.py create mode 100755 scripts/genaclperms.py create mode 100755 scripts/group-qemu-caps.py create mode 100755 scripts/hvsupport.py delete mode 100755 tests/check-file-access.pl delete mode 100755 tests/group-qemu-caps.pl -- 2.21.0
As part of a goal to eliminate Perl from libvirt build tools, rewrite the group-qemu-caps.pl tool in Python. This was a straight conversion, manually going line-by-line to change the syntax from Perl to Python. Thus the overall structure of the file and approach is the same. Tested-by: Cole Robinson <crobinso@redhat.com> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- Makefile.am | 1 + build-aux/syntax-check.mk | 3 +- scripts/group-qemu-caps.py | 121 ++++++++++++++++++++++++++++++++++++ tests/group-qemu-caps.pl | 124 ------------------------------------- 4 files changed, 124 insertions(+), 125 deletions(-) create mode 100755 scripts/group-qemu-caps.py delete mode 100755 tests/group-qemu-caps.pl diff --git a/Makefile.am b/Makefile.am index 15df6ed68e..545f2bdebe 100644 --- a/Makefile.am +++ b/Makefile.am @@ -59,6 +59,7 @@ EXTRA_DIST = \ scripts/esx_vi_generator.py \ scripts/genpolkit.py \ scripts/gensystemtap.py \ + scripts/group-qemu-caps.py \ scripts/header-ifdef.py \ scripts/hyperv_wmi_generator.py \ scripts/minimize-po.py \ diff --git a/build-aux/syntax-check.mk b/build-aux/syntax-check.mk index b29def5a30..5cc1e206c2 100644 --- a/build-aux/syntax-check.mk +++ b/build-aux/syntax-check.mk @@ -2194,7 +2194,8 @@ test-wrap-argv: $(PYTHON) $(top_srcdir)/scripts/test-wrap-argv.py --check group-qemu-caps: - $(AM_V_GEN)$(PERL) $(top_srcdir)/tests/group-qemu-caps.pl --check $(top_srcdir)/ + $(AM_V_GEN)$(RUNUTF8) $(PYTHON) $(top_srcdir)/scripts/group-qemu-caps.py \ + --check --prefix $(top_srcdir)/ # List all syntax-check exemptions: exclude_file_name_regexp--sc_avoid_strcase = ^tools/vsh\.h$$ diff --git a/scripts/group-qemu-caps.py b/scripts/group-qemu-caps.py new file mode 100755 index 0000000000..c53ff3ff6f --- /dev/null +++ b/scripts/group-qemu-caps.py @@ -0,0 +1,121 @@ +#!/usr/bin/env python3 +# +# This library is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 2.1 of the License, or (at your option) any later version. +# +# This library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this library. If not, see +# <http://www.gnu.org/licenses/>. +# +# +# Regroup array values into smaller groups separated by numbered comments. +# +# If --check is the first parameter, the script will return +# a non-zero value if a file is not grouped correctly. +# Otherwise the files are regrouped in place. + +import argparse +import re +import subprocess +import sys + + +def regroup_caps(check, filename, start_regex, end_regex, + trailing_newline, counter_prefix): + step = 5 + + original = [] + with open(filename, "r") as fh: + for line in fh: + original.append(line) + + fixed = [] + game_on = False + counter = 0 + for line in original: + line = line.rstrip("\n") + if game_on: + if re.search(r'''.*/\* [0-9]+ \*/.*''', line): + continue + if re.search(r'''^\s*$''', line): + continue + if counter % step == 0: + if counter != 0: + fixed.append("\n") + fixed.append("%s/* %d */\n" % (counter_prefix, counter)) + + if not (line.find("/*") != -1 and line.find("*/") == -1): + # count two-line comments as one line + counter = counter + 1 + + if re.search(start_regex, line): + game_on = True + elif game_on and re.search(end_regex, line): + if (counter - 1) % step == 0: + fixed = fixed[:-1] # /* $counter */ + if counter != 1: + fixed = fixed[:-1] # \n + + if trailing_newline: + fixed.append("\n") + + game_on = False + + fixed.append(line + "\n") + + if check: + orig = "".join(original) + new = "".join(fixed) + if new != orig: + diff = subprocess.Popen(["diff", "-u", filename, "-"], + stdin=subprocess.PIPE) + diff.communicate(input=new.encode('utf-8')) + + print("Incorrect line wrapping in $file", + file=sys.stderr) + print("Use group-qemu-caps.py to generate data files", + file=sys.stderr) + return False + else: + with open(filename, "w") as fh: + for line in fixed: + print(line, file=fh, end='') + + return True + + +parser = argparse.ArgumentParser(description='QEMU capabilities group formatter') +parser.add_argument('--check', action="store_true", + help='check existing files only') +parser.add_argument('--prefix', default='', + help='source code tree prefix') +args = parser.parse_args() + +errs = False + +if not regroup_caps(args.check, + args.prefix + 'src/qemu/qemu_capabilities.c', + r'virQEMUCaps grouping marker', + r'\);', + 0, + " "): + errs = True + +if not regroup_caps(args.check, + args.prefix + 'src/qemu/qemu_capabilities.h', + r'virQEMUCapsFlags grouping marker', + r'QEMU_CAPS_LAST \/\* this must', + 1, + " "): + errs = True + +if errs: + sys.exit(1) +sys.exit(0) diff --git a/tests/group-qemu-caps.pl b/tests/group-qemu-caps.pl deleted file mode 100755 index 829e63a562..0000000000 --- a/tests/group-qemu-caps.pl +++ /dev/null @@ -1,124 +0,0 @@ -#!/usr/bin/env perl -# -# This library is free software; you can redistribute it and/or -# modify it under the terms of the GNU Lesser General Public -# License as published by the Free Software Foundation; either -# version 2.1 of the License, or (at your option) any later version. -# -# This library is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# Lesser General Public License for more details. -# -# You should have received a copy of the GNU Lesser General Public -# License along with this library. If not, see -# <http://www.gnu.org/licenses/>. -# -# -# Regroup array values into smaller groups separated by numbered comments. -# -# If --check is the first parameter, the script will return -# a non-zero value if a file is not grouped correctly. -# Otherwise the files are regrouped in place. - -use strict; -use warnings; - -my $check = 0; - -if (defined $ARGV[0] && $ARGV[0] eq "--check") { - $check = 1; - shift @ARGV; -} - -my $prefix = ''; -if (defined $ARGV[0]) { - $prefix = $ARGV[0]; - shift @ARGV; -} - -my $ret = 0; -if (®roup_caps($prefix . 'src/qemu/qemu_capabilities.c', - 'virQEMUCaps grouping marker', - '\);', - 0, - " ") < 0) { - $ret = 1; -} -if (®roup_caps($prefix . 'src/qemu/qemu_capabilities.h', - 'virQEMUCapsFlags grouping marker', - 'QEMU_CAPS_LAST \/\* this must', - 1, - " ") < 0) { - $ret = 1; -} - -exit $ret; - -sub regroup_caps { - my $filename = shift; - my $start_regex = shift; - my $end_regex = shift; - my $trailing_newline = shift; - my $counter_prefix = shift; - my $step = 5; - - open FILE, '<', $filename or die "cannot open $filename: $!"; - my @original = <FILE>; - close FILE; - - my @fixed; - my $game_on = 0; - my $counter = 0; - foreach (@original) { - if ($game_on) { - next if ($_ =~ '/\* [0-9]+ \*/'); - next if (/^\s+$/); - if ($counter % $step == 0) { - if ($counter != 0) { - push @fixed, "\n"; - } - push @fixed, "$counter_prefix/* $counter */\n"; - } - if (!($_ =~ '/\*' && !($_ =~ '\*/'))) { - # count two-line comments as one line - $counter++; - } - } - if (/$start_regex/) { - $game_on = 1; - } elsif ($game_on && $_ =~ /$end_regex/) { - if (($counter -1) % $step == 0) { - pop @fixed; # /* $counter */ - if ($counter != 1) { - pop @fixed; # \n - } - } - if ($trailing_newline) { - push @fixed, "\n"; - } - $game_on = 0; - } - push @fixed, $_; - } - - if ($check) { - my $nl = join('', @fixed); - my $ol = join('', @original); - unless ($nl eq $ol) { - open DIFF, "| diff -u $filename -" or die "cannot run diff: $!"; - print DIFF $nl; - close DIFF; - - print STDERR "Incorrect array grouping in $filename\n"; - print STDERR "Use group-qemu-caps.pl to group long array members\n"; - return -1; - } - } else { - open FILE, '>', $filename or die "cannot open $filename: $!"; - foreach my $line (@fixed) { - print FILE $line; - } - close FILE; - } -} -- 2.21.0
As part of a goal to eliminate Perl from libvirt build tools, rewrite the check-file-access.pl tool in Python. This was a straight conversion, manually going line-by-line to change the syntax from Perl to Python. Thus the overall structure of the file and approach is the same. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- Makefile.am | 1 + scripts/check-file-access.py | 125 ++++++++++++++++++++++++++++++ tests/Makefile.am | 3 +- tests/check-file-access.pl | 130 -------------------------------- tests/file_access_whitelist.txt | 2 +- 5 files changed, 128 insertions(+), 133 deletions(-) create mode 100755 scripts/check-file-access.py delete mode 100755 tests/check-file-access.pl diff --git a/Makefile.am b/Makefile.am index 545f2bdebe..b9d33f5cf8 100644 --- a/Makefile.am +++ b/Makefile.am @@ -52,6 +52,7 @@ EXTRA_DIST = \ scripts/check-aclrules.py \ scripts/check-drivername.py \ scripts/check-driverimpls.py \ + scripts/check-file-access.py \ scripts/check-remote-protocol.py \ scripts/check-symfile.py \ scripts/check-symsorting.py \ diff --git a/scripts/check-file-access.py b/scripts/check-file-access.py new file mode 100755 index 0000000000..dd39de2d79 --- /dev/null +++ b/scripts/check-file-access.py @@ -0,0 +1,125 @@ +#!/usr/bin/env python3 +# +# Copyright (C) 2016-2019 Red Hat, Inc. +# +# This library is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 2.1 of the License, or (at your option) any later version. +# +# This library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this library. If not, see +# <http://www.gnu.org/licenses/>. +# +# This script is supposed to check test_file_access.txt file and +# warn about file accesses outside our working tree. +# +# + +import re +import sys + +if len(sys.argv) != 3: + print("syntax: %s ACCESS-FILE ACCESS-WHITELIST") + sys.exit(1) + +access_file = sys.argv[1] +whitelist_file = sys.argv[2] + +known_actions = ["open", "fopen", "access", "stat", "lstat", "connect"] + +files = [] +whitelist = [] + +with open(access_file, "r") as fh: + for line in fh: + line = line.rstrip("\n") + + m = re.search(r'''^(\S*):\s*(\S*):\s*(\S*)(\s*:\s*(.*))?$''', line) + if m is not None: + rec = { + "path": m.group(1), + "action": m.group(2), + "progname": m.group(3), + "testname": m.group(5), + } + files.append(rec) + else: + raise Exception("Malformed line %s" % line) + +with open(whitelist_file, "r") as fh: + for line in fh: + line = line.rstrip("\n") + + if re.search(r'''^\s*#.*$''', line): + continue # comment + if line == "": + continue + + m = re.search(r'''^(\S*):\s*(\S*)(:\s*(\S*)(\s*:\s*(.*))?)?$''', line) + if m is not None and m.group(2) in known_actions: + # $path: $action: $progname: $testname + rec = { + "path": m.group(1), + "action": m.group(3), + "progname": m.group(4), + "testname": m.group(6), + } + whitelist.append(rec) + else: + m = re.search(r'''^(\S*)(:\s*(\S*)(\s*:\s*(.*))?)?$''', line) + if m is not None: + # $path: $progname: $testname + rec = { + "path": m.group(1), + "action": None, + "progname": m.group(3), + "testname": m.group(5), + } + whitelist.append(rec) + else: + raise Exception("Malformed line %s" % line) + + +# Now we should check if %traces is included in $whitelist. For +# now checking just keys is sufficient +err = False +for file in files: + match = False + + for rule in whitelist: + if not re.match("^" + rule["path"] + "$", file["path"]): + continue + + if (rule["action"] is not None and + not re.match("^" + rule["action"] + "$", file["action"])): + continue + + if (rule["progname"] is not None and + not re.match("^" + rule["progname"] + "$", file["progname"])): + continue + + if (rule["testname"] is not None and + file["testname"] is not None and + not re.match("^" + rule["testname"] + "$", file["testname"])): + continue + + match = True + + if not match: + err = True + print("%s: %s: %s" % + (file["path"], file["action"], file["progname"]), + end="") + if file["testname"] is not None: + print(": %s" % file["testname"], end="") + print("") + +if err: + sys.exit(1) +sys.exit(0) diff --git a/tests/Makefile.am b/tests/Makefile.am index 9716d9d2be..1e364dad70 100644 --- a/tests/Makefile.am +++ b/tests/Makefile.am @@ -454,7 +454,7 @@ EXTRA_DIST += $(test_scripts) if WITH_LINUX check-access: file-access-clean VIR_TEST_FILE_ACCESS=1 $(MAKE) $(AM_MAKEFLAGS) check - $(PERL) $(abs_srcdir)/check-file-access.pl \ + $(RUNUTF8) $(PYTHON) $(top_srcdir)/scripts/check-file-access.py \ $(abs_builddir)/test_file_access.txt \ $(abs_srcdir)/file_access_whitelist.txt | sort -u @@ -463,7 +463,6 @@ file-access-clean: endif WITH_LINUX EXTRA_DIST += \ - check-file-access.pl \ file_access_whitelist.txt if WITH_TESTS diff --git a/tests/check-file-access.pl b/tests/check-file-access.pl deleted file mode 100755 index 2926126b14..0000000000 --- a/tests/check-file-access.pl +++ /dev/null @@ -1,130 +0,0 @@ -#!/usr/bin/env perl -# -# Copyright (C) 2016 Red Hat, Inc. -# -# This library is free software; you can redistribute it and/or -# modify it under the terms of the GNU Lesser General Public -# License as published by the Free Software Foundation; either -# version 2.1 of the License, or (at your option) any later version. -# -# This library is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# Lesser General Public License for more details. -# -# You should have received a copy of the GNU Lesser General Public -# License along with this library. If not, see -# <http://www.gnu.org/licenses/>. -# -# This script is supposed to check test_file_access.txt file and -# warn about file accesses outside our working tree. -# -# - -use strict; -use warnings; - -sub usage { - die "$0 access_file file_access_whitelist\n"; -} - -my $access_file = shift or usage(); -my $whitelist_file = shift or usage(); - -my @known_actions = ("open", "fopen", "access", "stat", "lstat", "connect"); - -my @files; -my @whitelist; - -open FILE, "<", $access_file or die "Unable to open $access_file: $!"; -while (<FILE>) { - chomp; - if (/^(\S*):\s*(\S*):\s*(\S*)(\s*:\s*(.*))?$/) { - my %rec; - ${rec}{path} = $1; - ${rec}{action} = $2; - ${rec}{progname} = $3; - if (defined $5) { - ${rec}{testname} = $5; - } - push (@files, \%rec); - } else { - die "Malformed line $_"; - } -} -close FILE; - -open FILE, "<", $whitelist_file or die "Unable to open $whitelist_file: $!"; -while (<FILE>) { - chomp; - if (/^\s*#.*$/) { - # comment - } elsif (/^(\S*):\s*(\S*)(:\s*(\S*)(\s*:\s*(.*))?)?$/ and - grep /^$2$/, @known_actions) { - # $path: $action: $progname: $testname - my %rec; - ${rec}{path} = $1; - ${rec}{action} = $3; - if (defined $4) { - ${rec}{progname} = $4; - } - if (defined $6) { - ${rec}{testname} = $6; - } - push (@whitelist, \%rec); - } elsif (/^(\S*)(:\s*(\S*)(\s*:\s*(.*))?)?$/) { - # $path: $progname: $testname - my %rec; - ${rec}{path} = $1; - if (defined $3) { - ${rec}{progname} = $3; - } - if (defined $5) { - ${rec}{testname} = $5; - } - push (@whitelist, \%rec); - } else { - die "Malformed line $_"; - } -} -close FILE; - -# Now we should check if %traces is included in $whitelist. For -# now checking just keys is sufficient -my $error = 0; -for my $file (@files) { - my $match = 0; - - for my $rule (@whitelist) { - if (not %${file}{path} =~ m/^$rule->{path}$/) { - next; - } - - if (defined %${rule}{action} and - not %${file}{action} =~ m/^$rule->{action}$/) { - next; - } - - if (defined %${rule}{progname} and - not %${file}{progname} =~ m/^$rule->{progname}$/) { - next; - } - - if (defined %${rule}{testname} and - defined %${file}{testname} and - not %${file}{testname} =~ m/^$rule->{testname}$/) { - next; - } - - $match = 1; - } - - if (not $match) { - $error = 1; - print "$file->{path}: $file->{action}: $file->{progname}"; - print ": $file->{testname}" if defined %${file}{testname}; - print "\n"; - } -} - -exit $error; diff --git a/tests/file_access_whitelist.txt b/tests/file_access_whitelist.txt index 3fb318cbab..5ec7ee63bb 100644 --- a/tests/file_access_whitelist.txt +++ b/tests/file_access_whitelist.txt @@ -5,7 +5,7 @@ # $path: $progname: $testname # $path: $action: $progname: $testname # -# All these variables are evaluated as perl RE. So to allow +# All these variables are evaluated as python RE. So to allow # /dev/sda and /dev/sdb, you can just '/dev/sd[a-b]', or to allow # /proc/$pid/status you can '/proc/\d+/status' and so on. # Moreover, $action, $progname and $testname can be empty, in which -- 2.21.0
As part of a goal to eliminate Perl from libvirt build tools, rewrite the hvsupport.pl tool in Python. This was a straight conversion, manually going line-by-line to change the syntax from Perl to Python. Thus the overall structure of the file and approach is the same. The new impl generates byte-for-byte identical output to the old impl. Tested-by: Cole Robinson <crobinso@redhat.com> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- Makefile.am | 1 + docs/Makefile.am | 7 +- docs/hvsupport.pl | 459 --------------------------------------- scripts/hvsupport.py | 502 +++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 506 insertions(+), 463 deletions(-) delete mode 100755 docs/hvsupport.pl create mode 100755 scripts/hvsupport.py diff --git a/Makefile.am b/Makefile.am index b9d33f5cf8..51ef567d3b 100644 --- a/Makefile.am +++ b/Makefile.am @@ -62,6 +62,7 @@ EXTRA_DIST = \ scripts/gensystemtap.py \ scripts/group-qemu-caps.py \ scripts/header-ifdef.py \ + scripts/hvsupport.py \ scripts/hyperv_wmi_generator.py \ scripts/minimize-po.py \ scripts/mock-noinline.py \ diff --git a/docs/Makefile.am b/docs/Makefile.am index 5f5dce28eb..18d17b3727 100644 --- a/docs/Makefile.am +++ b/docs/Makefile.am @@ -245,7 +245,6 @@ EXTRA_DIST= \ $(internals_html_in) $(internals_rst) $(fonts) \ $(kbase_html_in) $(kbase_rst) \ aclperms.htmlinc \ - hvsupport.pl \ $(schema_DATA) acl_generated = aclperms.htmlinc @@ -275,12 +274,12 @@ timestamp="$(shell if test -n "$$SOURCE_DATE_EPOCH"; \ hvsupport.html: hvsupport.html.in -hvsupport.html.in: $(srcdir)/hvsupport.pl $(api_DATA) \ +hvsupport.html.in: $(top_srcdir)/scripts/hvsupport.py $(api_DATA) \ $(top_srcdir)/src/libvirt_public.syms \ $(top_srcdir)/src/libvirt_qemu.syms $(top_srcdir)/src/libvirt_lxc.syms \ $(top_srcdir)/src/driver.h - $(AM_V_GEN)$(PERL) $(srcdir)/hvsupport.pl $(top_srcdir) $(top_builddir) > $@ \ - || { rm $@ && exit 1; } + $(AM_V_GEN)$(RUNUTF8) $(PYTHON) $(top_srcdir)/scripts/hvsupport.py \ + $(top_srcdir) $(top_builddir) > $@ || { rm $@ && exit 1; } news.html.in: \ $(srcdir)/news.xml \ diff --git a/docs/hvsupport.pl b/docs/hvsupport.pl deleted file mode 100755 index 0977098eac..0000000000 --- a/docs/hvsupport.pl +++ /dev/null @@ -1,459 +0,0 @@ -#!/usr/bin/env perl - -use strict; -use warnings; - -use File::Find; - -die "syntax: $0 SRCDIR BUILDDIR\n" unless int(@ARGV) == 2; - -my $srcdir = shift @ARGV; -my $builddir = shift @ARGV; - -my $symslibvirt = "$srcdir/src/libvirt_public.syms"; -my $symsqemu = "$srcdir/src/libvirt_qemu.syms"; -my $symslxc = "$srcdir/src/libvirt_lxc.syms"; -my @drivertable = ( - "$srcdir/src/driver-hypervisor.h", - "$srcdir/src/driver-interface.h", - "$srcdir/src/driver-network.h", - "$srcdir/src/driver-nodedev.h", - "$srcdir/src/driver-nwfilter.h", - "$srcdir/src/driver-secret.h", - "$srcdir/src/driver-state.h", - "$srcdir/src/driver-storage.h", - "$srcdir/src/driver-stream.h", - ); - -my %groupheaders = ( - "virHypervisorDriver" => "Hypervisor APIs", - "virNetworkDriver" => "Virtual Network APIs", - "virInterfaceDriver" => "Host Interface APIs", - "virNodeDeviceDriver" => "Host Device APIs", - "virStorageDriver" => "Storage Pool APIs", - "virSecretDriver" => "Secret APIs", - "virNWFilterDriver" => "Network Filter APIs", - ); - - -my @srcs; -find({ - wanted => sub { - if (m!$srcdir/src/.*/\w+_(driver|common|tmpl|monitor|hal|udev)\.c$!) { - push @srcs, $_ if $_ !~ /vbox_driver\.c/; - } - }, no_chdir => 1}, "$srcdir/src"); - -# Map API functions to the header and documentation files they're in -# so that we can generate proper hyperlinks to their documentation. -# -# The function names are grep'd from the XML output of apibuild.py. -sub getAPIFilenames { - my $filename = shift; - - my %files; - my $line; - - open FILE, "<", $filename or die "cannot read $filename: $!"; - - while (defined($line = <FILE>)) { - if ($line =~ /function name='([^']+)' file='([^']+)'/) { - $files{$1} = $2; - } - } - - close FILE; - - if (keys %files == 0) { - die "No functions found in $filename. Has the apibuild.py output changed?"; - } - return \%files; -} - -sub parseSymsFile { - my $apisref = shift; - my $prefix = shift; - my $filename = shift; - my $xmlfilename = shift; - - my $line; - my $vers; - my $prevvers; - - my $filenames = getAPIFilenames($xmlfilename); - - open FILE, "<$filename" - or die "cannot read $filename: $!"; - - while (defined($line = <FILE>)) { - chomp $line; - next if $line =~ /^\s*#/; - next if $line =~ /^\s*$/; - next if $line =~ /^\s*(global|local):/; - if ($line =~ /^\s*${prefix}_(\d+\.\d+\.\d+)\s*{\s*$/) { - if (defined $vers) { - die "malformed syms file"; - } - $vers = $1; - } elsif ($line =~ /\s*}\s*;\s*$/) { - if (defined $prevvers) { - die "malformed syms file"; - } - $prevvers = $vers; - $vers = undef; - } elsif ($line =~ /\s*}\s*${prefix}_(\d+\.\d+\.\d+)\s*;\s*$/) { - if ($1 ne $prevvers) { - die "malformed syms file $1 != $vers"; - } - $prevvers = $vers; - $vers = undef; - } elsif ($line =~ /\s*(\w+)\s*;\s*$/) { - $$apisref{$1} = {}; - $$apisref{$1}->{vers} = $vers; - $$apisref{$1}->{file} = $$filenames{$1}; - } else { - die "unexpected data $line\n"; - } - } - - close FILE; -} - -my %apis; -# Get the list of all public APIs and their corresponding version -parseSymsFile(\%apis, "LIBVIRT", $symslibvirt, "$builddir/docs/libvirt-api.xml"); - -# And the same for the QEMU specific APIs -parseSymsFile(\%apis, "LIBVIRT_QEMU", $symsqemu, "$builddir/docs/libvirt-qemu-api.xml"); - -# And the same for the LXC specific APIs -parseSymsFile(\%apis, "LIBVIRT_LXC", $symslxc, "$builddir/docs/libvirt-lxc-api.xml"); - - -# Some special things which aren't public APIs, -# but we want to report -$apis{virConnectSupportsFeature}->{vers} = "0.3.2"; -$apis{virDomainMigratePrepare}->{vers} = "0.3.2"; -$apis{virDomainMigratePerform}->{vers} = "0.3.2"; -$apis{virDomainMigrateFinish}->{vers} = "0.3.2"; -$apis{virDomainMigratePrepare2}->{vers} = "0.5.0"; -$apis{virDomainMigrateFinish2}->{vers} = "0.5.0"; -$apis{virDomainMigratePrepareTunnel}->{vers} = "0.7.2"; - -$apis{virDomainMigrateBegin3}->{vers} = "0.9.2"; -$apis{virDomainMigratePrepare3}->{vers} = "0.9.2"; -$apis{virDomainMigratePrepareTunnel3}->{vers} = "0.9.2"; -$apis{virDomainMigratePerform3}->{vers} = "0.9.2"; -$apis{virDomainMigrateFinish3}->{vers} = "0.9.2"; -$apis{virDomainMigrateConfirm3}->{vers} = "0.9.2"; - -$apis{virDomainMigrateBegin3Params}->{vers} = "1.1.0"; -$apis{virDomainMigratePrepare3Params}->{vers} = "1.1.0"; -$apis{virDomainMigratePrepareTunnel3Params}->{vers} = "1.1.0"; -$apis{virDomainMigratePerform3Params}->{vers} = "1.1.0"; -$apis{virDomainMigrateFinish3Params}->{vers} = "1.1.0"; -$apis{virDomainMigrateConfirm3Params}->{vers} = "1.1.0"; - - - -# Now we want to get the mapping between public APIs -# and driver struct fields. This lets us later match -# update the driver impls with the public APis. - -my $line; - -# Group name -> hash of APIs { fields -> api name } -my %groups; -my $ingrp; -foreach my $drivertable (@drivertable) { - open FILE, "<$drivertable" - or die "cannot read $drivertable: $!"; - - while (defined($line = <FILE>)) { - if ($line =~ /struct _(vir\w*Driver)/) { - my $grp = $1; - if ($grp ne "virStateDriver" && - $grp ne "virStreamDriver") { - $ingrp = $grp; - $groups{$ingrp} = { apis => {}, drivers => {} }; - } - } elsif ($ingrp) { - if ($line =~ /^\s*vir(?:Drv)(\w+)\s+(\w+);\s*$/) { - my $field = $2; - my $name = $1; - - my $api; - if (exists $apis{"vir$name"}) { - $api = "vir$name"; - } elsif ($name =~ /\w+(Open|Close|URIProbe)/) { - next; - } else { - die "driver $name does not have a public API"; - } - $groups{$ingrp}->{apis}->{$field} = $api; - } elsif ($line =~ /};/) { - $ingrp = undef; - } - } - } - - close FILE; -} - - -# Finally, we read all the primary driver files and extract -# the driver API tables from each one. - -foreach my $src (@srcs) { - open FILE, "<$src" or - die "cannot read $src: $!"; - - my $groups_regex = join("|", keys %groups); - $ingrp = undef; - my $impl; - while (defined($line = <FILE>)) { - if (!$ingrp) { - # skip non-matching lines early to save time - next if not $line =~ /$groups_regex/; - - if ($line =~ /^\s*(?:static\s+)?($groups_regex)\s+(\w+)\s*=\s*{/ || - $line =~ /^\s*(?:static\s+)?($groups_regex)\s+NAME\(\w+\)\s*=\s*{/) { - $ingrp = $1; - $impl = $src; - - if ($impl =~ m,.*/node_device_(\w+)\.c,) { - $impl = $1; - } else { - $impl =~ s,.*/(\w+?)_((\w+)_)?(\w+)\.c,$1,; - } - - if ($groups{$ingrp}->{drivers}->{$impl}) { - die "Group $ingrp already contains $impl"; - } - - $groups{$ingrp}->{drivers}->{$impl} = {}; - } - - } else { - if ($line =~ m!\s*\.(\w+)\s*=\s*(\w+)\s*,?\s*(?:/\*\s*(\d+\.\d+\.\d+)\s*(?:-\s*(\d+\.\d+\.\d+))?\s*\*/\s*)?$!) { - my $api = $1; - my $meth = $2; - my $vers = $3; - my $deleted = $4; - - next if $api eq "no" || $api eq "name"; - - if ($meth eq "NULL" && !defined $deleted) { - die "Method impl for $api is NULL, but no deleted version is provided"; - } - if ($meth ne "NULL" && defined $deleted) { - die "Method impl for $api is non-NULL, but deleted version is provided"; - } - - die "Method $meth in $src is missing version" unless defined $vers || $api eq "connectURIProbe"; - - if (!exists($groups{$ingrp}->{apis}->{$api})) { - next if $api =~ /\w(Open|Close|URIProbe)/; - - die "Found unexpected method $api in $ingrp\n"; - } - - $groups{$ingrp}->{drivers}->{$impl}->{$api} = {}; - $groups{$ingrp}->{drivers}->{$impl}->{$api}->{vers} = $vers; - $groups{$ingrp}->{drivers}->{$impl}->{$api}->{deleted} = $deleted; - if ($api eq "domainMigratePrepare" || - $api eq "domainMigratePrepare2" || - $api eq "domainMigratePrepare3") { - if (!$groups{$ingrp}->{drivers}->{$impl}->{"domainMigrate"}) { - $groups{$ingrp}->{drivers}->{$impl}->{"domainMigrate"} = {}; - $groups{$ingrp}->{drivers}->{$impl}->{"domainMigrate"}->{vers} = $vers; - } - } - - } elsif ($line =~ /}/) { - $ingrp = undef; - } - } - } - - close FILE; -} - - -# The '.open' driver method is used for 3 public APIs, so we -# have a bit of manual fixup todo with the per-driver versioning -# and support matrix - -$groups{virHypervisorDriver}->{apis}->{"openAuth"} = "virConnectOpenAuth"; -$groups{virHypervisorDriver}->{apis}->{"openReadOnly"} = "virConnectOpenReadOnly"; -$groups{virHypervisorDriver}->{apis}->{"domainMigrate"} = "virDomainMigrate"; - -my $openAuthVers = (0 * 1000 * 1000) + (4 * 1000) + 0; - -foreach my $drv (keys %{$groups{"virHypervisorDriver"}->{drivers}}) { - my $openVersStr = $groups{"virHypervisorDriver"}->{drivers}->{$drv}->{"connectOpen"}->{vers}; - my $openVers; - if ($openVersStr =~ /(\d+)\.(\d+)\.(\d+)/) { - $openVers = ($1 * 1000 * 1000) + ($2 * 1000) + $3; - } - - # virConnectOpenReadOnly always matches virConnectOpen version - $groups{"virHypervisorDriver"}->{drivers}->{$drv}->{"connectOpenReadOnly"} = - $groups{"virHypervisorDriver"}->{drivers}->{$drv}->{"connectOpen"}; - - $groups{"virHypervisorDriver"}->{drivers}->{$drv}->{"connectOpenAuth"} = {}; - - # virConnectOpenAuth is always 0.4.0 if the driver existed - # before this time, otherwise it matches the version of - # the driver's virConnectOpen entry - if ($openVersStr eq "Y" || - $openVers >= $openAuthVers) { - $groups{"virHypervisorDriver"}->{drivers}->{$drv}->{"connectOpenAuth"}->{vers} = $openVersStr; - } else { - $groups{"virHypervisorDriver"}->{drivers}->{$drv}->{"connectOpenAuth"}->{vers} = "0.4.0"; - } -} - - -# Another special case for the virDomainCreateLinux which was replaced -# with virDomainCreateXML -$groups{virHypervisorDriver}->{apis}->{"domainCreateLinux"} = "virDomainCreateLinux"; - -my $createAPIVers = (0 * 1000 * 1000) + (0 * 1000) + 3; - -foreach my $drv (keys %{$groups{"virHypervisorDriver"}->{drivers}}) { - my $createVersStr = $groups{"virHypervisorDriver"}->{drivers}->{$drv}->{"domainCreateXML"}->{vers}; - next unless defined $createVersStr; - my $createVers; - if ($createVersStr =~ /(\d+)\.(\d+)\.(\d+)/) { - $createVers = ($1 * 1000 * 1000) + ($2 * 1000) + $3; - } - - $groups{"virHypervisorDriver"}->{drivers}->{$drv}->{"domainCreateLinux"} = {}; - - # virCreateLinux is always 0.0.3 if the driver existed - # before this time, otherwise it matches the version of - # the driver's virCreateXML entry - if ($createVersStr eq "Y" || - $createVers >= $createAPIVers) { - $groups{"virHypervisorDriver"}->{drivers}->{$drv}->{"domainCreateLinux"}->{vers} = $createVersStr; - } else { - $groups{"virHypervisorDriver"}->{drivers}->{$drv}->{"domainCreateLinux"}->{vers} = "0.0.3"; - } -} - - -# Finally we generate the HTML file with the tables - -print <<EOF; -<?xml version="1.0" encoding="UTF-8"?> -<!DOCTYPE html> -<html xmlns="http://www.w3.org/1999/xhtml"> -<body class="hvsupport"> -<h1>libvirt API support matrix</h1> - -<ul id="toc"></ul> - -<p> -This page documents which <a href="html/">libvirt calls</a> work on -which libvirt drivers / hypervisors, and which version the API appeared -in. If a hypervisor driver later dropped support for the API, the version -when it was removed is also mentioned (highlighted in -<span class="removedhv">dark red</span>). -</p> - -EOF - - foreach my $grp (sort { $a cmp $b } keys %groups) { - print "<h2><a id=\"$grp\">", $groupheaders{$grp}, "</a></h2>\n"; - print <<EOF; -<table class="top_table"> -<thead> -<tr> -<th>API</th> -<th>Version</th> -EOF - - foreach my $drv (sort { $a cmp $b } keys %{$groups{$grp}->{drivers}}) { - print " <th>$drv</th>\n"; - } - - print <<EOF; -</tr> -</thead> -<tbody> -EOF - - my $row = 0; - foreach my $field (sort { - $groups{$grp}->{apis}->{$a} - cmp - $groups{$grp}->{apis}->{$b} - } keys %{$groups{$grp}->{apis}}) { - my $api = $groups{$grp}->{apis}->{$field}; - my $vers = $apis{$api}->{vers}; - my $htmlgrp = $apis{$api}->{file}; - print <<EOF; -<tr> -<td> -EOF - - if (defined $htmlgrp) { - print <<EOF; -<a href=\"html/libvirt-$htmlgrp.html#$api\">$api</a> -EOF - - } else { - print $api; - } - print <<EOF; -</td> -<td>$vers</td> -EOF - - foreach my $drv (sort {$a cmp $b } keys %{$groups{$grp}->{drivers}}) { - print "<td>"; - if (exists $groups{$grp}->{drivers}->{$drv}->{$field}) { - if ($groups{$grp}->{drivers}->{$drv}->{$field}->{vers}) { - print $groups{$grp}->{drivers}->{$drv}->{$field}->{vers}; - } - if ($groups{$grp}->{drivers}->{$drv}->{$field}->{deleted}) { - print " - <span class=\"removedhv\">", $groups{$grp}->{drivers}->{$drv}->{$field}->{deleted}, "</span>"; - } - } - print "</td>\n"; - } - - print <<EOF; -</tr> -EOF - - $row++; - if (($row % 15) == 0) { - print <<EOF; -<tr> -<th>API</th> -<th>Version</th> -EOF - - foreach my $drv (sort { $a cmp $b } keys %{$groups{$grp}->{drivers}}) { - print " <th>$drv</th>\n"; - } - - print <<EOF; -</tr> -EOF - } - - } - - print <<EOF; -</tbody> -</table> -EOF -} - -print <<EOF; -</body> -</html> -EOF diff --git a/scripts/hvsupport.py b/scripts/hvsupport.py new file mode 100755 index 0000000000..14c41da348 --- /dev/null +++ b/scripts/hvsupport.py @@ -0,0 +1,502 @@ +#!/usr/bin/env python3 +# +# Copyright (C) 2011-2019 Red Hat, Inc. +# +# This library is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 2.1 of the License, or (at your option) any later version. +# +# This library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this library. If not, see +# <http://www.gnu.org/licenses/>. + +import sys +import os.path +import re + +if len(sys.argv) != 3: + print("syntax: %s TOP-SRCDIR TOP-BUILDDIR\n" % sys.argv[0], file=sys.stderr) + +srcdir = sys.argv[1] +builddir = sys.argv[2] + +symslibvirt = os.path.join(srcdir, "src", "libvirt_public.syms") +symsqemu = os.path.join(srcdir, "src", "libvirt_qemu.syms") +symslxc = os.path.join(srcdir, "src", "libvirt_lxc.syms") +drivertablefiles = [ + os.path.join(srcdir, "src", "driver-hypervisor.h"), + os.path.join(srcdir, "src", "driver-interface.h"), + os.path.join(srcdir, "src", "driver-network.h"), + os.path.join(srcdir, "src", "driver-nodedev.h"), + os.path.join(srcdir, "src", "driver-nwfilter.h"), + os.path.join(srcdir, "src", "driver-secret.h"), + os.path.join(srcdir, "src", "driver-state.h"), + os.path.join(srcdir, "src", "driver-storage.h"), + os.path.join(srcdir, "src", "driver-stream.h"), +] + +groupheaders = { + "virHypervisorDriver": "Hypervisor APIs", + "virNetworkDriver": "Virtual Network APIs", + "virInterfaceDriver": "Host Interface APIs", + "virNodeDeviceDriver": "Host Device APIs", + "virStorageDriver": "Storage Pool APIs", + "virSecretDriver": "Secret APIs", + "virNWFilterDriver": "Network Filter APIs", +} + + +srcs = [] +for root, dirs, files in os.walk(os.path.join(srcdir, "src")): + for file in files: + if ((file.endswith("driver.c") and + not file.endswith("vbox_driver.c")) or + file.endswith("common.c") or + file.endswith("tmpl.c") or + file.endswith("monitor.c") or + file.endswith("hal.c") or + file.endswith("udev.c")): + srcs.append(os.path.join(root, file)) + + +# Map API functions to the header and documentation files they're in +# so that we can generate proper hyperlinks to their documentation. +# +# The function names are grep'd from the XML output of apibuild.py. +def getAPIFilenames(filename): + files = {} + + with open(filename) as fh: + for line in fh: + res = re.search(r"function name='([^']+)' file='([^']+)'", line) + if res is not None: + files[res.group(1)] = res.group(2) + + if len(files) == 0: + raise Exception(("No functions found in %s. " + + "Has the apibuild.py output changed?") % + filename) + + return files + + +def parseSymsFile(apisref, prefix, filename, xmlfilename): + vers = None + prevvers = None + + filenames = getAPIFilenames(xmlfilename) + + with open(filename) as fh: + for line in fh: + line = line.strip() + + if line == "": + continue + if line[0] == '#': + continue + if line.startswith("global:"): + continue + if line.startswith("local:"): + continue + + groupstartmatch = re.search(r"^\s*%s_(\d+\.\d+\.\d+)\s*{\s*$" % + prefix, line) + groupendmatch1 = re.search(r"^\s*}\s*;\s*$", line) + groupendmatch2 = re.search(r"^\s*}\s*%s_(\d+\.\d+\.\d+)\s*;\s*$" % + prefix, line) + symbolmatch = re.search(r"^\s*(\w+)\s*;\s*$", line) + if groupstartmatch is not None: + if vers is not None: + raise Exception("malformed syms file when starting group") + + vers = groupstartmatch.group(1) + elif groupendmatch1 is not None: + if prevvers is not None: + raise Exception("malformed syms file when ending group") + + prevvers = vers + vers = None + elif groupendmatch2 is not None: + if groupendmatch2.group(1) != prevvers: + raise Exception(("malformed syms file %s != %s " + + "when ending group") % + (groupendmatch2.group(1), prevvers)) + + prevvers = vers + vers = None + elif symbolmatch is not None: + name = symbolmatch.group(1) + apisref[name] = { + "vers": vers, + "file": filenames.get(name), + } + else: + raise Exception("unexpected data %s" % line) + + +apis = {} +# Get the list of all public APIs and their corresponding version +parseSymsFile(apis, "LIBVIRT", symslibvirt, + os.path.join(builddir, "docs", "libvirt-api.xml")) + +# And the same for the QEMU specific APIs +parseSymsFile(apis, "LIBVIRT_QEMU", symsqemu, + os.path.join(builddir, "docs", "libvirt-qemu-api.xml")) + +# And the same for the LXC specific APIs +parseSymsFile(apis, "LIBVIRT_LXC", symslxc, + os.path.join(builddir, "docs", "libvirt-lxc-api.xml")) + + +# Some special things which aren't public APIs, +# but we want to report +apis["virConnectSupportsFeature"] = { + "vers": "0.3.2" +} +apis["virDomainMigratePrepare"] = { + "vers": "0.3.2" +} +apis["virDomainMigratePerform"] = { + "vers": "0.3.2" +} +apis["virDomainMigrateFinish"] = { + "vers": "0.3.2" +} +apis["virDomainMigratePrepare2"] = { + "vers": "0.5.0" +} +apis["virDomainMigrateFinish2"] = { + "vers": "0.5.0" +} +apis["virDomainMigratePrepareTunnel"] = { + "vers": "0.7.2" +} + +apis["virDomainMigrateBegin3"] = { + "vers": "0.9.2" +} +apis["virDomainMigratePrepare3"] = { + "vers": "0.9.2" +} +apis["virDomainMigratePrepareTunnel3"] = { + "vers": "0.9.2" +} +apis["virDomainMigratePerform3"] = { + "vers": "0.9.2" +} +apis["virDomainMigrateFinish3"] = { + "vers": "0.9.2" +} +apis["virDomainMigrateConfirm3"] = { + "vers": "0.9.2" +} + +apis["virDomainMigrateBegin3Params"] = { + "vers": "1.1.0" +} +apis["virDomainMigratePrepare3Params"] = { + "vers": "1.1.0" +} +apis["virDomainMigratePrepareTunnel3Params"] = { + "vers": "1.1.0" +} +apis["virDomainMigratePerform3Params"] = { + "vers": "1.1.0" +} +apis["virDomainMigrateFinish3Params"] = { + "vers": "1.1.0" +} +apis["virDomainMigrateConfirm3Params"] = { + "vers": "1.1.0" +} + + +# Now we want to get the mapping between public APIs +# and driver struct fields. This lets us later match +# update the driver impls with the public APis. + +# Group name -> hash of APIs { fields -> api name } +groups = {} +ingrp = None +for drivertablefile in drivertablefiles: + with open(drivertablefile) as fh: + for line in fh: + starttablematch = re.search(r"struct _(vir\w*Driver)", line) + if starttablematch is not None: + grp = starttablematch.group(1) + if grp != "virStateDriver" and grp != "virStreamDriver": + ingrp = grp + groups[ingrp] = { + "apis": {}, + "drivers": {} + } + elif ingrp is not None: + callbackmatch = re.search(r"^\s*vir(?:Drv)(\w+)\s+(\w+);\s*$", + line) + if callbackmatch is not None: + name = callbackmatch.group(1) + field = callbackmatch.group(2) + + api = "vir" + name + if api in apis: + groups[ingrp]["apis"][field] = api + elif re.search(r"\w+(Open|Close|URIProbe)", api) is not None: + continue + else: + raise Exception(("driver %s does not have " + + "a public API") % name) + elif re.search(r"};", line): + ingrp = None + + +# Finally, we read all the primary driver files and extract +# the driver API tables from each one. + +for src in srcs: + with open(src) as fh: + groupsre = "|".join(groups.keys()) + + ingrp = None + impl = None + for line in fh: + if ingrp is None: + m = re.search(r"^\s*(static\s+)?(" + + groupsre + + r")\s+(\w+)\s*=\s*{", line) + if m is None: + m = re.search(r"^\s*(static\s+)?(" + + groupsre + + r")\s+NAME\(\w+\)\s*=\s*{", line) + if m is not None: + ingrp = m.group(2) + impl = src + + implmatch = re.search(r".*/node_device_(\w+)\.c", impl) + if implmatch is None: + implmatch = re.search(r".*/(\w+?)_((\w+)_)?(\w+)\.c", impl) + if implmatch is None: + raise Exception("Unexpected impl format '%s'" % impl) + impl = implmatch.group(1) + + if impl in groups[ingrp]["drivers"]: + raise Exception( + "Group %s already contains %s" % (ingrp, impl)) + + groups[ingrp]["drivers"][impl] = {} + else: + callbackmatch = re.search(r"\s*\.(\w+)\s*=\s*(\w+)\s*,?\s*" + + r"(?:/\*\s*(\d+\.\d+\.\d+)\s*" + + r"(?:-\s*(\d+\.\d+\.\d+))?\s*\*/\s*)?$", + line) + if callbackmatch is not None: + api = callbackmatch.group(1) + meth = callbackmatch.group(2) + vers = callbackmatch.group(3) + deleted = callbackmatch.group(4) + + if api == "no" or api == "name": + continue + + if meth == "NULL" and deleted is None: + raise Exception( + ("Method impl for %s is NULL, but " + + "no deleted version is provided") % api) + + if meth != "NULL" and deleted is not None: + raise Exception( + ("Method impl for %s is non-NULL, but " + + "deleted version is provided") % api) + + if vers is None and api != "connectURIProbe": + raise Exception( + "Method %s in %s is missing version" % + (meth, src)) + + if api not in groups[ingrp]["apis"]: + if re.search(r"\w+(Open|Close|URIProbe)", api): + continue + + raise Exception("Found unexpected method " + + "%s in %s" % (api, ingrp)) + + groups[ingrp]["drivers"][impl][api] = { + "vers": vers, + "deleted": deleted, + } + + if (api == "domainMigratePrepare" or + api == "domainMigratePrepare2" or + api == "domainMigratePrepare3"): + if ("domainMigrate" not in + groups[ingrp]["drivers"][impl]): + groups[ingrp]["drivers"][impl]["domainMigrate"] = { + "vers": vers, + } + elif line.find("}") != -1: + ingrp = None + + +# The '.open' driver method is used for 3 public APIs, so we +# have a bit of manual fixup todo with the per-driver versioning +# and support matrix + +groups["virHypervisorDriver"]["apis"]["openAuth"] = \ + "virConnectOpenAuth" +groups["virHypervisorDriver"]["apis"]["openReadOnly"] = \ + "virConnectOpenReadOnly" +groups["virHypervisorDriver"]["apis"]["domainMigrate"] = \ + "virDomainMigrate" + +openAuthVers = (0 * 1000 * 1000) + (4 * 1000) + 0 + +drivers = groups["virHypervisorDriver"]["drivers"] +for drv in drivers.keys(): + openVersStr = drivers[drv]["connectOpen"]["vers"] + openVers = 0 + if openVersStr != "Y": + openVersBits = openVersStr.split(".") + if len(openVersBits) != 3: + raise Exception("Expected 3 digit version for %s" % openVersStr) + openVers = ((int(openVersBits[0]) * 1000 * 1000) + + (int(openVersBits[1]) * 1000) + + int(openVersBits[2])) + + # virConnectOpenReadOnly always matches virConnectOpen version + drivers[drv]["connectOpenReadOnly"] = \ + drivers[drv]["connectOpen"] + + # virConnectOpenAuth is always 0.4.0 if the driver existed + # before this time, otherwise it matches the version of + # the driver's virConnectOpen entry + if openVersStr == "Y" or openVers >= openAuthVers: + vers = openVersStr + else: + vers = "0.4.0" + drivers[drv]["connectOpenAuth"] = { + "vers": vers, + } + + +# Another special case for the virDomainCreateLinux which was replaced +# with virDomainCreateXML +groups["virHypervisorDriver"]["apis"]["domainCreateLinux"] = \ + "virDomainCreateLinux" + +createAPIVers = (0 * 1000 * 1000) + (0 * 1000) + 3 + +for drv in drivers.keys(): + if "domainCreateXML" not in drivers[drv]: + continue + createVersStr = drivers[drv]["domainCreateXML"]["vers"] + createVers = 0 + if createVersStr != "Y": + createVersBits = createVersStr.split(".") + if len(createVersBits) != 3: + raise Exception("Expected 3 digit version for %s" % createVersStr) + createVers = ((int(createVersBits[0]) * 1000 * 1000) + + (int(createVersBits[1]) * 1000) + + int(createVersBits[2])) + + # virCreateLinux is always 0.0.3 if the driver existed + # before this time, otherwise it matches the version of + # the driver's virCreateXML entry + if createVersStr == "Y" or createVers >= createAPIVers: + vers = createVersStr + else: + vers = "0.0.3" + + drivers[drv]["domainCreateLinux"] = { + "vers": vers, + } + + +# Finally we generate the HTML file with the tables + +print('''<?xml version="1.0" encoding="UTF-8"?> +<!DOCTYPE html> +<html xmlns="http://www.w3.org/1999/xhtml"> +<body class="hvsupport"> +<h1>libvirt API support matrix</h1> + +<ul id="toc"></ul> + +<p> +This page documents which <a href="html/">libvirt calls</a> work on +which libvirt drivers / hypervisors, and which version the API appeared +in. If a hypervisor driver later dropped support for the API, the version +when it was removed is also mentioned (highlighted in +<span class="removedhv">dark red</span>). +</p> +''') + +for grp in sorted(groups.keys()): + print("<h2><a id=\"%s\">%s</a></h2>" % (grp, groupheaders[grp])) + print('''<table class="top_table"> +<thead> +<tr> +<th>API</th> +<th>Version</th>''') + + for drv in sorted(groups[grp]["drivers"].keys()): + print(" <th>%s</th>" % drv) + + print('''</tr> +</thead> +<tbody>''') + + row = 0 + + def sortkey(field): + return groups[grp]["apis"][field] + + for field in sorted(groups[grp]["apis"].keys(), key=sortkey): + api = groups[grp]["apis"][field] + vers = apis[api]["vers"] + htmlgrp = apis[api].get("file") + print("<tr>") + + if htmlgrp is not None: + print(('''<td>\n<a href=\"html/libvirt-%s.html#%s\">''' + + '''%s</a>\n</td>''') % + (htmlgrp, api, api)) + else: + print("<td>\n%s</td>" % api) + + print("<td>%s</td>" % vers) + + for drv in sorted(groups[grp]["drivers"].keys()): + info = "" + if field in groups[grp]["drivers"][drv]: + vers = groups[grp]["drivers"][drv][field]["vers"] + if vers is not None: + info = info + vers + + deleted = groups[grp]["drivers"][drv][field].get("deleted") + if deleted is not None: + info = info + (''' - <span class="removedhv">''' + + '''%s</span>''' % deleted) + + print("<td>%s</td>" % info) + + print("</tr>") + + row = row + 1 + if (row % 15) == 0: + print('''<tr> +<th>API</th> +<th>Version</th>''') + + for drv in sorted(groups[grp]["drivers"].keys()): + print(" <th>%s</th>" % drv) + + print("</tr>") + + print("</tbody>\n</table>") + +print("</body>\n</html>") -- 2.21.0
As part of a goal to eliminate Perl from libvirt build tools, rewrite the genaclperms.pl tool in Python. This was a straight conversion, manually going line-by-line to change the syntax from Perl to Python. Thus the overall structure of the file and approach is the same. Tested-by: Cole Robinson <crobinso@redhat.com> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- Makefile.am | 1 + docs/Makefile.am | 5 +- docs/genaclperms.pl | 125 ----------------------------------------- scripts/genaclperms.py | 121 +++++++++++++++++++++++++++++++++++++++ 4 files changed, 124 insertions(+), 128 deletions(-) delete mode 100755 docs/genaclperms.pl create mode 100755 scripts/genaclperms.py diff --git a/Makefile.am b/Makefile.am index 51ef567d3b..632281d9b6 100644 --- a/Makefile.am +++ b/Makefile.am @@ -58,6 +58,7 @@ EXTRA_DIST = \ scripts/check-symsorting.py \ scripts/dtrace2systemtap.py \ scripts/esx_vi_generator.py \ + scripts/genaclperms.py \ scripts/genpolkit.py \ scripts/gensystemtap.py \ scripts/group-qemu-caps.py \ diff --git a/docs/Makefile.am b/docs/Makefile.am index 18d17b3727..eb17da4299 100644 --- a/docs/Makefile.am +++ b/docs/Makefile.am @@ -236,7 +236,6 @@ schemadir = $(pkgdatadir)/schemas schema_DATA = $(wildcard $(srcdir)/schemas/*.rng) EXTRA_DIST= \ - genaclperms.pl \ site.xsl subsite.xsl newapi.xsl page.xsl \ wrapstring.xsl \ $(dot_html_in) $(dot_rst) $(gif) $(apipng) \ @@ -250,8 +249,8 @@ EXTRA_DIST= \ acl_generated = aclperms.htmlinc aclperms.htmlinc: $(top_srcdir)/src/access/viraccessperm.h \ - $(srcdir)/genaclperms.pl Makefile.am - $(AM_V_GEN)$(PERL) $(srcdir)/genaclperms.pl $< > $@ + $(top_srcdir)/scripts/genaclperms.py Makefile.am + $(AM_V_GEN)$(RUNUTF8) $(PYTHON) $(top_srcdir)/scripts/genaclperms.py $< > $@ CLEANFILES = \ $(dot_html) \ diff --git a/docs/genaclperms.pl b/docs/genaclperms.pl deleted file mode 100755 index 0de2cfad4d..0000000000 --- a/docs/genaclperms.pl +++ /dev/null @@ -1,125 +0,0 @@ -#!/usr/bin/env perl -# -# Copyright (C) 2013 Red Hat, Inc. -# -# This library is free software; you can redistribute it and/or -# modify it under the terms of the GNU Lesser General Public -# License as published by the Free Software Foundation; either -# version 2.1 of the License, or (at your option) any later version. -# -# This library is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# Lesser General Public License for more details. -# -# You should have received a copy of the GNU Lesser General Public -# License along with this library. If not, see -# <http://www.gnu.org/licenses/>. -# - -use strict; -use warnings; - -my @objects = ( - "CONNECT", "DOMAIN", "INTERFACE", - "NETWORK_PORT", "NETWORK", "NODE_DEVICE", - "NWFILTER_BINDING", "NWFILTER", - "SECRET", "STORAGE_POOL", "STORAGE_VOL", - ); - -my %class; - -foreach my $object (@objects) { - my $class = lc $object; - - $class =~ s/(^\w|_\w)/uc $1/eg; - $class =~ s/_//g; - $class =~ s/Nwfilter/NWFilter/; - $class = "vir" . $class . "Ptr"; - - $class{$object} = $class; -} - -my $objects = join ("|", @objects); - -my %opts; -my $in_opts = 0; - -my %perms; - -while (<>) { - if ($in_opts) { - if (m,\*/,) { - $in_opts = 0; - } elsif (/\*\s*\@(\w+):\s*(.*?)\s*$/) { - $opts{$1} = $2; - } - } elsif (m,/\*\*,) { - $in_opts = 1; - } elsif (/VIR_ACCESS_PERM_($objects)_((?:\w|_)+),/) { - my $object = $1; - my $perm = lc $2; - next if $perm eq "last"; - - $perm =~ s/_/-/g; - - $perms{$object} = {} unless exists $perms{$object}; - $perms{$object}->{$perm} = { - desc => $opts{desc}, - message => $opts{message}, - anonymous => $opts{anonymous} - }; - %opts = (); - } -} - -print <<EOF; -<?xml version="1.0" encoding="UTF-8"?> -<!DOCTYPE html> -<html xmlns="http://www.w3.org/1999/xhtml"> - <body> -EOF - -foreach my $object (sort { $a cmp $b } keys %perms) { - my $class = $class{$object}; - my $olink = lc "object_" . $object; - print <<EOF; -<h3><a id="$olink">$class</a></h3> -<table> - <thead> - <tr> - <th>Permission</th> - <th>Description</th> - </tr> - </thead> - <tbody> -EOF - - foreach my $perm (sort { $a cmp $b } keys %{$perms{$object}}) { - my $description = $perms{$object}->{$perm}->{desc}; - - die "missing description for $object.$perm" unless - defined $description; - - my $plink = lc "perm_" . $object . "_" . $perm; - $plink =~ s/-/_/g; - - print <<EOF; - <tr> - <td><a id="$plink">$perm</a></td> - <td>$description</td> - </tr> -EOF - - } - - print <<EOF; - </tbody> -</table> -EOF -} - -print <<EOF; - </body> -</html> -EOF diff --git a/scripts/genaclperms.py b/scripts/genaclperms.py new file mode 100755 index 0000000000..e228b3ef60 --- /dev/null +++ b/scripts/genaclperms.py @@ -0,0 +1,121 @@ +#!/usr/bin/env python3 +# +# Copyright (C) 2013-2019 Red Hat, Inc. +# +# This library is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 2.1 of the License, or (at your option) any later version. +# +# This library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this library. If not, see +# <http://www.gnu.org/licenses/>. +# + +import re +import sys + +objects = [ + "CONNECT", "DOMAIN", "INTERFACE", + "NETWORK_PORT", "NETWORK", "NODE_DEVICE", + "NWFILTER_BINDING", "NWFILTER", + "SECRET", "STORAGE_POOL", "STORAGE_VOL", +] + + +classes = {} + +for obj in objects: + klass = obj.lower() + + klass = re.sub(r'''(^\w|_\w)''', lambda a: a.group(1).upper(), klass) + klass = klass.replace("_", "") + klass = klass.replace("Nwfilter", "NWFilter") + klass = "vir" + klass + "Ptr" + + classes[obj] = klass + + +objectstr = "|".join(objects) + +opts = {} +in_opts = {} + +perms = {} + +aclfile = sys.argv[1] +with open(aclfile, "r") as fh: + for line in fh: + if in_opts: + if line.find("*/") != -1: + in_opts = False + else: + m = re.search(r'''\*\s*\@(\w+):\s*(.*?)\s*$''', line) + if m is not None: + opts[m.group(1)] = m.group(2) + elif line.find("**") != -1: + in_opts = True + else: + m = re.search(r'''VIR_ACCESS_PERM_(%s)_((?:\w|_)+),''' % + objectstr, line) + if m is not None: + obj = m.group(1) + perm = m.group(2).lower() + if perm == "last": + continue + + perm = perm.replace("_", "-") + + if obj not in perms: + perms[obj] = {} + perms[obj][perm] = { + "desc": opts.get("desc", None), + "message": opts.get("message", None), + "anonymous": opts.get("anonymous", None), + } + opts = {} + +print('<?xml version="1.0" encoding="UTF-8"?>') +print('<!DOCTYPE html>') +print('<html xmlns="http://www.w3.org/1999/xhtml">') +print(' <body>') + +for obj in sorted(perms.keys()): + klass = classes[obj] + + olink = "object_" + obj.lower() + + print(' <h3><a id="%s">%s</a></h3>' % (olink, klass)) + print(' <table>') + print(' <thead>') + print(' <tr>') + print(' <th>Permission</th>') + print(' <th>Description</th>') + print(' </tr>') + print(' </thead>') + print(' <tbody>') + + for perm in sorted(perms[obj].keys()): + description = perms[obj][perm]["desc"] + + if description is None: + raise Exception("missing description for %s.%s" % (obj, perm)) + + plink = "perm_" + obj.lower() + "_" + perm.lower() + plink = plink.replace("-", "_") + + print(' <tr>') + print(' <td><a id="%s">%s</a></td>' % (plink, perm)) + print(' <td>%s</td>' % description) + print(' </tr>') + + print(' </tbody>') + print(' </table>') + +print(' </body>') +print('</html>') -- 2.21.0
participants (2)
-
Cole Robinson -
Daniel P. Berrangé