[PATCH v2 0/9] docs: Fix migration.html generation and report such errors next time
This version refactors the XSLT transformation command a bit more carefully and thoroughly. A semantic difference in v2 is that the output HTMLs are no longer reformatted. Don't apply/revert first patch to see the build failure: [59/144] Generating migration.html with a meson_exe.py custom command FAILED: docs/migration.html /usr/bin/meson --internal exe --capture docs/migration.html -- /bin/xsltproc --stringparam pagesrc docs/migration.html.in --stringparam builddir /home/pipo/build/libvirt/gcc --stringparam timestamp 'Tue Oct 13 16:16:15 2020 UTC' --nonet ../../../libvirt/docs/site.xsl ../../../libvirt/docs/migration.html.in ../../../libvirt/docs/migration.html.in:664: parser error : Opening and ending tag mismatch: p line 649 and body </body> ^ ../../../libvirt/docs/migration.html.in:665: parser error : Opening and ending tag mismatch: body line 649 and html </html> ^ ../../../libvirt/docs/migration.html.in:666: parser error : EndTag: '</' not found ^ unable to parse ../../../libvirt/docs/migration.html.in [84/144] Generating hacking.html with a meson_exe.py custom command ninja: build stopped: subcommand failed. Peter Krempa (9): docs: migration: Fix syntax scripts/meson-html-gen.py: Don't rereformat output files docs: meson.build: Limit html files depending on 'aclperms.htmlinc' docs: meson.build: Generate HTML files directly by meson docs: meson.build: Prepare for use of identical code for XSLT processing of htmls docs/internals/meson.build: Use template code for XSLT processing docs/kbase/meson.build: Use template code for XSLT processing docs/manpages/meson.build: Use template code for XSLT processing scripts: meson-html-gen: Remove docs/internals/meson.build | 55 ++++++++++++++++---------- docs/kbase/meson.build | 56 ++++++++++++++++---------- docs/manpages/meson.build | 55 +++++++++++++++++--------- docs/meson.build | 80 +++++++++++++++++++++++--------------- docs/migration.html.in | 1 + scripts/meson-html-gen.py | 37 ------------------ scripts/meson.build | 1 - 7 files changed, 157 insertions(+), 128 deletions(-) delete mode 100755 scripts/meson-html-gen.py -- 2.26.2
One of the paragraphs added in f51cbe92c0d was not terminated thus making it invalid XML/XHTML. This was not caught by the build system as 'scripts/meson-html-gen.py' unnecessarily obscures and hides errors from 'xsltproc'. Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- docs/migration.html.in | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/migration.html.in b/docs/migration.html.in index 162c202227..dd5eddd6f4 100644 --- a/docs/migration.html.in +++ b/docs/migration.html.in @@ -653,6 +653,7 @@ virsh migrate --p2p --tunnelled web1 qemu+ssh://desthost/system qemu+ssh://10.0. daemons or forwarding connections to these sockets manually (using <code>socat</code>, <code>netcat</code> or a custom piece of software): + </p> <pre> virsh migrate web1 [--p2p] --copy-storage-all 'qemu+unix:///system?socket=/tmp/migdir/test-sock-driver' 'unix:///tmp/migdir/test-sock-qemu' --disks-uri unix:///tmp/migdir/test-sock-nbd </pre> -- 2.26.2
The output HTML files (especially those generated from rST files) don't look good even after reformatting. Skip the extra step and accept that no matter what we do HTMLs will not look great. This additionally makes it way simpler to remove meson-html-gen.py in the future (thus I've neglected to remove passing of xmllint). Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- scripts/meson-html-gen.py | 9 +-------- 1 file changed, 1 insertion(+), 8 deletions(-) diff --git a/scripts/meson-html-gen.py b/scripts/meson-html-gen.py index 2731d734a7..dcc11f37cf 100755 --- a/scripts/meson-html-gen.py +++ b/scripts/meson-html-gen.py @@ -14,7 +14,7 @@ parser.add_argument("htmlfile", type=str, help="path to generated HTML file") parser.add_argument("pagesrc", type=str, default="", nargs='?', help="(optional) path to source file used for edit this page") args = parser.parse_args() -html_tmp = subprocess.run( +html = subprocess.run( [ args.xsltproc, '--stringparam', 'pagesrc', args.pagesrc, @@ -26,12 +26,5 @@ html_tmp = subprocess.run( stderr=subprocess.PIPE, ) -html = subprocess.run( - [args.xmllint, '--nonet', '--format', '-'], - input=html_tmp.stdout, - stdout=subprocess.PIPE, - stderr=subprocess.PIPE, -) - with open(args.htmlfile, 'wb') as outfile: outfile.write(html.stdout) -- 2.26.2
Only 'acl.html' output file includes that file so there's no need to make everything depend on it. Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- docs/meson.build | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/docs/meson.build b/docs/meson.build index 400c1ca955..d18e5c1feb 100644 --- a/docs/meson.build +++ b/docs/meson.build @@ -32,7 +32,6 @@ docs_assets = [ docs_html_in_files = [ '404', - 'acl', 'aclpolkit', 'api_extension', 'api', @@ -199,6 +198,7 @@ docs_rst2html_gen = generator( # name - base file name (required) # file - generated file (required) # source - source filename relative to repository root (optional, if there is no source) +# depends - explicit dependency on other input (optional) docs_html_in_gen = [] foreach name : docs_html_in_files @@ -219,6 +219,13 @@ foreach name : docs_rst_files } endforeach +docs_html_in_gen += { + 'name': 'acl.html', + 'file': 'acl.html.in', + 'source': 'docs' / 'acl.html.in', + 'depends': aclperms_gen, +} + hvsupport_html_in = custom_target( 'hvsupport.html.in', output: 'hvsupport.html.in', @@ -272,7 +279,7 @@ foreach data : docs_html_in_gen '@OUTPUT@', data.get('source', []), ], - depends: [ aclperms_gen ], + depends: data.get('depends', []), depend_files: [ page_xsl ], install: true, install_dir: docs_html_dir, -- 2.26.2
Since we no longer reformat the XSLT-transformed files, there's no need to use an external script any more. Unfortunately this hid errors from 'xsltproc' as return value was not checked and the stderr was piped into xmllints stdin. The result was that any invalid input file would result into an empty output file. Since the script's only purpose was to prevent additional temporary files at the time we were reformatting the output in a pipeline we no longer need this. Moving the generation directly into the meson definition makes it more obvious what's happening and saves readers from having to parse what's going on. A free bonus is that errors are now properly caught and reported. This patch converts the main docs/ directory for now with cleanup of other comming later. Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- docs/meson.build | 15 ++++++--------- 1 file changed, 6 insertions(+), 9 deletions(-) diff --git a/docs/meson.build b/docs/meson.build index d18e5c1feb..8b7c63bc6f 100644 --- a/docs/meson.build +++ b/docs/meson.build @@ -267,20 +267,17 @@ foreach data : docs_html_in_gen input: data['file'], output: html_file, command: [ - meson_python_prog, - python3_prog.path(), - meson_html_gen_prog.path(), - xsltproc_prog.path(), - xmllint_prog.path(), - meson.build_root(), - docs_timestamp, + xsltproc_prog, + '--stringparam', 'pagesrc', data.get('source', ''), + '--stringparam', 'builddir', meson.build_root(), + '--stringparam', 'timestamp', docs_timestamp, + '--nonet', site_xsl, '@INPUT@', - '@OUTPUT@', - data.get('source', []), ], depends: data.get('depends', []), depend_files: [ page_xsl ], + capture: true, install: true, install_dir: docs_html_dir, ) -- 2.26.2
Meson unfortunately doesn't give us any means to share the code using xsltproc to output HTMLs processed by our template. This means we will have to resort to copy&paste engineering. To make things simpler, let's use the same block of code in docs/meson.build but also any of the subdirs which generate htmls. This will be achieved by making it configurable and wrapping it in a comment that instructs anybody editing it to keep it identical. We need to be able to configure the template file used and installation directory. The rest of the processing is same as we do in docs/meson.build. This code will then be copied to subdirs to refactor the current approach used there. Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- docs/meson.build | 60 +++++++++++++++++++++++++++++------------------- 1 file changed, 36 insertions(+), 24 deletions(-) diff --git a/docs/meson.build b/docs/meson.build index 8b7c63bc6f..a915d6252a 100644 --- a/docs/meson.build +++ b/docs/meson.build @@ -193,35 +193,37 @@ docs_rst2html_gen = generator( ) -# docs_html_in_gen: +# html_xslt_gen config + +html_xslt_gen_xslt = site_xsl +html_xslt_gen_install_dir = docs_html_dir + +html_xslt_gen = [] +# html_xslt_gen: # each entry is a dictionary with following items: -# name - base file name (required) -# file - generated file (required) +# name - base file name (required), output file will become 'name.html' +# file - input file (optional, 'name.html.in' assumed if missing) # source - source filename relative to repository root (optional, if there is no source) # depends - explicit dependency on other input (optional) -docs_html_in_gen = [] foreach name : docs_html_in_files - html_in_file = '@0@.html.in'.format(name) - docs_html_in_gen += { + html_xslt_gen += { 'name': name, - 'file': html_in_file, - 'source': 'docs' / html_in_file, + 'source': 'docs' / name + '.html.in', } endforeach foreach name : docs_rst_files rst_file = '@0@.rst'.format(name) - docs_html_in_gen += { + html_xslt_gen += { 'name': name, 'file': docs_rst2html_gen.process(rst_file), 'source': 'docs' / rst_file, } endforeach -docs_html_in_gen += { - 'name': 'acl.html', - 'file': 'acl.html.in', +html_xslt_gen += { + 'name': 'acl', 'source': 'docs' / 'acl.html.in', 'depends': aclperms_gen, } @@ -247,45 +249,55 @@ hvsupport_html_in = custom_target( docs_api_generated, ], ) -docs_html_in_gen += { +html_xslt_gen += { 'name': 'hvsupport', 'file': hvsupport_html_in, } news_html_in = docs_rst2html_gen.process(meson.source_root() / 'NEWS.rst') -docs_html_in_gen += { +html_xslt_gen += { 'name': 'news', 'file': news_html_in, 'source': 'NEWS.rst', } -foreach data : docs_html_in_gen - html_file = '@0@.html'.format(data['name']) +# The following code between the markers must be kept identical with the other +# copies of the code in various subdirs, since meson doesn't support any kind +# of functions. + +# --- begin of XSLT processing --- - out_file = custom_target( - html_file, - input: data['file'], - output: html_file, +foreach data : html_xslt_gen + html_filename = data['name'] + '.html' + + html_file = custom_target( + html_filename, + input: data.get('file', data['name'] + '.html.in'), + output: html_filename, command: [ xsltproc_prog, '--stringparam', 'pagesrc', data.get('source', ''), '--stringparam', 'builddir', meson.build_root(), '--stringparam', 'timestamp', docs_timestamp, '--nonet', - site_xsl, + html_xslt_gen_xslt, '@INPUT@', ], depends: data.get('depends', []), depend_files: [ page_xsl ], capture: true, install: true, - install_dir: docs_html_dir, + install_dir: html_xslt_gen_install_dir, ) - install_web_deps += out_file - install_web_files += '@0@:@1@'.format(out_file.full_path(), docs_html_dir) + install_web_deps += html_file + install_web_files += html_file.full_path() + ':' + html_xslt_gen_install_dir endforeach +html_xslt_gen = [] + +# --- end of XSLT processing --- + subdir('fonts') subdir('html') subdir('internals') -- 2.26.2
Replace the reimplementation of the XSLT processing custom target with an identical copy form docs/meson.build and an comment to keep them in sync. Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- docs/internals/meson.build | 55 ++++++++++++++++++++++++-------------- 1 file changed, 35 insertions(+), 20 deletions(-) diff --git a/docs/internals/meson.build b/docs/internals/meson.build index 5d008dec5b..5aa67ded5c 100644 --- a/docs/internals/meson.build +++ b/docs/internals/meson.build @@ -5,33 +5,48 @@ internals_in_files = [ 'rpc', ] +html_xslt_gen_xslt = subsite_xsl +html_xslt_gen_install_dir = docs_html_dir / 'internals' +html_xslt_gen = [] + foreach name : internals_in_files - html_in_file = '@0@.html.in'.format(name) - html_file = '@0@.html'.format(name) + html_xslt_gen += { + 'name': name, + 'source': 'docs/internals' / name + '.html.in', + } +endforeach + +# keep the XSLT processing code block in sync with docs/meson.build + +# --- begin of XSLT processing --- - out_file = custom_target( - html_file, - input: html_in_file, - output: html_file, +foreach data : html_xslt_gen + html_filename = data['name'] + '.html' + + html_file = custom_target( + html_filename, + input: data.get('file', data['name'] + '.html.in'), + output: html_filename, command: [ - meson_python_prog, - python3_prog.path(), - meson_html_gen_prog.path(), - xsltproc_prog.path(), - xmllint_prog.path(), - meson.build_root(), - docs_timestamp, - subsite_xsl, + xsltproc_prog, + '--stringparam', 'pagesrc', data.get('source', ''), + '--stringparam', 'builddir', meson.build_root(), + '--stringparam', 'timestamp', docs_timestamp, + '--nonet', + html_xslt_gen_xslt, '@INPUT@', - '@OUTPUT@', - 'docs/internals' / html_in_file, ], - depends: [ aclperms_gen ], + depends: data.get('depends', []), depend_files: [ page_xsl ], + capture: true, install: true, - install_dir: docs_html_dir / 'internals', + install_dir: html_xslt_gen_install_dir, ) - install_web_deps += out_file - install_web_files += '@0@:@1@'.format(out_file.full_path(), docs_html_dir / 'internals') + install_web_deps += html_file + install_web_files += html_file.full_path() + ':' + html_xslt_gen_install_dir endforeach + +html_xslt_gen = [] + +# --- end of XSLT processing --- -- 2.26.2
Replace the reimplementation of the XSLT processing custom target with an identical copy form docs/meson.build and an comment to keep them in sync. Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- docs/kbase/meson.build | 56 +++++++++++++++++++++++++++--------------- 1 file changed, 36 insertions(+), 20 deletions(-) diff --git a/docs/kbase/meson.build b/docs/kbase/meson.build index b1d1d7610b..c0fa72ff35 100644 --- a/docs/kbase/meson.build +++ b/docs/kbase/meson.build @@ -17,35 +17,51 @@ docs_kbase_files = [ 'virtiofs', ] +html_xslt_gen_xslt = subsite_xsl +html_xslt_gen_install_dir = docs_html_dir / 'kbase' +html_xslt_gen = [] + foreach name : docs_kbase_files rst_file = '@0@.rst'.format(name) - html_file = '@0@.html'.format(name) - html_in = docs_rst2html_gen.process(rst_file) + html_xslt_gen += { + 'name': name, + 'file': docs_rst2html_gen.process(rst_file), + 'source': 'docs/kbase' / rst_file, + } +endforeach + +# keep the XSLT processing code block in sync with docs/meson.build + +# --- begin of XSLT processing --- - out_file = custom_target( - html_file, - input: html_in, - output: html_file, +foreach data : html_xslt_gen + html_filename = data['name'] + '.html' + + html_file = custom_target( + html_filename, + input: data.get('file', data['name'] + '.html.in'), + output: html_filename, command: [ - meson_python_prog, - python3_prog.path(), - meson_html_gen_prog.path(), - xsltproc_prog.path(), - xmllint_prog.path(), - meson.build_root(), - docs_timestamp, - subsite_xsl, + xsltproc_prog, + '--stringparam', 'pagesrc', data.get('source', ''), + '--stringparam', 'builddir', meson.build_root(), + '--stringparam', 'timestamp', docs_timestamp, + '--nonet', + html_xslt_gen_xslt, '@INPUT@', - '@OUTPUT@', - 'docs/kbase' / rst_file, ], - depends: [ aclperms_gen ], + depends: data.get('depends', []), depend_files: [ page_xsl ], + capture: true, install: true, - install_dir: docs_html_dir / 'kbase', + install_dir: html_xslt_gen_install_dir, ) - install_web_deps += out_file - install_web_files += '@0@:@1@'.format(out_file.full_path(), docs_html_dir / 'kbase') + install_web_deps += html_file + install_web_files += html_file.full_path() + ':' + html_xslt_gen_install_dir endforeach + +html_xslt_gen = [] + +# --- end of XSLT processing --- -- 2.26.2
Replace the reimplementation of the XSLT processing custom target with an identical copy form docs/meson.build and an comment to keep them in sync. Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- docs/manpages/meson.build | 55 ++++++++++++++++++++++++++------------- 1 file changed, 37 insertions(+), 18 deletions(-) diff --git a/docs/manpages/meson.build b/docs/manpages/meson.build index 7ed1d304a4..ecc517e80e 100644 --- a/docs/manpages/meson.build +++ b/docs/manpages/meson.build @@ -1,3 +1,7 @@ +html_xslt_gen_xslt = subsite_xsl +html_xslt_gen_install_dir = docs_html_dir / 'manpages' +html_xslt_gen = [] + # docs_man_files # each entry is a dictionary with following items: # name - man page name (required) @@ -104,29 +108,44 @@ foreach data : docs_man_files capture: true, ) - out_file = custom_target( - html_file, - input: html_in, - output: html_file, + html_xslt_gen += { + 'name': data['name'], + 'file': html_in, + 'source': 'docs/manpages' / rst_in_file, + } +endforeach + +# keep the XSLT processing code block in sync with docs/meson.build + +# --- begin of XSLT processing --- + +foreach data : html_xslt_gen + html_filename = data['name'] + '.html' + + html_file = custom_target( + html_filename, + input: data.get('file', data['name'] + '.html.in'), + output: html_filename, command: [ - meson_python_prog, - python3_prog.path(), - meson_html_gen_prog.path(), - xsltproc_prog.path(), - xmllint_prog.path(), - meson.build_root(), - docs_timestamp, - subsite_xsl, + xsltproc_prog, + '--stringparam', 'pagesrc', data.get('source', ''), + '--stringparam', 'builddir', meson.build_root(), + '--stringparam', 'timestamp', docs_timestamp, + '--nonet', + html_xslt_gen_xslt, '@INPUT@', - '@OUTPUT@', - 'docs/manpages' / rst_in_file, ], - depends: [ aclperms_gen ], + depends: data.get('depends', []), depend_files: [ page_xsl ], + capture: true, install: true, - install_dir: docs_html_dir / 'manpages', + install_dir: html_xslt_gen_install_dir, ) - install_web_deps += out_file - install_web_files += '@0@:@1@'.format(out_file.full_path(), docs_html_dir / 'manpages') + install_web_deps += html_file + install_web_files += html_file.full_path() + ':' + html_xslt_gen_install_dir endforeach + +html_xslt_gen = [] + +# --- end of XSLT processing --- -- 2.26.2
The script was obscuring what's happening and not reporting errors properly. Remove it since it's no longer used now. Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- scripts/meson-html-gen.py | 30 ------------------------------ scripts/meson.build | 1 - 2 files changed, 31 deletions(-) delete mode 100755 scripts/meson-html-gen.py diff --git a/scripts/meson-html-gen.py b/scripts/meson-html-gen.py deleted file mode 100755 index dcc11f37cf..0000000000 --- a/scripts/meson-html-gen.py +++ /dev/null @@ -1,30 +0,0 @@ -#!/usr/bin/env python3 - -import argparse -import subprocess - -parser = argparse.ArgumentParser() -parser.add_argument("xsltproc", type=str, help="path to xsltproc bin") -parser.add_argument("xmllint", type=str, help="path to xmllint bin") -parser.add_argument("builddir", type=str, help="build root dir path") -parser.add_argument("timestamp", type=str, help="docs timestamp") -parser.add_argument("style", type=str, help="XSL stile file") -parser.add_argument("infile", type=str, help="path to source HTML file") -parser.add_argument("htmlfile", type=str, help="path to generated HTML file") -parser.add_argument("pagesrc", type=str, default="", nargs='?', help="(optional) path to source file used for edit this page") -args = parser.parse_args() - -html = subprocess.run( - [ - args.xsltproc, - '--stringparam', 'pagesrc', args.pagesrc, - '--stringparam', 'builddir', args.builddir, - '--stringparam', 'timestamp', args.timestamp, - '--nonet', args.style, args.infile, - ], - stdout=subprocess.PIPE, - stderr=subprocess.PIPE, -) - -with open(args.htmlfile, 'wb') as outfile: - outfile.write(html.stdout) diff --git a/scripts/meson.build b/scripts/meson.build index 59b3c9bacd..655ec0e0e2 100644 --- a/scripts/meson.build +++ b/scripts/meson.build @@ -22,7 +22,6 @@ scripts = [ 'meson-gen-authors.py', 'meson-gen-def.py', 'meson-gen-sym.py', - 'meson-html-gen.py', 'meson-install-dirs.py', 'meson-install-symlink.py', 'meson-install-web.py', -- 2.26.2
On Tue, Oct 13, 2020 at 18:18:04 +0200, Peter Krempa wrote:
This version refactors the XSLT transformation command a bit more carefully and thoroughly. A semantic difference in v2 is that the output HTMLs are no longer reformatted.
The output can be verified here: https://gitlab.com/pipo.sk/libvirt/-/jobs/788658848/artifacts/browse/website...
On Tue, Oct 13, 2020 at 06:30:32PM +0200, Peter Krempa wrote:
On Tue, Oct 13, 2020 at 18:18:04 +0200, Peter Krempa wrote:
This version refactors the XSLT transformation command a bit more carefully and thoroughly. A semantic difference in v2 is that the output HTMLs are no longer reformatted.
The output can be verified here:
https://gitlab.com/pipo.sk/libvirt/-/jobs/788658848/artifacts/browse/website...
Yeah, that shows me I need to do one other fix for the migration.html (see last code block) or our stylesheet (probably the former, though). Anyway this looks good except one tiny little thing (or three), so Reviewed-by: Martin Kletzander <mkletzan@redhat.com> if you fix s/an comment/a comment/ in commit messages for patches 6, 7 and 8 ;-) Thanks for fixing my error and making the build even better.
participants (2)
-
Martin Kletzander -
Peter Krempa