[libvirt] [PATCH] docs: attempt to document the general libvirt dev strategy
There are various ideas / plans floating around for future libvirt work, some of which is actively in progress. Historically we've never captured this kind of information anywhere, except in mailing list discussions. In particular guidelines in hacking.html.in don't appear until a policy is actively applied. This patch attempts to fill the documentation gap, by creating a new "strategy" page which outlines the general vision for some notable future changes. The key thing to note is that none of the stuff on this page is guaranteed, plans may change as new information arises. IOW this is a "best guess" as to the desired future. This doc has focused on three areas, related to the topic of language usage / consolidation - Use of non-C languages for the library, daemons or helper tools - Replacement of autotools with meson - Use of RST and Sphinx for documentation (website + man pages) Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- docs/docs.html.in | 3 + docs/strategy.html.in | 143 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 146 insertions(+) create mode 100644 docs/strategy.html.in diff --git a/docs/docs.html.in b/docs/docs.html.in index c1741c89b4..6cf09f51bc 100644 --- a/docs/docs.html.in +++ b/docs/docs.html.in @@ -128,6 +128,9 @@ <dt><a href="hacking.html">Contributor guidelines</a></dt> <dd>General hacking guidelines for contributors</dd> + <dt><a href="strategy.html">Project strategy</a></dt> + <dd>Sets a vision for future direction & technical choices</dd> + <dt><a href="bugs.html">Bug reports</a></dt> <dd>How and where to report bugs and request features</dd> diff --git a/docs/strategy.html.in b/docs/strategy.html.in new file mode 100644 index 0000000000..41bbebb8f9 --- /dev/null +++ b/docs/strategy.html.in @@ -0,0 +1,143 @@ +<?xml version="1.0" encoding="UTF-8"?> +<!DOCTYPE html> +<html xmlns="http://www.w3.org/1999/xhtml"> + <body> + <h1>Libvirt Project Strategy</h1> + + <p> + This document attempts to outline the libvirt project strategy for + the near future. Think of this is a high level vision or todo list + setting the direction for the project & its developers to take + </p> + + <h2>Language consolidation</h2> + + <p> + At time of writing libvirt uses the following languages + </p> + + <dl> + <dt>C</dt> + <dd>The core libvirt library, daemons, and helper tools are all written + in the C language.</dd> + <dt>Python</dt> + <dd>Various supporting build/test scripts are written in Python, with + compatibility for Python 2 and 3.</dd> + <dt>Perl</dt> + <dd>Various supporting build/test scripts are written in Python. It is + also used for many syntax-check inline rules</dd> + <dt>Shell</dt> + <dd>The autoconf configure script, is essentially shell script. Shell + is also used for some simple build/test scripts. At runtime libvirt + avoids shell except when using SSH tunnels to a remote host</dd> + <dt>XSLT</dt> + <dd>The website documentation is subsituted into templates using XSLT + scripts. The API documentation is also autogenerated from an XML + description using XSLT</dd> + <dt>HTML</dt> + <dd>The website documentation is all written in plain HTML. Some HTML + is also auto-generated for API documentation</dd> + <dt>M4</dt> + <dd>The autoconf configure script uses a large number of M4 macros to + generate its content</dd> + <dt>make</dt> + <dd>The core build system uses the traditional GNU make recipes</dd> + <dt>automake</dt> + <dd>The make recipes make use of automake's make language extensions + which are then turned into regular make rules</dd> + <dt>awk/sed</dt> + <dd>A number of the syntax-check inline rules involve use of awk/sed + scripts</dd> + <dt>POD</dt> + <dd>The command line manual pages are typically written in Perl's POD + format, and converted to troff</dt> + </dl> + + <p> + The wide range of languages used present a knowledge burden for + developers involved in libvirt, especially when there are multiple + languages all used in the same problem spaces. This is most notable + in the build system which uses a combination of shell, m4, make, + automake, awk, sed, perl and python, with debugging requiring + understanding of the interactions between many languages. The + relative popularity of the languages has implications for how easily + the project can attract new contributors, and the quality of the code + they'll be able to submit. This is most notable when comparing Perl + to Python, as since the start of the libvirt project, the popularity + and knowledge of Perl has declined, while Python has risen. + </p> + <p> + The C language has served libvirt well over the years, but its age shows + giving rise to limitations which negatively impact the project in terms + of code quality, reliability, and efficiency of development. Most notably + its lack of memory safety means that many code bugs become trivially + exploitable security flaws or denial of service. The lack of a high + level portable runtime, resulting in alot of effort being spent to + ensure cross platform portability. The modern languages Rust and Go + provide viable options for low level systems programming, in a way that + was not practical with other common languages such as Python and Java. + There is thus a desire to make use of either Rust or Go, or more likely + a combination of both, to incrementally replace existing use of C, and + also for greenfield development. + </p> + <p> + With this in mind the libvirt project has set a vision for language + usage in the future + </p> + + <dl> + <dt>C</dt> + <dd>The core libvirt library, daemons, and helper tools are all written + in the C language.</dd> + <dt>Rust</dt> + <dd>Parts of the core libvirt library / daemon are to be able to leverage + Rust to replace C. (TBD vs Go)</dd> + <dt>Go</dt> + <dd>Parts of the core libvirt library / daemon are to be able to leverage + Rust to replace C. (TBD vs Rust). Helper tools are able to be written + in Go.</dd> + <dt>Meson</dt> + <dd>The core build system is to be written in meson.</dd> + <dt>Python</dt> + <dd>Various supporting build/test scripts are written in Python 3 + compatible mode only.</dd> + <dt>RST</dt> + <dd>The website and command man pages are to be written in RST, using + Sphinx as the engine to convert to end user formats like HTML, troff, + etc</dd> + </dl> + + <p> + Some notable points from the above. Whether the core library / daemons + will use Rust or Go internally is still to be decided based on more + detailed evaluation of their pros and cons, to identify the best fit, + as the need to link & embed this functionality in other processes has + complex interactions both at a technical and non-technical level. For + standalone helper tools, Go is a good fit for rapid development as + there are no concerns around interactions with other in-process code + from 3rd parties. + </p> + + <p> + The meson build system is written in Python, at time of writing, + requiring version 3.4 or later. This directly informs the choice to use + Python 3.4 as the language for all supporting build scripts, re-inforcing + the other benefits of Python, over Perl, Shell, M4, automake, etc. + </p> + + <p> + Using the RST format for documentation allows for the use of XSLT to be + eliminated from the build process. RST and the Sphinx toolkit are widely + used, as seen by the huge repository of content on + https://readthedocs.org/. The ability embed raw HTML in the RST docs will + greatly facilitate its adoption, avoiding the need for a big bang + conversion of existing content. Given the desire to eliminate Perl + usage, replacing the use of POD documentation for manual pages is an + obvious followup task. RST is the obvious choice to given alignment + with the website, allowing the man pages to be easily published online + with other docs. It is further anticipated that the current API docs + generator which uses XSLT to convert the XML API description would be + converted to something which generates RST using Python instead of XSL. + </p> + </body> +</html> -- 2.21.0
On Fri, Sep 20, 2019 at 15:12:01 +0100, Daniel Berrange wrote:
There are various ideas / plans floating around for future libvirt work, some of which is actively in progress. Historically we've never captured this kind of information anywhere, except in mailing list discussions. In particular guidelines in hacking.html.in don't appear until a policy is actively applied.
This patch attempts to fill the documentation gap, by creating a new "strategy" page which outlines the general vision for some notable future changes. The key thing to note is that none of the stuff on this page is guaranteed, plans may change as new information arises. IOW this is a "best guess" as to the desired future.
This doc has focused on three areas, related to the topic of language usage / consolidation
- Use of non-C languages for the library, daemons or helper tools - Replacement of autotools with meson - Use of RST and Sphinx for documentation (website + man pages)
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- docs/docs.html.in | 3 + docs/strategy.html.in | 143 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 146 insertions(+) create mode 100644 docs/strategy.html.in
[...]
+ <p> + The C language has served libvirt well over the years, but its age shows + giving rise to limitations which negatively impact the project in terms + of code quality, reliability, and efficiency of development. Most notably + its lack of memory safety means that many code bugs become trivially + exploitable security flaws or denial of service. The lack of a high + level portable runtime, resulting in alot of effort being spent to + ensure cross platform portability. The modern languages Rust and Go + provide viable options for low level systems programming, in a way that + was not practical with other common languages such as Python and Java. + There is thus a desire to make use of either Rust or Go, or more likely + a combination of both, to incrementally replace existing use of C, and + also for greenfield development.
Please emphasise that senseless rewrite of old code, while being "memory safe", may introduce new logic bugs which are hard to find in many cases. It should be well justified that rewriting some functionality has clear maintenance and upkeep benefits even when we consider the cost logic bugs which will eventually be introduced in any refactor. This goes for any rewrite.
On Fri, 2019-09-20 at 15:12 +0100, Daniel P. Berrangé wrote: [...]
+ This document attempts to outline the libvirt project strategy for + the near future. Think of this is a high level vision or todo list
s/is a high/as a high/
+ setting the direction for the project & its developers to take
Missing period at the end of the sentence. Also: strategy.html.in:10: parser error : xmlParseEntityRef: no name setting the direction for the project & its developers to take ^ I would just use "and" anyway. [...]
+ <dt>Perl</dt> + <dd>Various supporting build/test scripts are written in Python. It is
s/Python/Perl/ [...]
+ <dt>XSLT</dt> + <dd>The website documentation is subsituted into templates using XSLT + scripts.
s/subsituted/substituted/ But I would probably reword it along the lines of The website uses XSLT for its templating system. [...]
+ <dt>automake</dt> + <dd>The make recipes make use of automake's make language extensions + which are then turned into regular make rules</dd>
Let's try to avoid using "make" every other word. How about something like automake extends the capabilities of GNU make by introducing a declarative syntax that can be used to describe recipes [...]
+ <dt>POD</dt> + <dd>The command line manual pages are typically written in Perl's POD + format, and converted to troff</dt>
strategy.html.in:53: parser error : Opening and ending tag mismatch: dd line 52 and dt format, and converted to troff</dt> ^ [...]
+ The C language has served libvirt well over the years, but its age shows + giving rise to limitations which negatively impact the project in terms + of code quality, reliability, and efficiency of development. Most notably + its lack of memory safety means that many code bugs become trivially + exploitable security flaws or denial of service. The lack of a high + level portable runtime, resulting in alot of effort being spent to
s/alot/a lot/
+ ensure cross platform portability. The modern languages Rust and Go + provide viable options for low level systems programming, in a way that + was not practical with other common languages such as Python and Java. + There is thus a desire to make use of either Rust or Go, or more likely + a combination of both, to incrementally replace existing use of C, and + also for greenfield development.
I would drop the "or more likely a combination of both" because I'm hoping that we'll be able to standardize on a single one: using a mix of both would once again increase the burden placed upon contributors, which is the very problem we're trying to address. Perhaps we're going to end up with both after all, but I would not include language encouraging this. [...]
+ <dt>C</dt> + <dd>The core libvirt library, daemons, and helper tools are all written + in the C language.</dd>
This reads to me as to say that all of the above are implemented using C only, which contradicts the points below. I would also single out the public API as the one part that will need to remain in C. So perhaps something like The core library, daemons and helper tools will continue to be written mostly in C for the foreseeable future. Even as other programming languages are introduced, libvirt will continue to offer a <a href="support.html">stable C API/ABI</a>.
+ <dt>Rust</dt> + <dd>Parts of the core libvirt library / daemon are to be able to leverage + Rust to replace C. (TBD vs Go)</dd> + <dt>Go</dt> + <dd>Parts of the core libvirt library / daemon are to be able to leverage + Rust to replace C. (TBD vs Rust). Helper tools are able to be written + in Go.</dd>
There's no reason Rust could be used for the library and daemons but not be suitable for helper tools. I'd just have a single section for the two languages, something like Rust and Go These languages should be usable as an alternative to C in all the areas listed above: new code should favor them over C, and existing C code may be ported after carefully considering the pros and cons of doing so. [...]
+ <dt>RST</dt>
I would s/RST/reStructuredText/ at least in this first occurrence. [...]
+ Some notable points from the above. Whether the core library / daemons + will use Rust or Go internally is still to be decided based on more + detailed evaluation of their pros and cons, to identify the best fit, + as the need to link & embed this functionality in other processes has
strategy.html.in:114: parser error : xmlParseEntityRef: no name as the need to link & embed this functionality in other processes has ^
+ complex interactions both at a technical and non-technical level. For + standalone helper tools, Go is a good fit for rapid development as + there are no concerns around interactions with other in-process code + from 3rd parties.
As mentioned above, if Rust is good enough to be used for the core library and the daemons it is also good enough for user-facing tools and helpers. [...]
+ Using the RST format for documentation allows for the use of XSLT to be + eliminated from the build process. RST and the Sphinx toolkit are widely + used, as seen by the huge repository of content on + https://readthedocs.org/. The ability embed raw HTML in the RST docs will
s/ability/ability to/
+ greatly facilitate its adoption, avoiding the need for a big bang + conversion of existing content. Given the desire to eliminate Perl + usage, replacing the use of POD documentation for manual pages is an + obvious followup task. RST is the obvious choice to given alignment
s/given/give/ Even then, the sentence doesn't quite flow. Perhaps use "achieve" instead?
+ with the website, allowing the man pages to be easily published online + with other docs. It is further anticipated that the current API docs + generator which uses XSLT to convert the XML API description would be + converted to something which generates RST using Python instead of XSL.
s/XSL/XSLT/ Overall the document is good and the overall strategy is IMHO solid. -- Andrea Bolognani / Red Hat / Virtualization
On Fri, Sep 20, 2019 at 03:12:01PM +0100, Daniel P. Berrangé wrote:
There are various ideas / plans floating around for future libvirt work, some of which is actively in progress. Historically we've never captured this kind of information anywhere, except in mailing list discussions. In particular guidelines in hacking.html.in don't appear until a policy is actively applied.
This patch attempts to fill the documentation gap, by creating a new "strategy" page which outlines the general vision for some notable future changes. The key thing to note is that none of the stuff on this page is guaranteed, plans may change as new information arises. IOW this is a "best guess" as to the desired future.
This doc has focused on three areas, related to the topic of language usage / consolidation
- Use of non-C languages for the library, daemons or helper tools - Replacement of autotools with meson - Use of RST and Sphinx for documentation (website + man pages)
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- docs/docs.html.in | 3 + docs/strategy.html.in | 143 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 146 insertions(+) create mode 100644 docs/strategy.html.in
[...]
+ <p> + The meson build system is written in Python, at time of writing, + requiring version 3.4 or later. This directly informs the choice to use + Python 3.4 as the language for all supporting build scripts, re-inforcing + the other benefits of Python, over Perl, Shell, M4, automake, etc. + </p>
In addition to all comments, Meson since version 0.45.0 requires Python 3.5 and we will most likely use at least 0.49.0. Pavel
On Fri, Sep 20, 2019 at 07:33:08PM +0200, Pavel Hrdina wrote:
On Fri, Sep 20, 2019 at 03:12:01PM +0100, Daniel P. Berrangé wrote:
There are various ideas / plans floating around for future libvirt work, some of which is actively in progress. Historically we've never captured this kind of information anywhere, except in mailing list discussions. In particular guidelines in hacking.html.in don't appear until a policy is actively applied.
This patch attempts to fill the documentation gap, by creating a new "strategy" page which outlines the general vision for some notable future changes. The key thing to note is that none of the stuff on this page is guaranteed, plans may change as new information arises. IOW this is a "best guess" as to the desired future.
This doc has focused on three areas, related to the topic of language usage / consolidation
- Use of non-C languages for the library, daemons or helper tools - Replacement of autotools with meson - Use of RST and Sphinx for documentation (website + man pages)
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- docs/docs.html.in | 3 + docs/strategy.html.in | 143 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 146 insertions(+) create mode 100644 docs/strategy.html.in
[...]
+ <p> + The meson build system is written in Python, at time of writing, + requiring version 3.4 or later. This directly informs the choice to use + Python 3.4 as the language for all supporting build scripts, re-inforcing + the other benefits of Python, over Perl, Shell, M4, automake, etc. + </p>
In addition to all comments, Meson since version 0.45.0 requires Python 3.5 and we will most likely use at least 0.49.0.
Perhaps I'll just say Python 3, otherwise we'll be chasing ever changing minor versions in this doc. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Mon, Sep 23, 2019 at 11:10:53AM +0100, Daniel P. Berrangé wrote:
On Fri, Sep 20, 2019 at 07:33:08PM +0200, Pavel Hrdina wrote:
On Fri, Sep 20, 2019 at 03:12:01PM +0100, Daniel P. Berrangé wrote:
There are various ideas / plans floating around for future libvirt work, some of which is actively in progress. Historically we've never captured this kind of information anywhere, except in mailing list discussions. In particular guidelines in hacking.html.in don't appear until a policy is actively applied.
This patch attempts to fill the documentation gap, by creating a new "strategy" page which outlines the general vision for some notable future changes. The key thing to note is that none of the stuff on this page is guaranteed, plans may change as new information arises. IOW this is a "best guess" as to the desired future.
This doc has focused on three areas, related to the topic of language usage / consolidation
- Use of non-C languages for the library, daemons or helper tools - Replacement of autotools with meson - Use of RST and Sphinx for documentation (website + man pages)
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- docs/docs.html.in | 3 + docs/strategy.html.in | 143 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 146 insertions(+) create mode 100644 docs/strategy.html.in
[...]
+ <p> + The meson build system is written in Python, at time of writing, + requiring version 3.4 or later. This directly informs the choice to use + Python 3.4 as the language for all supporting build scripts, re-inforcing + the other benefits of Python, over Perl, Shell, M4, automake, etc. + </p>
In addition to all comments, Meson since version 0.45.0 requires Python 3.5 and we will most likely use at least 0.49.0.
Perhaps I'll just say Python 3, otherwise we'll be chasing ever changing minor versions in this doc.
Sounds good, everyone can check requirements on Meson website or git repository. Pavel
participants (4)
-
Andrea Bolognani -
Daniel P. Berrangé -
Pavel Hrdina -
Peter Krempa