New subject: [libvirt] Re: [PATCH] Add some docs about the RPC protocol and APIs

Thursday, 11 August 2011

From: "Daniel P. Berrange" <berrange(a)redhat.com&gt;

* remote.html.in: Remove obsolete notes about internals of the
  RPC protocol
* internals/rpc.html.in: Extensive docs on RPC protocol/API
* sitemap.html.in: Add new page
---
 docs/internals/rpc.html.in |  876 ++++++++++++++++++++++++++++++++++++++++++++
 docs/remote.html.in        |   45 ---
 docs/sitemap.html.in       |    4 +
 3 files changed, 880 insertions(+), 45 deletions(-)
 create mode 100644 docs/internals/rpc.html.in

diff --git a/docs/internals/rpc.html.in b/docs/internals/rpc.html.in
new file mode 100644
index 0000000..761832a
--- /dev/null
+++ b/docs/internals/rpc.html.in
@@ -0,0 +1,876 @@
+<html>
+  <body>
+    <h1>libvirt RPC infrastructure</h1>
+
+    <ul id="toc"></ul>
+
+    <p>
+      libvirt includes a basic protocol and code to implement
+      an extensible, secure client/server RPC service. This was
+      originally designed for communication between the libvirt
+      client library and the libvirtd daemon. It is also also
+      used for communication to the virtlockd daemon and (soon)
+      for the libvirt_lxc controller process. This document
+      provides an overview of the protocol and structure / operation
+      of the internal RPC library APIs.
+    </p>
+
+
+    <h2><a name="protocol">RPC protocol</a></h2>
+
+    <p>
+      libvirt uses a simple, variable length, packet based RPC protocol.
+      All structured data within packets is encoded using the
+      <a
href="http://en.wikipedia.org/wiki/External_Data_Representation"...
standard</a>
+      as currently defined by <a
href="https://tools.ietf.org/html/rfc4506">RFC 4506</a>.
+      On any connection running the RPC protocol, there can be multiple
+      programs active, each supporting one or more versions. A program
+      defines a set of procedures that it supports. The procedures can
+      support call+reply method invocation, asynchronous events,
+      and generic data streams. Method invocations can be overlapped,
+      so waiting for a reply to one will not block the receipt of the
+      reply to another outstanding method. The protocol was loosely
+      inspired by the design of SunRPC. The definition of the RC
+      protocol is in the file <code>src/rpc/virnetprotocol.x</code>
+      in the libvirt source tree.
+    </p>
+
+    <h3><a href="protocolframing">Packet
framing</a></h3>
+
+    <p>
+      On the wire, there is no explicit packet framing marker. Instead
+      each packet is preceeded by an unsigned 32-bit integer giving
+      the total length of the packet in bytes. This length includes
+      the 4-bytes of the length word itself. Conceptually the framing
+      looks like this:
+    </p>
+
+<pre>
+|~~~   Packet 1   ~~~|~~~   Packet 2   ~~~|~~~  Packet 3    ~~~|~~~
+
++-------+------------+-------+------------+-------+------------+...
+| n=U32 | (n-4) * U8 | n=U32 | (n-4) * U8 | n=U32 | (n-4) * U8 |
++-------+------------+-------+------------+-------+------------+...
+
+|~ Len ~|~   Data   ~|~ Len ~|~   Data   ~|~ Len ~|~   Data   ~|~
+
+</pre>
+
+    <h3><a href="protocoldata">Packet data</a></h3>
+
+    <p>
+      The data in each packet is split into two parts, a short
+      fixed length header, followed by a variable length payload.
+      So a packet from the illustration above is more correctly
+      shown as
+    </p>
+
+<pre>
+
++-------+-------------+---------------....---+
+| n=U32 | 6*U32       | (n-(7*4))*U8         |
++-------+-------------+---------------....---+
+
+|~ Len ~|~  Header   ~|~  Payload     ....  ~|
+</pre>
+
+
+    <h3><a href="protocolheader">Packet
header</a></h3>
+    <p>
+      The header contains 6 fields, encoded as signed/unsigned 32-bit
+      integers.
+    </p>
+
+    <pre>
++---------------+
+| program=U32   |
++---------------+
+| version=U32   |
++---------------+
+| procedure=S32 |
++---------------+
+| type=S32      |
++---------------+
+| serial=U32    |
++---------------+
+| status=S32    |
++---------------+
+    </pre>
+
+    <dl>
+      <dt><code>program</code></dt>
+      <dd>
+        This is an arbitrarily chosen number that will uniquely
+        identify the "service" running over the stream.
+      </dd>
+      <dt><code>version</code></dt>
+      <dd>
+        This is the version number of the program, by convention
+        starting from '1'. When an incompatible change is made
+        to a program, the version number is incremented. Ideally
+        both versions will then be supported on the wire in
+        parallel for backwards compatibility.
+      </dd>
+      <dt><code>procedure</code></dt>
+      <dd>
+        This is an arbitrarily chosen number that will uniqely
+        identify the method call, or event associated with the
+        packet. By convention, procedure numbers start from 1
+        and are assigned monotonically thereafter.
+      </dd>
+      <dt><code>type</code></dt>
+      <dd>
+        <p>
+        This can be one of the following enumeration values
+        </p>
+        <ol>
+          <li>call: invocation of a method call</li>
+          <li>reply: completion of a method call</li>
+          <li>event: an asynchronous event</li>
+          <li>stream: control info or data from a stream</li>
+        </ol>
+      </dd>
+      <dt><code>serial</code></dt>
+      <dd>
+        This is an number that starts from 1 and increases
+        each time a method call packet is sent. A reply or
+        stream packet will have a serial number matching the
+        original method call packet serial. Events always
+        have the serial number set to 0.
+      </dd>
+      <dt><code>status</code></dt>
+      <dd>
+        <p>
+        This can one of the following enumeration values
+        </p>
+        <ol>
+          <li>ok: a normal packet. this is always set for method calls or events.
+            For replies it indicates succesful completion of the method. For
+            streams it indicates confirmation of the end of file on the
stream.</li>
+          <li>error: for replies this indicates that the method call failed
+            and error information is being returned. For streams this indicates
+            that not all data was sent and the stream has aborted</li>
+          <li>continue: for streams this indicates that further data packets
+            will be following</li>
+        </ol>
+    </dl>
+
+    <h3><a href="protocolpayload">Packet
payload</a></h3>
+
+    <p>
+      The payload of a packet will vary depending on the <code>type</code>
+      and <code>status</code> fields from the header.
+    </p>
+
+    <ul>
+      <li>type=call: the in parameters for the method call, XDR encoded</li>
+      <li>type=reply+status=ok: the return value and/or out parameters for the
method call, XDR encoded</li>
+      <li>type=reply+status=error: the error information for the method, a
virErrorPtr XDR encoded</li>
+      <li>type=event: the parameters for the event, XDR encoded</li>
+      <li>type=stream+status=ok: no payload</li>
+      <li>type=stream+status=error: the error information for the method, a
virErrorPtr XDR encoded</li>
+      <li>type=stream+status=continue: the raw bytes of data for the stream. No XDR
encoding</li>
+    </ul>
+
+    <p>
+      For the exact payload information for each procedure, consult the XDR protocol
+      definition for the program+version in question
+    </p>
+
+    <h3><a name="wireexamples">Wire examples</a></h3>
+
+    <p>
+      The following diagrams illustrate some example packet exchanges
+      between a client and server
+    </p>
+
+    <h4><a name="wireexamplescall">Method
call</a></h4>
+
+    <p>
+      A single method call and succesful
+      reply, for a program=8, version=1, procedure=3, which 10 bytes worth
+      of input args, and 4 bytes worth of return values. The overall input
+      packet length is is 4 + 24 + 10 == 38, and output packet length 32
+    </p>
+
+    <pre>
+          +--+-----------------------+-----------+
+   C --&gt;  |38| 8 | 1 | 3 | 0 | 1 | 0 | .o.oOo.o. |  --&gt; S  (call)
+          +--+-----------------------+-----------+
+
+          +--+-----------------------+--------+
+   C &lt;--  |32| 8 | 1 | 3 | 1 | 1 | 0 | .o.oOo |  &lt;-- S  (reply)
+          +--+-----------------------+--------+
+    </pre>
+
+    <h4><a name="wireexamplescallerr">Method call with
error</a></h4>
+
+    <p>
+      An unsuccessful method call will instead return an error object
+    </p>
+
+    <pre>
+          +--+-----------------------+-----------+
+   C --&gt;  |38| 8 | 1 | 3 | 0 | 1 | 0 | .o.oOo.o. |  --&gt; S   (call)
+          +--+-----------------------+-----------+
+
+          +--+-----------------------+--------------------------+
+   C &lt;--  |48| 8 | 1 | 3 | 2 | 1 | 0 | .o.oOo.o.oOo.o.oOo.o.oOo |  &lt;-- S 
(error)
+          +--+-----------------------+--------------------------+
+    </pre>
+
+    <h4><a name="wireexamplescallup">Method call with upload
stream</a></h4>
+
+    <p>
+      A method call which also involves uploading some data over
+      a stream will result in
+    </p>
+
+    <pre>
+          +--+-----------------------+-----------+
+   C --&gt;  |38| 8 | 1 | 3 | 0 | 1 | 0 | .o.oOo.o. |  --&gt; S  (call)
+          +--+-----------------------+-----------+
+
+          +--+-----------------------+--------+
+   C &lt;--  |32| 8 | 1 | 3 | 1 | 1 | 0 | .o.oOo |  &lt;-- S  (reply)
+          +--+-----------------------+--------+
+
+          +--+-----------------------+-------------....-------+
+   C --&gt;  |38| 8 | 1 | 3 | 3 | 1 | 2 | .o.oOo.o.oOo....o.oOo. |  --&gt; S 
(stream data up)
+          +--+-----------------------+-------------....-------+
+          +--+-----------------------+-------------....-------+
+   C --&gt;  |38| 8 | 1 | 3 | 3 | 1 | 2 | .o.oOo.o.oOo....o.oOo. |  --&gt; S 
(stream data up)
+          +--+-----------------------+-------------....-------+
+          +--+-----------------------+-------------....-------+
+   C --&gt;  |38| 8 | 1 | 3 | 3 | 1 | 2 | .o.oOo.o.oOo....o.oOo. |  --&gt; S 
(stream data up)
+          +--+-----------------------+-------------....-------+
+          ...
+          +--+-----------------------+-------------....-------+
+   C --&gt;  |38| 8 | 1 | 3 | 3 | 1 | 2 | .o.oOo.o.oOo....o.oOo. |  --&gt; S 
(stream data up)
+          +--+-----------------------+-------------....-------+
+          +--+-----------------------+
+   C --&gt;  |24| 8 | 1 | 3 | 3 | 1 | 0 | --&gt; S  (stream finish)
+          +--+-----------------------+
+          +--+-----------------------+
+   C &lt;--  |24| 8 | 1 | 3 | 3 | 1 | 0 | &lt;-- S  (stream finish)
+          +--+-----------------------+
+    </pre>
+
+    <h4><a name="wireexamplescallbi">Method call bidirectional
stream</a></h4>
+
+    <p>
+      A method call which also involves a bi-directional stream will
+      result in
+    </p>
+
+    <pre>
+          +--+-----------------------+-----------+
+   C --&gt;  |38| 8 | 1 | 3 | 0 | 1 | 0 | .o.oOo.o. |  --&gt; S  (call)
+          +--+-----------------------+-----------+
+
+          +--+-----------------------+--------+
+   C &lt;--  |32| 8 | 1 | 3 | 1 | 1 | 0 | .o.oOo |  &lt;-- S  (reply)
+          +--+-----------------------+--------+
+
+          +--+-----------------------+-------------....-------+
+   C --&gt;  |38| 8 | 1 | 3 | 3 | 1 | 2 | .o.oOo.o.oOo....o.oOo. |  --&gt; S 
(stream data up)
+          +--+-----------------------+-------------....-------+
+          +--+-----------------------+-------------....-------+
+   C --&gt;  |38| 8 | 1 | 3 | 3 | 1 | 2 | .o.oOo.o.oOo....o.oOo. |  --&gt; S 
(stream data up)
+          +--+-----------------------+-------------....-------+
+          +--+-----------------------+-------------....-------+
+   C &lt;--  |38| 8 | 1 | 3 | 3 | 1 | 2 | .o.oOo.o.oOo....o.oOo. |  &lt;-- S 
(stream data down)
+          +--+-----------------------+-------------....-------+
+          +--+-----------------------+-------------....-------+
+   C --&gt;  |38| 8 | 1 | 3 | 3 | 1 | 2 | .o.oOo.o.oOo....o.oOo. |  --&gt; S 
(stream data up)
+          +--+-----------------------+-------------....-------+
+          +--+-----------------------+-------------....-------+
+   C --&gt;  |38| 8 | 1 | 3 | 3 | 1 | 2 | .o.oOo.o.oOo....o.oOo. |  --&gt; S 
(stream data up)
+          +--+-----------------------+-------------....-------+
+          +--+-----------------------+-------------....-------+
+   C &lt;--  |38| 8 | 1 | 3 | 3 | 1 | 2 | .o.oOo.o.oOo....o.oOo. |  &lt;-- S 
(stream data down)
+          +--+-----------------------+-------------....-------+
+          +--+-----------------------+-------------....-------+
+   C &lt;--  |38| 8 | 1 | 3 | 3 | 1 | 2 | .o.oOo.o.oOo....o.oOo. |  &lt;-- S 
(stream data down)
+          +--+-----------------------+-------------....-------+
+          +--+-----------------------+-------------....-------+
+   C &lt;--  |38| 8 | 1 | 3 | 3 | 1 | 2 | .o.oOo.o.oOo....o.oOo. |  &lt;-- S 
(stream data down)
+          +--+-----------------------+-------------....-------+
+          +--+-----------------------+-------------....-------+
+   C --&gt;  |38| 8 | 1 | 3 | 3 | 1 | 2 | .o.oOo.o.oOo....o.oOo. |  --&gt; S 
(stream data up)
+          +--+-----------------------+-------------....-------+
+          ..
+          +--+-----------------------+-------------....-------+
+   C --&gt;  |38| 8 | 1 | 3 | 3 | 1 | 2 | .o.oOo.o.oOo....o.oOo. |  --&gt; S 
(stream data up)
+          +--+-----------------------+-------------....-------+
+          +--+-----------------------+
+   C --&gt;  |24| 8 | 1 | 3 | 3 | 1 | 0 | --&gt; S  (stream finish)
+          +--+-----------------------+
+          +--+-----------------------+
+   C &lt;--  |24| 8 | 1 | 3 | 3 | 1 | 0 | &lt;-- S  (stream finish)
+          +--+-----------------------+
+    </pre>
+
+
+    <h4><a name="wireexamplescallmany">Method calls
overlapping</a></h4>
+    <pre>
+          +--+-----------------------+-----------+
+   C --&gt;  |38| 8 | 1 | 3 | 0 | 1 | 0 | .o.oOo.o. |  --&gt; S  (call 1)
+          +--+-----------------------+-----------+
+          +--+-----------------------+-----------+
+   C --&gt;  |38| 8 | 1 | 3 | 0 | 2 | 0 | .o.oOo.o. |  --&gt; S  (call 2)
+          +--+-----------------------+-----------+
+          +--+-----------------------+--------+
+   C &lt;--  |32| 8 | 1 | 3 | 1 | 2 | 0 | .o.oOo |  &lt;-- S  (reply 2)
+          +--+-----------------------+--------+
+          +--+-----------------------+-----------+
+   C --&gt;  |38| 8 | 1 | 3 | 0 | 3 | 0 | .o.oOo.o. |  --&gt; S  (call 3)
+          +--+-----------------------+-----------+
+          +--+-----------------------+--------+
+   C &lt;--  |32| 8 | 1 | 3 | 1 | 3 | 0 | .o.oOo |  &lt;-- S  (reply 3)
+          +--+-----------------------+--------+
+          +--+-----------------------+-----------+
+   C --&gt;  |38| 8 | 1 | 3 | 0 | 4 | 0 | .o.oOo.o. |  --&gt; S  (call 4)
+          +--+-----------------------+-----------+
+          +--+-----------------------+--------+
+   C &lt;--  |32| 8 | 1 | 3 | 1 | 1 | 0 | .o.oOo |  &lt;-- S  (reply 1)
+          +--+-----------------------+--------+
+          +--+-----------------------+--------+
+   C &lt;--  |32| 8 | 1 | 3 | 1 | 4 | 0 | .o.oOo |  &lt;-- S  (reply 4)
+          +--+-----------------------+--------+
+    </pre>
+
+
+    <h2><a name="security">RPC security</a></h2>
+
+    <p>
+      There are various things to consider to ensure an implementation
+      of the RPC protocol can be satisfactorily secured
+    </p>
+
+    <h3><a
name="securitytls">Authentication/encryption</a></h3>
+
+    <p>
+      The basic RPC protocol does not define or require any specific
+      authentication/encryption capabilities. A generic solution to
+      providing encryption for the protocol is to run the protocol
+      over a TLS encrypted data stream. x509 certificate checks can
+      be done to form a crude authentication mechanism. It is also
+      possible for an RPC program to negotiate an encryption /
+      authentication capability, such as SASL, which may then also
+      provide per-packet data encryption. Finally the protocol data
+      stream can of course be tunnelled over transports such as SSH.
+    </p>
+
+    <h3><a name="securitylimits">Data limits</a></h3>
+
+    <p>
+      Although the protocol itself defines many arbitrary sized data values in the
+      payloads, to avoid denial of service attack there are a number of size limit
+      checks prior to encoding or decoding data. There is a limit on the maximum
+      size of a single RPC message, limit on the maximum string length, and limits
+      on any other parameter which uses a variable length array. These limits can
+      be raised, subject to agreement between client/server, without otherwise
+      breaking compatibility of the RPC data on the wire.
+    </p>
+
+    <h3><a name="securityvalidate">Data
validation</a></h3>
+
+    <p>
+      It is important that all data be fully validated before performing
+      any actions based on the data. When reading an RPC packet, the
+      first four bytes must be read and the max packet size limit validated,
+      before any attempt is made to read the variable length packet data.
+      After a complete packet has been read, the header must be decoded
+      and all 6 fields fully validated, before attempting to dispatch
+      the payload. Once dispatched, the payload can be decoded and passed
+      onto the appropriate API for execution. The RPC code must not take
+      any action based on the payload, since it has no way to validate
+      the semantics of the payload data. It must delegate this to the
+      execution API (eg corresponding libvirt public API).
+    </p>
+
+    <h2><a name="internals">RPC internal APIs</a></h2>
+
+    <p>
+      The generic internal RPC library code lives in the
<code>src/rpc/</code>
+      directory of the libvirt source tree. Unless otherwise noted, the
+      objects are all threadsafe. The core object types and their
+      purposes are:
+    </p>
+
+    <h3><a name="apioverview">Overview of RPC
objects</a></h3>
+
+    <p>
+      The following is a high level overview of the role of each
+      of the main RPC objects
+    </p>
+
+    <dl>
+      <dt><code>virNetSASLContextPtr</code>
(virnetsaslcontext.h)</dt>
+      <dd>The virNetSASLContext APIs maintain SASL state for a network
+        service (server or client). This is primarily used on the server
+        to provide a whitelist of allowed SASL usernames for clients.
+      </dd>
+
+      <dt><code>virNetSASLSessionPtr</code>
(virnetsaslcontext.h)</dt>
+      <dd>The virNetSASLSession APIs maintain SASL state for a single
+        network connection (socket). This is used to perform the multi-step
+        SASL handshake and perform encryption/decryption of data once
+        authenticated, via integration with virNetSocket.
+      </dd>
+
+      <dt><code>virNetTLSContextPtr</code>
(virnettlscontext.h)</dt>
+      <dd>The virNetTLSContext APIs maintain TLS state for a network
+        service (server or client). This is primarily used on the server
+        to provide a whitelist of allowed x509 distinguished names, as
+        well as diffie-hellman keys. It can also do validation of
+        x509 certificates prior to initiating a connection, in order
+        to improve detection of configuration errors.
+      </dd>
+
+      <dt><code>virNetTLSSessionPtr</code>
(virnettlscontext.h)</dt>
+      <dd>The virNetTLSSession APIs maintain TLS state for a single
+        network connection (socket). This is used to perform the multi-step
+        TLS handshake and perform encryption/decryption of data once
+        authenticated, via integration with virNetSocket.
+      </dd>
+
+      <dt><code>virNetSocketPtr</code> (virnetsocket.h)</dt>
+      <dd>The virNetSocket APIs provide a higher level wrapper around
+        the raw BSD sockets and getaddrinfo APIs. They allow for creation
+        of both server and client sockets. Data transports supported are
+        TCP, UNIX, SSH tunnel or external command tunnel. Internally the
+        TCP socket impl uses the getaddrinfo info APIs to ensure correct
+        protocol independant behaviour, thus supporting both IPv4 and IPv6.
+        The socket APIs can be associated with a virNetSASLSessionPtr or
+        virNetTLSSessionPtr object to allow seemless encryption/decryption
+        of all writes and reads. For UNIX sockets it is possible to obtain
+        the remote client user ID and process ID. Integration with the
+        libvirt event loop also allows use of callbacks for notification
+        of various I/O conditions
+      </dd>
+
+      <dt><code>virNetMessagePtr</code> (virnetmessage.h)</dt>
+      <dd>The virNetMessage APIs provide a wrapper around the libxdr
+        API calls, to facilitate processing and creation of RPC
+        packets. There are convenience APIs for encoding/encoding the
+        packet headers, encoding/decoding the payload using an XDR
+        filter, encoding/decoding a raw payload (for streams), and
+        encoding a virErrorPtr object. There is also a means to
+        add to/serve from a linked-list queue of messages.</dd>
+
+      <dt><code>virNetClientPtr</code> (virnetclient.h)</dt>
+      <dd>The virNetClient APIs provide a way to connect to a
+        remote server and run one or more RPC protocols over
+        the connection. Connections can be made over TCP, UNIX
+        sockets, SSH tunnels, or external command tunnels. There
+        is support for both TLS and SASL session encryption.
+        The client also supports management of multiple data streams
+        over each connection. Each client object can be used from
+        multiple threads concurrently, with method calls/replies
+        being interleaved on the wire as required.
+      </dd>
+
+      <dt><code>virNetClientProgramPtr</code>
(virnetclientprogram.h)</dt>
+      <dd>The virNetClientProgram APIs are used to register a
+        program+version with the connection. This then enables
+        invocation of method calls, receipt of asynchronous
+        events and use of data streams, within that program+version.
+        When created a set of callbacks must be supplied to take
+        care of dispatching any incoming asynchronous events.
+      </dd>
+
+      <dt><code>virNetClientStreamPtr</code>
(virnetclientstream.h)</dt>
+      <dd>The virNetClientStream APIs are used to control transmission and
+        receipt of data over a stream active on a client. Streams provide
+        a low latency, unlimited length, bi-directional raw data exchange
+        mechanism layered over the RPC connection
+      </dd>
+
+      <dt><code>virNetServerPtr</code> (virnetserver.h)</dt>
+      <dd>The virNetServer APIs are used to manage a network server. A
+        server exposed one or more programs, over one or more services.
+        It manages multiple client connections invoking multiple RPC
+        calls in parallel, with dispatch across multiple worker threads.
+      </dd>
+
+      <dt><code>virNetServerMDNSPtr</code>
(virnetservermdns.h)</dt>
+      <dd>The virNetServerMDNS APIs are used to advertize a server
+        across the local network, enabling clients to automatically
+        detect the existance of remote services. This is done by
+        interfacing with the Avahi mDNS advertisement service.
+      </dd>
+
+      <dt><code>virNetServerClientPtr</code>
(virnetserverclient.h)</dt>
+      <dd>The virNetServerClient APIs are used to manage I/O related
+        to a single client network connection. It handles initial
+        validation and routing of incoming RPC packets, and transmission
+        of outgoing packets.
+      </dd>
+
+      <dt><code>virNetServerProgramPtr</code>
(virnetserverprogram.h)</dt>
+      <dd>The virNetServerProgram APIs are used to provide the implementation
+        of a single program/version set. Primarily this includes a set of
+        callbacks used to actually invoke the APIs corresponding to
+        program procedure numbers. It is responsible for all the serialization
+        of payloads to/from XDR.</dd>
+
+      <dt><code>virNetServerServicePtr</code>
(virnetserverservice.h)</dt>
+      <dd>The virNetServerService APIs are used to connect the server to
+        one or more network protocols. A single service may involve multiple
+        sockets (ie both IPv4 and IPv6). A service also has an associated
+        authentication policy for incoming clients.
+      </dd>
+    </dl>
+
+    <h3><a name="apiclientdispatch">Client RPC
dispatch</a></h3>
+
+    <p>
+      The client RPC code must allow for multiple overlapping RPC method
+      calls to be invoked, transmission &amp; receipt of data for mutliple
+      streams and receipt of asynchronous events. Understandably this
+      involves coordination of multiple threads.
+    </p>
+
+    <p>
+      The core requirement in the client dispatch code is that only
+      one thread is allowed to be performing I/O on the socket at
+      any time. This thread is said to be "holding the buck". When
+      any other thread comes along and needs todo I/O it must place
+      its packets on a queue and delegate processing of them to the
+      thread that has the buck. This thread will send out the method
+      call, and if it sees a reply will pass it back to the waiting
+      thread. If the other thread's reply hasn't arrived, by the time
+      the main thread has got its own reply, then it will transfer
+      responsibility for I/O to the thread that has been waiting the
+      longest. It is said to be "passing the buck" for I/O.
+    </p>
+
+    <p>
+      When no thread is performing any RPC method call, or sending
+      stream data there is still a need to monitor the socket for
+      incoming I/O related to asynchronous events, or stream data
+      receipt. For this task, a watch is registered with the event
+      loop which triggers whenever the socket is readable. This
+      watch is automatically disabled whenever any other thread
+      grabs the buck, and re-enabled when the buck is released.
+    </p>
+
+    <h4><a name="apiclientdispatchex1">Example with buck
passing</a></h4>
+
+    <p>
+      In the first example, a second thread issues a API call
+      while the first thread holds the buck. The reply to the
+      first call arrives first, so the buck is passed to the
+      second thread.
+    </p>
+
+    <pre>
+        Thread-1
+           |
+           V
+       Call API1()
+           |
+           V
+       Grab Buck
+           |           Thread-2
+           V              |
+       Send method1       V
+           |          Call API2()
+           V              |
+        Wait I/O          V
+           |&lt;--------Queue method2
+           V              |
+       Send method2       V
+           |          Wait for buck
+           V              |
+        Wait I/O          |
+           |              |
+           V              |
+       Recv reply1        |
+           |              |
+           V              |
+       Pass the buck-----&gt;|
+           |              V
+           V           Wait I/O
+       Return API1()      |
+                          V
+                      Recv reply2
+                          |
+                          V
+                     Release the buck
+                          |
+                          V
+                      Return API2()
+    </pre>
+
+    <h4><a name="apiclientdispatchex2">Example without buck
passing</a></h4>
+
+    <p>
+      In this second example, a second thread issues an API call
+      which is sent and replied to, before the first thread's
+      API call has completed. The first thread thus notifies
+      the second that its reply is ready, and there is no need
+      to pass the buck
+    </p>
+
+    <pre>
+        Thread-1
+           |
+           V
+       Call API1()
+           |
+           V
+       Grab Buck
+           |           Thread-2
+           V              |
+       Send method1       V
+           |          Call API2()
+           V              |
+        Wait I/O          V
+           |&lt;--------Queue method2
+           V              |
+       Send method2       V
+           |          Wait for buck
+           V              |
+        Wait I/O          |
+           |              |
+           V              |
+       Recv reply2        |
+           |              |
+           V              |
+      Notify reply2------&gt;|
+           |              V
+           V          Return API2()
+        Wait I/O
+           |
+           V
+       Recv reply1
+           |
+           V
+     Release the buck
+           |
+           V
+       Return API1()
+    </pre>
+
+    <h4><a name="apiclientdispatchex3">Example with async
events</a></h4>
+
+    <p>
+      In this example, only one thread is present and it has to
+      deal with some async events arriving. The events are actually
+      dispatched to the application from the event loop thread
+    </p>
+
+    <pre>
+        Thread-1
+           |
+           V
+       Call API1()
+           |
+           V
+       Grab Buck
+           |
+           V
+       Send method1
+           |
+           V
+        Wait I/O
+           |          Event thread
+           V              ...
+       Recv event1         |
+           |               V
+           V          Wait for timer/fd
+       Queue event1        |
+           |               V
+           V           Timer fires
+        Wait I/O           |
+           |               V
+           V           Emit event1
+       Recv reply1         |
+           |               V
+           V          Wait for timer/fd
+       Return API1()       |
+                          ...
+    </pre>
+
+    <h3><a name="apiserverdispatch">Server RPC
dispatch</a></h3>
+
+    <p>
+      The RPC server code must support receipt of incoming RPC requests from
+      multiple client connections, and parallel processing of all RPC
+      requests, even many from a single client. This goal is achieved through
+      a combination of event driven I/O, and multiple processing threads.
+    </p>
+
+    <p>
+      The main libvirt event loop thread is responsible for performing all
+      socket I/O. It will read incoming packets from clients and willl
+      transmit outgoing packets to clients. It will handle the I/O to/from
+      streams associated with client API calls. When doing client I/O it
+      will also take pass the data through any applicable encryption layer
+      (through use of the virNetSocket / virNetTLSSession and virNetSASLSession
+      integration). What is paramount is that the event loop thread never
+      do any task that can take a non-trivial amount of time.
+    </p>
+
+    <p>
+      When reading packets, the event loop will first read the 4 byte length
+      word. This is validated to make sure it does not exceed the maximum
+      permissible packet size, and the client is set to allow receipt of the
+      rest of the packet data. Once a complete packet has been received, the
+      next step is to decode the RPC header. The header is validated to
+      ensure the request is sensible, ie the server should not receive a
+      method reply from a client. If the client has not yet authenticated,
+      a security check is also applied to make sure the procedure is on the
+      whitelist of those allowed prior to auth. If the packet is a method
+      call, it will be placed on a global processing queue. The event loop
+      thread is now done with the packet for the time being.
+    </p>
+
+    <p>
+      The server has a pool of worker threads, which wait for method call
+      packets to be queued. One of them will grab the new method call off
+      the queue for processing. The first step is to decode the payload of
+      the packet to extract the method call arguments. The worker does not
+      attempt todo any semantic validation of the arguments, except to make
+      sure the size of any variable length fields is below defined limits.
+    </p>
+
+    <p>
+      The worker now invokes the libvirt API call that corresponds to the
+      procedure number in the packet header. The worker is thus kept busy
+      until the API call completes. The implemementation of the API call
+      is responsible for doing semantic validation of parameters and any
+      MAC security checks on the objects affected.
+    </p>
+
+    <p>
+      Once the API call has completed, the worker thread will take the
+      return value and output parameters, or error object and encode
+      them into a reply packet. Again it does not attempt todo any
+      semantic validation of output data, aside from variable length
+      field limit checks. The worker thread puts the reply packet onto
+      the transmission queue for the client. The worker is now finished
+      and goes back to wait for another incoming method call.
+    </p>
+
+    <p>
+      The main event loop is back in charge and when the client socket
+      becomes writable, it will start sending the method reply packet
+      back to the client.
+    </p>
+
+    <p>
+      At any time the libvirt connection object can emit asynchronous
+      events. These are handled by callbacks in the main event thread.
+      The callback will simply encode the event parameters into a new
+      data packet and place the packet on the client transmission
+      queue.
+    </p>
+
+    <p>
+      Incoming and outgoing stream packets are also directly handled
+      by the main event thread. When an incoming stream packet is
+      received, instead of placing it in the global dispatch queue
+      for the worker threads, it is sidetracked into a per-stream
+      processing queue. When the stream becomes writable, queued
+      incoming stream packets will be processed, passing their data
+      payload onto the stream. Conversely when the stream becomes
+      readable, chunks of data will be read from it, encoded into
+      new outgoing packets, and placed on the client's transmit
+      queue
+    </p>
+
+    <h4><a name="apiserverdispatchex1">Example with overlapping
methods</a></h4>
+
+    <p>
+      This example illustrates processing of two incoming methods with
+      overlapping execution
+    </p>
+
+    <pre>
+   Event thread    Worker 1       Worker 2
+       |               |              |
+       V               V              V
+    Wait I/O       Wait Job       Wait Job
+       |               |              |
+       V               |              |
+   Recv method1        |              |
+       |               |              |
+       V               |              |
+   Queue method1       V              |
+       |          Serve method1       |
+       V               |              |
+    Wait I/O           V              |
+       |           Call API1()        |
+       V               |              |
+   Recv method2        |              |
+       |               |              |
+       V               |              |
+   Queue method2       |              V
+       |               |         Serve method2
+       V               V              |
+    Wait I/O      Return API1()       V
+       |               |          Call API2()
+       |               V              |
+       V         Queue reply1         |
+   Send reply1         |              |
+       |               V              V
+       V           Wait Job       Return API2()
+    Wait I/O           |              |
+       |              ...             V
+       V                          Queue reply2
+   Send reply2                        |
+       |                              V
+       V                          Wait Job
+    Wait I/O                          |
+       |                             ...
+      ...
+    </pre>
+
+    <h4><a name="apiserverdispatchex2">Example with stream
data</a></h4>
+
+    <p>
+      This example illustrates processing of stream data
+    </p>
+
+    <pre>
+   Event thread
+       |
+       V
+    Wait I/O
+       |
+       V
+   Recv stream1
+       |
+       V
+   Queue stream1
+       |
+       V
+    Wait I/O
+       |
+       V
+   Recv stream2
+       |
+       V
+   Queue stream2
+       |
+       V
+    Wait I/O
+       |
+       V
+   Write stream1
+       |
+       V
+   Write stream2
+       |
+       V
+    Wait I/O
+       |
+      ...
+    </pre>
+
+  </body>
+</html>
diff --git a/docs/remote.html.in b/docs/remote.html.in
index b554950..6a8e830 100644
--- a/docs/remote.html.in
+++ b/docs/remote.html.in
@@ -53,9 +53,6 @@ machines through authenticated and encrypted connections.
       <li>
         <a href="#Remote_limitations">Limitations</a>
       </li>
-      <li>
-        <a href="#Remote_implementation_notes">Implementation
notes</a>
-      </li>
     </ul>
     <h3>
       <a name="Remote_basic_usage">Basic usage</a>
@@ -880,47 +877,5 @@ just read-write/read-only as at present.
     <p>
 Please come and discuss these issues and more on <a
href="https://www.redhat.com/mailman/listinfo/libvir-list"
title="libvir-list mailing list">the mailing list</a>.
 </p>
-    <h3>
-      <a name="Remote_implementation_notes">Implementation
notes</a>
-    </h3>
-    <p>
-The current implementation uses <a
href="http://en.wikipedia.org/wiki/External_Data_Representation"
title="External Data Representation">XDR</a>-encoded packets with a
-simple remote procedure call implementation which also supports
-asynchronous messaging and asynchronous and out-of-order replies,
-although these latter features are not used at the moment.
-</p>
-    <p>
-The implementation should be considered <b>strictly internal</b> to
-libvirt and <b>subject to change at any time without notice</b>.  If
-you wish to talk to libvirtd, link to libvirt.  If there is a problem
-that means you think you need to use the protocol directly, please
-first discuss this on <a
href="https://www.redhat.com/mailman/listinfo/libvir-list"
title="libvir-list mailing list">the mailing list</a>.
-</p>
-    <p>
-The messaging protocol is described in
-<code>qemud/remote_protocol.x</code>.
-</p>
-    <p>
-Authentication and encryption (for TLS) is done using <a
href="http://www.gnu.org/software/gnutls/" title="GnuTLS
project&#10;page">GnuTLS</a> and the RPC protocol is unaware of this
layer.
-</p>
-    <p>
-Protocol messages are sent using a simple 32 bit length word (encoded
-XDR int) followed by the message header (XDR
-<code>remote_message_header</code>) followed by the message body.  The
-length count includes the length word itself, and is measured in
-bytes.  Maximum message size is <code>REMOTE_MESSAGE_MAX</code> and to
-avoid denial of services attacks on the XDR decoders strings are
-individually limited to <code>REMOTE_STRING_MAX</code> bytes.  In the
-TLS case, messages may be split over TLS records, but a TLS record
-cannot contain parts of more than one message.  In the common RPC case
-a single <code>REMOTE_CALL</code> message is sent from client to
-server, and the server then replies synchronously with a single
-<code>REMOTE_REPLY</code> message, but other forms of messaging are
-also possible.
-</p>
-    <p>
-The protocol contains support for multiple program types and protocol
-versioning, modelled after SunRPC.
-</p>
   </body>
 </html>
diff --git a/docs/sitemap.html.in b/docs/sitemap.html.in
index 897ee94..f50a8d2 100644
--- a/docs/sitemap.html.in
+++ b/docs/sitemap.html.in
@@ -289,6 +289,10 @@
                 <span>Spawning commands from libvirt driver code</span>
               </li>
               <li>
+                <a href="internals/rpc.html">RPC protocol &amp;
APIs</a>
+                <span>RPC protocol information and API / dispatch
guide</span>
+              </li>
+              <li>
                 <a href="internals/locking.html">Lock managers</a>
                 <span>Use lock managers to protect disk content</span>
               </li>
-- 
1.7.6


    

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

[libvirt] [PATCH] Add some docs about the RPC protocol and APIs