9:01 a.m.
After conversion the table doesn't have to custom colors, but otherwise
seems to hold well.
Signed-off-by: Peter Krempa <pkrempa(a)redhat.com>
---
docs/meson.build | 2 +-
docs/migration.html.in | 688 -----------------------------------------
docs/migration.rst | 458 +++++++++++++++++++++++++++
3 files changed, 459 insertions(+), 689 deletions(-)
delete mode 100644 docs/migration.html.in
create mode 100644 docs/migration.rst
diff --git a/docs/meson.build b/docs/meson.build
index fb6e0029d0..3aabb52950 100644
--- a/docs/meson.build
+++ b/docs/meson.build
@@ -77,7 +77,6 @@ docs_html_in_files = [
'internals',
'java',
'logging',
- 'migration',
'nss',
'pci-hotplug',
'php',
@@ -121,6 +120,7 @@ docs_rst_files = [
'hacking',
'libvirt-go',
'libvirt-go-xml',
+ 'migration',
'newreposetup',
'pci-addresses',
'platforms',
diff --git a/docs/migration.html.in b/docs/migration.html.in
deleted file mode 100644
index 627200f96a..0000000000
--- a/docs/migration.html.in
+++ /dev/null
@@ -1,688 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<!DOCTYPE html>
-<html xmlns="http://www.w3.org/1999/xhtml">
- <body>
- <h1>Guest migration</h1>
-
- <ul id="toc"></ul>
-
- <p>
- Migration of guests between hosts is a complicated problem with many possible
- solutions, each with their own positive and negative points. For maximum
- flexibility of both hypervisor integration, and administrator deployment,
- libvirt implements several options for migration.
- </p>
-
- <h2><a id="transport">Network data
transports</a></h2>
-
- <p>
- There are two options for the data transport used during migration, either
- the hypervisor's own <strong>native</strong> transport, or
<strong>tunnelled</strong>
- over a libvirtd connection.
- </p>
-
- <h3><a id="transportnative">Hypervisor native
transport</a></h3>
- <p>
- <em>Native</em> data transports may or may not support encryption,
depending
- on the hypervisor in question, but will typically have the lowest computational
costs
- by minimising the number of data copies involved. The native data transports will
also
- require extra hypervisor-specific network configuration steps by the administrator
when
- deploying a host. For some hypervisors, it might be necessary to open up a large
range
- of ports on the firewall to allow multiple concurrent migration operations.
- </p>
-
- <p>
- Modern hypervisors support TLS for encryption and authentication of the
- migration connections which can be enabled using the
- <code>VIR_MIGRATE_TLS</code> flag. The <em>qemu</em>
hypervisor driver
- allows users to force use of TLS via the
<code>migrate_tls_force</code>
- knob configured in <code>/etc/libvirt/qemu.conf</code>.
- </p>
-
- <p>
- <img class="diagram" src="migration-native.png"
alt="Migration native path"/>
- </p>
-
- <h3><a id="transporttunnel">libvirt tunnelled
transport</a></h3>
- <p>
- <em>Tunnelled</em> data transports will always be capable of strong
encryption
- since they are able to leverage the capabilities built in to the libvirt RPC
protocol.
- The downside of a tunnelled transport, however, is that there will be extra data
copies
- involved on both the source and destinations hosts as the data is moved between
libvirtd
- and the hypervisor. This is likely to be a more significant problem for guests
with
- very large RAM sizes, which dirty memory pages quickly. On the deployment side,
tunnelled
- transports do not require any extra network configuration over and above what's
already
- required for general libvirtd <a href="remote.html">remote
access</a>, and there is only
- need for a single port to be open on the firewall to support multiple concurrent
- migration operations.
- </p>
-
- <p>
- <em>Note:</em> Certain features such as migration of non-shared
storage
- (<code>VIR_MIGRATE_NON_SHARED_DISK</code>), the multi-connection
migration
- (<code>VIR_MIGRATE_PARALLEL</code>), or post-copy migration
- (<code>VIR_MIGRATE_POSTCOPY</code>) may not be available when using
- libvirt's tunnelling.
- </p>
-
- <p>
- <img class="diagram" src="migration-tunnel.png"
alt="Migration tunnel path"/>
- </p>
-
- <h2><a id="flow">Communication control
paths/flows</a></h2>
-
- <p>
- Migration of virtual machines requires close co-ordination of the two
- hosts involved, as well as the application invoking the migration,
- which may be on the source, the destination, or a third host.
- </p>
-
- <h3><a id="flowmanageddirect">Managed direct
migration</a></h3>
-
- <p>
- With <em>managed direct</em> migration, the libvirt client process
- controls the various phases of migration. The client application must
- be able to connect and authenticate with the libvirtd daemons on both
- the source and destination hosts. There is no need for the two libvirtd
- daemons to communicate with each other. If the client application
- crashes, or otherwise loses its connection to libvirtd during the
- migration process, an attempt will be made to abort the migration and
- restart the guest CPUs on the source host. There may be scenarios
- where this cannot be safely done, in which cases the guest will be
- left paused on one or both of the hosts.
- </p>
-
- <p>
- <img class="diagram" src="migration-managed-direct.png"
alt="Migration direct, managed"/>
- </p>
-
-
- <h3><a id="flowpeer2peer">Managed peer to peer
migration</a></h3>
-
- <p>
- With <em>peer to peer</em> migration, the libvirt client process only
- talks to the libvirtd daemon on the source host. The source libvirtd
- daemon controls the entire migration process itself, by directly
- connecting the destination host libvirtd. If the client application crashes,
- or otherwise loses its connection to libvirtd, the migration process
- will continue uninterrupted until completion. Note that the
- source libvirtd uses its own credentials (typically root) to
- connect to the destination, rather than the credentials used
- by the client to connect to the source; if these differ, it is
- common to run into a situation where a client can connect to the
- destination directly but the source cannot make the connection to
- set up the peer-to-peer migration.
- </p>
-
- <p>
- <img class="diagram" src="migration-managed-p2p.png"
alt="Migration peer-to-peer"/>
- </p>
-
-
- <h3><a id="flowunmanageddirect">Unmanaged direct
migration</a></h3>
-
- <p>
- With <em>unmanaged direct</em> migration, neither the libvirt client
- or libvirtd daemon control the migration process. Control is instead
- delegated to the hypervisor's over management services (if any). The
- libvirt client merely initiates the migration via the hypervisor's
- management layer. If the libvirt client or libvirtd crash, the
- migration process will continue uninterrupted until completion.
- </p>
-
- <p>
- <img class="diagram" src="migration-unmanaged-direct.png"
alt="Migration direct, unmanaged"/>
- </p>
-
-
- <h2><a id="security">Data security</a></h2>
-
- <p>
- Since the migration data stream includes a complete copy of the guest
- OS RAM, snooping of the migration data stream may allow compromise
- of sensitive guest information. If the virtualization hosts have
- multiple network interfaces, or if the network switches support
- tagged VLANs, then it is very desirable to separate guest network
- traffic from migration or management traffic.
- </p>
-
- <p>
- In some scenarios, even a separate network for migration data may
- not offer sufficient security. In this case it is possible to apply
- encryption to the migration data stream. If the hypervisor does not
- itself offer encryption, then the libvirt tunnelled migration
- facility should be used.
- </p>
-
- <h2><a id="offline">Offline migration</a></h2>
-
- <p>
- Offline migration transfers the inactive definition of a domain
- (which may or may not be active). After successful completion, the
- domain remains in its current state on the source host and is defined
- but inactive on the destination host. It's a bit more clever than
- <code>virsh dumpxml</code> on source host followed by
- <code>virsh define</code> on destination host, as offline migration
- will run the pre-migration hook to update the domain XML on
- destination host. Currently, copying non-shared storage or other file
- based storages (e.g. UEFI variable storage) is not supported during
- offline migration.
- </p>
-
- <h2><a id="uris">Migration URIs</a></h2>
-
- <p>
- Initiating a guest migration requires the client application to
- specify up to three URIs, depending on the choice of control
- flow and/or APIs used. The first URI is that of the libvirt
- connection to the source host, where the virtual guest is
- currently running. The second URI is that of the libvirt
- connection to the destination host, where the virtual guest
- will be moved to (and in peer-to-peer migrations, this is from
- the perspective of the source, not the client). The third URI is
- a hypervisor specific
- URI used to control how the guest will be migrated. With
- any managed migration flow, the first and second URIs are
- compulsory, while the third URI is optional. With the
- unmanaged direct migration mode, the first and third URIs are
- compulsory and the second URI is not used.
- </p>
-
- <p>
- Ordinarily management applications only need to care about the
- first and second URIs, which are both in the normal libvirt
- connection URI format. Libvirt will then automatically determine
- the hypervisor specific URI, by looking up the target host's
- configured hostname. There are a few scenarios where the management
- application may wish to have direct control over the third URI.
- </p>
-
- <ol>
- <li>The configured hostname is incorrect, or DNS is broken. If a
- host has a hostname which will not resolve to match one of its
- public IP addresses, then libvirt will generate an incorrect
- URI. In this case the management application should specify the
- hypervisor specific URI explicitly, using an IP address, or a
- correct hostname.</li>
- <li>The host has multiple network interfaces. If a host has multiple
- network interfaces, it might be desirable for the migration data
- stream to be sent over a specific interface for either security
- or performance reasons. In this case the management application
- should specify the hypervisor specific URI, using an IP address
- associated with the network to be used.</li>
- <li>The firewall restricts what ports are available. When libvirt
- generates a migration URI it will pick a port number using hypervisor
- specific rules. Some hypervisors only require a single port to be
- open in the firewalls, while others require a whole range of port
- numbers. In the latter case the management application may wish
- to choose a specific port number outside the default range in order
- to comply with local firewall policies.</li>
- <li>The second URI uses UNIX transport method. In this advanced case
- libvirt should not guess a *migrateuri* and it should be specified using
- UNIX socket path URI:
<code>unix:///path/to/socket</code>.</li>
- </ol>
-
- <h2><a id="config">Configuration file
handling</a></h2>
-
- <p>
- There are two types of virtual machines known to libvirt. A
<em>transient</em>
- guest only exists while it is running, and has no configuration file stored
- on disk. A <em>persistent</em> guest maintains a configuration file on
disk
- even when it is not running.
- </p>
-
- <p>
- By default, a migration operation will not attempt to modify any configuration
- files that may be stored on either the source or destination host. It is the
- administrator, or management application's, responsibility to manage
distribution
- of configuration files (if desired). It is important to note that the
<code>/etc/libvirt</code>
- directory <strong>MUST NEVER BE SHARED BETWEEN HOSTS</strong>. There
are some
- typical scenarios that might be applicable:
- </p>
-
- <ul>
- <li>Centralized configuration files outside libvirt, in shared storage. A
cluster
- aware management application may maintain all the master guest configuration
- files in a cluster filesystem. When attempting to start a guest, the config
- will be read from the cluster FS and used to deploy a persistent guest.
- For migration the configuration will need to be copied to the destination
- host and removed on the original.
- </li>
- <li>Centralized configuration files outside libvirt, in a database. A data
center
- management application may not store configuration files at all. Instead it
- may generate libvirt XML on the fly when a guest is booted. It will typically
- use transient guests, and thus not have to consider configuration files during
- migration.
- </li>
- <li>Distributed configuration inside libvirt. The configuration file for
each
- guest is copied to every host where the guest is able to run. Upon migration
- the existing config merely needs to be updated with any changes.
- </li>
- <li>Ad-hoc configuration management inside libvirt. Each guest is tied to a
- specific host and rarely migrated. When migration is required, the config
- is moved from one host to the other.
- </li>
- </ul>
-
- <p>
- As mentioned above, libvirt will not modify configuration files during
- migration by default. The <code>virsh</code> command has two flags to
- influence this behaviour. The <code>--undefinesource</code> flag
- will cause the configuration file to be removed on the source host
- after a successful migration. The <code>--persistent</code> flag will
- cause a configuration file to be created on the destination host
- after a successful migration. The following table summarizes the
- configuration file handling in all possible state and flag
- combinations.
- </p>
-
- <table>
- <thead>
- <tr class="head">
- <th colspan="3">Before migration</th>
- <th colspan="2">Flags</th>
- <th colspan="3">After migration</th>
- </tr>
- <tr class="subhead">
- <th>Source type</th>
- <th>Source config</th>
- <th>Dest config</th>
- <th>--undefinesource</th>
- <th>--persistent</th>
- <th>Dest type</th>
- <th>Source config</th>
- <th>Dest config</th>
- </tr>
- </thead>
- <tbody>
- <!-- src:N, dst:N -->
- <tr>
- <td>Transient</td>
- <td class="n">N</td>
- <td class="n">N</td>
- <td class="n">N</td>
- <td class="n">N</td>
- <td>Transient</td>
- <td class="n">N</td>
- <td class="n">N</td>
- </tr>
- <tr>
- <td>Transient</td>
- <td class="n">N</td>
- <td class="n">N</td>
- <td class="y">Y</td>
- <td class="n">N</td>
- <td>Transient</td>
- <td class="n">N</td>
- <td class="n">N</td>
- </tr>
- <tr>
- <td>Transient</td>
- <td class="n">N</td>
- <td class="n">N</td>
- <td class="n">N</td>
- <td class="y">Y</td>
- <td>Persistent</td>
- <td class="n">N</td>
- <td class="y">Y</td>
- </tr>
- <tr>
- <td>Transient</td>
- <td class="n">N</td>
- <td class="n">N</td>
- <td class="y">Y</td>
- <td class="y">Y</td>
- <td>Persistent</td>
- <td class="n">N</td>
- <td class="y">Y</td>
- </tr>
-
- <!-- src:N, dst:Y -->
- <tr>
- <td>Transient</td>
- <td class="n">N</td>
- <td class="y">Y</td>
- <td class="n">N</td>
- <td class="n">N</td>
- <td>Persistent</td>
- <td class="n">N</td>
- <td class="y">Y<br/>(unchanged dest config)</td>
- </tr>
- <tr>
- <td>Transient</td>
- <td class="n">N</td>
- <td class="y">Y</td>
- <td class="y">Y</td>
- <td class="n">N</td>
- <td>Persistent</td>
- <td class="n">N</td>
- <td class="y">Y<br/>(unchanged dest config)</td>
- </tr>
- <tr>
- <td>Transient</td>
- <td class="n">N</td>
- <td class="y">Y</td>
- <td class="n">N</td>
- <td class="y">Y</td>
- <td>Persistent</td>
- <td class="n">N</td>
- <td class="y">Y<br/>(replaced with source)</td>
- </tr>
- <tr>
- <td>Transient</td>
- <td class="n">N</td>
- <td class="y">Y</td>
- <td class="y">Y</td>
- <td class="y">Y</td>
- <td>Persistent</td>
- <td class="n">N</td>
- <td class="y">Y<br/>(replaced with source)</td>
- </tr>
-
- <!-- src:Y dst:N -->
- <tr>
- <td>Persistent</td>
- <td class="y">Y</td>
- <td class="n">N</td>
- <td class="n">N</td>
- <td class="n">N</td>
- <td>Transient</td>
- <td class="y">Y</td>
- <td class="n">N</td>
- </tr>
- <tr>
- <td>Persistent</td>
- <td class="y">Y</td>
- <td class="n">N</td>
- <td class="y">Y</td>
- <td class="n">N</td>
- <td>Transient</td>
- <td class="n">N</td>
- <td class="n">N</td>
- </tr>
- <tr>
- <td>Persistent</td>
- <td class="y">Y</td>
- <td class="n">N</td>
- <td class="n">N</td>
- <td class="y">Y</td>
- <td>Persistent</td>
- <td class="y">Y</td>
- <td class="y">Y</td>
- </tr>
- <tr>
- <td>Persistent</td>
- <td class="y">Y</td>
- <td class="n">N</td>
- <td class="y">Y</td>
- <td class="y">Y</td>
- <td>Persistent</td>
- <td class="n">N</td>
- <td class="y">Y</td>
- </tr>
-
- <!-- src:Y dst:Y -->
- <tr>
- <td>Persistent</td>
- <td class="y">Y</td>
- <td class="y">Y</td>
- <td class="n">N</td>
- <td class="n">N</td>
- <td>Persistent</td>
- <td class="y">Y</td>
- <td class="y">Y<br/>(unchanged dest config)</td>
- </tr>
- <tr>
- <td>Persistent</td>
- <td class="y">Y</td>
- <td class="y">Y</td>
- <td class="y">Y</td>
- <td class="n">N</td>
- <td>Persistent</td>
- <td class="n">N</td>
- <td class="y">Y<br/>(unchanged dest config)</td>
- </tr>
- <tr>
- <td>Persistent</td>
- <td class="y">Y</td>
- <td class="y">Y</td>
- <td class="n">N</td>
- <td class="y">Y</td>
- <td>Persistent</td>
- <td class="y">Y</td>
- <td class="y">Y<br/>(replaced with source)</td>
- </tr>
- <tr>
- <td>Persistent</td>
- <td class="y">Y</td>
- <td class="y">Y</td>
- <td class="y">Y</td>
- <td class="y">Y</td>
- <td>Persistent</td>
- <td class="n">N</td>
- <td class="y">Y<br/>(replaced with source)</td>
- </tr>
- </tbody>
- </table>
-
- <h2><a id="scenarios">Migration scenarios</a></h2>
-
-
- <h3><a id="scenarionativedirect">Native migration, client to
two libvirtd servers</a></h3>
-
- <p>
- At an API level this requires use of virDomainMigrate, without the
- VIR_MIGRATE_PEER2PEER flag set. The destination libvirtd server
- will automatically determine the native hypervisor URI for migration
- based off the primary hostname. To force migration over an alternate
- network interface the optional hypervisor specific URI must be provided
- </p>
-
- <pre>
-syntax: virsh migrate GUESTNAME DEST-LIBVIRT-URI [HV-URI]
-
-
-eg using default network interface
-
-virsh migrate web1 qemu+ssh://desthost/system
-virsh migrate web1 xen+tls://desthost/system
-
-
-eg using secondary network interface
-
-virsh migrate web1 qemu://desthost/system tcp://10.0.0.1/
- </pre>
-
- <p>
- Supported by Xen, QEMU, VMware and VirtualBox drivers
- </p>
-
- <h3><a id="scenarionativepeer2peer">Native migration, client to
and peer2peer between, two libvirtd servers</a></h3>
-
- <p>
- virDomainMigrate, with the VIR_MIGRATE_PEER2PEER flag set,
- using the libvirt URI format for the 'uri' parameter. The
- destination libvirtd server will automatically determine
- the native hypervisor URI for migration, based off the
- primary hostname. The optional uri parameter controls how
- the source libvirtd connects to the destination libvirtd,
- in case it is not accessible using the same address that
- the client uses to connect to the destination, or a different
- encryption/auth scheme is required. There is no
- scope for forcing an alternative network interface for the
- native migration data with this method.
- </p>
-
- <p>
- This mode cannot be invoked from virsh
- </p>
-
- <p>
- Supported by QEMU driver
- </p>
-
- <h3><a id="scenariotunnelpeer2peer1">Tunnelled migration,
client and peer2peer between two libvirtd servers</a></h3>
-
- <p>
- virDomainMigrate, with the VIR_MIGRATE_PEER2PEER & VIR_MIGRATE_TUNNELLED
- flags set, using the libvirt URI format for the 'uri' parameter. The
- destination libvirtd server will automatically determine
- the native hypervisor URI for migration, based off the
- primary hostname. The optional uri parameter controls how
- the source libvirtd connects to the destination libvirtd,
- in case it is not accessible using the same address that
- the client uses to connect to the destination, or a different
- encryption/auth scheme is required. The native hypervisor URI
- format is not used at all.
- </p>
-
- <p>
- This mode cannot be invoked from virsh
- </p>
-
- <p>
- Supported by QEMU driver
- </p>
-
- <h3><a id="nativedirectunmanaged">Native migration, client to
one libvirtd server</a></h3>
-
- <p>
- virDomainMigrateToURI, without the VIR_MIGRATE_PEER2PEER flag set,
- using a hypervisor specific URI format for the 'uri' parameter.
- There is no use or requirement for a destination libvirtd instance
- at all. This is typically used when the hypervisor has its own
- native management daemon available to handle incoming migration
- attempts on the destination.
- </p>
-
- <pre>
-syntax: virsh migrate GUESTNAME HV-URI
-
-
-eg using same libvirt URI for all connections
-
- </pre>
-
- <h3><a id="nativepeer2peer">Native migration, peer2peer between
two libvirtd servers</a></h3>
-
- <p>
- virDomainMigrateToURI, with the VIR_MIGRATE_PEER2PEER flag set,
- using the libvirt URI format for the 'uri' parameter. The
- destination libvirtd server will automatically determine
- the native hypervisor URI for migration, based off the
- primary hostname. There is no scope for forcing an alternative
- network interface for the native migration data with this
- method. The destination URI must be reachable using the source
- libvirtd credentials (which are not necessarily the same as the
- credentials of the client in connecting to the source).
- </p>
-
- <pre>
-syntax: virsh migrate GUESTNAME DEST-LIBVIRT-URI [ALT-DEST-LIBVIRT-URI]
-
-
-eg using same libvirt URI for all connections
-
-virsh migrate --p2p web1 qemu+ssh://desthost/system
-
-
-eg using different libvirt URI auth scheme for peer2peer connections
-
-virsh migrate --p2p web1 qemu+ssh://desthost/system qemu+tls:/desthost/system
-
-
-eg using different libvirt URI hostname for peer2peer connections
-
-virsh migrate --p2p web1 qemu+ssh://desthost/system qemu+ssh://10.0.0.1/system
- </pre>
-
- <p>
- Supported by the QEMU driver
- </p>
-
- <h3><a id="scenariotunnelpeer2peer2">Tunnelled migration,
peer2peer between two libvirtd servers</a></h3>
-
- <p>
- virDomainMigrateToURI, with the VIR_MIGRATE_PEER2PEER &
VIR_MIGRATE_TUNNELLED
- flags set, using the libvirt URI format for the 'uri' parameter. The
- destination libvirtd server will automatically determine
- the native hypervisor URI for migration, based off the
- primary hostname. The optional uri parameter controls how
- the source libvirtd connects to the destination libvirtd,
- in case it is not accessible using the same address that
- the client uses to connect to the destination, or a different
- encryption/auth scheme is required. The native hypervisor URI
- format is not used at all. The destination URI must be
- reachable using the source libvirtd credentials (which are not
- necessarily the same as the credentials of the client in
- connecting to the source).
- </p>
-
- <pre>
-syntax: virsh migrate GUESTNAME DEST-LIBVIRT-URI [ALT-DEST-LIBVIRT-URI]
-
-
-eg using same libvirt URI for all connections
-
-virsh migrate --p2p --tunnelled web1 qemu+ssh://desthost/system
-
-
-eg using different libvirt URI auth scheme for peer2peer connections
-
-virsh migrate --p2p --tunnelled web1 qemu+ssh://desthost/system
qemu+tls:/desthost/system
-
-
-eg using different libvirt URI hostname for peer2peer connections
-
-virsh migrate --p2p --tunnelled web1 qemu+ssh://desthost/system
qemu+ssh://10.0.0.1/system
- </pre>
-
- <p>
- Supported by QEMU driver
- </p>
-
-
- <h3><a id="scenariounixsocket">Migration using only UNIX
sockets</a></h3>
-
- <p>
- In niche scenarios where libvirt daemon does not have access to the
- network (e.g. running in a restricted container on a host that has
- accessible network), when a management application wants to have complete
- control over the transfer or when migrating between two containers on the
- same host all the communication can be done using UNIX sockets. This
- includes connecting to non-standard socket path for the destination
- daemon, using UNIX sockets for hypervisor's communication or for the NBD
- data transfer. All of that can be used with both peer2peer and direct
- migration options.
- </p>
-
- <p>
- Example using <code>/tmp/migdir</code> as a directory representing the
- same path visible from both libvirt daemons. That can be achieved by
- bind-mounting the same directory to different containers running separate
- daemons or forwarding connections to these sockets manually
- (using <code>socat</code>, <code>netcat</code> or a custom
piece of
- software):
- </p>
- <pre>
-virsh migrate --domain web1 [--p2p] --copy-storage-all
- --desturi 'qemu+unix:///system?socket=/tmp/migdir/test-sock-driver'
- --migrateuri 'unix:///tmp/migdir/test-sock-qemu'
- --disks-uri unix:///tmp/migdir/test-sock-nbd
- </pre>
-
- <p>
- One caveat is that on SELinux-enabled systems all the sockets that the
- hypervisor is going to connect to needs to have the proper context and
- that is chosen before its creation by the process that creates it. That
- is usually done by using <code>setsockcreatecon{,raw}()</code>
functions.
- Generally *system_r:system_u:svirt_socket_t:s0* should do the trick, but
- check the SELinux rules and settings of your system.
- </p>
-
- <p>
- Supported by QEMU driver
- </p>
-
- </body>
-</html>
diff --git a/docs/migration.rst b/docs/migration.rst
new file mode 100644
index 0000000000..0a40600462
--- /dev/null
+++ b/docs/migration.rst
@@ -0,0 +1,458 @@
+===============
+Guest migration
+===============
+
+.. contents::
+
+Migration of guests between hosts is a complicated problem with many possible
+solutions, each with their own positive and negative points. For maximum
+flexibility of both hypervisor integration, and administrator deployment,
+libvirt implements several options for migration.
+
+Network data transports
+-----------------------
+
+There are two options for the data transport used during migration, either the
+hypervisor's own **native** transport, or **tunnelled** over a libvirtd
+connection.
+
+Hypervisor native transport
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+*Native* data transports may or may not support encryption, depending on the
+hypervisor in question, but will typically have the lowest computational costs
+by minimising the number of data copies involved. The native data transports
+will also require extra hypervisor-specific network configuration steps by the
+administrator when deploying a host. For some hypervisors, it might be necessary
+to open up a large range of ports on the firewall to allow multiple concurrent
+migration operations.
+
+Modern hypervisors support TLS for encryption and authentication of the
+migration connections which can be enabled using the ``VIR_MIGRATE_TLS`` flag.
+The *qemu* hypervisor driver allows users to force use of TLS via the
+``migrate_tls_force`` knob configured in ``/etc/libvirt/qemu.conf``.
+
+|Migration native path|
+
+libvirt tunnelled transport
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+*Tunnelled* data transports will always be capable of strong encryption since
+they are able to leverage the capabilities built in to the libvirt RPC protocol.
+The downside of a tunnelled transport, however, is that there will be extra data
+copies involved on both the source and destinations hosts as the data is moved
+between libvirtd and the hypervisor. This is likely to be a more significant
+problem for guests with very large RAM sizes, which dirty memory pages quickly.
+On the deployment side, tunnelled transports do not require any extra network
+configuration over and above what's already required for general libvirtd
+`remote access <remote.html>`__, and there is only need for a single port to be
+open on the firewall to support multiple concurrent migration operations.
+
+*Note:* Certain features such as migration of non-shared storage
+(``VIR_MIGRATE_NON_SHARED_DISK``), the multi-connection migration
+(``VIR_MIGRATE_PARALLEL``), or post-copy migration (``VIR_MIGRATE_POSTCOPY``)
+may not be available when using libvirt's tunnelling.
+
+|Migration tunnel path|
+
+Communication control paths/flows
+---------------------------------
+
+Migration of virtual machines requires close co-ordination of the two hosts
+involved, as well as the application invoking the migration, which may be on the
+source, the destination, or a third host.
+
+Managed direct migration
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+With *managed direct* migration, the libvirt client process controls the various
+phases of migration. The client application must be able to connect and
+authenticate with the libvirtd daemons on both the source and destination hosts.
+There is no need for the two libvirtd daemons to communicate with each other. If
+the client application crashes, or otherwise loses its connection to libvirtd
+during the migration process, an attempt will be made to abort the migration and
+restart the guest CPUs on the source host. There may be scenarios where this
+cannot be safely done, in which cases the guest will be left paused on one or
+both of the hosts.
+
+|Migration direct, managed|
+
+Managed peer to peer migration
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+With *peer to peer* migration, the libvirt client process only talks to the
+libvirtd daemon on the source host. The source libvirtd daemon controls the
+entire migration process itself, by directly connecting the destination host
+libvirtd. If the client application crashes, or otherwise loses its connection
+to libvirtd, the migration process will continue uninterrupted until completion.
+Note that the source libvirtd uses its own credentials (typically root) to
+connect to the destination, rather than the credentials used by the client to
+connect to the source; if these differ, it is common to run into a situation
+where a client can connect to the destination directly but the source cannot
+make the connection to set up the peer-to-peer migration.
+
+|Migration peer-to-peer|
+
+Unmanaged direct migration
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+With *unmanaged direct* migration, neither the libvirt client or libvirtd daemon
+control the migration process. Control is instead delegated to the hypervisor's
+over management services (if any). The libvirt client merely initiates the
+migration via the hypervisor's management layer. If the libvirt client or
+libvirtd crash, the migration process will continue uninterrupted until
+completion.
+
+|Migration direct, unmanaged|
+
+Data security
+-------------
+
+Since the migration data stream includes a complete copy of the guest OS RAM,
+snooping of the migration data stream may allow compromise of sensitive guest
+information. If the virtualization hosts have multiple network interfaces, or if
+the network switches support tagged VLANs, then it is very desirable to separate
+guest network traffic from migration or management traffic.
+
+In some scenarios, even a separate network for migration data may not offer
+sufficient security. In this case it is possible to apply encryption to the
+migration data stream. If the hypervisor does not itself offer encryption, then
+the libvirt tunnelled migration facility should be used.
+
+Offline migration
+-----------------
+
+Offline migration transfers the inactive definition of a domain (which may or
+may not be active). After successful completion, the domain remains in its
+current state on the source host and is defined but inactive on the destination
+host. It's a bit more clever than ``virsh dumpxml`` on source host followed by
+``virsh define`` on destination host, as offline migration will run the
+pre-migration hook to update the domain XML on destination host. Currently,
+copying non-shared storage or other file based storages (e.g. UEFI variable
+storage) is not supported during offline migration.
+
+Migration URIs
+--------------
+
+Initiating a guest migration requires the client application to specify up to
+three URIs, depending on the choice of control flow and/or APIs used. The first
+URI is that of the libvirt connection to the source host, where the virtual
+guest is currently running. The second URI is that of the libvirt connection to
+the destination host, where the virtual guest will be moved to (and in
+peer-to-peer migrations, this is from the perspective of the source, not the
+client). The third URI is a hypervisor specific URI used to control how the
+guest will be migrated. With any managed migration flow, the first and second
+URIs are compulsory, while the third URI is optional. With the unmanaged direct
+migration mode, the first and third URIs are compulsory and the second URI is
+not used.
+
+Ordinarily management applications only need to care about the first and second
+URIs, which are both in the normal libvirt connection URI format. Libvirt will
+then automatically determine the hypervisor specific URI, by looking up the
+target host's configured hostname. There are a few scenarios where the
+management application may wish to have direct control over the third URI.
+
+#. The configured hostname is incorrect, or DNS is broken. If a host has a
+ hostname which will not resolve to match one of its public IP addresses, then
+ libvirt will generate an incorrect URI. In this case the management
+ application should specify the hypervisor specific URI explicitly, using an
+ IP address, or a correct hostname.
+#. The host has multiple network interfaces. If a host has multiple network
+ interfaces, it might be desirable for the migration data stream to be sent
+ over a specific interface for either security or performance reasons. In this
+ case the management application should specify the hypervisor specific URI,
+ using an IP address associated with the network to be used.
+#. The firewall restricts what ports are available. When libvirt generates a
+ migration URI it will pick a port number using hypervisor specific rules.
+ Some hypervisors only require a single port to be open in the firewalls,
+ while others require a whole range of port numbers. In the latter case the
+ management application may wish to choose a specific port number outside the
+ default range in order to comply with local firewall policies.
+#. The second URI uses UNIX transport method. In this advanced case libvirt
+ should not guess a \*migrateuri\* and it should be specified using UNIX
+ socket path URI: ``unix:///path/to/socket``.
+
+Configuration file handling
+---------------------------
+
+There are two types of virtual machines known to libvirt. A *transient* guest
+only exists while it is running, and has no configuration file stored on disk. A
+*persistent* guest maintains a configuration file on disk even when it is not
+running.
+
+By default, a migration operation will not attempt to modify any configuration
+files that may be stored on either the source or destination host. It is the
+administrator, or management application's, responsibility to manage
+distribution of configuration files (if desired). It is important to note that
+the ``/etc/libvirt`` directory **MUST NEVER BE SHARED BETWEEN HOSTS**. There are
+some typical scenarios that might be applicable:
+
+- Centralized configuration files outside libvirt, in shared storage. A cluster
+ aware management application may maintain all the master guest configuration
+ files in a cluster filesystem. When attempting to start a guest, the config
+ will be read from the cluster FS and used to deploy a persistent guest. For
+ migration the configuration will need to be copied to the destination host
+ and removed on the original.
+- Centralized configuration files outside libvirt, in a database. A data center
+ management application may not store configuration files at all. Instead it
+ may generate libvirt XML on the fly when a guest is booted. It will typically
+ use transient guests, and thus not have to consider configuration files
+ during migration.
+- Distributed configuration inside libvirt. The configuration file for each
+ guest is copied to every host where the guest is able to run. Upon migration
+ the existing config merely needs to be updated with any changes.
+- Ad-hoc configuration management inside libvirt. Each guest is tied to a
+ specific host and rarely migrated. When migration is required, the config is
+ moved from one host to the other.
+
+As mentioned above, libvirt will not modify configuration files during migration
+by default. The ``virsh`` command has two flags to influence this behaviour. The
+``--undefinesource`` flag will cause the configuration file to be removed on the
+source host after a successful migration. The ``--persistent`` flag will cause a
+configuration file to be created on the destination host after a successful
+migration. The following table summarizes the configuration file handling in all
+possible state and flag combinations.
+
++-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+
+| Before migration | Flags
| After migration |
++-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+
+| Source type | Source config | Dest config | --undefinesource |
--persistent | Dest type | Source config | Dest config |
++===================+===================+===================+===================+===================+===================+===================+===================+
+| Transient | N | N | N | N
| Transient | N | N |
++-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+
+| Transient | N | N | Y | N
| Transient | N | N |
++-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+
+| Transient | N | N | N | Y
| Persistent | N | Y |
++-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+
+| Transient | N | N | Y | Y
| Persistent | N | Y |
++-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+
+| Transient | N | Y | N | N
| Persistent | N | Y |
+| | | | |
| | | (unchanged dest |
+| | | | |
| | | config) |
++-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+
+| Transient | N | Y | Y | N
| Persistent | N | Y |
+| | | | |
| | | (unchanged dest |
+| | | | |
| | | config) |
++-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+
+| Transient | N | Y | N | Y
| Persistent | N | Y |
+| | | | |
| | | (replaced with |
+| | | | |
| | | source) |
++-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+
+| Transient | N | Y | Y | Y
| Persistent | N | Y |
+| | | | |
| | | (replaced with |
+| | | | |
| | | source) |
++-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+
+| Persistent | Y | N | N | N
| Transient | Y | N |
++-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+
+| Persistent | Y | N | Y | N
| Transient | N | N |
++-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+
+| Persistent | Y | N | N | Y
| Persistent | Y | Y |
++-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+
+| Persistent | Y | N | Y | Y
| Persistent | N | Y |
++-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+
+| Persistent | Y | Y | N | N
| Persistent | Y | Y |
+| | | | |
| | | (unchanged dest |
+| | | | |
| | | config) |
++-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+
+| Persistent | Y | Y | Y | N
| Persistent | N | Y |
+| | | | |
| | | (unchanged dest |
+| | | | |
| | | config) |
++-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+
+| Persistent | Y | Y | N | Y
| Persistent | Y | Y |
+| | | | |
| | | (replaced with |
+| | | | |
| | | source) |
++-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+
+| Persistent | Y | Y | Y | Y
| Persistent | N | Y |
+| | | | |
| | | (replaced with |
+| | | | |
| | | source) |
++-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+
+
+Migration scenarios
+-------------------
+
+Native migration, client to two libvirtd servers
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+At an API level this requires use of virDomainMigrate, without the
+VIR_MIGRATE_PEER2PEER flag set. The destination libvirtd server will
+automatically determine the native hypervisor URI for migration based off the
+primary hostname. To force migration over an alternate network interface the
+optional hypervisor specific URI must be provided
+
+::
+
+ syntax: virsh migrate GUESTNAME DEST-LIBVIRT-URI [HV-URI]
+
+
+ eg using default network interface
+
+ virsh migrate web1 qemu+ssh://desthost/system
+ virsh migrate web1 xen+tls://desthost/system
+
+
+ eg using secondary network interface
+
+ virsh migrate web1 qemu://desthost/system tcp://10.0.0.1/
+
+Supported by Xen, QEMU, VMware and VirtualBox drivers
+
+Native migration, client to and peer2peer between, two libvirtd servers
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+virDomainMigrate, with the VIR_MIGRATE_PEER2PEER flag set, using the libvirt URI
+format for the 'uri' parameter. The destination libvirtd server will
+automatically determine the native hypervisor URI for migration, based off the
+primary hostname. The optional uri parameter controls how the source libvirtd
+connects to the destination libvirtd, in case it is not accessible using the
+same address that the client uses to connect to the destination, or a different
+encryption/auth scheme is required. There is no scope for forcing an alternative
+network interface for the native migration data with this method.
+
+This mode cannot be invoked from virsh
+
+Supported by QEMU driver
+
+Tunnelled migration, client and peer2peer between two libvirtd servers
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+virDomainMigrate, with the VIR_MIGRATE_PEER2PEER & VIR_MIGRATE_TUNNELLED flags
+set, using the libvirt URI format for the 'uri' parameter. The destination
+libvirtd server will automatically determine the native hypervisor URI for
+migration, based off the primary hostname. The optional uri parameter controls
+how the source libvirtd connects to the destination libvirtd, in case it is not
+accessible using the same address that the client uses to connect to the
+destination, or a different encryption/auth scheme is required. The native
+hypervisor URI format is not used at all.
+
+This mode cannot be invoked from virsh
+
+Supported by QEMU driver
+
+Native migration, client to one libvirtd server
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+virDomainMigrateToURI, without the VIR_MIGRATE_PEER2PEER flag set, using a
+hypervisor specific URI format for the 'uri' parameter. There is no use or
+requirement for a destination libvirtd instance at all. This is typically used
+when the hypervisor has its own native management daemon available to handle
+incoming migration attempts on the destination.
+
+::
+
+ syntax: virsh migrate GUESTNAME HV-URI
+
+
+ eg using same libvirt URI for all connections
+
+
+Native migration, peer2peer between two libvirtd servers
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+virDomainMigrateToURI, with the VIR_MIGRATE_PEER2PEER flag set, using the
+libvirt URI format for the 'uri' parameter. The destination libvirtd server will
+automatically determine the native hypervisor URI for migration, based off the
+primary hostname. There is no scope for forcing an alternative network interface
+for the native migration data with this method. The destination URI must be
+reachable using the source libvirtd credentials (which are not necessarily the
+same as the credentials of the client in connecting to the source).
+
+::
+
+ syntax: virsh migrate GUESTNAME DEST-LIBVIRT-URI [ALT-DEST-LIBVIRT-URI]
+
+
+ eg using same libvirt URI for all connections
+
+ virsh migrate --p2p web1 qemu+ssh://desthost/system
+
+
+ eg using different libvirt URI auth scheme for peer2peer connections
+
+ virsh migrate --p2p web1 qemu+ssh://desthost/system qemu+tls:/desthost/system
+
+
+ eg using different libvirt URI hostname for peer2peer connections
+
+ virsh migrate --p2p web1 qemu+ssh://desthost/system qemu+ssh://10.0.0.1/system
+
+Supported by the QEMU driver
+
+Tunnelled migration, peer2peer between two libvirtd servers
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+virDomainMigrateToURI, with the VIR_MIGRATE_PEER2PEER & VIR_MIGRATE_TUNNELLED
+flags set, using the libvirt URI format for the 'uri' parameter. The destination
+libvirtd server will automatically determine the native hypervisor URI for
+migration, based off the primary hostname. The optional uri parameter controls
+how the source libvirtd connects to the destination libvirtd, in case it is not
+accessible using the same address that the client uses to connect to the
+destination, or a different encryption/auth scheme is required. The native
+hypervisor URI format is not used at all. The destination URI must be reachable
+using the source libvirtd credentials (which are not necessarily the same as the
+credentials of the client in connecting to the source).
+
+::
+
+ syntax: virsh migrate GUESTNAME DEST-LIBVIRT-URI [ALT-DEST-LIBVIRT-URI]
+
+
+ eg using same libvirt URI for all connections
+
+ virsh migrate --p2p --tunnelled web1 qemu+ssh://desthost/system
+
+
+ eg using different libvirt URI auth scheme for peer2peer connections
+
+ virsh migrate --p2p --tunnelled web1 qemu+ssh://desthost/system
qemu+tls:/desthost/system
+
+
+ eg using different libvirt URI hostname for peer2peer connections
+
+ virsh migrate --p2p --tunnelled web1 qemu+ssh://desthost/system
qemu+ssh://10.0.0.1/system
+
+Supported by QEMU driver
+
+Migration using only UNIX sockets
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+In niche scenarios where libvirt daemon does not have access to the network
+(e.g. running in a restricted container on a host that has accessible network),
+when a management application wants to have complete control over the transfer
+or when migrating between two containers on the same host all the communication
+can be done using UNIX sockets. This includes connecting to non-standard socket
+path for the destination daemon, using UNIX sockets for hypervisor's
+communication or for the NBD data transfer. All of that can be used with both
+peer2peer and direct migration options.
+
+Example using ``/tmp/migdir`` as a directory representing the same path visible
+from both libvirt daemons. That can be achieved by bind-mounting the same
+directory to different containers running separate daemons or forwarding
+connections to these sockets manually (using ``socat``, ``netcat`` or a custom
+piece of software):
+
+::
+
+ virsh migrate --domain web1 [--p2p] --copy-storage-all
+ --desturi 'qemu+unix:///system?socket=/tmp/migdir/test-sock-driver'
+ --migrateuri 'unix:///tmp/migdir/test-sock-qemu'
+ --disks-uri unix:///tmp/migdir/test-sock-nbd
+
+One caveat is that on SELinux-enabled systems all the sockets that the
+hypervisor is going to connect to needs to have the proper context and that is
+chosen before its creation by the process that creates it. That is usually done
+by using ``setsockcreatecon{,raw}()`` functions. Generally
+\*system_r:system_u:svirt_socket_t:s0\* should do the trick, but check the
+SELinux rules and settings of your system.
+
+Supported by QEMU driver
+
+.. |Migration native path| image:: migration-native.png
+ :class: diagram
+.. |Migration tunnel path| image:: migration-tunnel.png
+ :class: diagram
+.. |Migration direct, managed| image:: migration-managed-direct.png
+ :class: diagram
+.. |Migration peer-to-peer| image:: migration-managed-p2p.png
+ :class: diagram
+.. |Migration direct, unmanaged| image:: migration-unmanaged-direct.png
+ :class: diagram
--
2.31.1