5:59 a.m.
In preparation for adding docs about virtlockd, split out
the sanlock setup docs into a separate page.
Signed-off-by: Daniel P. Berrange <berrange(a)redhat.com>
---
docs/locking-sanlock.html.in | 247 ++++++++++++++++++++++++++++++++++++++
docs/locking.html.in | 277 ++++++-------------------------------------
docs/sitemap.html.in | 6 +
3 files changed, 286 insertions(+), 244 deletions(-)
create mode 100644 docs/locking-sanlock.html.in
diff --git a/docs/locking-sanlock.html.in b/docs/locking-sanlock.html.in
new file mode 100644
index 0000000..84b1fb3
--- /dev/null
+++ b/docs/locking-sanlock.html.in
@@ -0,0 +1,247 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
+<html xmlns="http://www.w3.org/1999/xhtml">
+ <body>
+ <h1>Virtual machine lock manager, sanlock plugin</h1>
+
+ <ul id="toc"></ul>
+
+ <p>
+ This page describes use of the
+ <a href="https://fedorahosted.org/sanlock/">sanlock</a>
+ service as a <a href="locking.html">lock driver</a>
+ plugin for virtual machine disk mutual exclusion.
+ </p>
+
+ <h2><a name="sanlock">Sanlock daemon
setup</a></h2>
+
+ <p>
+ On many operating systems, the <strong>sanlock</strong> plugin
+ is distributed in a sub-package which needs to be installed
+ separately from the main libvirt RPM. On a Fedora/RHEL host
+ this can be done with the <code>yum</code> command
+ </p>
+
+ <pre>
+ $ su - root
+ # yum install libvirt-lock-sanlock
+ </pre>
+
+ <p>
+ The next step is to start the sanlock daemon. For maximum
+ safety sanlock prefers to have a connection to a watchdog
+ daemon. This will cause the entire host to be rebooted in
+ the event that sanlock crashes / terminates abnormally.
+ To start the watchdog daemon on a Fedora/RHEL host
+ the following commands can be run:
+ </p>
+
+ <pre>
+ $ su - root
+ # chkconfig wdmd on
+ # service wdmd start
+ </pre>
+
+ <p>
+ Once the watchdog is running, sanlock can be started
+ as follows
+ </p>
+
+ <pre>
+ # chkconfig sanlock on
+ # service sanlock start
+ </pre>
+
+ <p>
+ <em>Note:</em> if you wish to avoid the use of the
+ watchdog, add the following line to
<code>/etc/sysconfig/sanlock</code>
+ before starting it
+ </p>
+
+ <pre>
+ SANLOCKOPTS="-w 0"
+ </pre>
+
+ <p>
+ The sanlock daemon must be started on every single host
+ that will be running virtual machines. So repeat these
+ steps as necessary.
+ </p>
+
+ <h2><a name="sanlockplugin">libvirt sanlock plugin
configuration</a></h2>
+
+ <p>
+ Once the sanlock daemon is running, the next step is to
+ configure the libvirt sanlock plugin. There is a separate
+ configuration file for each libvirt driver that is using
+ sanlock. For QEMU, we will edit
<code>/etc/libvirt/qemu-sanlock.conf</code>
+ There is one mandatory parameter that needs to be set,
+ the <code>host_id</code>. This is a integer between
+ 1 and 2000, which must be set to a <strong>unique</strong>
+ value on each host running virtual machines.
+ </p>
+
+ <pre>
+ $ su - root
+ # augtool -s set /files/etc/libvirt/qemu-sanlock.conf/host_id 1
+ </pre>
+
+ <p>
+ Repeat this on every host, changing <strong>1</strong> to a
+ unique value for the host.
+ </p>
+
+ <h2><a name="sanlockstorage">libvirt sanlock storage
configuration</a></h2>
+
+ <p>
+ The sanlock plugin needs to create leases in a directory
+ that is on a filesystem shared between all hosts running
+ virtual machines. Obvious choices for this include NFS
+ or GFS2. The libvirt sanlock plugin expects its lease
+ directory be at <code>/var/lib/libvirt/sanlock</code>
+ so update the host's <code>/etc/fstab</code> to mount
+ a suitable shared/cluster filesystem at that location
+ </p>
+
+ <pre>
+ $ su - root
+ # echo "some.nfs.server:/export/sanlock /var/lib/libvirt/sanlock nfs
hard,nointr 0 0" >> /etc/fstab
+ # mount /var/lib/libvirt/sanlock
+ </pre>
+
+ <p>
+ If your sanlock daemon happen to run under non-root
+ privileges, you need to tell this to libvirt so it
+ chowns created files correctly. This can be done by
+ setting <code>user</code> and/or <code>group</code>
+ variables in the configuration file. Accepted values
+ range is specified in description to the same
+ variables in <code>/etc/libvirt/qemu.conf</code>. For
+ example:
+ </p>
+
+ <pre>
+ augtool -s set /files/etc/libvirt/qemu-sanlock.conf/user sanlock
+ augtool -s set /files/etc/libvirt/qemu-sanlock.conf/group sanlock
+ </pre>
+
+ <p>
+ But remember, that if this is NFS share, you need a
+ no_root_squash-ed one for chown (and chmod possibly)
+ to succeed.
+ </p>
+
+ <p>
+ In terms of storage requirements, if the filesystem
+ uses 512 byte sectors, you need to allow for <code>1MB</code>
+ of storage for each guest disk. So if you have a network
+ with 20 virtualization hosts, each running 50 virtual
+ machines and an average of 2 disks per guest, you will
+ need <code>20*50*2 == 2000 MB</code> of storage for
+ sanlock.
+ </p>
+
+
+ <p>
+ On one of the hosts on the network is it wise to setup
+ a cron job which runs the <code>virt-sanlock-cleanup</code>
+ script periodically. This scripts deletes any lease
+ files which are not currently in use by running virtual
+ machines, freeing up disk space on the shared filesystem.
+ Unless VM disks are very frequently created + deleted
+ it should be sufficient to run the cleanup once a week.
+ </p>
+
+ <h2><a name="qemuconfig">QEMU/KVM driver
configuration</a></h2>
+
+ <p>
+ The QEMU/KVM driver is fully integrated with the lock
+ manager framework as of release <span>0.9.3</span>.
+ The out of the box configuration, however, currently
+ uses the <strong>nop</strong> lock manager plugin.
+ To get protection for disks, it is thus necessary
+ to reconfigure QEMU to activate the <strong>sanlock</strong>
+ driver. This is achieved by editing the QEMU driver
+ configuration file (<code>/etc/libvirt/qemu.conf</code>)
+ and changing the <code>lock_manager</code> configuration
+ tunable.
+ </p>
+
+ <pre>
+ $ su - root
+ # augtool -s set /files/etc/libvirt/qemu.conf/lock_manager sanlock
+ # service libvirtd restart
+ </pre>
+
+ <p>
+ If all went well, libvirtd will have talked to sanlock
+ and created the basic lockspace. This can be checked
+ by looking for existence of the following file
+ </p>
+
+ <pre>
+ # ls /var/lib/libvirt/sanlock/
+ __LIBVIRT__DISKS__
+ </pre>
+
+ <p>
+ Every time you start a guest, additional lease files will appear
+ in this directory, one for each virtual disk. The lease
+ files are named based on the MD5 checksum of the fully qualified
+ path of the virtual disk backing file. So if the guest is given
+ a disk backed by <code>/var/lib/libvirt/images/demo.img</code>
+ expect to see a lease
<code>/var/lib/libvirt/sanlock/bfa0240911bc17753e0b473688822159</code>
+ </p>
+
+ <p>
+ It should be obvious that for locking to work correctly, every
+ host running virtual machines should have storage configured
+ in the same way. The easiest way to do this is to use the libvirt
+ storage pool capability to configure any NFS volumes, iSCSI targets,
+ or SCSI HBAs used for guest storage. Simply replicate the same
+ storage pool XML across every host. It is important that any
+ storage pools exposing block devices are configured to create
+ volume paths under <code>/dev/disks/by-path</code> to ensure
+ stable paths across hosts. An example iSCSI configuration
+ which ensures this is:
+ </p>
+
+ <pre>
+<pool type='iscsi'>
+ <name>myiscsipool</name>
+ <source>
+ <host name='192.168.254.8'/>
+ <device path='your-iscsi-target-iqn'/>
+ </source>
+ <target>
+ <path>/dev/disk/by-path</path>
+ </target>
+</pool>
+ </pre>
+
+ <h2><a name="domainconfig">Domain
configuration</a></h2>
+
+ <p>
+ In case sanlock loses access to disk locks for some reason, it will
+ kill all domains that lost their locks. This default behavior may
+ be changed using
+ <a href="formatdomain.html#elementsEvents">on_lockfailure
+ element</a> in domain XML. When this element is present, sanlock
+ will call <code>sanlock_helper</code> (provided by libvirt) with
+ the specified action. This helper binary will connect to libvirtd
+ and thus it may need to authenticate if libvirtd was configured to
+ require that on the read-write UNIX socket. To provide the
+ appropriate credentials to sanlock_helper, a
+ <a href="auth.html#Auth_client_config">client authentication
+ file</a> needs to contain something like the following:
+ </p>
+ <pre>
+[auth-libvirt-localhost]
+credentials=sanlock
+
+[credentials-sanlock]
+authname=login
+password=password
+ </pre>
+ </body>
+</html>
diff --git a/docs/locking.html.in b/docs/locking.html.in
index 8e1e72c..cf72eb0 100644
--- a/docs/locking.html.in
+++ b/docs/locking.html.in
@@ -2,258 +2,47 @@
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<body>
- <h1>Virtual machine disk locking</h1>
+ <h1>Virtual machine lock manager</h1>
<ul id="toc"></ul>
<p>
- This page describes how to ensure a single disk cannot be
- used by more than one running VM at a time, across any
- host in a network. This is critical to avoid data corruption
- of guest files systems that are not cluster aware.
+ Libvirt includes a framework for ensuring mutual exclusion
+ between virtual machines using host resources. Typically
+ this is used to prevent two VM processes from having concurrent
+ write access to the same disk image, as this would result in
+ data corruption if the guest was not using a cluster
+ aware filesystem.
</p>
<h2><a name="plugins">Lock manager
plugins</a></h2>
<p>
- libvirt includes a pluggable framework for lock managers,
- which hypervisor drivers can use to ensure safety for
- guest domain disks, and potentially other resources.
- At this time there are only two plugin implementations,
- a "no op" implementation which does absolutely nothing,
- and a <a href="https://fedorahosted.org/sanlock/">sanlock</a>
implementation which uses
- the Disk Paxos algorithm to ensure safety.
- </p>
-
- <h2><a name="sanlock">Sanlock daemon
setup</a></h2>
-
- <p>
- On many operating systems, the <strong>sanlock</strong> plugin
- is distributed in a sub-package which needs to be installed
- separately from the main libvirt RPM. On a Fedora/RHEL host
- this can be done with the <code>yum</code> command
- </p>
-
- <pre>
- $ su - root
- # yum install libvirt-lock-sanlock
- </pre>
-
- <p>
- The next step is to start the sanlock daemon. For maximum
- safety sanlock prefers to have a connection to a watchdog
- daemon. This will cause the entire host to be rebooted in
- the event that sanlock crashes / terminates abnormally.
- To start the watchdog daemon on a Fedora/RHEL host
- the following commands can be run:
- </p>
-
- <pre>
- $ su - root
- # chkconfig wdmd on
- # service wdmd start
- </pre>
-
- <p>
- Once the watchdog is running, sanlock can be started
- as follows
- </p>
-
- <pre>
- # chkconfig sanlock on
- # service sanlock start
- </pre>
-
- <p>
- <em>Note:</em> if you wish to avoid the use of the
- watchdog, add the following line to
<code>/etc/sysconfig/sanlock</code>
- before starting it
- </p>
-
- <pre>
- SANLOCKOPTS="-w 0"
- </pre>
-
- <p>
- The sanlock daemon must be started on every single host
- that will be running virtual machines. So repeat these
- steps as necessary.
- </p>
-
- <h2><a name="sanlockplugin">libvirt sanlock plugin
configuration</a></h2>
-
- <p>
- Once the sanlock daemon is running, the next step is to
- configure the libvirt sanlock plugin. There is a separate
- configuration file for each libvirt driver that is using
- sanlock. For QEMU, we will edit
<code>/etc/libvirt/qemu-sanlock.conf</code>
- There is one mandatory parameter that needs to be set,
- the <code>host_id</code>. This is a integer between
- 1 and 2000, which must be set to a <strong>unique</strong>
- value on each host running virtual machines.
- </p>
-
- <pre>
- $ su - root
- # augtool -s set /files/etc/libvirt/qemu-sanlock.conf/host_id 1
- </pre>
-
- <p>
- Repeat this on every host, changing <strong>1</strong> to a
- unique value for the host.
- </p>
-
- <h2><a name="sanlockstorage">libvirt sanlock storage
configuration</a></h2>
-
- <p>
- The sanlock plugin needs to create leases in a directory
- that is on a filesystem shared between all hosts running
- virtual machines. Obvious choices for this include NFS
- or GFS2. The libvirt sanlock plugin expects its lease
- directory be at <code>/var/lib/libvirt/sanlock</code>
- so update the host's <code>/etc/fstab</code> to mount
- a suitable shared/cluster filesystem at that location
- </p>
-
- <pre>
- $ su - root
- # echo "some.nfs.server:/export/sanlock /var/lib/libvirt/sanlock nfs
hard,nointr 0 0" >> /etc/fstab
- # mount /var/lib/libvirt/sanlock
- </pre>
-
- <p>
- If your sanlock daemon happen to run under non-root
- privileges, you need to tell this to libvirt so it
- chowns created files correctly. This can be done by
- setting <code>user</code> and/or <code>group</code>
- variables in the configuration file. Accepted values
- range is specified in description to the same
- variables in <code>/etc/libvirt/qemu.conf</code>. For
- example:
- </p>
-
- <pre>
- augtool -s set /files/etc/libvirt/qemu-sanlock.conf/user sanlock
- augtool -s set /files/etc/libvirt/qemu-sanlock.conf/group sanlock
- </pre>
-
- <p>
- But remember, that if this is NFS share, you need a
- no_root_squash-ed one for chown (and chmod possibly)
- to succeed.
- </p>
-
- <p>
- In terms of storage requirements, if the filesystem
- uses 512 byte sectors, you need to allow for <code>1MB</code>
- of storage for each guest disk. So if you have a network
- with 20 virtualization hosts, each running 50 virtual
- machines and an average of 2 disks per guest, you will
- need <code>20*50*2 == 2000 MB</code> of storage for
- sanlock.
- </p>
-
-
- <p>
- On one of the hosts on the network is it wise to setup
- a cron job which runs the <code>virt-sanlock-cleanup</code>
- script periodically. This scripts deletes any lease
- files which are not currently in use by running virtual
- machines, freeing up disk space on the shared filesystem.
- Unless VM disks are very frequently created + deleted
- it should be sufficient to run the cleanup once a week.
- </p>
-
- <h2><a name="qemuconfig">QEMU/KVM driver
configuration</a></h2>
-
- <p>
- The QEMU/KVM driver is fully integrated with the lock
- manager framework as of release <span>0.9.3</span>.
- The out of the box configuration, however, currently
- uses the <strong>nop</strong> lock manager plugin.
- To get protection for disks, it is thus necessary
- to reconfigure QEMU to activate the <strong>sanlock</strong>
- driver. This is achieved by editing the QEMU driver
- configuration file (<code>/etc/libvirt/qemu.conf</code>)
- and changing the <code>lock_manager</code> configuration
- tunable.
- </p>
-
- <pre>
- $ su - root
- # augtool -s set /files/etc/libvirt/qemu.conf/lock_manager sanlock
- # service libvirtd restart
- </pre>
-
- <p>
- If all went well, libvirtd will have talked to sanlock
- and created the basic lockspace. This can be checked
- by looking for existence of the following file
- </p>
-
- <pre>
- # ls /var/lib/libvirt/sanlock/
- __LIBVIRT__DISKS__
- </pre>
-
- <p>
- Every time you start a guest, additional lease files will appear
- in this directory, one for each virtual disk. The lease
- files are named based on the MD5 checksum of the fully qualified
- path of the virtual disk backing file. So if the guest is given
- a disk backed by <code>/var/lib/libvirt/images/demo.img</code>
- expect to see a lease
<code>/var/lib/libvirt/sanlock/bfa0240911bc17753e0b473688822159</code>
- </p>
-
- <p>
- It should be obvious that for locking to work correctly, every
- host running virtual machines should have storage configured
- in the same way. The easiest way to do this is to use the libvirt
- storage pool capability to configure any NFS volumes, iSCSI targets,
- or SCSI HBAs used for guest storage. Simply replicate the same
- storage pool XML across every host. It is important that any
- storage pools exposing block devices are configured to create
- volume paths under <code>/dev/disks/by-path</code> to ensure
- stable paths across hosts. An example iSCSI configuration
- which ensures this is:
- </p>
-
- <pre>
-<pool type='iscsi'>
- <name>myiscsipool</name>
- <source>
- <host name='192.168.254.8'/>
- <device path='your-iscsi-target-iqn'/>
- </source>
- <target>
- <path>/dev/disk/by-path</path>
- </target>
-</pool>
- </pre>
-
- <h2><a name="domainconfig">Domain
configuration</a></h2>
-
- <p>
- In case sanlock loses access to disk locks for some reason, it will
- kill all domains that lost their locks. This default behavior may
- be changed using
- <a href="formatdomain.html#elementsEvents">on_lockfailure
- element</a> in domain XML. When this element is present, sanlock
- will call <code>sanlock_helper</code> (provided by libvirt) with
- the specified action. This helper binary will connect to libvirtd
- and thus it may need to authenticate if libvirtd was configured to
- require that on the read-write UNIX socket. To provide the
- appropriate credentials to sanlock_helper, a
- <a href="auth.html#Auth_client_config">client authentication
- file</a> needs to contain something like the following:
- </p>
- <pre>
-[auth-libvirt-localhost]
-credentials=sanlock
-
-[credentials-sanlock]
-authname=login
-password=password
- </pre>
+ The lock manager framework has a pluggable architecture,
+ to allow different locking technologies to be used.
+ </p>
+
+ <dl>
+ <dt>nop</dt>
+ <dd>This is a "no op" implementation which does absolutely
+ nothing. This can be used if mutual exclusion between
+ virtual machines is not required, or if it is being
+ solved at another level in the management stack.</dd>
+ <dt>lockd</dt>
+ <dd>This is the current preferred implementation shipped
+ with libvirt. It uses the <code>virtlockd</code> daemon
+ to manage locks using the POSIX fcntl() advisory locking
+ capability. As such it requires a shared filesystem of
+ some kind be accessible to all hosts which share the
+ same image storage.</dd>
+ <dt><a
href="locking-sanlock.html">sanlock</a></dt>
+ <dd>This is an alternative implementation preferred by
+ the oVirt project. It uses a disk paxos algorithm for
+ maintaining continuously renewed leases. In the default
+ setup it requires some shared filesystem, but it is
+ possible to use it in a manual mode where the management
+ application creates leases in SAN storage volumes.
+ </dd>
+ </dl>
</body>
</html>
diff --git a/docs/sitemap.html.in b/docs/sitemap.html.in
index fad5358..5b911ce 100644
--- a/docs/sitemap.html.in
+++ b/docs/sitemap.html.in
@@ -101,6 +101,12 @@
<li>
<a href="locking.html">Disk locking</a>
<span>Ensuring exclusive guest access to disks</span>
+ <ul>
+ <li>
+ <a href="locking-sanlock.html">Sanlock</a>
+ <span>Sanlock lock manager plugin</span>
+ </li>
+ </ul>
</li>
<li>
<a href="cgroups.html">CGroups</a>
--
2.1.0