[libvirt] [PATCH 0/1] qemu: Add Secure Shell (ssh) network block device.
This patch allows you to use the qemu Secure Shell (ssh) block device. This is not upstream yet, but you can find my latest version here: http://lists.nongnu.org/archive/html/qemu-devel/2013-04/threads.html#01703 This patch lets you specify a ssh device like this: <disk type='network' device='disk'> <source protocol='ssh' name='/remote/path/to/disk/image'> <host name='remote-server.example.com'/> </source> <driver name='qemu' type='raw'/> <target dev='vda' bus='virtio'/> </disk> Patched qemu will connect to remote-server.example.com using libssh2, and access /remote/path/to/disk/image using the sftp protocol. This works for both read and write. Of course, since you'll have to use a patched qemu, you will also need to fiddle with the <emulator> setting. One current problem with this patch is that you have to manually set the SSH_AUTH_SOCK environment variable to point at your ssh-agent (since qemu's ssh block device requires ssh-agent authentication). I added the following to my XML, your value will be different: <qemu:commandline> <qemu:env name="SSH_AUTH_SOCK" value="/tmp/ssh-DThteVfEeOq3/agent.1773" /> </qemu:commandline> Some shortcomings: - Does not allow you to specify the host_key_check parameter. - No tests. - Not sure how best to deal with the ssh-agent authentication socket problem. Use libvirt secrets? If so, how? - I did not test if you can specify an alternate remote user. - I did not test (or care) if parsing qemu command lines works. Rich.
From: "Richard W.M. Jones" <rjones@redhat.com> --- docs/formatdomain.html.in | 10 ++++++++-- docs/schemas/domaincommon.rng | 1 + src/conf/domain_conf.c | 3 ++- src/conf/domain_conf.h | 1 + src/qemu/qemu_command.c | 44 +++++++++++++++++++++++++++++++++++++++++-- src/util/virstoragefile.c | 5 +++-- 6 files changed, 57 insertions(+), 7 deletions(-) diff --git a/docs/formatdomain.html.in b/docs/formatdomain.html.in index d400e35..c86f023 100644 --- a/docs/formatdomain.html.in +++ b/docs/formatdomain.html.in @@ -1450,8 +1450,8 @@ to the directory to use as the disk. If the disk <code>type</code> is "network", then the <code>protocol</code> attribute specifies the protocol to access to the requested image; possible values - are "nbd", "iscsi", "rbd", "sheepdog" or "gluster". If the - <code>protocol</code> attribute is "rbd", "sheepdog" or "gluster", an + are "nbd", "iscsi", "rbd", "sheepdog", "gluster" or "ssh". If the + <code>protocol</code> attribute is "rbd", "sheepdog", "gluster" or "ssh", an additional attribute <code>name</code> is mandatory to specify which volume/image will be used; for "nbd" it is optional. For "iscsi", the <code>name</code> attribute may include a logical unit number, @@ -1746,6 +1746,12 @@ <td> only one </td> <td> 24007 </td> </tr> + <tr> + <td> ssh </td> + <td> the ssh server </td> + <td> only one </td> + <td> 22 </td> + </tr> </table> gluster supports "tcp", "rdma", "unix" as valid values for the transport attribute. nbd supports "tcp" and "unix". Others only diff --git a/docs/schemas/domaincommon.rng b/docs/schemas/domaincommon.rng index 2c31f76..c9e4e5b 100644 --- a/docs/schemas/domaincommon.rng +++ b/docs/schemas/domaincommon.rng @@ -1143,6 +1143,7 @@ <value>sheepdog</value> <value>gluster</value> <value>iscsi</value> + <value>ssh</value> </choice> </attribute> <optional> diff --git a/src/conf/domain_conf.c b/src/conf/domain_conf.c index e00a532..5ccaac2 100644 --- a/src/conf/domain_conf.c +++ b/src/conf/domain_conf.c @@ -255,7 +255,8 @@ VIR_ENUM_IMPL(virDomainDiskProtocol, VIR_DOMAIN_DISK_PROTOCOL_LAST, "rbd", "sheepdog", "gluster", - "iscsi") + "iscsi", + "ssh") VIR_ENUM_IMPL(virDomainDiskProtocolTransport, VIR_DOMAIN_DISK_PROTO_TRANS_LAST, "tcp", diff --git a/src/conf/domain_conf.h b/src/conf/domain_conf.h index 08b8e48..7e3e162 100644 --- a/src/conf/domain_conf.h +++ b/src/conf/domain_conf.h @@ -506,6 +506,7 @@ enum virDomainDiskProtocol { VIR_DOMAIN_DISK_PROTOCOL_SHEEPDOG, VIR_DOMAIN_DISK_PROTOCOL_GLUSTER, VIR_DOMAIN_DISK_PROTOCOL_ISCSI, + VIR_DOMAIN_DISK_PROTOCOL_SSH, VIR_DOMAIN_DISK_PROTOCOL_LAST }; diff --git a/src/qemu/qemu_command.c b/src/qemu/qemu_command.c index 41a2dfd..4a5d0bf 100644 --- a/src/qemu/qemu_command.c +++ b/src/qemu/qemu_command.c @@ -2500,6 +2500,18 @@ error: } static int +qemuParseSSHString(virDomainDiskDefPtr disk) +{ + virURIPtr uri; + + uri = virURIParse(disk->src); + if (!uri) + return -1; + + return qemuParseDriveURIString(disk, uri, "ssh"); +} + +static int qemuBuildDriveURIString(virConnectPtr conn, virDomainDiskDefPtr disk, virBufferPtr opt, const char *scheme, virSecretUsageType secretType) @@ -2617,8 +2629,6 @@ qemuBuildGlusterString(virConnectPtr conn, virDomainDiskDefPtr disk, virBufferPt VIR_SECRET_USAGE_TYPE_NONE); } -#define QEMU_DEFAULT_NBD_PORT "10809" - static int qemuBuildISCSIString(virConnectPtr conn, virDomainDiskDefPtr disk, virBufferPtr opt) { @@ -2627,6 +2637,16 @@ qemuBuildISCSIString(virConnectPtr conn, virDomainDiskDefPtr disk, virBufferPtr } static int +qemuBuildSSHString(virConnectPtr conn, virDomainDiskDefPtr disk, + virBufferPtr opt) +{ + return qemuBuildDriveURIString(conn, disk, opt, "ssh", + VIR_SECRET_USAGE_TYPE_NONE); +} + +#define QEMU_DEFAULT_NBD_PORT "10809" + +static int qemuBuildNBDString(virConnectPtr conn, virDomainDiskDefPtr disk, virBufferPtr opt) { const char *transp; @@ -2828,6 +2848,12 @@ qemuBuildDriveStr(virConnectPtr conn ATTRIBUTE_UNUSED, virBufferEscape(&opt, ',', ",", "%s,", disk->src); } break; + + case VIR_DOMAIN_DISK_PROTOCOL_SSH: + if (qemuBuildSSHString(conn, disk, &opt) < 0) + goto error; + virBufferAddChar(&opt, ','); + break; } } else if (disk->type == VIR_DOMAIN_DISK_TYPE_VOLUME) { switch (disk->srcpool->voltype) { @@ -8120,6 +8146,13 @@ qemuParseCommandLineDisk(virDomainXMLOptionPtr xmlopt, } VIR_FREE(p); + } else if (STRPREFIX(def->src, "ssh:")) { + def->type = VIR_DOMAIN_DISK_TYPE_NETWORK; + def->protocol = VIR_DOMAIN_DISK_PROTOCOL_SSH; + + if (qemuParseSSHString(def) < 0) + goto error; + } else def->type = VIR_DOMAIN_DISK_TYPE_FILE; } else { @@ -9296,6 +9329,9 @@ virDomainDefPtr qemuParseCommandLine(virCapsPtr qemuCaps, disk->type = VIR_DOMAIN_DISK_TYPE_NETWORK; disk->protocol = VIR_DOMAIN_DISK_PROTOCOL_SHEEPDOG; val += strlen("sheepdog:"); + } else if (STRPREFIX(val, "ssh:")) { + disk->type = VIR_DOMAIN_DISK_TYPE_NETWORK; + disk->protocol = VIR_DOMAIN_DISK_PROTOCOL_SSH; } else disk->type = VIR_DOMAIN_DISK_TYPE_FILE; if (STREQ(arg, "-cdrom")) { @@ -9372,6 +9408,10 @@ virDomainDefPtr qemuParseCommandLine(virCapsPtr qemuCaps, goto error; break; + case VIR_DOMAIN_DISK_PROTOCOL_SSH: + if (qemuParseSSHString(disk) < 0) + goto error; + break; } } diff --git a/src/util/virstoragefile.c b/src/util/virstoragefile.c index aabb5c8..2c701c3 100644 --- a/src/util/virstoragefile.c +++ b/src/util/virstoragefile.c @@ -622,8 +622,9 @@ virStorageFileMatchesVersion(int format, static bool virBackingStoreIsFile(const char *backing) { - /* Backing store is a network block device or Rados block device */ - if (STRPREFIX(backing, "nbd:") || STRPREFIX(backing, "rbd:")) + /* Backing store is a network block device, Rados block device, or ssh */ + if (STRPREFIX(backing, "nbd:") || STRPREFIX(backing, "rbd:") || + STRPREFIX(backing, "ssh:")) return false; return true; } -- 1.8.1.4
On Wed, Apr 10, 2013 at 03:09:18PM +0100, Richard W.M. Jones wrote:
This patch allows you to use the qemu Secure Shell (ssh) block device. This is not upstream yet, but you can find my latest version here:
http://lists.nongnu.org/archive/html/qemu-devel/2013-04/threads.html#01703
This patch lets you specify a ssh device like this:
<disk type='network' device='disk'> <source protocol='ssh' name='/remote/path/to/disk/image'> <host name='remote-server.example.com'/> </source>
Ok, looks sensible.
<driver name='qemu' type='raw'/> <target dev='vda' bus='virtio'/> </disk>
Patched qemu will connect to remote-server.example.com using libssh2, and access /remote/path/to/disk/image using the sftp protocol. This works for both read and write.
Of course, since you'll have to use a patched qemu, you will also need to fiddle with the <emulator> setting.
One current problem with this patch is that you have to manually set the SSH_AUTH_SOCK environment variable to point at your ssh-agent (since qemu's ssh block device requires ssh-agent authentication). I added the following to my XML, your value will be different:
<qemu:commandline> <qemu:env name="SSH_AUTH_SOCK" value="/tmp/ssh-DThteVfEeOq3/agent.1773" /> </qemu:commandline>
Hmm, yes, that's a big problem, particularly from an sVirt POV. You certainly do not want a compromised QEMU to be able to use the SSH agent to obtain your keys for logging into arbitrary remote systems. So exposing the SSH agent to QEMU directly seems like a no-go.
Some shortcomings:
- Does not allow you to specify the host_key_check parameter.
- No tests.
- Not sure how best to deal with the ssh-agent authentication socket problem. Use libvirt secrets? If so, how?
The way that works, is that the application creating the guest has to pre-populate secrets with keys/passwords/whatever. The guest XML refers to which secrets it needs, and at guest startup Libvirt would push the secrets into QEMU. IIUC, the SSH agent protocol is fundamentally designed around the idea of "application pull", where as we want an approach which I describe as "manager push". Using secrets for passing authentication information in for disks is the right thing todo - we already do this for RBD for example. What is missing is a way to securely pass this into QEMU - for RBD we currently pass this on the command line. The guys who added RBD support to libvirt were supposed to be adding ability to pass the passwords via the monitor but that never seems to have happened. I wish we'd never allowed the patches for RBD for passing stuff on the command line, still it removed the motivation for them todo a better job. Anyway for SSH, I think we need to have this bit of QEMU finished, so we can pass in authentication credentials via the monitor. There is an existing 'block_password' command, but that was used for the decryption keys and I don't think it is wise to overload that for authentication keys too, since it is in theory possible to access an encrypted QCow2 image over the SSH protocol, in which case we would need both decryption keys and authetnication keys at the same time. The only other alternative would be to actally make libvirtd implement an SSH agent service, exposing per-VM UNIX to the QEMU process(es) which needed SSH keys. I don't much like this though, since it would only be useful to the SSH disk protocol, and not help address the problems we already face for RBD and other network block devices.
- I did not test if you can specify an alternate remote user.
- I did not test (or care) if parsing qemu command lines works.
Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On Mon, Apr 15, 2013 at 11:11:10AM +0100, Daniel P. Berrange wrote:
On Wed, Apr 10, 2013 at 03:09:18PM +0100, Richard W.M. Jones wrote:
- Not sure how best to deal with the ssh-agent authentication socket problem. Use libvirt secrets? If so, how?
The way that works, is that the application creating the guest has to pre-populate secrets with keys/passwords/whatever. The guest XML refers to which secrets it needs, and at guest startup Libvirt would push the secrets into QEMU.
IIUC, the SSH agent protocol is fundamentally designed around the idea of "application pull", where as we want an approach which I describe as "manager push".
Using secrets for passing authentication information in for disks is the right thing todo - we already do this for RBD for example. What is missing is a way to securely pass this into QEMU - for RBD we currently pass this on the command line. The guys who added RBD support to libvirt were supposed to be adding ability to pass the passwords via the monitor but that never seems to have happened. I wish we'd never allowed the patches for RBD for passing stuff on the command line, still it removed the motivation for them todo a better job.
Anyway for SSH, I think we need to have this bit of QEMU finished, so we can pass in authentication credentials via the monitor. There is an existing 'block_password' command, but that was used for the decryption keys and I don't think it is wise to overload that for authentication keys too, since it is in theory possible to access an encrypted QCow2 image over the SSH protocol, in which case we would need both decryption keys and authetnication keys at the same time.
The only other alternative would be to actally make libvirtd implement an SSH agent service, exposing per-VM UNIX to the QEMU process(es) which needed SSH keys. I don't much like this though, since it would only be useful to the SSH disk protocol, and not help address the problems we already face for RBD and other network block devices.
Would this be any easier if we implemented password authentication (in qemu's ssh block driver). Then the secret stored by libvirt would be the password, which it could pass to qemu. There is still the question of overloading the block_password monitor command, but that problem presumably applies to other drivers as well. In fact, you seem to have summarised the solution (I assume this was never implemented) back in 2011: http://lists.gnu.org/archive/html/qemu-devel/2011-10/msg02495.html Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top
On Mon, Apr 15, 2013 at 11:18:26AM +0100, Richard W.M. Jones wrote:
On Mon, Apr 15, 2013 at 11:11:10AM +0100, Daniel P. Berrange wrote:
On Wed, Apr 10, 2013 at 03:09:18PM +0100, Richard W.M. Jones wrote:
- Not sure how best to deal with the ssh-agent authentication socket problem. Use libvirt secrets? If so, how?
The way that works, is that the application creating the guest has to pre-populate secrets with keys/passwords/whatever. The guest XML refers to which secrets it needs, and at guest startup Libvirt would push the secrets into QEMU.
IIUC, the SSH agent protocol is fundamentally designed around the idea of "application pull", where as we want an approach which I describe as "manager push".
Using secrets for passing authentication information in for disks is the right thing todo - we already do this for RBD for example. What is missing is a way to securely pass this into QEMU - for RBD we currently pass this on the command line. The guys who added RBD support to libvirt were supposed to be adding ability to pass the passwords via the monitor but that never seems to have happened. I wish we'd never allowed the patches for RBD for passing stuff on the command line, still it removed the motivation for them todo a better job.
Anyway for SSH, I think we need to have this bit of QEMU finished, so we can pass in authentication credentials via the monitor. There is an existing 'block_password' command, but that was used for the decryption keys and I don't think it is wise to overload that for authentication keys too, since it is in theory possible to access an encrypted QCow2 image over the SSH protocol, in which case we would need both decryption keys and authetnication keys at the same time.
The only other alternative would be to actally make libvirtd implement an SSH agent service, exposing per-VM UNIX to the QEMU process(es) which needed SSH keys. I don't much like this though, since it would only be useful to the SSH disk protocol, and not help address the problems we already face for RBD and other network block devices.
Would this be any easier if we implemented password authentication (in qemu's ssh block driver). Then the secret stored by libvirt would be the password, which it could pass to qemu.
Yep, that would make it easier. Also if you did SSH key auth, but allowed passphrases to be passed in, instead of pulled from an agent (in same way SSH does if no agent is running).
There is still the question of overloading the block_password monitor command, but that problem presumably applies to other drivers as well. In fact, you seem to have summarised the solution (I assume this was never implemented) back in 2011:
http://lists.gnu.org/archive/html/qemu-devel/2011-10/msg02495.html
Yep, not implemented afaik. Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On Mon, Apr 15, 2013 at 11:31:08AM +0100, Daniel P. Berrange wrote:
Yep, that would make it easier. Also if you did SSH key auth, but allowed passphrases to be passed in, instead of pulled from an agent (in same way SSH does if no agent is running).
Because qemu is running as a different user (qemu.qemu) it most likely won't have access to $HOME/.ssh/id_rsa, even assuming it knew which $HOME to go to. For ssh key auth, it would be helpful if both the raw key file contents and the passphrase could be stored as libvirt secrets. Is that possible? If so, qemu can pass both to libssh2_userauth_publickey. Almost as in this example: http://libssh2.org/examples/ssh2.html , combined with looking at how libssh2_userauth_publickey_fromfile is implemented: http://git.libssh2.org/?p=libssh2.git;a=blob;f=src/userauth.c;h=a0733d5da05f... I'm also a bit concerned that the solution should be usable for ordinary users. qemu -drive file=ssh://... currently Just Works. ---- Next questions: - How should host_key_check be modelled via the libvirt XML / API? - We want the user to be able to select different authentication methods (at least, password, publickey, agent [insecurely]). How would you see these being modelled in the API? Particularly since these may require associated secret(s). Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming blog: http://rwmj.wordpress.com Fedora now supports 80 OCaml packages (the OPEN alternative to F#)
On Mon, Apr 15, 2013 at 12:18:43PM +0100, Richard W.M. Jones wrote:
On Mon, Apr 15, 2013 at 11:31:08AM +0100, Daniel P. Berrange wrote:
Yep, that would make it easier. Also if you did SSH key auth, but allowed passphrases to be passed in, instead of pulled from an agent (in same way SSH does if no agent is running).
Because qemu is running as a different user (qemu.qemu) it most likely won't have access to $HOME/.ssh/id_rsa, even assuming it knew which $HOME to go to.
True, and actally we'd be blocking access to that file with sVirt anyway.
For ssh key auth, it would be helpful if both the raw key file contents and the passphrase could be stored as libvirt secrets. Is that possible?
That would imply multiple secrets, but yes, it should be possible.
If so, qemu can pass both to libssh2_userauth_publickey. Almost as in this example: http://libssh2.org/examples/ssh2.html , combined with looking at how libssh2_userauth_publickey_fromfile is implemented:
http://git.libssh2.org/?p=libssh2.git;a=blob;f=src/userauth.c;h=a0733d5da05f...
I'm also a bit concerned that the solution should be usable for ordinary users. qemu -drive file=ssh://... currently Just Works.
If you want "just works" then you don't really want to have the separate "load credentials into libvirt" step that using the secrets APIs implies. This pretty much says you need to let it use the SSH agent. You don't want to be including the SSH agent socket path in the XML though, since that is randomly generated on each login. In the session libvirtd instance, libvirtd iself will already see the SSH agent socket, so could just whitelist the SSH_AUTH_SOCK env variable when starting QEMU. For system libvirtd, I think connecting to the user's SSH agent is a dead end, because we can't assume the 'qemu' user has access to the SSH agent socket - beyond file permissions, /tmp can be in a separate namespace, so qemu can't even assume to see the same /tmp as the user does.
- How should host_key_check be modelled via the libvirt XML / API?
I'm not sure - what does that do ?
- We want the user to be able to select different authentication methods (at least, password, publickey, agent [insecurely]). How would you see these being modelled in the API? Particularly since these may require associated secret(s).
Well for password/publickey the existance of the neccessary secrets in the XML will indicate whether each of them can be enabled. Not sure what to suggest for the ssh agent though. Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On Mon, Apr 15, 2013 at 12:39:30PM +0100, Daniel P. Berrange wrote:
On Mon, Apr 15, 2013 at 12:18:43PM +0100, Richard W.M. Jones wrote:
- How should host_key_check be modelled via the libvirt XML / API?
I'm not sure - what does that do ?
Three settings: (1) host_key_check=no turns off known_hosts checking (2) host_key_check=yes turns it on (the default). In this case, qemu will check the remote host's key against the contents of $HOME/.ssh/known_hosts or /root/.ssh/known_hosts if $HOME is not set. In the libvirt case this is of course troublesome because $HOME is not set and $HOME/.ssh is not likely to be accessible. Although, this worked for me in my tests, so I assume libvirtd now passes $HOME through in the session case? (3) host_key_check={md5,sha1}:<hash> Check that hash(host key) = <hash>, where the hash function is md5 or sha1. In practice you'd write something like: ssh://remote/disk.img?host_key_check=md5:78:45:8e:14:57:4f:d5:45:83:0a:0e:f3:49:82:c9:c8 where '78:45:8e:14:57:4f:d5:45:83:0a:0e:f3:49:82:c9:c8' is the output of: ssh-keygen -l -f /etc/ssh/ssh_host_rsa_key.pub | awk '{print $2}' on the remote host. In future there could be a fourth case: (4) host_key_check=key:<key> where you actually pass the whole remote host public key (base64- encoded). The problem with this is it runs into URL length limits quite easily.
- We want the user to be able to select different authentication methods (at least, password, publickey, agent [insecurely]). How would you see these being modelled in the API? Particularly since these may require associated secret(s).
Well for password/publickey the existance of the neccessary secrets in the XML will indicate whether each of them can be enabled. Not sure what to suggest for the ssh agent though.
I suspect we do need to specify the alternatives, and there may be a sequence to be tried in turn. Accidentally sending a passphrase as password-interactive, for example, could be dangerous. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW
participants (2)
-
Daniel P. Berrange -
Richard W.M. Jones