Network disks and replacing qemu-block-curl|ssh with nbdkit
As mentioned in https://bugzilla.redhat.com/show_bug.cgi?id=2016527, RHEL is planning to remove dependencies on the qemu-block-curl and qemu-block-ssh plugins from the main qemu package. This creates issues for libvirt for supporting network disk sources. So I've been looking into using nbdkit from libvirt to proxy these network disks to qemu as NBD disks. The basic idea is that libvirt will spin up an nbdkit instance for e.g. an https network disk source, and will provide the resulting unix socket to qemu as an nbd disk. This allows libvirt to continue supporting http/ftp/ssh disk sources regardless of whether the qemu block plugins are installed or not. However, there are a couple of issues and feature gaps that I've run into that I'd like to discuss. 1. secrets There is some code in libvirt[1] which seems to expect that it is possible for http(s) disk sources to have a username and password specified. However, I can't find any valid xml schema for specifying an http username and password, and my reading of the code suggests that there shouldn't be any way for these to be set for http(s)/ftp(s) disk sources either since auth is only supported for ISCSI and RBD protocols [2]. Am I missing something? [1] https://gitlab.com/libvirt/libvirt/-/blob/6be7beb3bdb9ad611a5598dad7edfbd2a8... [2] https://gitlab.com/libvirt/libvirt/-/blob/6be7beb3bdb9ad611a5598dad7edfbd2a8... If it *is* possible for the username/password to be set for these disks, then we have the issue that these sensitive pieces of data have so far been passed as encrypted data to qemu using qemu secrets. But if nbdkit is handling the http requests, we need to pass this data to nbdkit rather than qemu and so we can no longer use qemu secrets. The same issue applies to http cookies, which could potentially include security-sensitive data such as login credentials, etc. Fortunately, nbdkit provides a method for reading cookies and passwords from a file, which should be secure if the file has permissions set properly. So I'm currently planning to write a file containing the cookies and pass them to nbdkit by specifying the filename. But I'm still confused about the username/password possibility. 2. readahead The libvirt xml format allows to specify a readahead size for disks that are handled by the qemu-block-curl plugin. Unfortunately, nbdkit doesn't currently support any readahead configuration. In nbdkit, readahead is handled by an nbkit "filter" that takes no configuration options (`nbdkit --filter=readahead ...`). In theory, this filter tries to adaptively read ahead. But when I discussed it with Rich, he suggested that he had stopped using it in virt-v2v because it was causing more trouble than it was worth. He also suggested that this readahead filter might need a complete rewrite, and presumably the rewrite could include the ability to configure a readahead buffer size. But I'm not sure what the timeframe might be for that. 3. blockdev-create There is support in libvirt[3] for creating ssh network disks by sending a 'blockdev-create' command to qemu. If qemu is no longer handling ssh network disk sources directly, this feature becomes significantly more complicated. I don't yet know enough about this part of the libvirt code to know what further complications might pop up here. From my reading of the code, this is mostly triggered by things like `virsh blockcopy` `virsh backup-begin`, etc. But nbdkit cannot currently do this. Rich pointed me to a recent commit[4] where he added disk creation to the nbdkit vddk plugin, and suggested that something similar could be added for the nbdkit ssh plugin. [3] https://gitlab.com/libvirt/libvirt/-/blob/6be7beb3bdb9ad611a5598dad7edfbd2a8... [4] https://gitlab.com/nbdkit/nbdkit/-/commit/a39d5773afc3ebab7e5768118a2bccb89a... It seems to me that it's essential that we resolve #3 before we can move forward with nbdkit support in libvirt. (Although I admit that I have no idea how common it is for people to use ssh disks so I suppose there's a slim possibility that we could just disable the 'create disk' feature for ssh disks without any practical loss of functionality?) But it's less obvious to me whether we could move ahead despite missing readahead size configuration, etc. Thoughts? Jonathon
On Thu, Apr 14, 2022 at 05:02:46PM -0500, Jonathon Jongsma wrote:
1. secrets [...] Fortunately, nbdkit provides a method for reading cookies and passwords from a file, which should be secure if the file has permissions set properly. So I'm currently planning to write a file containing the cookies and pass them to nbdkit by specifying the filename. But I'm still confused about the username/password possibility.
You can also send the password or cookie over an inherited file descriptor, which has the possible advantage that the secret will never hit the disk at all. For completeness I should say that we found HTTP authentication against some servers to be quite slow (presumably because validating a password involves a lot of machinery so doing it on every request is slow). For those servers we implemented a complicated scheme where you could make an authenticated request, fetch the cookie that the server sends back, send back the cookie, _and_ autorenew the cookie if it times out. (Did I say this was complicated?) This is required for at least VMware servers and Docker registries. https://libguestfs.org/nbdkit-curl-plugin.1.html#HEADER-AND-COOKIE-SCRIPTS I wouldn't try implementing this through libvirt ...
2. readahead
3. blockdev-create
See also: https://listman.redhat.com/archives/libguestfs/2022-April/028674.html I agree we should implement creation for ssh disks (not sure if it's possible or even makes sense for curl). Shouldn't be too difficult. You might also want to think about VDDK disk support, ie. is it possible to make the nbdkit stuff generic enough that nbdkit-vddk-plugin can be slotted in later? It would allow a libvirt domain to be backed with remote disks stored on VMFS, or to use VMware's own proprietary drivers to open local VMDK files, both significant enhancements. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com nbdkit - Flexible, fast NBD server with plugins https://gitlab.com/nbdkit/nbdkit
On Fri, Apr 15, 2022 at 10:09:59AM +0100, Richard W.M. Jones wrote:
I agree we should implement creation for ssh disks (not sure if it's possible or even makes sense for curl). Shouldn't be too difficult.
https://listman.redhat.com/archives/libguestfs/2022-April/028680.html Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW
On Thu, Apr 14, 2022 at 05:02:46PM -0500, Jonathon Jongsma wrote:
As mentioned in https://bugzilla.redhat.com/show_bug.cgi?id=2016527, RHEL is planning to remove dependencies on the qemu-block-curl and qemu-block-ssh plugins from the main qemu package. This creates issues for libvirt for supporting network disk sources. So I've been looking into using nbdkit from libvirt to proxy these network disks to qemu as NBD disks.
The basic idea is that libvirt will spin up an nbdkit instance for e.g. an https network disk source, and will provide the resulting unix socket to qemu as an nbd disk. This allows libvirt to continue supporting http/ftp/ssh disk sources regardless of whether the qemu block plugins are installed or not.
However, there are a couple of issues and feature gaps that I've run into that I'd like to discuss.
1. secrets
There is some code in libvirt[1] which seems to expect that it is possible for http(s) disk sources to have a username and password specified. However, I can't find any valid xml schema for specifying an http username and password, and my reading of the code suggests that there shouldn't be any way for these to be set for http(s)/ftp(s) [disk sources either since auth is only supported for ISCSI and RBD protocols [2]. Am I missing something?
[1] https://gitlab.com/libvirt/libvirt/-/blob/6be7beb3bdb9ad611a5598dad7edfbd2a8...
[2] https://gitlab.com/libvirt/libvirt/-/blob/6be7beb3bdb9ad611a5598dad7edfbd2a8...
I'm unclear if this is a regression, or a long term bug. Clearly the intention of the code is to support auth, so if it doesn't work right now we need to fix that, regardless of adding the nbdkit support. I do notice we don't have any XML in tests/qemuxml2argvdata/ that exercises auth with https, which is likely why we have this bug lurking.
If it *is* possible for the username/password to be set for these disks, then we have the issue that these sensitive pieces of data have so far been passed as encrypted data to qemu using qemu secrets. But if nbdkit is handling the http requests, we need to pass this data to nbdkit rather than qemu and so we can no longer use qemu secrets. The same issue applies to http cookies, which could potentially include security-sensitive data such as login credentials, etc.
Fortunately, nbdkit provides a method for reading cookies and passwords from a file, which should be secure if the file has permissions set properly. So I'm currently planning to write a file containing the cookies and pass them to nbdkit by specifying the filename. But I'm still confused about the username/password possibility.
2. readahead
The libvirt xml format allows to specify a readahead size for disks that are handled by the qemu-block-curl plugin. Unfortunately, nbdkit doesn't currently support any readahead configuration. In nbdkit, readahead is handled by an nbkit "filter" that takes no configuration options (`nbdkit --filter=readahead ...`). In theory, this filter tries to adaptively read ahead. But when I discussed it with Rich, he suggested that he had stopped using it in virt-v2v because it was causing more trouble than it was worth. He also suggested that this readahead filter might need a complete rewrite, and presumably the rewrite could include the ability to configure a readahead buffer size. But I'm not sure what the timeframe might be for that.
In terms of libvirt upstream policy, there is no requirement for a given hypervisor implementation to support all the features possible in the XML. If readahead isn't supported then just report VIR_ERR_CONFIG_UNSUPPORTED if the user provides it in the XML. If there is a choice of backends though, we should prefer the impl that avoids regressions. IOW, if both qemu-block-curl and nbdkit are installed, we must prefer qemu-block-curl. It is upto RHEL downstream if they are happy to ship only nbdkit given the feature differences.
3. blockdev-create
There is support in libvirt[3] for creating ssh network disks by sending a 'blockdev-create' command to qemu. If qemu is no longer handling ssh network disk sources directly, this feature becomes significantly more complicated. I don't yet know enough about this part of the libvirt code to know what further complications might pop up here. From my reading of the code, this is mostly triggered by things like `virsh blockcopy` `virsh backup-begin`, etc. But nbdkit cannot currently do this. Rich pointed me to a recent commit[4] where he added disk creation to the nbdkit vddk plugin, and suggested that something similar could be added for the nbdkit ssh plugin.
[3] https://gitlab.com/libvirt/libvirt/-/blob/6be7beb3bdb9ad611a5598dad7edfbd2a8...
[4] https://gitlab.com/nbdkit/nbdkit/-/commit/a39d5773afc3ebab7e5768118a2bccb89a...
It seems to me that it's essential that we resolve #3 before we can move forward with nbdkit support in libvirt. (Although I admit that I have no idea how common it is for people to use ssh disks so I suppose there's a slim possibility that we could just disable the 'create disk' feature for ssh disks without any practical loss of functionality?) But it's less obvious to me whether we could move ahead despite missing readahead size configuration, etc.
Again, from an upstream POV #3 isn't a blocker, as long as we prefer qemu-block-curl when available. It is upto RHEL to decide whether #3 is a blocker for their downstream dropping of qemu-block-curl. With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On 4/19/22 9:05 AM, Daniel P. Berrangé wrote:
On Thu, Apr 14, 2022 at 05:02:46PM -0500, Jonathon Jongsma wrote:
As mentioned in https://bugzilla.redhat.com/show_bug.cgi?id=2016527, RHEL is planning to remove dependencies on the qemu-block-curl and qemu-block-ssh plugins from the main qemu package. This creates issues for libvirt for supporting network disk sources. So I've been looking into using nbdkit from libvirt to proxy these network disks to qemu as NBD disks.
The basic idea is that libvirt will spin up an nbdkit instance for e.g. an https network disk source, and will provide the resulting unix socket to qemu as an nbd disk. This allows libvirt to continue supporting http/ftp/ssh disk sources regardless of whether the qemu block plugins are installed or not.
However, there are a couple of issues and feature gaps that I've run into that I'd like to discuss.
1. secrets
There is some code in libvirt[1] which seems to expect that it is possible for http(s) disk sources to have a username and password specified. However, I can't find any valid xml schema for specifying an http username and password, and my reading of the code suggests that there shouldn't be any way for these to be set for http(s)/ftp(s) [disk sources either since auth is only supported for ISCSI and RBD protocols [2]. Am I missing something?
[1] https://gitlab.com/libvirt/libvirt/-/blob/6be7beb3bdb9ad611a5598dad7edfbd2a8...
[2] https://gitlab.com/libvirt/libvirt/-/blob/6be7beb3bdb9ad611a5598dad7edfbd2a8...
I'm unclear if this is a regression, or a long term bug. Clearly the intention of the code is to support auth, so if it doesn't work right now we need to fix that, regardless of adding the nbdkit support.
I do notice we don't have any XML in tests/qemuxml2argvdata/ that exercises auth with https, which is likely why we have this bug lurking.
Well, As far as I can tell, there is no valid XML for exercising http auth. The schema for http(s) sources does not include any <auth> element [1]. And the schema for the <auth> element [2] requires a <secret> element with a required 'type' attribute, and the only choices for 'type' are 'ceph' or 'iscsi', neither of which apply to http authentication. So it looks to me like http auth was never supported, rather than a regression. It also seems that it would require some additional schema changes to support this. [1] https://gitlab.com/libvirt/libvirt/-/blob/136b821f183deb0b58c571211f6917985b... [2]https://gitlab.com/libvirt/libvirt/-/blob/136b821f183deb0b58c571211f6917985b...
If it *is* possible for the username/password to be set for these disks, then we have the issue that these sensitive pieces of data have so far been passed as encrypted data to qemu using qemu secrets. But if nbdkit is handling the http requests, we need to pass this data to nbdkit rather than qemu and so we can no longer use qemu secrets. The same issue applies to http cookies, which could potentially include security-sensitive data such as login credentials, etc.
Fortunately, nbdkit provides a method for reading cookies and passwords from a file, which should be secure if the file has permissions set properly. So I'm currently planning to write a file containing the cookies and pass them to nbdkit by specifying the filename. But I'm still confused about the username/password possibility.
2. readahead
The libvirt xml format allows to specify a readahead size for disks that are handled by the qemu-block-curl plugin. Unfortunately, nbdkit doesn't currently support any readahead configuration. In nbdkit, readahead is handled by an nbkit "filter" that takes no configuration options (`nbdkit --filter=readahead ...`). In theory, this filter tries to adaptively read ahead. But when I discussed it with Rich, he suggested that he had stopped using it in virt-v2v because it was causing more trouble than it was worth. He also suggested that this readahead filter might need a complete rewrite, and presumably the rewrite could include the ability to configure a readahead buffer size. But I'm not sure what the timeframe might be for that.
In terms of libvirt upstream policy, there is no requirement for a given hypervisor implementation to support all the features possible in the XML. If readahead isn't supported then just report VIR_ERR_CONFIG_UNSUPPORTED if the user provides it in the XML.
If there is a choice of backends though, we should prefer the impl that avoids regressions.
IOW, if both qemu-block-curl and nbdkit are installed, we must prefer qemu-block-curl.
It is upto RHEL downstream if they are happy to ship only nbdkit given the feature differences.
3. blockdev-create
There is support in libvirt[3] for creating ssh network disks by sending a 'blockdev-create' command to qemu. If qemu is no longer handling ssh network disk sources directly, this feature becomes significantly more complicated. I don't yet know enough about this part of the libvirt code to know what further complications might pop up here. From my reading of the code, this is mostly triggered by things like `virsh blockcopy` `virsh backup-begin`, etc. But nbdkit cannot currently do this. Rich pointed me to a recent commit[4] where he added disk creation to the nbdkit vddk plugin, and suggested that something similar could be added for the nbdkit ssh plugin.
[3] https://gitlab.com/libvirt/libvirt/-/blob/6be7beb3bdb9ad611a5598dad7edfbd2a8...
[4] https://gitlab.com/nbdkit/nbdkit/-/commit/a39d5773afc3ebab7e5768118a2bccb89a...
It seems to me that it's essential that we resolve #3 before we can move forward with nbdkit support in libvirt. (Although I admit that I have no idea how common it is for people to use ssh disks so I suppose there's a slim possibility that we could just disable the 'create disk' feature for ssh disks without any practical loss of functionality?) But it's less obvious to me whether we could move ahead despite missing readahead size configuration, etc.
Again, from an upstream POV #3 isn't a blocker, as long as we prefer qemu-block-curl when available. It is upto RHEL to decide whether #3 is a blocker for their downstream dropping of qemu-block-curl.
And now I notice that we do not actually have support for 'ssh' network disks in our xml schema either [3]. The VIR_STORAGE_NET_PROTOCOL_SSH enum value was originally added with a comment that it allows using the ssh protocol in backing chains. I've also seen some commits indicating that some of the support for ssh disk sources may be related to libguestfs usage in some way. cc'ing Rich and Peter in case they can add any historical context here. [3] https://gitlab.com/libvirt/libvirt/-/blob/136b821f183deb0b58c571211f6917985b...
On Tue, Apr 19, 2022 at 11:40:49AM -0500, Jonathon Jongsma wrote:
On 4/19/22 9:05 AM, Daniel P. Berrangé wrote:
On Thu, Apr 14, 2022 at 05:02:46PM -0500, Jonathon Jongsma wrote:
As mentioned in https://bugzilla.redhat.com/show_bug.cgi?id=2016527, RHEL is planning to remove dependencies on the qemu-block-curl and qemu-block-ssh plugins from the main qemu package. This creates issues for libvirt for supporting network disk sources. So I've been looking into using nbdkit from libvirt to proxy these network disks to qemu as NBD disks.
The basic idea is that libvirt will spin up an nbdkit instance for e.g. an https network disk source, and will provide the resulting unix socket to qemu as an nbd disk. This allows libvirt to continue supporting http/ftp/ssh disk sources regardless of whether the qemu block plugins are installed or not.
However, there are a couple of issues and feature gaps that I've run into that I'd like to discuss.
1. secrets
There is some code in libvirt[1] which seems to expect that it is possible for http(s) disk sources to have a username and password specified. However, I can't find any valid xml schema for specifying an http username and password, and my reading of the code suggests that there shouldn't be any way for these to be set for http(s)/ftp(s) [disk sources either since auth is only supported for ISCSI and RBD protocols [2]. Am I missing something?
[1] https://gitlab.com/libvirt/libvirt/-/blob/6be7beb3bdb9ad611a5598dad7edfbd2a8...
[2] https://gitlab.com/libvirt/libvirt/-/blob/6be7beb3bdb9ad611a5598dad7edfbd2a8...
I'm unclear if this is a regression, or a long term bug. Clearly the intention of the code is to support auth, so if it doesn't work right now we need to fix that, regardless of adding the nbdkit support.
I do notice we don't have any XML in tests/qemuxml2argvdata/ that exercises auth with https, which is likely why we have this bug lurking.
Well, As far as I can tell, there is no valid XML for exercising http auth. The schema for http(s) sources does not include any <auth> element [1]. And the schema for the <auth> element [2] requires a <secret> element with a required 'type' attribute, and the only choices for 'type' are 'ceph' or 'iscsi', neither of which apply to http authentication.
So it looks to me like http auth was never supported, rather than a regression. It also seems that it would require some additional schema changes to support this.
Note the schema is not considered the source of truth, the actual XML parsing code is law. It wouldn't be the first time we had schema bugs. Usually such bugs are found when we add XML examples to tests/qemuxml2argvdata, since we validate all those test files against the schema. Since we're lacking test files, it is unsurprising if the schema is also broken. I've not actually checked whether the XML parsing code support this or not, but that's what matters most of all.
Again, from an upstream POV #3 isn't a blocker, as long as we prefer qemu-block-curl when available. It is upto RHEL to decide whether #3 is a blocker for their downstream dropping of qemu-block-curl.
And now I notice that we do not actually have support for 'ssh' network disks in our xml schema either [3]. The VIR_STORAGE_NET_PROTOCOL_SSH enum value was originally added with a comment that it allows using the ssh protocol in backing chains. I've also seen some commits indicating that some of the support for ssh disk sources may be related to libguestfs usage in some way. cc'ing Rich and Peter in case they can add any historical context here.
Again, thue XML parsing code is law, the schema can be buggy if we don't have enough example test XML files With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Tue, Apr 19, 2022 at 11:40:49AM -0500, Jonathon Jongsma wrote:
Well, As far as I can tell, there is no valid XML for exercising http auth. The schema for http(s) sources does not include any <auth> element [1]. And the schema for the <auth> element [2] requires a <secret> element with a required 'type' attribute, and the only choices for 'type' are 'ceph' or 'iscsi', neither of which apply to http authentication.
So it looks to me like http auth was never supported, rather than a regression. It also seems that it would require some additional schema changes to support this.
[1] https://gitlab.com/libvirt/libvirt/-/blob/136b821f183deb0b58c571211f6917985b...
[2]https://gitlab.com/libvirt/libvirt/-/blob/136b821f183deb0b58c571211f6917985b...
As noted in my other reply, simple http auth is not practically very useful for the kinds of sources we want to access, so if libvirt never supported it I wouldn't bother adding that complexity now.
3. blockdev-create ...
nbdkit support for creating SSH remote files is upstream now: https://gitlab.com/nbdkit/nbdkit/-/commit/0793f30b1071753532362b2ebf9cb8156a...
And now I notice that we do not actually have support for 'ssh' network disks in our xml schema either [3]. The VIR_STORAGE_NET_PROTOCOL_SSH enum value was originally added with a comment that it allows using the ssh protocol in backing chains. I've also seen some commits indicating that some of the support for ssh disk sources may be related to libguestfs usage in some way. cc'ing Rich and Peter in case they can add any historical context here.
[3] https://gitlab.com/libvirt/libvirt/-/blob/136b821f183deb0b58c571211f6917985b...
Peter will remember this better, but IIRC the history was that virt-v2v used ssh protocol in backing files when accessing certain disks, ie: - qemu-img create -b ssh:blah overlay.qcow2 - use libvirt to open overlay.qcow2 Originally libvirt didn't care about this detail, but later libvirt was changed so that it validated the backing chain. Peter added support for ssh in the backing chain. Now actual support for protocol='ssh' (as in, the main drive, not only in the backing chain), while not currently documented, seems to be sort of working. libguestfs will try to create a protocol='ssh' drive through libvirt if you ask it: $ guestfish --format=raw -a ssh://foo/var/tmp/newdisk run libguestfs: error: could not create appliance through libvirt. ... Original error from libvirt: internal error: qemu unexpectedly closed the monitor: 2022-04-19T17:19:47.096384Z qemu-kvm: -blockdev {"driver":"ssh","path":"/var/tmp/newdisk","server":{"host":"foo","port":"22"},"node-name":"libvirt-2-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}: failed to authenticate using publickey authentication and the identities held by your ssh-agent [code=1 int1=-1] If you add -vx to the guestfish command you can see the libvirt XML fragment that was used: <disk device="disk" type="network"> <source protocol="ssh" name="/var/tmp/newdisk"> <host name="foo"/> </source> <target dev="sda" bus="scsi"/> <driver name="qemu" type="raw" cache="writeback"/> <address type="drive" controller="0" bus="0" target="0" unit="0"/> </disk> which looks plausible. I don't understand why it doesn't work, because I've got my agent set up correctly. I wonder if it's not passing SSH_AUTH_SOCK through to session virtqemud? Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org
On 4/19/22 12:31 PM, Richard W.M. Jones wrote:
On Tue, Apr 19, 2022 at 11:40:49AM -0500, Jonathon Jongsma wrote:
Well, As far as I can tell, there is no valid XML for exercising http auth. The schema for http(s) sources does not include any <auth> element [1]. And the schema for the <auth> element [2] requires a <secret> element with a required 'type' attribute, and the only choices for 'type' are 'ceph' or 'iscsi', neither of which apply to http authentication.
So it looks to me like http auth was never supported, rather than a regression. It also seems that it would require some additional schema changes to support this.
[1] https://gitlab.com/libvirt/libvirt/-/blob/136b821f183deb0b58c571211f6917985b...
[2]https://gitlab.com/libvirt/libvirt/-/blob/136b821f183deb0b58c571211f6917985b...
As noted in my other reply, simple http auth is not practically very useful for the kinds of sources we want to access, so if libvirt never supported it I wouldn't bother adding that complexity now.
As far as I can tell, we haven't ever supported it. But it is possible that I missed something in my digging, so if anybody has any recollection of http auth being supported in the past, please chime in.
3. blockdev-create ...
nbdkit support for creating SSH remote files is upstream now: https://gitlab.com/nbdkit/nbdkit/-/commit/0793f30b1071753532362b2ebf9cb8156a...
And now I notice that we do not actually have support for 'ssh' network disks in our xml schema either [3]. The VIR_STORAGE_NET_PROTOCOL_SSH enum value was originally added with a comment that it allows using the ssh protocol in backing chains. I've also seen some commits indicating that some of the support for ssh disk sources may be related to libguestfs usage in some way. cc'ing Rich and Peter in case they can add any historical context here.
[3] https://gitlab.com/libvirt/libvirt/-/blob/136b821f183deb0b58c571211f6917985b...
Peter will remember this better, but IIRC the history was that virt-v2v used ssh protocol in backing files when accessing certain disks, ie:
- qemu-img create -b ssh:blah overlay.qcow2 - use libvirt to open overlay.qcow2
Originally libvirt didn't care about this detail, but later libvirt was changed so that it validated the backing chain. Peter added support for ssh in the backing chain.
Now actual support for protocol='ssh' (as in, the main drive, not only in the backing chain), while not currently documented, seems to be sort of working. libguestfs will try to create a protocol='ssh' drive through libvirt if you ask it:
$ guestfish --format=raw -a ssh://foo/var/tmp/newdisk run libguestfs: error: could not create appliance through libvirt. ... Original error from libvirt: internal error: qemu unexpectedly closed the monitor: 2022-04-19T17:19:47.096384Z qemu-kvm: -blockdev {"driver":"ssh","path":"/var/tmp/newdisk","server":{"host":"foo","port":"22"},"node-name":"libvirt-2-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}: failed to authenticate using publickey authentication and the identities held by your ssh-agent [code=1 int1=-1]
If you add -vx to the guestfish command you can see the libvirt XML fragment that was used:
<disk device="disk" type="network"> <source protocol="ssh" name="/var/tmp/newdisk"> <host name="foo"/> </source> <target dev="sda" bus="scsi"/> <driver name="qemu" type="raw" cache="writeback"/> <address type="drive" controller="0" bus="0" target="0" unit="0"/> </disk>
which looks plausible. I don't understand why it doesn't work, because I've got my agent set up correctly. I wonder if it's not passing SSH_AUTH_SOCK through to session virtqemud?
Rich.
This looks like the same thing: https://bugzilla.redhat.com/show_bug.cgi?id=1140166 (and associated duplicates). It was finally closed deferred last year. Does this mean that the ssh disk source never properly worked in libvirt? Jonathon
On Tue, Apr 19, 2022 at 03:00:58PM -0500, Jonathon Jongsma wrote:
On 4/19/22 12:31 PM, Richard W.M. Jones wrote:
On Tue, Apr 19, 2022 at 11:40:49AM -0500, Jonathon Jongsma wrote:
And now I notice that we do not actually have support for 'ssh' network disks in our xml schema either [3]. The VIR_STORAGE_NET_PROTOCOL_SSH enum value was originally added with a comment that it allows using the ssh protocol in backing chains. I've also seen some commits indicating that some of the support for ssh disk sources may be related to libguestfs usage in some way. cc'ing Rich and Peter in case they can add any historical context here.
[3] https://gitlab.com/libvirt/libvirt/-/blob/136b821f183deb0b58c571211f6917985b...
Peter will remember this better, but IIRC the history was that virt-v2v used ssh protocol in backing files when accessing certain disks, ie:
- qemu-img create -b ssh:blah overlay.qcow2 - use libvirt to open overlay.qcow2
Originally libvirt didn't care about this detail, but later libvirt was changed so that it validated the backing chain. Peter added support for ssh in the backing chain.
Now actual support for protocol='ssh' (as in, the main drive, not only in the backing chain), while not currently documented, seems to be sort of working. libguestfs will try to create a protocol='ssh' drive through libvirt if you ask it:
$ guestfish --format=raw -a ssh://foo/var/tmp/newdisk run libguestfs: error: could not create appliance through libvirt. ... Original error from libvirt: internal error: qemu unexpectedly closed the monitor: 2022-04-19T17:19:47.096384Z qemu-kvm: -blockdev {"driver":"ssh","path":"/var/tmp/newdisk","server":{"host":"foo","port":"22"},"node-name":"libvirt-2-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}: failed to authenticate using publickey authentication and the identities held by your ssh-agent [code=1 int1=-1]
If you add -vx to the guestfish command you can see the libvirt XML fragment that was used:
<disk device="disk" type="network"> <source protocol="ssh" name="/var/tmp/newdisk"> <host name="foo"/> </source> <target dev="sda" bus="scsi"/> <driver name="qemu" type="raw" cache="writeback"/> <address type="drive" controller="0" bus="0" target="0" unit="0"/> </disk>
which looks plausible. I don't understand why it doesn't work, because I've got my agent set up correctly. I wonder if it's not passing SSH_AUTH_SOCK through to session virtqemud?
Rich.
This looks like the same thing: https://bugzilla.redhat.com/show_bug.cgi?id=1140166 (and associated duplicates). It was finally closed deferred last year.
Does this mean that the ssh disk source never properly worked in libvirt?
It seems possible. That bug was closed after we switched virt-v2v to using nbdkit-ssh-plugin in these commits: https://github.com/libguestfs/virt-v2v/commit/8312c03dc0f688b8e99e5602658fa3... https://github.com/libguestfs/virt-v2v/commit/10f8324bf8b97466092f9a43951a77... https://github.com/libguestfs/virt-v2v/commit/d4f8e5a4a01b0fbbb0158d0acc11f1... https://github.com/libguestfs/virt-v2v/commit/7a6f6113a25f96c813d2e73ee7e6cb... Be nice to support ssh in libvirt if it's not too hard, since ssh/sftp is probably the most widely available remote protocol after http. Almost every server is running sshd. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top
On Tue, Apr 19, 2022 at 22:30:51 +0100, Richard W.M. Jones wrote:
On Tue, Apr 19, 2022 at 03:00:58PM -0500, Jonathon Jongsma wrote:
On 4/19/22 12:31 PM, Richard W.M. Jones wrote:
[..]
Now actual support for protocol='ssh' (as in, the main drive, not only in the backing chain), while not currently documented, seems to be sort of working. libguestfs will try to create a protocol='ssh' drive through libvirt if you ask it:
$ guestfish --format=raw -a ssh://foo/var/tmp/newdisk run libguestfs: error: could not create appliance through libvirt. ... Original error from libvirt: internal error: qemu unexpectedly closed the monitor: 2022-04-19T17:19:47.096384Z qemu-kvm: -blockdev {"driver":"ssh","path":"/var/tmp/newdisk","server":{"host":"foo","port":"22"},"node-name":"libvirt-2-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}: failed to authenticate using publickey authentication and the identities held by your ssh-agent [code=1 int1=-1]
If you add -vx to the guestfish command you can see the libvirt XML fragment that was used:
<disk device="disk" type="network"> <source protocol="ssh" name="/var/tmp/newdisk"> <host name="foo"/> </source> <target dev="sda" bus="scsi"/> <driver name="qemu" type="raw" cache="writeback"/> <address type="drive" controller="0" bus="0" target="0" unit="0"/> </disk>
which looks plausible. I don't understand why it doesn't work, because I've got my agent set up correctly. I wonder if it's not passing SSH_AUTH_SOCK through to session virtqemud?
Rich.
This looks like the same thing: https://bugzilla.redhat.com/show_bug.cgi?id=1140166 (and associated duplicates). It was finally closed deferred last year.
Does this mean that the ssh disk source never properly worked in libvirt?
SSH disk sources indeed never worked properly. The existing infrastructure we have was added for the sole purpose to allow existing libguestfs images to work. The problem why it never got implemented was that the socket to the ssh agent was to be passed as an environment variable to qemu so it made it very unwieldy to actually implement it properly. At the time I've re-wrote to use -blockdev it became necessary to be able to parse the backing image definition at least to some extend and thus the partial support for 'ssh' protocol was added and the support for 'slice'-ing of the storage was added (libguestfs was accessing a disk image inside a tar archive without actually extracting it).
On Tue, Apr 19, 2022 at 15:00:58 -0500, Jonathon Jongsma wrote:
On 4/19/22 12:31 PM, Richard W.M. Jones wrote:
On Tue, Apr 19, 2022 at 11:40:49AM -0500, Jonathon Jongsma wrote:
Well, As far as I can tell, there is no valid XML for exercising http auth. The schema for http(s) sources does not include any <auth> element [1]. And the schema for the <auth> element [2] requires a <secret> element with a required 'type' attribute, and the only choices for 'type' are 'ceph' or 'iscsi', neither of which apply to http authentication.
So it looks to me like http auth was never supported, rather than a regression. It also seems that it would require some additional schema changes to support this.
[1] https://gitlab.com/libvirt/libvirt/-/blob/136b821f183deb0b58c571211f6917985b...
[2]https://gitlab.com/libvirt/libvirt/-/blob/136b821f183deb0b58c571211f6917985b...
As noted in my other reply, simple http auth is not practically very useful for the kinds of sources we want to access, so if libvirt never supported it I wouldn't bother adding that complexity now.
As far as I can tell, we haven't ever supported it. But it is possible that I missed something in my digging, so if anybody has any recollection of http auth being supported in the past, please chime in.
I don't think it was really supported. There were some bits present which made me to think that it was supported in the -drive code, so I've forward ported the functionality to the -blockdev code, but when I actually try it we don't setup the 'secret' object which is supposed to store the password and thus end up crashing in qemuBlockStorageSourceGetCURLProps I'll post patches to address that, but the question is whether we want to bother with actually supporting the password authentication or not, because the simpler approach to fixing the bug is to simply allow it.
On Wed, Apr 20, 2022 at 09:36:29AM +0200, Peter Krempa wrote:
I'll post patches to address that, but the question is whether we want to bother with actually supporting the password authentication or not, because the simpler approach to fixing the bug is to simply allow it.
Did you mean: Simply _not_ allow it? Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-p2v converts physical machines to virtual machines. Boot with a live CD or over the network (PXE) and turn machines into KVM guests. http://libguestfs.org/virt-v2v
On Wed, Apr 20, 2022 at 09:52:34 +0100, Richard W.M. Jones wrote:
On Wed, Apr 20, 2022 at 09:36:29AM +0200, Peter Krempa wrote:
I'll post patches to address that, but the question is whether we want to bother with actually supporting the password authentication or not, because the simpler approach to fixing the bug is to simply allow it.
Did you mean: Simply _not_ allow it?
No actually code-wise everything seems to be in place. The parser parses it, the -blockdev formatter supports it. The only thing that prevents from use of the authentication with the curl driver backends is a check in a helper function which limits the protocols we instantiate the 'secret' object for. Removing that limit actually makes us pass the secret to qemu and our validator shows that it's valid definition. Adding the schema bits should be easy too.
participants (4)
-
Daniel P. Berrangé -
Jonathon Jongsma -
Peter Krempa -
Richard W.M. Jones