Network disks and replacing qemu-block-curl|ssh with nbdkit

As mentioned in https://bugzilla.redhat.com/show_bug.cgi?id=2016527, RHEL is planning to remove dependencies on the qemu-block-curl and qemu-block-ssh plugins from the main qemu package. This creates issues for libvirt for supporting network disk sources. So I've been looking into using nbdkit from libvirt to proxy these network disks to qemu as NBD disks. The basic idea is that libvirt will spin up an nbdkit instance for e.g. an https network disk source, and will provide the resulting unix socket to qemu as an nbd disk. This allows libvirt to continue supporting http/ftp/ssh disk sources regardless of whether the qemu block plugins are installed or not. However, there are a couple of issues and feature gaps that I've run into that I'd like to discuss. 1. secrets There is some code in libvirt[1] which seems to expect that it is possible for http(s) disk sources to have a username and password specified. However, I can't find any valid xml schema for specifying an http username and password, and my reading of the code suggests that there shouldn't be any way for these to be set for http(s)/ftp(s) disk sources either since auth is only supported for ISCSI and RBD protocols [2]. Am I missing something? [1] https://gitlab.com/libvirt/libvirt/-/blob/6be7beb3bdb9ad611a5598dad7edfbd... [2] https://gitlab.com/libvirt/libvirt/-/blob/6be7beb3bdb9ad611a5598dad7edfbd... If it *is* possible for the username/password to be set for these disks, then we have the issue that these sensitive pieces of data have so far been passed as encrypted data to qemu using qemu secrets. But if nbdkit is handling the http requests, we need to pass this data to nbdkit rather than qemu and so we can no longer use qemu secrets. The same issue applies to http cookies, which could potentially include security-sensitive data such as login credentials, etc. Fortunately, nbdkit provides a method for reading cookies and passwords from a file, which should be secure if the file has permissions set properly. So I'm currently planning to write a file containing the cookies and pass them to nbdkit by specifying the filename. But I'm still confused about the username/password possibility. 2. readahead The libvirt xml format allows to specify a readahead size for disks that are handled by the qemu-block-curl plugin. Unfortunately, nbdkit doesn't currently support any readahead configuration. In nbdkit, readahead is handled by an nbkit "filter" that takes no configuration options (`nbdkit --filter=readahead ...`). In theory, this filter tries to adaptively read ahead. But when I discussed it with Rich, he suggested that he had stopped using it in virt-v2v because it was causing more trouble than it was worth. He also suggested that this readahead filter might need a complete rewrite, and presumably the rewrite could include the ability to configure a readahead buffer size. But I'm not sure what the timeframe might be for that. 3. blockdev-create There is support in libvirt[3] for creating ssh network disks by sending a 'blockdev-create' command to qemu. If qemu is no longer handling ssh network disk sources directly, this feature becomes significantly more complicated. I don't yet know enough about this part of the libvirt code to know what further complications might pop up here. From my reading of the code, this is mostly triggered by things like `virsh blockcopy` `virsh backup-begin`, etc. But nbdkit cannot currently do this. Rich pointed me to a recent commit[4] where he added disk creation to the nbdkit vddk plugin, and suggested that something similar could be added for the nbdkit ssh plugin. [3] https://gitlab.com/libvirt/libvirt/-/blob/6be7beb3bdb9ad611a5598dad7edfbd... [4] https://gitlab.com/nbdkit/nbdkit/-/commit/a39d5773afc3ebab7e5768118a2bccb... It seems to me that it's essential that we resolve #3 before we can move forward with nbdkit support in libvirt. (Although I admit that I have no idea how common it is for people to use ssh disks so I suppose there's a slim possibility that we could just disable the 'create disk' feature for ssh disks without any practical loss of functionality?) But it's less obvious to me whether we could move ahead despite missing readahead size configuration, etc. Thoughts? Jonathon