Question/comment on nbdkit support in libvirt.spec
In this section (under %package libvirt-daemon-driver-qemu): %if %{with_nbdkit} Recommends: nbdkit Recommends: nbdkit-curl-plugin Recommends: nbdkit-ssh-plugin %endif You could save a bit of space by removing "Recommends: nbdkit" and replacing it with: Recommends: nbdkit-server Recommends: nbdkit-readahead-filter In Fedora packaging "nbdkit" is a meta-package which pulls in nbdkit-server, nbdkit-basic-plugins and nbdkit-basic-filters. This meta-package is intended to get a useful, functional nbdkit, but it's not very minimal. The basic plugins/filters pull in about an extra 1.3MB. nbdkit-basic-{plugins,filters} also "Provides" the names of the individual plugins and filters, so if (for example) you wanted the file plugin then you could just "Recommends: nbdkit-file-plugin" which would pull in nbdkit-basic-plugins (or something else if we ever moved that plugin to another package). nbdkit-curl-plugin / nbdkit-ssh-plugin are packaged separately because they have non-trivial dependencies on external libraries, and using Recommends for those is absolutely correct. They depend on nbdkit-server so it's not strictly needed to also "Recommends: nbdkit-server", but it is better for clarity. nbdkit-server includes nbdkit-null-plugin so you can still use that for probing filters. (I asked another question on a Jira ticket about if you are using --readonly since that improves performance with nbdkit-curl-plugin. Checking the source code I see you are, so that's great.) Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-p2v converts physical machines to virtual machines. Boot with a live CD or over the network (PXE) and turn machines into KVM guests. http://libguestfs.org/virt-v2v
Few more things ... (1) Looking at the parameters to nbdkit in qemuNbdkitProcessBuildCommand, it might make sense to consider --exit-with-parent. What this does on Linux is it causes nbdkit to call: prctl(PR_SET_PDEATHSIG, SIGTERM); which causes the kernel to send SIGTERM to nbdkit if the parent (virtqemud, presumably?) ever goes away. This is just a belt and braces approach to ensuring that nbdkit is always cleaned up, even if libvirt dies without managing to clean up. So a couple of potential problems: (a) --exit-with-parent is only supported on a subset of platforms (currently Linux, BSD and macOS). You have to check: $ nbdkit --dump-config | grep exit exit_with_parent=yes if you might use nbdkit outside those platforms. (Using --exit-with-parent on a non-supported platform will give an error.) (b) Is virtqemud actually persistent? I'm not totally sure what happens for long-running libvirt domains if you restart libvirtd. Of course if the parent process did go away in this case then it would kill nbdkit which would be bad presumably ... (2) I'm fairly sure you'll find you need to use --selinux-label at some point. This does some SELinux/sVirt voodoo on the socket. We found that this was necessary: nbdkit -U /tmp/sock --selinux-label=system_u:object_r:svirt_socket_t:s0 ... chcon system_u:object_r:svirt_image_t:s0 /tmp/sock to allow qemu clients to connect to nbdkit when SELinux is enabled, but only some of the time (like, it works fine without this on either Fedora or RHEL, but not the other one, I forget which way round now). (3) I'm wondering if, when libvirt's own debugging is enabled, we should also add -v (verbose) and maybe even these -D flags: -v -D curl.verbose=1 # for nbdkit-curl-plugin -v -D ssh.log=2 # for nbdkit-ssh-plugin nbdkit will send debug messages to stderr or syslog (see --log=stderr|syslog option), which may or may not be useful. (4) case VIR_STORAGE_NET_PROTOCOL_NBD gives an error. Technically nbdkit does support NBD forwarding (nbdkit-nbd-plugin), but I think you're correct not to use this. It's better to have qemu connect directly using its own NBD support. (5) Now we have the infrastructure, I think a massively valuable feature would be VDDK support (nbdkit-vddk-plugin). Having libvirt being able to boot qemu VMs backed directly by VMware servers would be huge. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-p2v converts physical machines to virtual machines. Boot with a live CD or over the network (PXE) and turn machines into KVM guests. http://libguestfs.org/virt-v2v
On Fri, Jan 19, 2024 at 12:01:55 +0000, Richard W.M. Jones wrote:
Few more things ...
(1) Looking at the parameters to nbdkit in qemuNbdkitProcessBuildCommand, it might make sense to consider --exit-with-parent. What this does on Linux is it causes nbdkit to call:
prctl(PR_SET_PDEATHSIG, SIGTERM);
which causes the kernel to send SIGTERM to nbdkit if the parent (virtqemud, presumably?) ever goes away.
We definitely do not want this. The VM will continue running if virtqemud is restarted, thus nbdkit must continue running too.
This is just a belt and braces approach to ensuring that nbdkit is always cleaned up, even if libvirt dies without managing to clean up.
So a couple of potential problems:
(a) --exit-with-parent is only supported on a subset of platforms (currently Linux, BSD and macOS). You have to check:
$ nbdkit --dump-config | grep exit exit_with_parent=yes
if you might use nbdkit outside those platforms. (Using --exit-with-parent on a non-supported platform will give an error.)
(b) Is virtqemud actually persistent? I'm not totally sure what happens for long-running libvirt domains if you restart libvirtd. Of course if the parent process did go away in this case then it would kill nbdkit which would be bad presumably ...
(2) I'm fairly sure you'll find you need to use --selinux-label at some point. This does some SELinux/sVirt voodoo on the socket. We found that this was necessary:
nbdkit -U /tmp/sock --selinux-label=system_u:object_r:svirt_socket_t:s0 ... chcon system_u:object_r:svirt_image_t:s0 /tmp/sock
to allow qemu clients to connect to nbdkit when SELinux is enabled, but only some of the time (like, it works fine without this on either Fedora or RHEL, but not the other one, I forget which way round now).
IIRC this is handled by libvirt's security labelling code anyways.
(3) I'm wondering if, when libvirt's own debugging is enabled, we should also add -v (verbose) and maybe even these -D flags:
-v -D curl.verbose=1 # for nbdkit-curl-plugin -v -D ssh.log=2 # for nbdkit-ssh-plugin
nbdkit will send debug messages to stderr or syslog (see --log=stderr|syslog option), which may or may not be useful.
(4) case VIR_STORAGE_NET_PROTOCOL_NBD gives an error. Technically nbdkit does support NBD forwarding (nbdkit-nbd-plugin), but I think you're correct not to use this. It's better to have qemu connect directly using its own NBD support.
That doesn't matter because nbdkit won't be even used for VIR_STORAGE_NET_PROTOCOL_NBD disks and I don't see a point in doing so.
(5) Now we have the infrastructure, I think a massively valuable feature would be VDDK support (nbdkit-vddk-plugin). Having libvirt being able to boot qemu VMs backed directly by VMware servers would be huge.
Rich.
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-p2v converts physical machines to virtual machines. Boot with a live CD or over the network (PXE) and turn machines into KVM guests. http://libguestfs.org/virt-v2v _______________________________________________ Devel mailing list -- devel@lists.libvirt.org To unsubscribe send an email to devel-leave@lists.libvirt.org
On Fri, Jan 19, 2024 at 01:11:19PM +0100, Peter Krempa wrote:
On Fri, Jan 19, 2024 at 12:01:55 +0000, Richard W.M. Jones wrote:
(2) I'm fairly sure you'll find you need to use --selinux-label at some point. This does some SELinux/sVirt voodoo on the socket. We found that this was necessary:
nbdkit -U /tmp/sock --selinux-label=system_u:object_r:svirt_socket_t:s0 ... chcon system_u:object_r:svirt_image_t:s0 /tmp/sock
to allow qemu clients to connect to nbdkit when SELinux is enabled, but only some of the time (like, it works fine without this on either Fedora or RHEL, but not the other one, I forget which way round now).
IIRC this is handled by libvirt's security labelling code anyways.
I think that's the labelling of the socket, but there's also the labelling of the process (nbdkit) side of the socket? They are two separate things, IIRC. Anyway just noting that we had trouble with this in the past and the above ^^^ was found to be the solution. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-builder quickly builds VMs from scratch http://libguestfs.org/virt-builder.1.html
participants (2)
-
Peter Krempa -
Richard W.M. Jones