consuming pre-created tap - with multiqueue
Hello, On KubeVirt, we are trying to pre-create a tap device, then instruct libvirt to consume it (via the type=ethernet , managed='no' attributes). It works as expected, **unless** when we create a multi-queue tap device. The difference when creating the tap device is that we set the multi-queue flag; libvirt throws the following error when consuming it: ``` LibvirtError(Code=38, Domain=0, Message='Unable to create tap device tap0: Invalid argument') ``` After digging a bit on the libvirt code (we're using libvirt 6.0.0), I see this on the logs (immediately before the error): {"component":"virt-launcher","level":"info","msg":"Enabling IFF_VNET_HDR","pos":"virNetDevProbeVnetHdr:190","subcomponent":"libvirt","thread":"33" ,"timestamp":"2020-09-22T10:34:29.335000Z"} I do not understand how it can try to set the VNET_HDR flag, since I have not set it when I created it, which, as per [0] should only happen when requested. Here's the tap device I'm creating: (output of `ip tuntap show`) - tap0: tap persist0x100 user 107 group 107 I'm confused by this, since on [1] it says that only one flag can be used (persist *or* vnet_hdr). Also, from my limited understanding, in order to open a pre-created tap device, it must be set to persistent. Am I right (in the sense that these flags cannot co-exist) ? If so, why would libvirt be setting the VNET_HDR flag on the tap device ? [0] - https://github.com/libvirt/libvirt/blob/v6.0.0/src/util/virnetdevtap.c#L261 [1] - https://github.com/libvirt/libvirt/blob/v6.0.0/src/util/virnetdevtap.c#L209
On Tue, Sep 22, 2020 at 01:48:08PM +0200, Miguel Duarte de Mora Barroso wrote:
Hello,
On KubeVirt, we are trying to pre-create a tap device, then instruct libvirt to consume it (via the type=ethernet , managed='no' attributes).
It works as expected, **unless** when we create a multi-queue tap device.
The difference when creating the tap device is that we set the multi-queue flag; libvirt throws the following error when consuming it:
``` LibvirtError(Code=38, Domain=0, Message='Unable to create tap device tap0: Invalid argument') ```
After digging a bit on the libvirt code (we're using libvirt 6.0.0), I see this on the logs (immediately before the error): {"component":"virt-launcher","level":"info","msg":"Enabling IFF_VNET_HDR","pos":"virNetDevProbeVnetHdr:190","subcomponent":"libvirt","thread":"33" ,"timestamp":"2020-09-22T10:34:29.335000Z"}
I do not understand how it can try to set the VNET_HDR flag, since I have not set it when I created it, which, as per [0] should only happen when requested. Here's the tap device I'm creating: (output of `ip tuntap show`) - tap0: tap persist0x100 user 107 group 107
IIUC the kernel code correctly, the VNET_HDR flag is not required to be set when you first create the tap device - it appears to be permitted any time you open a file descriptor for it. AFAIK, there is no problem with VNET_HDR, as it is a standard flag we've set on all tap devices on Linux for 10 years.
I'm confused by this, since on [1] it says that only one flag can be used (persist *or* vnet_hdr). Also, from my limited understanding, in order to open a pre-created tap device, it must be set to persistent.
That comment you're pointing to is saying that the method only has one named flag defined in the enum. It should have been updated when we added the second PERSIST flag. They are not mutually exclusive from either libvirt or kernel POV AFAICT.
Am I right (in the sense that these flags cannot co-exist) ? If so, why would libvirt be setting the VNET_HDR flag on the tap device ?
[0] - https://github.com/libvirt/libvirt/blob/v6.0.0/src/util/virnetdevtap.c#L261 [1] - https://github.com/libvirt/libvirt/blob/v6.0.0/src/util/virnetdevtap.c#L209
Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Wed, Sep 23, 2020 at 05:44:28PM +0100, Daniel P. Berrangé wrote:
On Tue, Sep 22, 2020 at 01:48:08PM +0200, Miguel Duarte de Mora Barroso wrote:
Hello,
On KubeVirt, we are trying to pre-create a tap device, then instruct libvirt to consume it (via the type=ethernet , managed='no' attributes).
It works as expected, **unless** when we create a multi-queue tap device.
The difference when creating the tap device is that we set the multi-queue flag; libvirt throws the following error when consuming it:
``` LibvirtError(Code=38, Domain=0, Message='Unable to create tap device tap0: Invalid argument') ```
After digging a bit on the libvirt code (we're using libvirt 6.0.0), I see this on the logs (immediately before the error): {"component":"virt-launcher","level":"info","msg":"Enabling IFF_VNET_HDR","pos":"virNetDevProbeVnetHdr:190","subcomponent":"libvirt","thread":"33" ,"timestamp":"2020-09-22T10:34:29.335000Z"}
I do not understand how it can try to set the VNET_HDR flag, since I have not set it when I created it, which, as per [0] should only happen when requested. Here's the tap device I'm creating: (output of `ip tuntap show`) - tap0: tap persist0x100 user 107 group 107
IIUC the kernel code correctly, the VNET_HDR flag is not required to be set when you first create the tap device - it appears to be permitted any time you open a file descriptor for it.
AFAIK, there is no problem with VNET_HDR, as it is a standard flag we've set on all tap devices on Linux for 10 years.
Looking at the kernel code, you need to set the MULTI_QUEUE flag at time you create the device and also set it when opening the device. In tun_set_iff(): if (!!(ifr->ifr_flags & IFF_MULTI_QUEUE) != !!(tun->flags & IFF_MULTI_QUEUE)) return -EINVAL; so if you've configured QEMU to use multiqueue, the you need to use: $ ip tuntap add dev mytap mode tap vnet_hdr multi_queue actually vnet_hdr doesn't matter as that can be set on the fly but multi_queue is mandatory. Without it, I get the same EINVAL error as you mention. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Wed, Sep 23, 2020 at 6:59 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
On Wed, Sep 23, 2020 at 05:44:28PM +0100, Daniel P. Berrangé wrote:
On Tue, Sep 22, 2020 at 01:48:08PM +0200, Miguel Duarte de Mora Barroso wrote:
Hello,
On KubeVirt, we are trying to pre-create a tap device, then instruct libvirt to consume it (via the type=ethernet , managed='no' attributes).
It works as expected, **unless** when we create a multi-queue tap device.
The difference when creating the tap device is that we set the multi-queue flag; libvirt throws the following error when consuming it:
``` LibvirtError(Code=38, Domain=0, Message='Unable to create tap device tap0: Invalid argument') ```
After digging a bit on the libvirt code (we're using libvirt 6.0.0), I see this on the logs (immediately before the error): {"component":"virt-launcher","level":"info","msg":"Enabling IFF_VNET_HDR","pos":"virNetDevProbeVnetHdr:190","subcomponent":"libvirt","thread":"33" ,"timestamp":"2020-09-22T10:34:29.335000Z"}
I do not understand how it can try to set the VNET_HDR flag, since I have not set it when I created it, which, as per [0] should only happen when requested. Here's the tap device I'm creating: (output of `ip tuntap show`) - tap0: tap persist0x100 user 107 group 107
IIUC the kernel code correctly, the VNET_HDR flag is not required to be set when you first create the tap device - it appears to be permitted any time you open a file descriptor for it.
AFAIK, there is no problem with VNET_HDR, as it is a standard flag we've set on all tap devices on Linux for 10 years.
Looking at the kernel code, you need to set the MULTI_QUEUE flag at time you create the device and also set it when opening the device. In tun_set_iff():
if (!!(ifr->ifr_flags & IFF_MULTI_QUEUE) != !!(tun->flags & IFF_MULTI_QUEUE)) return -EINVAL;
so if you've configured QEMU to use multiqueue, the you need to use:
$ ip tuntap add dev mytap mode tap vnet_hdr multi_queue
actually vnet_hdr doesn't matter as that can be set on the fly but multi_queue is mandatory. Without it, I get the same EINVAL error as you mention.
Right, sorry for the noise; I found out one of our tests was abusing the API, which caused the tap device to be created with multi-queue, then when attempting to consume it, we were requesting a single vcpu. My bad for not spotting this sooner. Thanks for the reply.
Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
participants (2)
-
Daniel P. Berrangé -
Miguel Duarte de Mora Barroso