Re: [libvirt-users] Need to increase the rx and tx buffer size of my interface
Hi Yalan, In the previous email you mentioned "tx_queue_size='512' will not work in the guest with direct type interface, in fact, no matter what you set, it will not work and guest will get the default '256'. " So if I am using macvtap for my interfaces, then the device type will always be direct type. Does it mean that there is no way I can increase the buffer size with the macvtap interfaces? Best Regards, Ashish Kurian On Thu, Oct 26, 2017 at 9:04 AM, Ashish Kurian <ashishbnv@gmail.com> wrote:
Hi Yalan,
Thank you for your comment on qemu-kvm-rhev
I am waiting for a response about my previous email with the logs attached. I do not understand what is the problem.
On Oct 26, 2017 8:58 AM, "Yalan Zhang" <yalzhang@redhat.com> wrote:
Hi Ashish,
Please never mind for qemu-kvm-rhev. qemu with the code base 2.10.0 will support the tx_queue_size and rx_queue_size.
Thank you~
------- Best Regards, Yalan Zhang IRC: yalzhang Internal phone: 8389413
On Thu, Oct 26, 2017 at 2:22 PM, Yalan Zhang <yalzhang@redhat.com> wrote:
Hi Ashish,
Are these packages available for free? How can I install them? => You did have vhost backend driver. Do not set <driver name='qemu'...>, by default it will use vhost as backend driver.
Is it possible to have my interfaces with an IP address inside the VM to be bridged to the physical interfaces on the host? => Yes, you can create a linux bridge with physical interface connected, and use bridge type interface. Refer to https://libvirt.org/formatd omain.html#elementsNICSBridge direct type is also ok (but your host and guest have no access to each other).
Is it also a possibility that I change the rx and tx buffer on the physical interface on the host and it is reflected automatically inside the VM as you said it will always receive the default value of the host? => No, it do not receive the default value of the host. It's the default value related with the virtual device driver on the guest. hostdev type interface will passthrough the physical interface or VF of the host to guest, it will get the device's parameters for rx and tx buffer.
------- Best Regards, Yalan Zhang IRC: yalzhang Internal phone: 8389413
On Thu, Oct 26, 2017 at 1:30 PM, Ashish Kurian <ashishbnv@gmail.com> wrote:
Hi Yalan,
Thank you for your response. I do not have the following packages installed
vhost backend driver qemu-kvm-rhev package
Are these packages available for free? How can I install them?
In my KVM VM, I must have an IP address to the interfaces that I am trying to increasing the buffers. That is the reason I was using macvtap (direct type interface). Is it possible to have my interfaces with an IP address inside the VM to be bridged to the physical interfaces on the host?
Is it also a possibility that I change the rx and tx buffer on the physical interface on the host and it is reflected automatically inside the VM as you said it will always receive the default value of the host?
Best Regards, Ashish Kurian
On Thu, Oct 26, 2017 at 6:45 AM, Yalan Zhang <yalzhang@redhat.com> wrote:
Hi Ashish,
I have tested with your xml in the first mail, and it works for rx_queue_size(see below). multiqueue need to work with vhost backend driver. And when you set "queues=1" it will ignored.
Please check your qemu-kvm-rhev package, should be newer than qemu-kvm-rhev-2.9.0-16.el7_4.2 And the logs?
tx_queue_size='512' will not work in the guest with direct type interface, in fact, no matter what you set, it will not work and guest will get the default '256'. We only support vhost-user backend to have more than 256. refer to https://libvirt.org/formatdomain.html#elementsNICSEthernet
tx_queue_size The optional tx_queue_size attribute controls the size of virtio ring for each queue as described above. The default value is hypervisor dependent and may change across its releases. Moreover, some hypervisors may pose some restrictions on actual value. For instance, QEMU v2.9 requires value to be a power of two from [256, 1024] range. In addition to that, this may work only for a subset of interface types, e.g. aforementioned QEMU enables this option only for vhostuser type. Since 3.7.0 (QEMU and KVM only) multiqueue only supports vhost as backend driver.
# rpm -q libvirt qemu-kvm-rhev libvirt-3.2.0-14.el7_4.3.x86_64 qemu-kvm-rhev-2.9.0-16.el7_4.9.x86_64
1. the xml as below <interface type='direct'> <mac address='52:54:00:00:b5:99'/> <source dev='eno1' mode='vepa'/> <model type='virtio'/> <driver name='vhost' queues='5' rx_queue_size='512' tx_queue_size='512'> <host csum='off' gso='off' tso4='off' tso6='off' ecn='off' ufo='off' mrg_rxbuf='off'/> <guest csum='off' tso4='off' tso6='off' ecn='off' ufo='off'/> </driver> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface>
2. after start the vm, check the qemu command line: *-netdev tap,fds=26:28:29:30:31,id=hostnet0,vhost=on,vhostfds=32:33:34:35:36* -device virtio-net-pci,csum=off,gso=off,host_tso4=off,host_tso6=off, host_ecn=off,host_ufo=off,mrg_rxbuf=off,guest_csum=off,guest _tso4=off,guest_tso6=off,guest_ecn=off,guest_ufo=off, *mq=on,vectors=12,rx_queue_size=512,tx_queue_size=512*,netdev=hostnet 0,id=net0,mac=52:54:00:00:b5:99,bus=pci.0,addr=0x3
3. check on guest # ethtool -g eth0 Ring parameters for eth0: Pre-set maximums: RX: *512 ==> rx_queue_size works* RX Mini: 0 RX Jumbo: 0 TX: *256 ===> no change* Current hardware settings: RX: *512 **==> rx_queue_size works* RX Mini: 0 RX Jumbo: 0 TX: *256 ===> no change*
# ethtool -l eth0 Channel parameters for eth0: Pre-set maximums: RX: 0 TX: 0 Other: 0 Combined: *5 ==> queues what we set* Current hardware settings: RX: 0 TX: 0 Other: 0 Combined: 1
If change to qemu as driver, # virsh edit rhel7 .. <interface type='direct'> <mac address='52:54:00:00:b5:99'/> <source dev='eno1' mode='vepa'/> <model type='virtio'/> <driver name='qemu' queues='5' rx_queue_size='512' tx_queue_size='512'> <host csum='off' gso='off' tso4='off' tso6='off' ecn='off' ufo='off' mrg_rxbuf='off'/> <guest csum='off' tso4='off' tso6='off' ecn='off' ufo='off'/> </driver> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> .. Domain rhel7 XML configuration edited. ==> the xml can validate and save
# virsh start rhel7 Domain rhel7 started
# virsh dumpxml rhel7 | grep /interface -B9 <source dev='eno1' mode='vepa'/> <target dev='macvtap0'/> <model type='virtio'/> *<driver name='qemu' queues='5' rx_queue_size='512' tx_queue_size='512'>* <host csum='off' gso='off' tso4='off' tso6='off' ecn='off' ufo='off' mrg_rxbuf='off'/> <guest csum='off' tso4='off' tso6='off' ecn='off' ufo='off'/> </driver> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface>
* -netdev tap,fds=26:28:29:30:31*,id=hostnet0 -device virtio-net-pci,csum=off,gso=off,host_tso4=off,host_tso6=off, host_ecn=off,host_ufo=off,mrg_rxbuf=off,guest_csum=off,guest _tso4=off,guest_tso6=off,guest_ecn=off,guest_ufo=off, *rx_queue_size=512,tx_queue_size=512*,netdev=hostnet0,id=net0,mac=52: 54:00:00:b5:99,bus=pci.0,addr=0x3
*"mq=on,vectors=12" is missing*, indicates there is no multiqueue
and check on guest
# ethtool -l eth0 Channel parameters for eth0: Pre-set maximums: RX: 0 TX: 0 Other: 0 Combined: 1 ==> no multiqueue Current hardware settings: RX: 0 TX: 0 Other: 0 Combined: 1
# ethtool -g eth0 Ring parameters for eth0: Pre-set maximums: RX: *512* RX Mini: 0 RX Jumbo: 0 TX: 256 Current hardware settings: RX: *512* RX Mini: 0 RX Jumbo: 0 TX: 256
------- Best Regards, Yalan Zhang IRC: yalzhang Internal phone: 8389413
On Thu, Oct 26, 2017 at 2:33 AM, Ashish Kurian <ashishbnv@gmail.com> wrote:
Hi Michal,
An update to what I have already said : when I try adding <driver name='qemu' txmode='iothread' ioeventfd='on' event_idx='off' queues='1' rx_queue_size='512' tx_queue_size='512'> although it showed me the error as mentioned, when I checked the xml again I saw that <driver name='qemu' txmode='iothread' ioeventfd='on' event_idx='off' > is added to the interface.
The missing parameters are : queues='1' rx_queue_size='512' tx_queue_size='512'
Best Regards, Ashish Kurian
On Wed, Oct 25, 2017 at 5:07 PM, Ashish Kurian <ashishbnv@gmail.com> wrote:
Hi Michal,
What I found was that when I restarted the machine and did a virsh edit command to see the xml config, I see that it is was not actually changed. This suggests why I saw 256 again after restarting.
So now I tried again to edit the xml via virsh edit command and used the following to set the parameters.
<driver name='qemu' txmode='iothread' ioeventfd='on' event_idx='off' queues='1' rx_queue_size='512' tx_queue_size='512'> </driver>
It was not accepted and I got the error saying :
error: XML document failed to validate against schema: Unable to validate doc against /usr/share/libvirt/schemas/domain.rng Extra element devices in interleave Element domain failed to validate content
What does this imply? I have two more other interfaces and do I have to the same to them also?
Btw, there are now logs generated now in the domain log or libvirtd log
Best Regards, Ashish Kurian
On Wed, Oct 25, 2017 at 2:50 PM, Michal Privoznik < mprivozn@redhat.com> wrote:
> On 10/25/2017 01:53 PM, Ashish Kurian wrote: > > Dear Users/Developers, > > > > I am using a KVM Ubuntu VM as a degrader to apply specific delays > to > > incoming packets. As the delay for my packets can be higher than > 7.5 > > seconds, there is not enough buffer on my interface to buffer all > the > > packets. Therefore those overflowing packets are dropped in the > machine and > > not forwarded. > > > > When I tried to use the command ethtool -G ens8 rx 512 to > increase the > > buffer size, I get the following error. > > > > Cannot set device ring parameters: Operation not permitted > > > > I have kept the VM xml files as specified in the link : > > https://libvirt.org/formatdomain.html. The value that I kept in > my xml file > > is as follows. > > > > <interface type='direct'> > > <mac address='52:54:00:72:f9:eb'/> > > <source dev='enp7s0f0' mode='vepa'/> > > <model type='virtio'/> > > <driver name='vhost' queues='5' rx_queue_size='512' > > tx_queue_size='512'> > > <host csum='off' gso='off' tso4='off' tso6='off' ecn='off' > ufo='off' > > mrg_rxbuf='off'/> > > <guest csum='off' tso4='off' tso6='off' ecn='off' ufo='off'/> > > </driver> > > <address type='pci' domain='0x0000' bus='0x00' slot='0x08' > > function='0x0'/> > > </interface> > > <interface type='direct'> > > <mac address='52:54:00:00:b5:99'/> > > <source dev='enp7s0f1' mode='vepa'/> > > <model type='virtio'/> > > <driver name='vhost' queues='5' rx_queue_size='512' > > tx_queue_size='512'> > > <host csum='off' gso='off' tso4='off' tso6='off' ecn='off' > ufo='off' > > mrg_rxbuf='off'/> > > <guest csum='off' tso4='off' tso6='off' ecn='off' ufo='off'/> > > </driver> > > <address type='pci' domain='0x0000' bus='0x00' slot='0x09' > > function='0x0'/> > > So what does the qemu command line look like? You can find it in > either > libvirtd log or domain log. > > http://wiki.libvirt.org/page/DebugLogs > > Michal >
_______________________________________________ libvirt-users mailing list libvirt-users@redhat.com https://www.redhat.com/mailman/listinfo/libvirt-users
On 10/26/2017 09:38 AM, Ashish Kurian wrote:
Hi Yalan,
In the previous email you mentioned "tx_queue_size='512' will not work in the guest with direct type interface, in fact, no matter what you set, it will not work and guest will get the default '256'. "
So if I am using macvtap for my interfaces, then the device type will always be direct type. Does it mean that there is no way I can increase the buffer size with the macvtap interfaces?
Correct. However, this is limitation of qemu. You should talk to them to find out if there are plans to extend the implementation. BTW: rx_queue_size should work with any backend as long as the model is virtio (obviously). Michal
Hi Ashish, IMO, it is yes, no way to increase tx_queue_size for direct type interface ------- Best Regards, Yalan Zhang IRC: yalzhang Internal phone: 8389413 On Thu, Oct 26, 2017 at 3:38 PM, Ashish Kurian <ashishbnv@gmail.com> wrote:
Hi Yalan,
In the previous email you mentioned "tx_queue_size='512' will not work in the guest with direct type interface, in fact, no matter what you set, it will not work and guest will get the default '256'. "
So if I am using macvtap for my interfaces, then the device type will always be direct type. Does it mean that there is no way I can increase the buffer size with the macvtap interfaces?
Best Regards, Ashish Kurian
On Thu, Oct 26, 2017 at 9:04 AM, Ashish Kurian <ashishbnv@gmail.com> wrote:
Hi Yalan,
Thank you for your comment on qemu-kvm-rhev
I am waiting for a response about my previous email with the logs attached. I do not understand what is the problem.
On Oct 26, 2017 8:58 AM, "Yalan Zhang" <yalzhang@redhat.com> wrote:
Hi Ashish,
Please never mind for qemu-kvm-rhev. qemu with the code base 2.10.0 will support the tx_queue_size and rx_queue_size.
Thank you~
------- Best Regards, Yalan Zhang IRC: yalzhang Internal phone: 8389413
On Thu, Oct 26, 2017 at 2:22 PM, Yalan Zhang <yalzhang@redhat.com> wrote:
Hi Ashish,
Are these packages available for free? How can I install them? => You did have vhost backend driver. Do not set <driver name='qemu'...>, by default it will use vhost as backend driver.
Is it possible to have my interfaces with an IP address inside the VM to be bridged to the physical interfaces on the host? => Yes, you can create a linux bridge with physical interface connected, and use bridge type interface. Refer to https://libvirt.org/formatd omain.html#elementsNICSBridge direct type is also ok (but your host and guest have no access to each other).
Is it also a possibility that I change the rx and tx buffer on the physical interface on the host and it is reflected automatically inside the VM as you said it will always receive the default value of the host? => No, it do not receive the default value of the host. It's the default value related with the virtual device driver on the guest. hostdev type interface will passthrough the physical interface or VF of the host to guest, it will get the device's parameters for rx and tx buffer.
------- Best Regards, Yalan Zhang IRC: yalzhang Internal phone: 8389413
On Thu, Oct 26, 2017 at 1:30 PM, Ashish Kurian <ashishbnv@gmail.com> wrote:
Hi Yalan,
Thank you for your response. I do not have the following packages installed
vhost backend driver qemu-kvm-rhev package
Are these packages available for free? How can I install them?
In my KVM VM, I must have an IP address to the interfaces that I am trying to increasing the buffers. That is the reason I was using macvtap (direct type interface). Is it possible to have my interfaces with an IP address inside the VM to be bridged to the physical interfaces on the host?
Is it also a possibility that I change the rx and tx buffer on the physical interface on the host and it is reflected automatically inside the VM as you said it will always receive the default value of the host?
Best Regards, Ashish Kurian
On Thu, Oct 26, 2017 at 6:45 AM, Yalan Zhang <yalzhang@redhat.com> wrote:
Hi Ashish,
I have tested with your xml in the first mail, and it works for rx_queue_size(see below). multiqueue need to work with vhost backend driver. And when you set "queues=1" it will ignored.
Please check your qemu-kvm-rhev package, should be newer than qemu-kvm-rhev-2.9.0-16.el7_4.2 And the logs?
tx_queue_size='512' will not work in the guest with direct type interface, in fact, no matter what you set, it will not work and guest will get the default '256'. We only support vhost-user backend to have more than 256. refer to https://libvirt.org/formatdomain.html#elementsNICSEthernet
tx_queue_size The optional tx_queue_size attribute controls the size of virtio ring for each queue as described above. The default value is hypervisor dependent and may change across its releases. Moreover, some hypervisors may pose some restrictions on actual value. For instance, QEMU v2.9 requires value to be a power of two from [256, 1024] range. In addition to that, this may work only for a subset of interface types, e.g. aforementioned QEMU enables this option only for vhostuser type. Since 3.7.0 (QEMU and KVM only) multiqueue only supports vhost as backend driver.
# rpm -q libvirt qemu-kvm-rhev libvirt-3.2.0-14.el7_4.3.x86_64 qemu-kvm-rhev-2.9.0-16.el7_4.9.x86_64
1. the xml as below <interface type='direct'> <mac address='52:54:00:00:b5:99'/> <source dev='eno1' mode='vepa'/> <model type='virtio'/> <driver name='vhost' queues='5' rx_queue_size='512' tx_queue_size='512'> <host csum='off' gso='off' tso4='off' tso6='off' ecn='off' ufo='off' mrg_rxbuf='off'/> <guest csum='off' tso4='off' tso6='off' ecn='off' ufo='off'/> </driver> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface>
2. after start the vm, check the qemu command line: *-netdev tap,fds=26:28:29:30:31,id=hostnet0,vhost=on,vhostfds=32:33:34:35:36* -device virtio-net-pci,csum=off,gso=off,host_tso4=off,host_tso6=off, host_ecn=off,host_ufo=off,mrg_rxbuf=off,guest_csum=off,guest _tso4=off,guest_tso6=off,guest_ecn=off,guest_ufo=off, *mq=on,vectors=12,rx_queue_size=512,tx_queue_size=512*,netdev=hostnet 0,id=net0,mac=52:54:00:00:b5:99,bus=pci.0,addr=0x3
3. check on guest # ethtool -g eth0 Ring parameters for eth0: Pre-set maximums: RX: *512 ==> rx_queue_size works* RX Mini: 0 RX Jumbo: 0 TX: *256 ===> no change* Current hardware settings: RX: *512 **==> rx_queue_size works* RX Mini: 0 RX Jumbo: 0 TX: *256 ===> no change*
# ethtool -l eth0 Channel parameters for eth0: Pre-set maximums: RX: 0 TX: 0 Other: 0 Combined: *5 ==> queues what we set* Current hardware settings: RX: 0 TX: 0 Other: 0 Combined: 1
If change to qemu as driver, # virsh edit rhel7 .. <interface type='direct'> <mac address='52:54:00:00:b5:99'/> <source dev='eno1' mode='vepa'/> <model type='virtio'/> <driver name='qemu' queues='5' rx_queue_size='512' tx_queue_size='512'> <host csum='off' gso='off' tso4='off' tso6='off' ecn='off' ufo='off' mrg_rxbuf='off'/> <guest csum='off' tso4='off' tso6='off' ecn='off' ufo='off'/> </driver> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> .. Domain rhel7 XML configuration edited. ==> the xml can validate and save
# virsh start rhel7 Domain rhel7 started
# virsh dumpxml rhel7 | grep /interface -B9 <source dev='eno1' mode='vepa'/> <target dev='macvtap0'/> <model type='virtio'/> *<driver name='qemu' queues='5' rx_queue_size='512' tx_queue_size='512'>* <host csum='off' gso='off' tso4='off' tso6='off' ecn='off' ufo='off' mrg_rxbuf='off'/> <guest csum='off' tso4='off' tso6='off' ecn='off' ufo='off'/> </driver> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface>
* -netdev tap,fds=26:28:29:30:31*,id=hostnet0 -device virtio-net-pci,csum=off,gso=off,host_tso4=off,host_tso6=off, host_ecn=off,host_ufo=off,mrg_rxbuf=off,guest_csum=off,guest _tso4=off,guest_tso6=off,guest_ecn=off,guest_ufo=off, *rx_queue_size=512,tx_queue_size=512*,netdev=hostnet0,id=net0,mac=52: 54:00:00:b5:99,bus=pci.0,addr=0x3
*"mq=on,vectors=12" is missing*, indicates there is no multiqueue
and check on guest
# ethtool -l eth0 Channel parameters for eth0: Pre-set maximums: RX: 0 TX: 0 Other: 0 Combined: 1 ==> no multiqueue Current hardware settings: RX: 0 TX: 0 Other: 0 Combined: 1
# ethtool -g eth0 Ring parameters for eth0: Pre-set maximums: RX: *512* RX Mini: 0 RX Jumbo: 0 TX: 256 Current hardware settings: RX: *512* RX Mini: 0 RX Jumbo: 0 TX: 256
------- Best Regards, Yalan Zhang IRC: yalzhang Internal phone: 8389413
On Thu, Oct 26, 2017 at 2:33 AM, Ashish Kurian <ashishbnv@gmail.com> wrote:
Hi Michal,
An update to what I have already said : when I try adding <driver name='qemu' txmode='iothread' ioeventfd='on' event_idx='off' queues='1' rx_queue_size='512' tx_queue_size='512'> although it showed me the error as mentioned, when I checked the xml again I saw that <driver name='qemu' txmode='iothread' ioeventfd='on' event_idx='off' > is added to the interface.
The missing parameters are : queues='1' rx_queue_size='512' tx_queue_size='512'
Best Regards, Ashish Kurian
On Wed, Oct 25, 2017 at 5:07 PM, Ashish Kurian <ashishbnv@gmail.com> wrote:
> Hi Michal, > > What I found was that when I restarted the machine and did a virsh > edit command to see the xml config, I see that it is was not actually > changed. This suggests why I saw 256 again after restarting. > > So now I tried again to edit the xml via virsh edit command and used > the following to set the parameters. > > <driver name='qemu' txmode='iothread' ioeventfd='on' event_idx='off' > queues='1' rx_queue_size='512' tx_queue_size='512'> > </driver> > > It was not accepted and I got the error saying : > > > error: XML document failed to validate against schema: Unable to > validate doc against /usr/share/libvirt/schemas/domain.rng > Extra element devices in interleave > Element domain failed to validate content > > What does this imply? I have two more other interfaces and do I have > to the same to them also? > > Btw, there are now logs generated now in the domain log or libvirtd > log > > > > > > Best Regards, > Ashish Kurian > > On Wed, Oct 25, 2017 at 2:50 PM, Michal Privoznik < > mprivozn@redhat.com> wrote: > >> On 10/25/2017 01:53 PM, Ashish Kurian wrote: >> > Dear Users/Developers, >> > >> > I am using a KVM Ubuntu VM as a degrader to apply specific delays >> to >> > incoming packets. As the delay for my packets can be higher than >> 7.5 >> > seconds, there is not enough buffer on my interface to buffer all >> the >> > packets. Therefore those overflowing packets are dropped in the >> machine and >> > not forwarded. >> > >> > When I tried to use the command ethtool -G ens8 rx 512 to >> increase the >> > buffer size, I get the following error. >> > >> > Cannot set device ring parameters: Operation not permitted >> > >> > I have kept the VM xml files as specified in the link : >> > https://libvirt.org/formatdomain.html. The value that I kept in >> my xml file >> > is as follows. >> > >> > <interface type='direct'> >> > <mac address='52:54:00:72:f9:eb'/> >> > <source dev='enp7s0f0' mode='vepa'/> >> > <model type='virtio'/> >> > <driver name='vhost' queues='5' rx_queue_size='512' >> > tx_queue_size='512'> >> > <host csum='off' gso='off' tso4='off' tso6='off' ecn='off' >> ufo='off' >> > mrg_rxbuf='off'/> >> > <guest csum='off' tso4='off' tso6='off' ecn='off' >> ufo='off'/> >> > </driver> >> > <address type='pci' domain='0x0000' bus='0x00' slot='0x08' >> > function='0x0'/> >> > </interface> >> > <interface type='direct'> >> > <mac address='52:54:00:00:b5:99'/> >> > <source dev='enp7s0f1' mode='vepa'/> >> > <model type='virtio'/> >> > <driver name='vhost' queues='5' rx_queue_size='512' >> > tx_queue_size='512'> >> > <host csum='off' gso='off' tso4='off' tso6='off' ecn='off' >> ufo='off' >> > mrg_rxbuf='off'/> >> > <guest csum='off' tso4='off' tso6='off' ecn='off' >> ufo='off'/> >> > </driver> >> > <address type='pci' domain='0x0000' bus='0x00' slot='0x09' >> > function='0x0'/> >> >> So what does the qemu command line look like? You can find it in >> either >> libvirtd log or domain log. >> >> http://wiki.libvirt.org/page/DebugLogs >> >> Michal >> > >
_______________________________________________ libvirt-users mailing list libvirt-users@redhat.com https://www.redhat.com/mailman/listinfo/libvirt-users
Hi Yalan and Michal, Thank you for your response. So what I understand is that I can change rx_queue size even if I use direct type interface and qemu driver as long as the driver is virtio. Am I right? If that is the case why am I getting the error saying that error: XML document failed to validate against schema: Unable to validate doc against /usr/share/libvirt/schemas/domain.rng Extra element devices in interleave Element domain failed to validate content for the following xml config? Btw I changed vepa to bridged mode hoping it would help me. Also note that I tried the following without any driver name and it still failed. <interface type='direct'> <mac address='52:54:00:72:f9:eb'/> <source dev='enp7s0f0' mode='bridge'/> <model type='virtio'/> <driver name='qemu' queues='5' rx_queue_size='512'> <host csum='off' gso='off' tso4='off' tso6='off' ecn='off' ufo='off' mrg_rxbuf='off'/> <guest csum='off' tso4='off' tso6='off' ecn='off' ufo='off'/> </driver> Best Regards, Ashish Kurian On Thu, Oct 26, 2017 at 10:01 AM, Yalan Zhang <yalzhang@redhat.com> wrote:
Hi Ashish,
IMO, it is yes, no way to increase tx_queue_size for direct type interface
------- Best Regards, Yalan Zhang IRC: yalzhang Internal phone: 8389413
On Thu, Oct 26, 2017 at 3:38 PM, Ashish Kurian <ashishbnv@gmail.com> wrote:
Hi Yalan,
In the previous email you mentioned "tx_queue_size='512' will not work in the guest with direct type interface, in fact, no matter what you set, it will not work and guest will get the default '256'. "
So if I am using macvtap for my interfaces, then the device type will always be direct type. Does it mean that there is no way I can increase the buffer size with the macvtap interfaces?
Best Regards, Ashish Kurian
On Thu, Oct 26, 2017 at 9:04 AM, Ashish Kurian <ashishbnv@gmail.com> wrote:
Hi Yalan,
Thank you for your comment on qemu-kvm-rhev
I am waiting for a response about my previous email with the logs attached. I do not understand what is the problem.
On Oct 26, 2017 8:58 AM, "Yalan Zhang" <yalzhang@redhat.com> wrote:
Hi Ashish,
Please never mind for qemu-kvm-rhev. qemu with the code base 2.10.0 will support the tx_queue_size and rx_queue_size.
Thank you~
------- Best Regards, Yalan Zhang IRC: yalzhang Internal phone: 8389413
On Thu, Oct 26, 2017 at 2:22 PM, Yalan Zhang <yalzhang@redhat.com> wrote:
Hi Ashish,
Are these packages available for free? How can I install them? => You did have vhost backend driver. Do not set <driver name='qemu'...>, by default it will use vhost as backend driver.
Is it possible to have my interfaces with an IP address inside the VM to be bridged to the physical interfaces on the host? => Yes, you can create a linux bridge with physical interface connected, and use bridge type interface. Refer to https://libvirt.org/formatdomain.html#elementsNICSBridge direct type is also ok (but your host and guest have no access to each other).
Is it also a possibility that I change the rx and tx buffer on the physical interface on the host and it is reflected automatically inside the VM as you said it will always receive the default value of the host? => No, it do not receive the default value of the host. It's the default value related with the virtual device driver on the guest. hostdev type interface will passthrough the physical interface or VF of the host to guest, it will get the device's parameters for rx and tx buffer.
------- Best Regards, Yalan Zhang IRC: yalzhang Internal phone: 8389413
On Thu, Oct 26, 2017 at 1:30 PM, Ashish Kurian <ashishbnv@gmail.com> wrote:
Hi Yalan,
Thank you for your response. I do not have the following packages installed
vhost backend driver qemu-kvm-rhev package
Are these packages available for free? How can I install them?
In my KVM VM, I must have an IP address to the interfaces that I am trying to increasing the buffers. That is the reason I was using macvtap (direct type interface). Is it possible to have my interfaces with an IP address inside the VM to be bridged to the physical interfaces on the host?
Is it also a possibility that I change the rx and tx buffer on the physical interface on the host and it is reflected automatically inside the VM as you said it will always receive the default value of the host?
Best Regards, Ashish Kurian
On Thu, Oct 26, 2017 at 6:45 AM, Yalan Zhang <yalzhang@redhat.com> wrote:
Hi Ashish,
I have tested with your xml in the first mail, and it works for rx_queue_size(see below). multiqueue need to work with vhost backend driver. And when you set "queues=1" it will ignored.
Please check your qemu-kvm-rhev package, should be newer than qemu-kvm-rhev-2.9.0-16.el7_4.2 And the logs?
tx_queue_size='512' will not work in the guest with direct type interface, in fact, no matter what you set, it will not work and guest will get the default '256'. We only support vhost-user backend to have more than 256. refer to https://libvirt.org/formatdomain.html#elementsNICSEthernet
tx_queue_size The optional tx_queue_size attribute controls the size of virtio ring for each queue as described above. The default value is hypervisor dependent and may change across its releases. Moreover, some hypervisors may pose some restrictions on actual value. For instance, QEMU v2.9 requires value to be a power of two from [256, 1024] range. In addition to that, this may work only for a subset of interface types, e.g. aforementioned QEMU enables this option only for vhostuser type. Since 3.7.0 (QEMU and KVM only) multiqueue only supports vhost as backend driver.
# rpm -q libvirt qemu-kvm-rhev libvirt-3.2.0-14.el7_4.3.x86_64 qemu-kvm-rhev-2.9.0-16.el7_4.9.x86_64
1. the xml as below <interface type='direct'> <mac address='52:54:00:00:b5:99'/> <source dev='eno1' mode='vepa'/> <model type='virtio'/> <driver name='vhost' queues='5' rx_queue_size='512' tx_queue_size='512'> <host csum='off' gso='off' tso4='off' tso6='off' ecn='off' ufo='off' mrg_rxbuf='off'/> <guest csum='off' tso4='off' tso6='off' ecn='off' ufo='off'/> </driver> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface>
2. after start the vm, check the qemu command line: *-netdev tap,fds=26:28:29:30:31,id=hostnet0,vhost=on,vhostfds=32:33:34:35:36* -device virtio-net-pci,csum=off,gso=off,host_tso4=off,host_tso6=off, host_ecn=off,host_ufo=off,mrg_rxbuf=off,guest_csum=off,guest _tso4=off,guest_tso6=off,guest_ecn=off,guest_ufo=off, *mq=on,vectors=12,rx_queue_size=512,tx_queue_size=512*,netdev=hostnet 0,id=net0,mac=52:54:00:00:b5:99,bus=pci.0,addr=0x3
3. check on guest # ethtool -g eth0 Ring parameters for eth0: Pre-set maximums: RX: *512 ==> rx_queue_size works* RX Mini: 0 RX Jumbo: 0 TX: *256 ===> no change* Current hardware settings: RX: *512 **==> rx_queue_size works* RX Mini: 0 RX Jumbo: 0 TX: *256 ===> no change*
# ethtool -l eth0 Channel parameters for eth0: Pre-set maximums: RX: 0 TX: 0 Other: 0 Combined: *5 ==> queues what we set* Current hardware settings: RX: 0 TX: 0 Other: 0 Combined: 1
If change to qemu as driver, # virsh edit rhel7 .. <interface type='direct'> <mac address='52:54:00:00:b5:99'/> <source dev='eno1' mode='vepa'/> <model type='virtio'/> <driver name='qemu' queues='5' rx_queue_size='512' tx_queue_size='512'> <host csum='off' gso='off' tso4='off' tso6='off' ecn='off' ufo='off' mrg_rxbuf='off'/> <guest csum='off' tso4='off' tso6='off' ecn='off' ufo='off'/> </driver> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> .. Domain rhel7 XML configuration edited. ==> the xml can validate and save
# virsh start rhel7 Domain rhel7 started
# virsh dumpxml rhel7 | grep /interface -B9 <source dev='eno1' mode='vepa'/> <target dev='macvtap0'/> <model type='virtio'/> *<driver name='qemu' queues='5' rx_queue_size='512' tx_queue_size='512'>* <host csum='off' gso='off' tso4='off' tso6='off' ecn='off' ufo='off' mrg_rxbuf='off'/> <guest csum='off' tso4='off' tso6='off' ecn='off' ufo='off'/> </driver> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface>
* -netdev tap,fds=26:28:29:30:31*,id=hostnet0 -device virtio-net-pci,csum=off,gso=off,host_tso4=off,host_tso6=off, host_ecn=off,host_ufo=off,mrg_rxbuf=off,guest_csum=off,guest _tso4=off,guest_tso6=off,guest_ecn=off,guest_ufo=off, *rx_queue_size=512,tx_queue_size=512*,netdev=hostnet0,id=net0,mac=52: 54:00:00:b5:99,bus=pci.0,addr=0x3
*"mq=on,vectors=12" is missing*, indicates there is no multiqueue
and check on guest
# ethtool -l eth0 Channel parameters for eth0: Pre-set maximums: RX: 0 TX: 0 Other: 0 Combined: 1 ==> no multiqueue Current hardware settings: RX: 0 TX: 0 Other: 0 Combined: 1
# ethtool -g eth0 Ring parameters for eth0: Pre-set maximums: RX: *512* RX Mini: 0 RX Jumbo: 0 TX: 256 Current hardware settings: RX: *512* RX Mini: 0 RX Jumbo: 0 TX: 256
------- Best Regards, Yalan Zhang IRC: yalzhang Internal phone: 8389413
On Thu, Oct 26, 2017 at 2:33 AM, Ashish Kurian <ashishbnv@gmail.com> wrote:
> Hi Michal, > > An update to what I have already said : when I try adding <driver > name='qemu' txmode='iothread' ioeventfd='on' event_idx='off' queues='1' > rx_queue_size='512' tx_queue_size='512'> although it showed me the error as > mentioned, when I checked the xml again I saw that <driver > name='qemu' txmode='iothread' ioeventfd='on' event_idx='off' > is added to > the interface. > > The missing parameters are : queues='1' rx_queue_size='512' > tx_queue_size='512' > > Best Regards, > Ashish Kurian > > On Wed, Oct 25, 2017 at 5:07 PM, Ashish Kurian <ashishbnv@gmail.com> > wrote: > >> Hi Michal, >> >> What I found was that when I restarted the machine and did a virsh >> edit command to see the xml config, I see that it is was not actually >> changed. This suggests why I saw 256 again after restarting. >> >> So now I tried again to edit the xml via virsh edit command and >> used the following to set the parameters. >> >> <driver name='qemu' txmode='iothread' ioeventfd='on' >> event_idx='off' queues='1' rx_queue_size='512' tx_queue_size='512'> >> </driver> >> >> It was not accepted and I got the error saying : >> >> >> error: XML document failed to validate against schema: Unable to >> validate doc against /usr/share/libvirt/schemas/domain.rng >> Extra element devices in interleave >> Element domain failed to validate content >> >> What does this imply? I have two more other interfaces and do I >> have to the same to them also? >> >> Btw, there are now logs generated now in the domain log or libvirtd >> log >> >> >> >> >> >> Best Regards, >> Ashish Kurian >> >> On Wed, Oct 25, 2017 at 2:50 PM, Michal Privoznik < >> mprivozn@redhat.com> wrote: >> >>> On 10/25/2017 01:53 PM, Ashish Kurian wrote: >>> > Dear Users/Developers, >>> > >>> > I am using a KVM Ubuntu VM as a degrader to apply specific >>> delays to >>> > incoming packets. As the delay for my packets can be higher than >>> 7.5 >>> > seconds, there is not enough buffer on my interface to buffer >>> all the >>> > packets. Therefore those overflowing packets are dropped in the >>> machine and >>> > not forwarded. >>> > >>> > When I tried to use the command ethtool -G ens8 rx 512 to >>> increase the >>> > buffer size, I get the following error. >>> > >>> > Cannot set device ring parameters: Operation not permitted >>> > >>> > I have kept the VM xml files as specified in the link : >>> > https://libvirt.org/formatdomain.html. The value that I kept in >>> my xml file >>> > is as follows. >>> > >>> > <interface type='direct'> >>> > <mac address='52:54:00:72:f9:eb'/> >>> > <source dev='enp7s0f0' mode='vepa'/> >>> > <model type='virtio'/> >>> > <driver name='vhost' queues='5' rx_queue_size='512' >>> > tx_queue_size='512'> >>> > <host csum='off' gso='off' tso4='off' tso6='off' ecn='off' >>> ufo='off' >>> > mrg_rxbuf='off'/> >>> > <guest csum='off' tso4='off' tso6='off' ecn='off' >>> ufo='off'/> >>> > </driver> >>> > <address type='pci' domain='0x0000' bus='0x00' slot='0x08' >>> > function='0x0'/> >>> > </interface> >>> > <interface type='direct'> >>> > <mac address='52:54:00:00:b5:99'/> >>> > <source dev='enp7s0f1' mode='vepa'/> >>> > <model type='virtio'/> >>> > <driver name='vhost' queues='5' rx_queue_size='512' >>> > tx_queue_size='512'> >>> > <host csum='off' gso='off' tso4='off' tso6='off' ecn='off' >>> ufo='off' >>> > mrg_rxbuf='off'/> >>> > <guest csum='off' tso4='off' tso6='off' ecn='off' >>> ufo='off'/> >>> > </driver> >>> > <address type='pci' domain='0x0000' bus='0x00' slot='0x09' >>> > function='0x0'/> >>> >>> So what does the qemu command line look like? You can find it in >>> either >>> libvirtd log or domain log. >>> >>> http://wiki.libvirt.org/page/DebugLogs >>> >>> Michal >>> >> >> > > _______________________________________________ > libvirt-users mailing list > libvirt-users@redhat.com > https://www.redhat.com/mailman/listinfo/libvirt-users >
On 10/26/2017 10:38 AM, Ashish Kurian wrote:
Hi Yalan and Michal,
Thank you for your response. So what I understand is that I can change rx_queue size even if I use direct type interface and qemu driver as long as the driver is virtio. Am I right?
Yes.
If that is the case why am I getting the error saying that
error: XML document failed to validate against schema: Unable to validate doc against /usr/share/libvirt/schemas/domain.rng Extra element devices in interleave Element domain failed to validate content
What's your libvirt version? For rx_queue_size you need 2.3.0 or newer. For tx_queue_size you need 3.7.0 or newer (not that there's one yet).
for the following xml config? Btw I changed vepa to bridged mode hoping it would help me. Also note that I tried the following without any driver name and it still failed.
<interface type='direct'> <mac address='52:54:00:72:f9:eb'/> <source dev='enp7s0f0' mode='bridge'/> <model type='virtio'/> <driver name='qemu' queues='5' rx_queue_size='512'> <host csum='off' gso='off' tso4='off' tso6='off' ecn='off' ufo='off' mrg_rxbuf='off'/> <guest csum='off' tso4='off' tso6='off' ecn='off' ufo='off'/> </driver>
This works for me. I'm running recent git. Michal
Hi Michal, Mine is iplab@iceman:~$ libvirtd --version libvirtd (libvirt) 1.3.1 I think this can be the issue. I will update and try again Best Regards, Ashish Kurian On Thu, Oct 26, 2017 at 10:46 AM, Michal Privoznik <mprivozn@redhat.com> wrote:
On 10/26/2017 10:38 AM, Ashish Kurian wrote:
Hi Yalan and Michal,
Thank you for your response. So what I understand is that I can change rx_queue size even if I use direct type interface and qemu driver as long as the driver is virtio. Am I right?
Yes.
If that is the case why am I getting the error saying that
error: XML document failed to validate against schema: Unable to validate doc against /usr/share/libvirt/schemas/domain.rng Extra element devices in interleave Element domain failed to validate content
What's your libvirt version? For rx_queue_size you need 2.3.0 or newer. For tx_queue_size you need 3.7.0 or newer (not that there's one yet).
for the following xml config? Btw I changed vepa to bridged mode hoping
it
would help me. Also note that I tried the following without any driver name and it still failed.
<interface type='direct'> <mac address='52:54:00:72:f9:eb'/> <source dev='enp7s0f0' mode='bridge'/> <model type='virtio'/> <driver name='qemu' queues='5' rx_queue_size='512'> <host csum='off' gso='off' tso4='off' tso6='off' ecn='off' ufo='off' mrg_rxbuf='off'/> <guest csum='off' tso4='off' tso6='off' ecn='off' ufo='off'/> </driver>
This works for me. I'm running recent git.
Michal
Hi Michal, I installed the latest version from the git and did run the libvirt from the new directory using /home/to/your/checkout/daemon/libvirtd But I still see that the version is 1.3.1. I have also stopped the already running libvirt daemon and ran the new daemon from the directory and it is showing 1.3.1 Best Regards, Ashish Kurian On Thu, Oct 26, 2017 at 10:50 AM, Ashish Kurian <ashishbnv@gmail.com> wrote:
Hi Michal,
Mine is
iplab@iceman:~$ libvirtd --version libvirtd (libvirt) 1.3.1
I think this can be the issue. I will update and try again
Best Regards, Ashish Kurian
On Thu, Oct 26, 2017 at 10:46 AM, Michal Privoznik <mprivozn@redhat.com> wrote:
On 10/26/2017 10:38 AM, Ashish Kurian wrote:
Hi Yalan and Michal,
Thank you for your response. So what I understand is that I can change rx_queue size even if I use direct type interface and qemu driver as long as the driver is virtio. Am I right?
Yes.
If that is the case why am I getting the error saying that
error: XML document failed to validate against schema: Unable to validate doc against /usr/share/libvirt/schemas/domain.rng Extra element devices in interleave Element domain failed to validate content
What's your libvirt version? For rx_queue_size you need 2.3.0 or newer. For tx_queue_size you need 3.7.0 or newer (not that there's one yet).
for the following xml config? Btw I changed vepa to bridged mode hoping
it
would help me. Also note that I tried the following without any driver name and it still failed.
<interface type='direct'> <mac address='52:54:00:72:f9:eb'/> <source dev='enp7s0f0' mode='bridge'/> <model type='virtio'/> <driver name='qemu' queues='5' rx_queue_size='512'> <host csum='off' gso='off' tso4='off' tso6='off' ecn='off' ufo='off' mrg_rxbuf='off'/> <guest csum='off' tso4='off' tso6='off' ecn='off' ufo='off'/> </driver>
This works for me. I'm running recent git.
Michal
On 10/26/2017 12:18 PM, Ashish Kurian wrote:
Hi Michal,
I installed the latest version from the git and did run the libvirt from the new directory using /home/to/your/checkout/daemon/libvirtd
Try: libvirt.git $ ./autogen.sh --system && make;
From a root terminal then:
libvirt.git # ./run ./daemon/libvirtd Now, if you check the version (virsh version --daemon) you should see the new version. Although, it'd be probably better if you build the package for your distro and install it. Michal
participants (3)
-
Ashish Kurian -
Michal Privoznik -
Yalan Zhang