[libvirt] vhost-net configuration

Hi everyone, I'm trying to test out the virtio driver's "tx=bh" option, or iothread in libvirt. When I create a machine, I get the "unsupported" error message: # virsh create web101.xml error: Failed to create domain from web101.xml error: unsupported configuration: vhost-net is not supported with this QEMU binary My interface XML looks like this: <interface type='bridge'> <mac address='52:54:00:6f:51:43'/> <source bridge='vnet0'/> <model type='virtio'/> <driver name='vhost' txmode='iothread'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> I've compiled libvirt (0.9.1) and QEMU (0.14.1) from source, but am still using my distro's kernel, 2.6.32-71.29.1.el6.x86_64. (Scientific Linux 6) Thanks, and please let me know if you need any more info! --Cal

I should also add that I did compile qemu with vhost-net enabled: # /usr/local/bin/qemu-system-x86_64 -device virtio-net-pci,? virtio-net-pci.ioeventfd=on/off virtio-net-pci.vectors=uint32 virtio-net-pci.indirect_desc=on/off virtio-net-pci.csum=on/off virtio-net-pci.guest_csum=on/off virtio-net-pci.gso=on/off virtio-net-pci.guest_tso4=on/off virtio-net-pci.guest_tso6=on/off virtio-net-pci.guest_ecn=on/off virtio-net-pci.guest_ufo=on/off virtio-net-pci.host_tso4=on/off virtio-net-pci.host_tso6=on/off virtio-net-pci.host_ecn=on/off virtio-net-pci.host_ufo=on/off virtio-net-pci.mrg_rxbuf=on/off virtio-net-pci.status=on/off virtio-net-pci.ctrl_vq=on/off virtio-net-pci.ctrl_rx=on/off virtio-net-pci.ctrl_vlan=on/off virtio-net-pci.ctrl_rx_extra=on/off virtio-net-pci.mac=macaddr virtio-net-pci.vlan=vlan virtio-net-pci.netdev=netdev virtio-net-pci.bootindex=int32 virtio-net-pci.x-txtimer=uint32 virtio-net-pci.x-txburst=int32 virtio-net-pci.tx=string On Fri, May 20, 2011 at 11:15 AM, Cal Heldenbrand <cal@fbsdata.com> wrote:
Hi everyone,
I'm trying to test out the virtio driver's "tx=bh" option, or iothread in libvirt. When I create a machine, I get the "unsupported" error message:
# virsh create web101.xml error: Failed to create domain from web101.xml error: unsupported configuration: vhost-net is not supported with this QEMU binary
My interface XML looks like this:
<interface type='bridge'> <mac address='52:54:00:6f:51:43'/> <source bridge='vnet0'/> <model type='virtio'/> <driver name='vhost' txmode='iothread'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface>
I've compiled libvirt (0.9.1) and QEMU (0.14.1) from source, but am still using my distro's kernel, 2.6.32-71.29.1.el6.x86_64. (Scientific Linux 6)
Thanks, and please let me know if you need any more info!
--Cal

On 05/20/2011 12:15 PM, Cal Heldenbrand wrote:
Hi everyone,
I'm trying to test out the virtio driver's "tx=bh" option, or iothread in libvirt. When I create a machine, I get the "unsupported" error message:
# virsh create web101.xml error: Failed to create domain from web101.xml error: unsupported configuration: vhost-net is not supported with this QEMU binary
First look in /var/log/libvirt/qemu/<domainname>.log and see which qemu binary is being executed by libvirt - it may not be the one you've built. Then, run "qemu-kvm --help" (replacing "qemu-kvm" with the path to the binary that libvirt is executing, found in the previous paragraph), and search through that output for the string ",vhost=" - if it's not there, then your qemu doesn't support vhost-net. If that string *is* there, then we need to do some more investigating, starting with you sending the output of "qemu-kvm --help".
My interface XML looks like this:
<interface type='bridge'> <mac address='52:54:00:6f:51:43'/> <source bridge='vnet0'/> <model type='virtio'/> <driver name='vhost' txmode='iothread'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface>
I've compiled libvirt (0.9.1) and QEMU (0.14.1) from source, but am still using my distro's kernel, 2.6.32-71.29.1.el6.x86_64. (Scientific Linux 6)
Thanks, and please let me know if you need any more info!
--Cal
-- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list

First look in /var/log/libvirt/qemu/<domainname>.log and see which qemu
binary is being executed by libvirt - it may not be the one you've built.
It looks correct... I pulled out the driver tag from my config and fired it up. Here is the full command line for it: LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin:/usr/local/sbin:/usr/local/bin QEMU_AUDIO_DRV=none /usr/local/bin/qemu-system-x86_64 -S -M pc-0.14 -enable-kvm -m 4096 -smp 7,sockets=7,cores=1,threads=1 -name web101 -uuid bda63821-ccc4-6607-67c0-db6712526cbd -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/web101.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=readline -rtc base=utc -boot c -drive file=/var/lib/libvirt/images/web101.img,if=none,id=drive-virtio-disk0,format=raw,cache=writethrough -device virtio-blk-pci,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0 -drive file=/home/httpd.img,if=none,id=drive-virtio-disk1,format=raw,cache=writethrough -device virtio-blk-pci,bus=pci.0,addr=0x7,drive=drive-virtio-disk1,id=virtio-disk1 -drive if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -fsdev local,security_model=passthrough,id=fsdev-fs0,path=/home/httpd -device virtio-9p-pci,id=fs0,fsdev=fsdev-fs0,mount_tag=/home/httpd,bus=pci.0,addr=0x8 -netdev tap,fd=15,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:6f:51:43,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -usb -device usb-tablet,id=input0 -vnc 127.0.0.1:0 -vga cirrus -device AC97,id=sound0,bus=pci.0,addr=0x4 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6
Then, run "qemu-kvm --help" (replacing "qemu-kvm" with the path to the binary that libvirt is executing, found in the previous paragraph), and search through that output for the string ",vhost=" - if it's not there, then your qemu doesn't support vhost-net.
This also looks correct. Is it wrong that my qemu compilation didn't create a binary named "qemu-kvm"? (Even though it said "KVM support yes" on the configure output?) # /usr/local/bin/qemu-system-x86_64 --help |grep vhost -net tap[,vlan=n][,name=str][,fd=h][,ifname=name][,script=file][,downscript=dfile][,sndbuf=nbytes][,vnet_hdr=on|off][,vhost=on|off][,vhostfd=h][,vhostforce=on|off] use vhost=on to enable experimental in kernel accelerator use vhostforce=on to force vhost on for non-MSIX virtio guests use 'vhostfd=h' to connect to an already opened vhost net device
If that string *is* there, then we need to do some more investigating, starting with you sending the output of "qemu-kvm --help".
Here's the full output for you. # /usr/local/bin/qemu-system-x86_64 --help QEMU emulator version 0.14.1, Copyright (c) 2003-2008 Fabrice Bellard usage: qemu [options] [disk_image] 'disk_image' is a raw hard disk image for IDE hard disk 0 Standard options: -h or -help display this help and exit -version display version information and exit -M machine select emulated machine (-M ? for list) -cpu cpu select CPU (-cpu ? for list) -smp n[,maxcpus=cpus][,cores=cores][,threads=threads][,sockets=sockets] set the number of CPUs to 'n' [default=1] maxcpus= maximum number of total cpus, including offline CPUs for hotplug, etc cores= number of CPU cores on one socket threads= number of threads on one CPU core sockets= number of discrete sockets in the system -numa node[,mem=size][,cpus=cpu[-cpu]][,nodeid=node] -fda/-fdb file use 'file' as floppy disk 0/1 image -hda/-hdb file use 'file' as IDE hard disk 0/1 image -hdc/-hdd file use 'file' as IDE hard disk 2/3 image -cdrom file use 'file' as IDE cdrom image (cdrom is ide1 master) -drive [file=file][,if=type][,bus=n][,unit=m][,media=d][,index=i] [,cyls=c,heads=h,secs=s[,trans=t]][,snapshot=on|off] [,cache=writethrough|writeback|none|unsafe][,format=f] [,serial=s][,addr=A][,id=name][,aio=threads|native] [,readonly=on|off] use 'file' as a drive image -set group.id.arg=value set <arg> parameter for item <id> of type <group> i.e. -set drive.$id.file=/path/to/image -global driver.property=value set a global default for a driver property -mtdblock file use 'file' as on-board Flash memory image -sd file use 'file' as SecureDigital card image -pflash file use 'file' as a parallel flash image -boot [order=drives][,once=drives][,menu=on|off] 'drives': floppy (a), hard disk (c), CD-ROM (d), network (n) -snapshot write to temporary files instead of disk image files -m megs set virtual RAM size to megs MB [default=128] -mem-path FILE provide backing storage for guest RAM -mem-prealloc preallocate guest memory (use with -mem-path) -k language use keyboard layout (for example 'fr' for French) -audio-help print list of audio drivers and their options -soundhw c1,... enable audio support and only specified sound cards (comma separated list) use -soundhw ? to get the list of supported cards use -soundhw all to enable all of them -usb enable the USB driver (will be the default soon) -usbdevice name add the host or guest USB device 'name' -device driver[,prop[=value][,...]] add device (based on driver) prop=value,... sets driver properties use -device ? to print all possible drivers use -device driver,? to print all possible properties File system options: -fsdev local,id=id,path=path,security_model=[mapped|passthrough|none] Virtual File system pass-through options: -virtfs local,path=path,mount_tag=tag,security_model=[mapped|passthrough|none] -name string1[,process=string2] set the name of the guest string1 sets the window title and string2 the process name (on Linux) -uuid %08x-%04x-%04x-%04x-%012x specify machine UUID Display options: -nographic disable graphical output and redirect serial I/Os to console -spice <args> enable spice -portrait rotate graphical output 90 deg left (only PXA LCD) -vga [std|cirrus|vmware|qxl|xenfb|none] select video card type -full-screen start in full screen -g WxH[xDEPTH] Set the initial graphical resolution and depth -vnc display start a VNC server on display i386 target only: -win2k-hack use it when installing Windows 2000 to avoid a disk full bug -no-fd-bootchk disable boot signature checking for floppy disks -no-acpi disable ACPI -no-hpet disable HPET -balloon none disable balloon device -balloon virtio[,addr=str] enable virtio balloon device (default) -acpitable [sig=str][,rev=n][,oem_id=str][,oem_table_id=str][,oem_rev=n][,asl_compiler_id=str][,asl_compiler_rev=n][,data=file1[:file2]...] ACPI table description -smbios file=binary load SMBIOS entry from binary file -smbios type=0[,vendor=str][,version=str][,date=str][,release=%d.%d] specify SMBIOS type 0 fields -smbios type=1[,manufacturer=str][,product=str][,version=str][,serial=str] [,uuid=uuid][,sku=str][,family=str] specify SMBIOS type 1 fields Network options: -net nic[,vlan=n][,macaddr=mac][,model=type][,name=str][,addr=str][,vectors=v] create a new Network Interface Card and connect it to VLAN 'n' -net user[,vlan=n][,name=str][,net=addr[/mask]][,host=addr][,restrict=y|n] [,hostname=host][,dhcpstart=addr][,dns=addr][,tftp=dir][,bootfile=f] [,hostfwd=rule][,guestfwd=rule][,smb=dir[,smbserver=addr]] connect the user mode network stack to VLAN 'n', configure its DHCP server and enabled optional services -net tap[,vlan=n][,name=str][,fd=h][,ifname=name][,script=file][,downscript=dfile][,sndbuf=nbytes][,vnet_hdr=on|off][,vhost=on|off][,vhostfd=h][,vhostforce=on|off] connect the host TAP network interface to VLAN 'n' and use the network scripts 'file' (default=/etc/qemu-ifup) and 'dfile' (default=/etc/qemu-ifdown) use '[down]script=no' to disable script execution use 'fd=h' to connect to an already opened TAP interface use 'sndbuf=nbytes' to limit the size of the send buffer (the default is disabled 'sndbuf=0' to enable flow control set 'sndbuf=1048576') use vnet_hdr=off to avoid enabling the IFF_VNET_HDR tap flag use vnet_hdr=on to make the lack of IFF_VNET_HDR support an error condition use vhost=on to enable experimental in kernel accelerator (only has effect for virtio guests which use MSIX) use vhostforce=on to force vhost on for non-MSIX virtio guests use 'vhostfd=h' to connect to an already opened vhost net device -net socket[,vlan=n][,name=str][,fd=h][,listen=[host]:port][,connect=host:port] connect the vlan 'n' to another VLAN using a socket connection -net socket[,vlan=n][,name=str][,fd=h][,mcast=maddr:port[,localaddr=addr]] connect the vlan 'n' to multicast maddr and port use 'localaddr=addr' to specify the host address to send packets from -net dump[,vlan=n][,file=f][,len=n] dump traffic on vlan 'n' to file 'f' (max n bytes per packet) -net none use it alone to have zero network devices. If no -net option is provided, the default is '-net nic -net user' -netdev [user|tap|socket],id=str[,option][,option][,...] Character device options: -chardev null,id=id[,mux=on|off] -chardev socket,id=id[,host=host],port=host[,to=to][,ipv4][,ipv6][,nodelay] [,server][,nowait][,telnet][,mux=on|off] (tcp) -chardev socket,id=id,path=path[,server][,nowait][,telnet],[mux=on|off] (unix) -chardev udp,id=id[,host=host],port=port[,localaddr=localaddr] [,localport=localport][,ipv4][,ipv6][,mux=on|off] -chardev msmouse,id=id[,mux=on|off] -chardev vc,id=id[[,width=width][,height=height]][[,cols=cols][,rows=rows]] [,mux=on|off] -chardev file,id=id,path=path[,mux=on|off] -chardev pipe,id=id,path=path[,mux=on|off] -chardev pty,id=id[,mux=on|off] -chardev stdio,id=id[,mux=on|off][,signal=on|off] -chardev tty,id=id,path=path[,mux=on|off] -chardev parport,id=id,path=path[,mux=on|off] Bluetooth(R) options: -bt hci,null dumb bluetooth HCI - doesn't respond to commands -bt hci,host[:id] use host's HCI with the given name -bt hci[,vlan=n] emulate a standard HCI in virtual scatternet 'n' -bt vhci[,vlan=n] add host computer to virtual scatternet 'n' using VHCI -bt device:dev[,vlan=n] emulate a bluetooth device 'dev' in scatternet 'n' Linux/Multiboot boot specific: -kernel bzImage use 'bzImage' as kernel image -append cmdline use 'cmdline' as kernel command line -initrd file use 'file' as initial ram disk Debug/Expert options: -serial dev redirect the serial port to char device 'dev' -parallel dev redirect the parallel port to char device 'dev' -monitor dev redirect the monitor to char device 'dev' -qmp dev like -monitor but opens in 'control' mode -mon chardev=[name][,mode=readline|control][,default] -debugcon dev redirect the debug console to char device 'dev' -pidfile file write PID to 'file' -singlestep always run in singlestep mode -S freeze CPU at startup (use 'c' to start execution) -gdb dev wait for gdb connection on 'dev' -s shorthand for -gdb tcp::1234 -d item1,... output log to /tmp/qemu.log (use -d ? for a list of log items) -hdachs c,h,s[,t] force hard disk 0 physical geometry and the optional BIOS translation (t=none or lba) (usually qemu can guess them) -L path set the directory for the BIOS, VGA BIOS and keymaps -bios file set the filename for the BIOS -enable-kvm enable KVM full virtualization support -xen-domid id specify xen guest domain id -xen-create create domain using xen hypercalls, bypassing xend warning: should not be used when xend is in use -xen-attach attach to existing xen domain xend will use this when starting qemu -no-reboot exit instead of rebooting -no-shutdown stop before shutdown -loadvm [tag|id] start right away with a saved state (loadvm in monitor) -daemonize daemonize QEMU after initializing -option-rom rom load a file, rom, into the option ROM space -clock force the use of the given methods for timer alarm. To see what timers are available use -clock ? -rtc [base=utc|localtime|date][,clock=host|vm][,driftfix=none|slew] set the RTC base and clock, enable drift fix for clock ticks (x86 only) -icount [N|auto] enable virtual instruction counter with 2^N clock ticks per instruction -watchdog i6300esb|ib700 enable virtual hardware watchdog [default=none] -watchdog-action reset|shutdown|poweroff|pause|debug|none action when watchdog fires [default=reset] -echr chr set terminal escape character instead of ctrl-a -virtioconsole c set virtio console -show-cursor show cursor -tb-size n set TB size -incoming p prepare for incoming migration, listen on port p -nodefaults don't create default devices -chroot dir chroot to dir just before starting the VM -runas user change to user id user just before starting the VM -prom-env variable=value set OpenBIOS nvram variables -semihosting semihosting mode -old-param old param mode -readconfig <file> -writeconfig <file> read/write config file -nodefconfig do not load default config files at startup During emulation, the following keys are useful: ctrl-alt-f toggle full screen ctrl-alt-n switch to virtual console 'n' ctrl-alt toggle mouse and keyboard grab When using -nographic, press 'ctrl-a h' to get some help. Thanks for the help! --Cal

On 05/20/2011 01:12 PM, Cal Heldenbrand wrote:
First look in /var/log/libvirt/qemu/<domainname>.log and see which qemu binary is being executed by libvirt - it may not be the one you've built.
It looks correct... I pulled out the driver tag from my config and fired it up. Here is the full command line for it:
LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin:/usr/local/sbin:/usr/local/bin QEMU_AUDIO_DRV=none /usr/local/bin/qemu-system-x86_64 -S -M pc-0.14 -enable-kvm -m 4096 -smp 7,sockets=7,cores=1,threads=1 -name web101 -uuid bda63821-ccc4-6607-67c0-db6712526cbd -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/web101.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=readline -rtc base=utc -boot c -drive file=/var/lib/libvirt/images/web101.img,if=none,id=drive-virtio-disk0,format=raw,cache=writethrough -device virtio-blk-pci,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0 -drive file=/home/httpd.img,if=none,id=drive-virtio-disk1,format=raw,cache=writethrough -device virtio-blk-pci,bus=pci.0,addr=0x7,drive=drive-virtio-disk1,id=virtio-disk1 -drive if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -fsdev local,security_model=passthrough,id=fsdev-fs0,path=/home/httpd -device virtio-9p-pci,id=fs0,fsdev=fsdev-fs0,mount_tag=/home/httpd,bus=pci.0,addr=0x8 -netdev tap,fd=15,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:6f:51:43,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -usb -device usb-tablet,id=input0 -vnc 127.0.0.1:0 <http://127.0.0.1:0> -vga cirrus -device AC97,id=sound0,bus=pci.0,addr=0x4 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6
Then, run "qemu-kvm --help" (replacing "qemu-kvm" with the path to the binary that libvirt is executing, found in the previous paragraph), and search through that output for the string ",vhost=" - if it's not there, then your qemu doesn't support vhost-net.
This also looks correct. Is it wrong that my qemu compilation didn't create a binary named "qemu-kvm"? (Even though it said "KVM support yes" on the configure output?)
Not exactly, but close. In the qemu capabilities code, there is a variable called "is_kvm" that is only set if it finds the string "(qemu-kvm-" or "(kvm-" in the help output. It seems to me that this used to be used for a lot more things, but when I look now I see that it's used for very few things (maybe a result of the capabilities code refactoring, or maybe my memory is faulty :-). However, one of the things still in place is that QEMU_CAPS_VNET_HOST isn't set unless is_kvm is true. I believe this dependency on is_kvm is there because someone said that the feature was only in kvm, but retrospectively that doesn't make sense. Can you try building with the following patch, and see if that works? diff --git a/src/qemu/qemu_capabilities.c b/src/qemu/qemu_capabilities.c index ea55df5..71a54a5 100644 --- a/src/qemu/qemu_capabilities.c +++ b/src/qemu/qemu_capabilities.c @@ -982,7 +982,7 @@ qemuCapsComputeCmdFlags(const char *help, if (is_kvm && (version >= 10000 || kvm_version >= 74)) qemuCapsSet(flags, QEMU_CAPS_VNET_HDR); - if (is_kvm && strstr(help, ",vhost=")) { + if (strstr(help, ",vhost=")) { qemuCapsSet(flags, QEMU_CAPS_VNET_HOST); }

Hi Laine, It appears that patch worked! I grabbed the latest git source for both libvirt and qemu, applied your patch and reinstalled. After running through a few more setup stuff, I managed to get my web machine up and running with tx=bh. However, I didn't notice any latency improvements compared to timer mode. My test is pretty basic: I'm connecting to a mysql server, running a simple query, then disconnecting. On the host guy, this process takes around 5ms. On the virtual machine, it's around 15ms for both iothread and timer modes. I'm also comparing this with my other Xen machines in production, which do the same task in around 12 - 16ms. I know it sounds like small potatoes, but over the course of running our somewhat-inefficient webapp, it adds up to around 100ms of overhead for KVM vs a bare hardware machine. Any suggestions on that? Thanks! --Cal
Not exactly, but close. In the qemu capabilities code, there is a variable called "is_kvm" that is only set if it finds the string "(qemu-kvm-" or "(kvm-" in the help output. It seems to me that this used to be used for a lot more things, but when I look now I see that it's used for very few things (maybe a result of the capabilities code refactoring, or maybe my memory is faulty :-). However, one of the things still in place is that QEMU_CAPS_VNET_HOST isn't set unless is_kvm is true.
I believe this dependency on is_kvm is there because someone said that the feature was only in kvm, but retrospectively that doesn't make sense. Can you try building with the following patch, and see if that works?
diff --git a/src/qemu/qemu_capabilities.c b/src/qemu/qemu_capabilities.c index ea55df5..71a54a5 100644 --- a/src/qemu/qemu_capabilities.c +++ b/src/qemu/qemu_capabilities.c @@ -982,7 +982,7 @@ qemuCapsComputeCmdFlags(const char *help, if (is_kvm && (version >= 10000 || kvm_version >= 74)) qemuCapsSet(flags, QEMU_CAPS_VNET_HDR);
- if (is_kvm && strstr(help, ",vhost=")) { + if (strstr(help, ",vhost=")) { qemuCapsSet(flags, QEMU_CAPS_VNET_HOST); }
participants (2)
-
Cal Heldenbrand
-
Laine Stump