- Users - libvirt List Archives

[libvirt-users] NUMA issues on virtualized hosts
by Lukas Hejtmanek 21 Sep '18

21 Sep '18

Hello, I have cluster with AMD EPYC 7351 cpu. Two CPUs per node. I have performance 8-NUMA configuration: This is from hypervizor: [root@hde10 ~]# lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 64 On-line CPU(s) list: 0-63 Thread(s) per core: 2 Core(s) per socket: 16 Socket(s): 2 NUMA node(s): 8 Vendor ID: AuthenticAMD CPU family: 23 Model: 1 Model name: AMD EPYC 7351 16-Core Processor Stepping: 2 CPU MHz: 1800.000 CPU max MHz: 2400.0000 CPU min MHz: 1200.0000 BogoMIPS: 4800.05 Virtualization: AMD-V L1d cache: 32K L1i cache: 64K L2 cache: 512K L3 cache: 8192K NUMA node0 CPU(s): 0-3,32-35 NUMA node1 CPU(s): 4-7,36-39 NUMA node2 CPU(s): 8-11,40-43 NUMA node3 CPU(s): 12-15,44-47 NUMA node4 CPU(s): 16-19,48-51 NUMA node5 CPU(s): 20-23,52-55 NUMA node6 CPU(s): 24-27,56-59 NUMA node7 CPU(s): 28-31,60-63 I'm running one big virtual on this hypervizor - almost whole memory + all physical CPUs. This is what I'm seeing inside: root@zenon10:~# lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 32 On-line CPU(s) list: 0-31 Thread(s) per core: 1 Core(s) per socket: 4 Socket(s): 8 NUMA node(s): 8 Vendor ID: AuthenticAMD CPU family: 23 Model: 1 Model name: AMD EPYC 7351 16-Core Processor Stepping: 2 CPU MHz: 2400.000 BogoMIPS: 4800.00 Virtualization: AMD-V Hypervisor vendor: KVM Virtualization type: full L1d cache: 64K L1i cache: 64K L2 cache: 512K NUMA node0 CPU(s): 0-3 NUMA node1 CPU(s): 4-7 NUMA node2 CPU(s): 8-11 NUMA node3 CPU(s): 12-15 NUMA node4 CPU(s): 16-19 NUMA node5 CPU(s): 20-23 NUMA node6 CPU(s): 24-27 NUMA node7 CPU(s): 28-31 This is virtual node configuration: (i tried different numatune settings but it was still the same) <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> <name>one-55782</name> <vcpu><![CDATA[32]]></vcpu> <cputune> <shares>32768</shares> </cputune> <memory>507904000</memory> <os> <type arch='x86_64'>hvm</type> </os> <devices> <emulator><![CDATA[/usr/bin/kvm]]></emulator> <disk type='file' device='disk'> <source file='/opt/opennebula/var/datastores/108/55782/disk.0'/> <target dev='vda'/> <driver name='qemu' type='qcow2' cache='unsafe'/> </disk> <disk type='file' device='disk'> <source file='/opt/opennebula/var/datastores/108/55782/disk.1'/> <target dev='vdc'/> <driver name='qemu' type='raw' cache='unsafe'/> </disk> <disk type='file' device='disk'> <source file='/opt/opennebula/var/datastores/108/55782/disk.2'/> <target dev='vdd'/> <driver name='qemu' type='raw' cache='unsafe'/> </disk> <disk type='file' device='disk'> <source file='/opt/opennebula/var/datastores/108/55782/disk.3'/> <target dev='vde'/> <driver name='qemu' type='raw' cache='unsafe'/> </disk> <disk type='file' device='cdrom'> <source file='/opt/opennebula/var/datastores/108/55782/disk.4'/> <target dev='vdb'/> <readonly/> <driver name='qemu' type='raw'/> </disk> <interface type='bridge'> <source bridge='br0'/> <mac address='02:00:93:fb:3b:78'/> <target dev='one-55782-0'/> <model type='virtio'/> <filterref filter='no-arp-mac-spoofing'> <parameter name='IP' value='147.251.59.120'/> </filterref> </interface> </devices> <features> <pae/> <acpi/> </features>  <cpu mode='host-passthrough'><topology sockets='8' cores='4' threads='1'/><numa><cell cpus='0-3' memory='62000000' /><cell cpus='4-7' memory='62000000' /><cell cpus='8-11' memory='62000000' /><cell cpus='12-15' memory='62000000' /><cell cpus='16-19' memory='62000000' /><cell cpus='20-23' memory='62000000' /><cell cpus='24-27' memory='62000000' /><cell cpus='28-31' memory='62000000' /></numa></cpu> <cputune><vcpupin vcpu='0' cpuset='0' /><vcpupin vcpu='1' cpuset='2' /><vcpupin vcpu='2' cpuset='4' /><vcpupin vcpu='3' cpuset='6' /><vcpupin vcpu='4' cpuset='8' /><vcpupin vcpu='5' cpuset='10' /><vcpupin vcpu='6' cpuset='12' /><vcpupin vcpu='7' cpuset='14' /><vcpupin vcpu='8' cpuset='16' /><vcpupin vcpu='9' cpuset='18' /><vcpupin vcpu='10' cpuset='20' /><vcpupin vcpu='11' cpuset='22' /><vcpupin vcpu='12' cpuset='24' /><vcpupin vcpu='13' cpuset='26' /><vcpupin vcpu='14' cpuset='28' /><vcpupin vcpu='15' cpuset='30' /><vcpupin vcpu='16' cpuset='1' /><vcpupin vcpu='17' cpuset='3' /><vcpupin vcpu='18' cpuset='5' /><vcpupin vcpu='19' cpuset='7' /><vcpupin vcpu='20' cpuset='9' /><vcpupin vcpu='21' cpuset='11' /><vcpupin vcpu='22' cpuset='13' /><vcpupin vcpu='23' cpuset='15' /><vcpupin vcpu='24' cpuset='17' /><vcpupin vcpu='25' cpuset='19' /><vcpupin vcpu='26' cpuset='21' /><vcpupin vcpu='27' cpuset='23' /><vcpupin vcpu='28' cpuset='25' /><vcpupin vcpu='29' cpuset='27' /><vcpupin vcpu='30' cpuset='29' /><vcpupin vcpu='31' cpuset='31' /></cputune> <numatune><memory mode='preferred' nodeset='0'/></numatune>) <devices><serial type='pty'><target port='0'/></serial><console type='pty'><target type='serial' port='0'/></console><channel type='pty'><target type='virtio' name='org.qemu.guest_agent.0'/></channel></devices> <devices><hostdev mode='subsystem' type='pci' managed='yes'><source><address domain='0x0' bus='0x11' slot='0x0' function='0x1'/></source></hostdev></devices> <devices><controller type='pci' index='1' model='pci-bridge'/><controller type='pci' index='2' model='pci-bridge'/><controller type='pci' index='3' model='pci-bridge'/><controller type='pci' index='4' model='pci-bridge'/><controller type='pci' index='5' model='pci-bridge'/></devices> <metadata> <system_datastore><![CDATA[/opt/opennebula/var/datastores/108/55782]]> </system_datastore> </metadata> </domain> If I run e.g., spec2017 on the virtual, I can see: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1350 root 20 0 843136 830068 2524 R 78.1 0.2 513:16.16 bwaves_r_base.m 2456 root 20 0 804608 791264 2524 R 76.6 0.2 491:39.92 bwaves_r_base.m 4631 root 20 0 843136 829892 2344 R 75.8 0.2 450:16.04 bwaves_r_base.m 6441 root 20 0 802580 790212 2532 R 75.0 0.2 120:37.54 bwaves_r_base.m 7991 root 20 0 784676 772092 2576 R 75.0 0.2 387:15.39 bwaves_r_base.m 8142 root 20 0 843136 830044 2496 R 75.0 0.2 384:39.02 bwaves_r_base.m 8234 root 20 0 843136 830064 2524 R 75.0 0.2 99:04.48 bwaves_r_base.m 8578 root 20 0 749240 736604 2468 R 73.4 0.2 375:45.66 bwaves_r_base.m 9974 root 20 0 784676 771984 2468 R 73.4 0.2 348:01.36 bwaves_r_base.m 10396 root 20 0 802580 790264 2576 R 73.4 0.2 340:08.40 bwaves_r_base.m 12932 root 20 0 843136 830024 2480 R 73.4 0.2 288:39.76 bwaves_r_base.m 13113 root 20 0 784676 771864 2348 R 71.9 0.2 284:47.34 bwaves_r_base.m 13518 root 20 0 784676 762816 2540 R 71.9 0.2 276:31.58 bwaves_r_base.m 14443 root 20 0 784676 771984 2468 R 71.9 0.2 260:01.82 bwaves_r_base.m 12791 root 20 0 784676 772060 2544 R 70.3 0.2 291:43.96 bwaves_r_base.m 10544 root 20 0 843136 830068 2520 R 68.8 0.2 336:47.43 bwaves_r_base.m 15464 root 20 0 784676 762880 2608 R 60.9 0.2 239:19.14 bwaves_r_base.m 15487 root 20 0 784676 772048 2532 R 60.2 0.2 238:37.07 bwaves_r_base.m 16824 root 20 0 784676 772120 2604 R 55.5 0.2 212:10.92 bwaves_r_base.m 17255 root 20 0 843136 830012 2468 R 54.7 0.2 203:22.89 bwaves_r_base.m 17962 root 20 0 784676 772004 2488 R 54.7 0.2 188:26.07 bwaves_r_base.m 17505 root 20 0 843136 830068 2520 R 53.1 0.2 198:04.25 bwaves_r_base.m 27767 root 20 0 784676 771860 2344 R 52.3 0.2 592:25.95 bwaves_r_base.m 24458 root 20 0 843136 829888 2344 R 50.8 0.2 658:23.70 bwaves_r_base.m 30746 root 20 0 747376 735160 2604 R 43.0 0.2 556:47.67 bwaves_r_base.m The CPU TIME should be roughly the same but huge differences are obvious. This is what I see on the hypervizor: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 18201 oneadmin 20 0 474.0g 473.3g 1732 S 2459 94.0 33332:54 kvm 369 root 20 0 0 0 0 R 100.0 0.0 768:12.85 kswapd1 368 root 20 0 0 0 0 R 94.1 0.0 869:05.61 kswapd0 i.e., kswapd is eating whole CPU. Swap is turned off. [root@hde10 ~]# free total used free shared buff/cache available Mem: 528151432 503432580 1214048 34740 23504804 21907800 Swap: 0 0 0 Hypervisor is [root@hde10 ~]# cat /etc/redhat-release CentOS Linux release 7.5.1804 (Core) qemu-kvm-1.5.3-156.el7_5.5.x86_64 Virtual is Debian 9. Moreover, I'm using this type of disks for virtuals: <disk type='file' device='disk'> <source file='/opt/opennebula/var/datastores/108/55782/disk.3'/> <target dev='vde'/> <driver name='qemu' type='raw' cache='unsafe'/> </disk> If I keep cache='unsafe' and if I run iozone test on really big files (e.g., 8x 100GB), I can see huge cache pressure on the hypervizor - all 8 kswapd are running on 100 % percent and slowing things down. The disk under datastore is NVME SSD Intel 4500. If I set cache='none', kswaps are on idle, disk writes are pretty fast, however, with 8-NUMA configuration, writes slow down to less than 10MB/s as soon as the size of written data is roughly the same as memory size in the virtual node. iozone has 100 % CPU usage thereafter and it seems that it is traversing page lists. If I do the same with 1-NUMA configuration, everything is ok except performance penalty about 25 %. -- Lukáš Hejtmánek Linux Administrator only because Full Time Multitasking Ninja is not an official job title

2 7

[libvirt-users] Which objects does dynamic_ownership apply to?
by Milan Zamazal 20 Sep '18

20 Sep '18

Hi, I'm playing with dynamic ownership and not all objects have their owners changed. Is dynamic_ownership and its scope documented somewhere, besides the comment in qemu.conf? And what kinds of objects are handled by dynamic ownership? While some objects seem to be handled, other objects are apparently unaffected. For instance /dev/hwrng or a USB host device keep their root owners and are inaccessible to the VM. Is that expected or do I have anything wrong? Thanks, Milan

3 4

[libvirt-users] How to assign IP via virt-install
by Devis Reagan Ponraj -ERS, HCL Tech 18 Sep '18

18 Sep '18

Hi Team , I am new to virt-install , need your help pls . With below command I able to create a VM & I can see the mac address as well . virt-install --virt-type=kvm --name newclicentos --ram 6144 --vcpus=2 --import --network=bridge=br0,model=virtio --graphics vnc --disk path=centos7.0.qcow2,size=60,bus=virtio,format=qcow2 --os-variant centos7.0 --mac=00:00:00:00:01:01 With below command I am trying to assign an IP address , however it throws error . I dont want to give --location as I already have the vm build in centos7.0.qcow2 Is there a was I can assign static IP to my VM ? [root@localhost tmp]# virt-install --virt-type=kvm --name newclicentos --ram 6144 --vcpus=2 --import --network=bridge=br0,model=virtio --extra-args "ip=10.254.173.99::10.254.172.1::255.255.254.0::newclicentos:eth0:none" --mac=00:00:00:00:01:01 --graphics vnc --disk path=centos7.0.qcow2,size=60,bus=virtio,format=qcow2 ERROR --extra-args only work if specified with --location. Let me know if you need any info from me Thanks, Devis Reagan Regards, Devis Reagan HCL Technologies Ltd. www.hcl.com<http://www.hcl.com/> ::DISCLAIMER:: -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only. E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e mail and its contents (with or without referred errors) shall therefore not attach any liability on the originator or HCL or its affiliates. Views or opinions, if any, presented in this email are solely those of the author and may not necessarily reflect the views or opinions of HCL or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of this message without the prior written consent of authorized representative of HCL is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately. Before opening any email and/or attachments, please check them for viruses and other defects. --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

2 2

[libvirt-users] Save VGA console log
by Naruto Nguyen 16 Sep '18

16 Sep '18

Hi everyone, Currently I boot my guest through VGA as LABEL vga KERNEL vmlinuz APPEND initrd=initrd console=tty0 rootfstype=ramfs If I boot through VGA tty0, is there any way to modify libvirt xml config that when a user type command "reboot" in VGA console, I can log all messages types from user in "console", could you please help? Thanks a lot Brs, Naruto

1 0

[libvirt-users] Secondary disk unattached if guest unresponsive and killed with 'virsh destroy'
by Mathew Crane 12 Sep '18

12 Sep '18

I have a very strange issue that has plagued our KVM-based hypervisor infrastructure. Environment: - All EL7-based libvirt releases since EL7.0 - Fully patched EL7-based KVM guest on a fully patched EL7-based KVM host. - Secondary disk attached via 'virsh attach-device guest /vm/guest-with-secondary-disk-01/disk2.xml --persistent' (note --persistent flag here) Problem Description: - Guest becomes unresponsive/hangs due to issue on guest. Typically the scenario is that PID1 hangs or goes into a bad state due to a misbehaving application. - Before virsh destroy, 'virsh dumpxml guest | grep vd[ab]' returns the expected filesystems. - Guest is destroyed from the hypervisor via 'virsh destroy guest'. - With guest still powered off, 'virsh dumpxml guest | grep vd[ab]' only shows 'vda'. 'vdb' now unattached. - To fix, requires re-run of 'virsh attach-device guest guest-with-secondary-disk-01/disk2.xml --persistent' Expected Behavior: - 'virsh destroy guest' does not drop the secondary disk attached to the guest if the node is in a hung state. Other notes: - If guest is not in a hung state, 'virsh destroy' does not drop the secondary disk. - 'virsh reset guest' does not exhibit the same behavior when guest is hung - vdb remains attached. - No logs produced in /var/log/libvirt indicating an issue, even with debug logging enabled. Is there any obvious I'm missing? Is this expected behavior? Guest XML via 'virsh dumpxml guest': <domain type='kvm' id='17'> <name>guest-with-secondary-disk-01</name> <uuid>ab578b51-3afd-4d01-86ab-a16b1b953919</uuid> <memory unit='KiB'>4194304</memory> <currentMemory unit='KiB'>4194304</currentMemory> <vcpu placement='static'>1</vcpu> <resource> <partition>/machine</partition> </resource> <os> <type arch='x86_64' machine='pc-i440fx-rhel7.0.0'>hvm</type> <boot dev='hd'/> <bootmenu enable='yes'/> </os> <features> <acpi/> <apic/> <pae/> </features> <clock offset='utc'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/libexec/qemu-kvm</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2' cache='none'/> <source file='/vm/guest-with-secondary-disk-01/guest-with-secondary-disk-01.img'/> <backingStore/> <target dev='vda' bus='virtio'/> <alias name='virtio-disk0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='qcow2' cache='none'/> <source file='/vm/guest-with-secondary-disk-01/guest-with-secondary-disk-01_2.img'/> <backingStore/> <target dev='vdb' bus='virtio'/> <alias name='virtio-disk1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> </disk> <controller type='usb' index='0' model='piix3-uhci'> <alias name='usb'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> </controller> <controller type='pci' index='0' model='pci-root'> <alias name='pci.0'/> </controller> <controller type='virtio-serial' index='0'> <alias name='virtio-serial0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </controller> <interface type='bridge'> <mac address='01:02:03:99:99:99'/> <source bridge='bridge_99'/> <target dev='vnet0'/> <model type='virtio'/> <alias name='eth0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </interface> <channel type='unix'> <source mode='bind' path='/var/lib/libvirt/qemu/guest-with-secondary-disk-01.agent'/> <target type='virtio' name='org.qemu.guest_agent.0' state='connected'/> <alias name='channel0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='mouse' bus='ps2'> <alias name='input0'/> </input> <input type='keyboard' bus='ps2'> <alias name='input1'/> </input> <graphics type='vnc' port='5900' autoport='yes' listen='127.0.0.1' keymap='en-us'> <listen type='address' address='127.0.0.1'/> </graphics> <video> <model type='cirrus' vram='16384' heads='1' primary='yes'/> <alias name='video0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </video> <memballoon model='virtio'> <alias name='balloon0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </memballoon> <rng model='virtio'> <backend model='random'>/dev/random</backend> <alias name='rng0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </rng> </devices> <seclabel type='dynamic' model='selinux' relabel='yes'> <label>system_u:system_r:svirt_t:s0:c259,c495</label> <imagelabel>system_u:object_r:svirt_image_t:s0:c259,c495</imagelabel> </seclabel> <seclabel type='dynamic' model='dac' relabel='yes'> <label>+107:+107</label> <imagelabel>+107:+107</imagelabel> </seclabel> </domain> Secondary disk XML: <disk type='file' device='disk'> <driver name='qemu' type='qcow2' cache='none'/> <source file='/vm/guest-with-secondary-disk-01/guest-with-secondary-disk-01_2.img'/> <target dev='vdb'/> </disk>

1 0

[libvirt-users] Running libvirtd outside of a shell results in qemu defunct processes
by Roman Mohr 12 Sep '18

12 Sep '18

Hi, In KubeVirt we are right now starting libvirt via a wrapper shell script. I want to remove that wrapper and directly start libvirtd from our golang code. This means that our golang code is PID 1 in the container and we start libvirtd as a fork from there. When I did this, strange things started to happen. Libvirt was still able to start and manage qemu processes, but stopping the qemu processes stopped working. I can see that all qemu processes, the ones with VMs and the ones used by libvirt to do some host capability detection, stay in the process table as zombies: qemu 54 0.0 0.0 0 0 ? Zs 11:44 0:00 [qemu-system-i38] <defunct> qemu 56 0.2 0.0 0 0 ? Z 11:44 0:00 [qemu-system-i38] <defunct> qemu 61 0.0 0.0 0 0 ? Zs 11:44 0:00 [qemu-system-i38] <defunct> qemu 63 0.1 0.0 0 0 ? Z 11:44 0:00 [qemu-system-i38] <defunct> qemu 70 0.0 0.0 0 0 ? Zs 11:44 0:00 [qemu-system-x86] <defunct> qemu 72 0.2 0.0 0 0 ? Z 11:44 0:00 [qemu-system-x86] <defunct> qemu 79 0.0 0.0 0 0 ? Zs 11:44 0:00 [qemu-system-x86] <defunct> qemu 81 0.2 0.0 0 0 ? Z 11:44 0:00 [qemu-system-x86] <defunct> qemu 86 0.0 0.0 0 0 ? Zs 11:44 0:00 [qemu-system-x86] <defunct> qemu 88 0.3 0.0 0 0 ? Z 11:44 0:00 [qemu-system-x86] <defunct> qemu 93 0.0 0.0 0 0 ? Zs 11:44 0:00 [qemu-system-x86] <defunct> qemu 95 0.1 0.0 0 0 ? Z 11:44 0:00 [qemu-system-x86] <defunct> So to sum up, when I start our golang wrapper around libvirtd inside a bash script without "exec" everything works as usual. If I start in the container pid namespace the golang wrapper directly, or use "exec" from the bash script, things stop working. I also see the libvirt repeatedly tries to kill the qemu processes, but they are stuck. Maybe someone has a clue why things stop working. Thanks and Best Regards, Roman

1 1

[libvirt-users] Shared clipboard not working
by Filipe Laíns 10 Sep '18

10 Sep '18

Hi! I have been plaing arround with libvirt and I've ran into a problem that I don't seem to able to fix, sharing the clipboard. Both my host and my guest are running Arch Linux. I have properly installed, enabled and started spice-vdagent on both machines and I don't see any error in the logs. Am I doing anything wrong? Do I need to have anything else in consideration? Any help is appreciated :) Thank you, Filipe Laíns 3DCE 51D6 0930 EBA4 7858 BA41 46F6 33CB B0EB 4BF2

2 1

[libvirt-users] Immutable backing files
by Gionatan Danti 08 Sep '18

08 Sep '18

Hi list, suppose I have an immutable (ie: due to a read-only snapshot) backing file. After creating an overlay file with "qemu-img create -f qcow2 -o backing_file=/path/to/immutable/file.img current.qcow2", libvirt refuse to start the virtual machine and exits with an error stating "Could not open backing file /path/to/immutable/file.img: Permission denied". From my understanding, libvirtd is trying to change the owner/permissions/selinux context of the backing file but, being it immutable, fails to start the VM. Is that the expected behavior? Thanks. -- Danti Gionatan Supporto Tecnico Assyoma S.r.l. - www.assyoma.it email: g.danti(a)assyoma.it - info(a)assyoma.it GPG public key ID: FF5F32A8

2 5

[libvirt-users] Domain vCPU threads affinity
by Nikos Anastopoulos 06 Sep '18

06 Sep '18

Hello, According to the docs, vcpupin will use either cgroups or sched_setaffinity to pin vcpu threads to cpus. How is this decision made? I observe differences even on different hosts featuring the same version of libvirtd (1.3.1): on one host vcpupin affects cpuset.cpus (cgroup), and on the other it affects vcpu threads affinity (observed through taskset). Thanks, Nikos

2 2

[libvirt-users] How to specify a x509-dir from XML config file?
by 陈炤 05 Sep '18

05 Sep '18

Hi Everyone, I'd like to ask a question about libvirt xml config. I am using kvm with tls certification. For some reason I need to specify a unique certificate file for every instance, so my kvm command would be like: /usr/libexec/qemu-kvm -spice port=5900,tls-port=5901,addr=0.0.0.0,disable-ticketing,x509-dir=/openstack/etc/pki/libvirt-spice the argument x509-dir=/openstack/etc/pki/libvirt-spice is specified in /etc/libvirt/qemu.conf. It's a global argument. Can I specify it from xml? I searched the xml doc but unable to find useful infomation. BR Don

2 1

[libvirt-users] Intel's latest L1TF vulnerability and libvirt
by Paul O'Rorke 05 Sep '18

05 Sep '18

Hi, with regards Intels L1TF vulnerabilities, it seems they are somewhat non-committal on whether turning off HyperThreading is required, suggesting people > Consult with your hypervisor vendor for more guidance. https://www.intel.com/content/www/us/en/architecture-and-technology/l1tf.ht… What is the consensus in the Libvirt community about the risks (or not) of leaving Hyperthreading enabled? After updates my hosts are showing they have conditional cache flushing enabled yet still report as "SMT vulnerable": root@trk-kvm-03:~# cat /sys/devices/system/cpu/vulnerabilities/l1tf Mitigation: PTE Inversion; VMX: conditional cache flushes, SMT vulnerable Thoughts? -- *Paul O'Rorke* *Tracker Software Products (Canada) Limited * www.tracker-software.com <http://www.tracker-software.com/> Tel: +1 (250) 324 1621 Fax: +1 (250) 324 1623 <http://www.tracker-software.com/> Support: http://www.tracker-software.com/support Download latest Releases http://www.tracker-software.com/downloads/

2 1

[libvirt-users] So Close - Port forwarding in user mode.
by Rhys Ferris 03 Sep '18

03 Sep '18

People of the interwebs please help me! I am so close, but can't quite get it. I think due to my limited understanding of iptables. Here's what’s up: I'm trying to port forward from my host to a VM on usermode NAT with libvirt. Using the hook script provided on the wiki, it doesn't work (connection refused) but it if give the command "sudo iptables -t nat -D POSTROUTING -j MASQUERADE" the port forwarding works, but name resolution on my host breaks. I think maybe I need to be more specific in the Masquerade command, but I have no idea what to do. Here’s my iptables (without the above command): https://pastebin.com/J2vdKkLU The source interface is eno1 with an IP of 172.16.99.170 and the destination is virbr1 with a VM IP of 10.128.10.100 Thanks in advance! Rhys Sent from Mail for Windows 10

1 0

[libvirt-users] Snapshot overlay-file deletion
by Gionatan Danti 01 Sep '18

01 Sep '18

Hi list, issuing the command "virsh blockcommit --delete --pivot" (on a previously created external snapshot/overlay) returns an error regarding an unsupported flag warning stating "error: unsupported flags (0x2)". I need to manually "rm" the affected file after the blockcommit. A similar issue was reported many years ago here [1]. Am I missing something, or is the "--delete" flag not supported? Thanks. [1] https://bugzilla.redhat.com/show_bug.cgi?id=1001475 -- Danti Gionatan Supporto Tecnico Assyoma S.r.l. - www.assyoma.it email: g.danti(a)assyoma.it - info(a)assyoma.it GPG public key ID: FF5F32A8

2 2

[libvirt-users] Setting up port forwarding to guests on nat network
by Rhys Ferris 31 Aug '18

31 Aug '18

Hello all, I’m currently trying to figure out how to forward ports to guests that are on a NAT Network. I have followed the directions on https://wiki.libvirt.org/page/Networking under the “Forwarding Incoming Connections” Section and get connection refused when attempting to connect. System: Ubuntu Server 18.04.1 Virsh / LibVirtd Version: 4.0.0 Here’s the contents of /etc/libvirt/hooks/qemu #!/bin/bash # IMPORTANT: Change the "VM NAME" string to match your actual VM Name. # In order to create rules to other VMs, just duplicate the below block and configure # it accordingly. if [ "${1}" = "testy" ]; then # Update the following variables to fit your setup GUEST_IP='10.128.10.100' GUEST_PORT='22' HOST_PORT='2588' if [ "${2}" = "stopped" ] || [ "${2}" = "reconnect" ]; then /sbin/iptables -D FORWARD -o virbr0 -d $GUEST_IP -j ACCEPT /sbin/iptables -t nat -D PREROUTING -p tcp --dport $HOST_PORT -j DNAT --to $GUEST_IP:$GUEST_PORT fi if [ "${2}" = "start" ] || [ "${2}" = "reconnect" ]; then /sbin/iptables -I FORWARD -o virbr0 -d $GUEST_IP -j ACCEPT /sbin/iptables -t nat -I PREROUTING -p tcp --dport $HOST_PORT -j DNAT --to $GUEST_IP:$GUEST_PORT fi fi Here’s my network XML <network> <name>olympus</name> <uuid>3b0d968c-8166-42f7-8109-e5f0317cab42</uuid> <forward mode='nat'> <nat> <port start='1024' end='65535'/> </nat> </forward> <bridge name='virbr1' stp='on' delay='0'/> <mac address='52:54:00:bb:18:6b'/> <ip address='10.128.10.1' netmask='255.255.255.0'> <dhcp> <range start='10.128.10.2' end='10.128.10.254'/> <host mac='52:54:00:8d:f5:0c' name='testy' ip='10.128.10.100'/> </dhcp> </ip> </network> And here’s the results of iptables -L -vt nat: BEFORE VM BOOT: Chain PREROUTING (policy ACCEPT 46615 packets, 6618K bytes) pkts bytes target prot opt in out source destination Chain INPUT (policy ACCEPT 46615 packets, 6618K bytes) pkts bytes target prot opt in out source destination Chain OUTPUT (policy ACCEPT 198K packets, 18M bytes) pkts bytes target prot opt in out source destination Chain POSTROUTING (policy ACCEPT 198K packets, 18M bytes) pkts bytes target prot opt in out source destination 24 1812 RETURN all -- any any 10.128.10.0/24 base-address.mcast.net/24 0 0 RETURN all -- any any 10.128.10.0/24 255.255.255.255 17 1020 MASQUERADE tcp -- any any 10.128.10.0/24 !10.128.10.0/24 masq ports: 1024-65535 15 1700 MASQUERADE udp -- any any 10.128.10.0/24 !10.128.10.0/24 masq ports: 1024-65535 0 0 MASQUERADE all -- any any 10.128.10.0/24 !10.128.10.0/24 22 1666 RETURN all -- any any 192.168.122.0/24 base-address.mcast.net/24 0 0 RETURN all -- any any 192.168.122.0/24 255.255.255.255 0 0 MASQUERADE tcp -- any any 192.168.122.0/24 !192.168.122.0/24 masq ports: 1024-65535 8 1168 MASQUERADE udp -- any any 192.168.122.0/24 !192.168.122.0/24 masq ports: 1024-65535 0 0 MASQUERADE all -- any any 192.168.122.0/24 !192.168.122.0/24 AFTER VM BOOT Chain PREROUTING (policy ACCEPT 2 packets, 120 bytes) pkts bytes target prot opt in out source destination 0 0 DNAT tcp -- any any anywhere anywhere tcp dpt:2588 to:10.128.10.100:22 Chain INPUT (policy ACCEPT 2 packets, 120 bytes) pkts bytes target prot opt in out source destination Chain OUTPUT (policy ACCEPT 18 packets, 1263 bytes) pkts bytes target prot opt in out source destination Chain POSTROUTING (policy ACCEPT 18 packets, 1263 bytes) pkts bytes target prot opt in out source destination 24 1812 RETURN all -- any any 10.128.10.0/24 base-address.mcast.net/24 0 0 RETURN all -- any any 10.128.10.0/24 255.255.255.255 17 1020 MASQUERADE tcp -- any any 10.128.10.0/24 !10.128.10.0/24 masq ports: 1024-65535 15 1700 MASQUERADE udp -- any any 10.128.10.0/24 !10.128.10.0/24 masq ports: 1024-65535 0 0 MASQUERADE all -- any any 10.128.10.0/24 !10.128.10.0/24 22 1666 RETURN all -- any any 192.168.122.0/24 base-address.mcast.net/24 0 0 RETURN all -- any any 192.168.122.0/24 255.255.255.255 0 0 MASQUERADE tcp -- any any 192.168.122.0/24 !192.168.122.0/24 masq ports: 1024-65535 8 1168 MASQUERADE udp -- any any 192.168.122.0/24 !192.168.122.0/24 masq ports: 1024-65535 0 0 MASQUERADE all -- any any 192.168.122.0/24 !192.168.122.0/24 And lastly heres what actually happens on attempt to SSH: rhys@odin:~$ ssh rhys(a)172.16.99.170 -p 2258 ssh: connect to host 172.16.99.170 port 2258: Connection refused rhys@odin:~$ The connection refused is instant, not a timeout. I’ve ensured that ufw is disabled. Any help appreciated. I just can’t figure this out. Sent from Mail for Windows 10

2 3

[libvirt-users] ceph rbd pool and libvirt manageability (virt-install)
by Jelle de Jong 29 Aug '18

29 Aug '18

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hello everybody, I created a rbd pool and activated it, but I can't seem to create volumes in it with virsh or virt-install? # virsh pool-dumpxml myrbdpool <pool type='rbd'> <name>myrbdpool</name> <uuid>2d786f7a-2df3-4d79-ae60-1535bcf1c6b5</uuid> <capacity unit='bytes'>6997998301184</capacity> <allocation unit='bytes'>10309227031</allocation> <available unit='bytes'>6977204658176</available> <source> <host name='ceph01.powercraft.nl' port='6789'/> <host name='ceph02.powercraft.nl' port='6789'/> <host name='ceph03.powercraft.nl' port='6789'/> <name>libvirt-pool</name> <auth type='ceph' username='libvirt'> <secret uuid='029a334e-ed57-4293-bb99-ffafa8867122'/> </auth> </source> </pool> # virt-install --version 1.0.1 # virsh --version 1.2.9 I ended using virsh edit ceph-test.powercraft.nl and making creating the disk manually. <disk type='network' device='disk'> <auth username='libvirt'> <secret type='ceph' uuid='029a334e-ed57-4293-bb99-ffafa8867122'/> </auth> <source protocol='rbd' name='libvirt-pool/kvm01-storage'> <host name='ceph01.powercraft.nl' port='6789'/> <host name='ceph02.powercraft.nl' port='6789'/> <host name='ceph03.powercraft.nl' port='6789'/> </source> <target dev='vdc' bus='virtio'/> </disk> I use virt-install a lot to define, import and undefine domains, how can I use virt-install to manage my rdb disks? Kind regards, Jelle de Jong -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iJwEAQECAAYFAlV1xlQACgkQ1WclBW9j5HkbPQP+PjNrzvlqysslOp2Yk7wH4Mxy 2sh2dn96G0KOAHEeEn3BN6IWlnD1TADZbHdpTtMwkdv48Xwn0sP1s+3QDM4pb3gP n+z+dVxS8FouDIy/eiso3IBCj3g4TWbEX8ZHqs3jKqe0lZgAXBzB9xYSUowcEBrZ ddkPbr8p8ozWyOG+9V8= =lkK7 -----END PGP SIGNATURE-----

2 3

[libvirt-users] OVMF / UEFI boot abnormally
by Allence 29 Aug '18

29 Aug '18

Hey guys, today I use ovmf to start win10 guests, my monitor is not bright But is it normal for me to use seabios? why? The next message is my detailed settings. ovmf: os: <loader readonly='yes' type='pflash'>/root/OVMF_CODE.fd</loader> <nvram>/root/OVMF_VARS.fd</nvram> vga: <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=01:00.0,romfile=/root/10de:1b06:19da:1474.dump,multifunction=on'/> bios: os: <loader readonly='yes' type='rom'>/usr/share/qemu/bios.bin</loader> vga: <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=01:00.0,romfile=/root/us.dump,multifunction=on,x-vga=on'/> And this is my kvm environment: qemu: 2.12.0 libvirt: 4.14 kernel: 4.17.2 host: Lfs-8.2(as similar as ubuntu)

1 0

[libvirt-users] Download image to libvirt storage pool
by Shashwat shagun 27 Aug '18

27 Aug '18

Is there any function in libvirt (golang SDK) to download image to libvirt storage pool from path?

2 1

[libvirt-users] libvirt and virt-manager - Unable to complete install: 'internal error: unsupported input bus usb'
by Eduardo Lúcio Amorim Costa 24 Aug '18

24 Aug '18

I'm trying create a virtual machine using Xen as hypervisor and virt-manager (libvirt) as the management module. When trying to create the virtual machine I am getting the following error: " Unable to complete install: 'internal error: unsupported input bus usb' " ERROR DETAILS: Unable to complete install: 'internal error: unsupported input bus usb' Traceback (most recent call last): File "/usr/share/virt-manager/virtManager/asyncjob.py", line 75, in cb_wrapper callback(asyncjob, *args, **kwargs) File "/usr/share/virt-manager/virtManager/create.py", line 2276, in _do_async_install guest.start_install(meter=meter) File "/usr/share/virt-manager/virtinst/guest.py", line 461, in start_install doboot, transient) File "/usr/share/virt-manager/virtinst/guest.py", line 397, in _create_guest domain = self.conn.createXML(install_xml or final_xml, 0) File "/usr/lib/python3.7/site-packages/libvirt.py", line 3717, in createXML if ret is None:raise libvirtError('virDomainCreateXML() failed', conn=self) libvirt.libvirtError: internal error: unsupported input bus usb NOTE I: The deploy process follows the instructions here https://www.youtube.com/watch?v=BwkmDM-Gpzc and here https://wiki.centos.org/HowTos/Xen/Xen4QuickStart . NOTE II: Xen Hypervisor uses a CentOS 7 as "domu". MORE INFORMATION ABOUT THE PROBLEM HERE: https://unix.stackexchange.com/q/464186/61742 Thanks! =D

2 3

[libvirt-users] Guest startup delay options ignored
by Inception Hosting 24 Aug '18

24 Aug '18

Hi Folks, been searching around for a while on this and see similar issues reported going back a number of years without solution. The START_DELAY=(number) seems to be completely ignored in /etc/sysconfig/libvirt-guests which is unfortunate as it means everything starts at once without control. This is fine on NVMe based servers however on the older spinning disks the I/O is creating significant issues. Is anyone aware of any other methods to achieve a start up delay for guest virtual machines on boot? Thanks. Anthony.

2 2

[libvirt-users] Efficacy of jitterentropy RNG on qemu-kvm Guests
by procmem 18 Aug '18

18 Aug '18

Hello. I'm a distro maintainer and was wondering about the efficacy of entropy daemons like haveged and jitterentropyd in qemu-kvm. One of the authors of haveged [0] pointed out if the hardware cycles counter is emulated and deterministic, and thus predictible. He therefore does not recommend using HAVEGE on those systems. Is this the case with KVM's counters? PS. I will be setting VM CPU settings to host-passthrough. Bonus: Also if anyone knows the answer to this question about Xen please let me know because its the other main platform we support and they don't have the luxury of virtio-rng in PVH mode. Thanks. [0] https://github.com/BetterCrypto/Applied-Crypto-Hardening/commit/cf7cef7a870…

3 5

[libvirt-users] qemu guest agent
by Cobin Bluth 15 Aug '18

15 Aug '18

Hello Libvirt-Users! I have a quick question about the qemu guest agent. Is it possible to use the guest agent from inside the guest in order to query the name of its own domain? For example, I use a base-image.qcow2 with a baked-in hostname. I would like to include a little tool in my guest image to change the hostname to the name of the domain. Is this sort of thing possible? -Cobin

2 1

[libvirt-users] New Application to add to web site
by Matthew 12 Aug '18

12 Aug '18

Hello, Could you please add the OpenVM Dashboard to the list of applications that use libvirt. Below is the URL and description. Also, to all the developers who put libvirt-php together...THANK YOU! Thanks, Matthew Penning URL: https://openvm.tech <https://openvm.tech/> Description: The OpenVM Dashboard is an open source HTML5 and PHP based web interface for the KVM and QEMU hypervisor. It is designed to be a easy-to-use management platform allowing users to create and manage domains (virtual machines) using the libvirt-php module.

1 0

Re: [libvirt-users] Windows Guest I/O performance issues (already using virtio) (Matt Schumacher)
by Allence 10 Aug '18

10 Aug '18

I think performance is not just about your xml, the host system will have a bigger impact. Maybe you can see this link: Https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/ht… >Date: Wed, 8 Aug 2018 17:35:11 +0000 >From: Matt Schumacher <matt.s(a)aptalaska.com> >To: "libvirt-users(a)redhat.com" <libvirt-users(a)redhat.com> >Subject: [libvirt-users] Windows Guest I/O performance issues (already > using virtio) >Message-ID: <30EE4896-CF82-4E23-82D5-0C56B38F5E09(a)aptalaska.com> >Content-Type: text/plain; charset="utf-8" > >List, > >I have a number of Windows 2016 servers I am deploying, but I?m having some I/O performance issues. I have done all of the obvious things like virtio drivers, but am finding there is more performance to be found with hyper-v extensions, how we virtualize the hardware clock, and iothreads. I?m using ZVOLs to back the VM, and I?m using 4k block sizes, which seems to offer the best 4k random read/write performance (mail and database workloads), but maybe I?m missing something at this layer too. > >Questions: > > > 1. Does my VM config look reasonable for the latest releases of windows? Are there features I should be using that will help performance? > 2. Why does the hypervclock timer make so much performance difference in windows VMs? > 3. Does my virtualized CPU model make sense? I defined Haswell-noTSX-IBRS and libvirt added the features. > 4. Which kernel branch offers the best stability and performance? > 5. Are there performance gains in using UEFI booting the windows guest and defining ?<blockio logical_block_size='4096' physical_block_size='4096'/>?? Perhaps better block size consistency through to the zvol? > > >Here is my setup: > >48 core Haswell CPU >192G Ram >Linux 4.14.61 or 4.9.114 (testing both) >ZFS file system on optane SSD drive or ZFS file system on dumb HBA with 8 spindles of 15k disks (testing both) >4k block size zvol for virtual machines >32G arc cache > >Here is my VM: > ><domain type='kvm' id='12'> > <name>testvm</name> > <memory unit='KiB'>33554432</memory> > <currentMemory unit='KiB'>33554432</currentMemory> > <vcpu placement='static'>12</vcpu> > <iothreads>1</iothreads> > <os> > <type arch='x86_64' machine='pc-i440fx-2.12'>hvm</type> > <boot dev='cdrom'/> > <boot dev='hd'/> > </os> > <features> > <acpi/> > <hyperv> > <relaxed state='on'/> > <vapic state='on'/> > <spinlocks state='on' retries='8191'/> > <vpindex state='on'/> > <runtime state='on'/> > <synic state='on'/> > <reset state='on'/> > <vendor_id state='on' value='KVM Hv'/> > </hyperv> > </features> > <cpu mode='custom' match='exact' check='full'> > <model fallback='forbid'>Haswell-noTSX-IBRS</model> > <topology sockets='1' cores='6' threads='2'/> > <feature policy='require' name='vme'/> > <feature policy='require' name='f16c'/> > <feature policy='require' name='rdrand'/> > <feature policy='require' name='hypervisor'/> > <feature policy='require' name='arat'/> > <feature policy='disable' name='spec-ctrl'/> > <feature policy='require' name='xsaveopt'/> > <feature policy='require' name='abm'/> > </cpu> > <clock offset='localtime'> > <timer name='rtc' tickpolicy='catchup'/> > <timer name='pit' tickpolicy='delay'/> > <timer name='hpet' present='yes'/> > <timer name='hypervclock' present='yes'/> > </clock> > <on_poweroff>destroy</on_poweroff> > <on_reboot>restart</on_reboot> > <on_crash>destroy</on_crash> > <devices> > <emulator>/usr/bin/qemu-system-x86_64</emulator> > <disk type='block' device='disk'> > <driver name='qemu' type='raw' cache='none' io='native' ioeventfd='on' iothread='1'/> > <source dev='/dev/zvol/datastore/vm/testvm-vda'/> > <backingStore/> > <target dev='vda' bus='virtio'/> > <alias name='virtio-disk0'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> > </disk> > <disk type='file' device='cdrom'> > <driver name='qemu'/> > <target dev='hdc' bus='ide'/> > <readonly/> > <alias name='ide0-1-0'/> > <address type='drive' controller='0' bus='1' target='0' unit='0'/> > </disk> > <controller type='ide' index='0'> > <alias name='ide'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> > </controller> > <controller type='usb' index='0' model='piix3-uhci'> > <alias name='usb'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> > </controller> > <controller type='pci' index='0' model='pci-root'> > <alias name='pci.0'/> > </controller> > <controller type='virtio-serial' index='0'> > <alias name='virtio-serial0'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> > </controller> > <interface type='bridge'> > <source bridge='lan'/> > <target dev='vnet0'/> > <model type='virtio'/> > <alias name='net0'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> > </interface> > <channel type='unix'> > <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-12-testvm/org.qemu.guest_agent.0'/> > <target type='virtio' name='org.qemu.guest_agent.0' state='connected'/> > <alias name='channel0'/> > <address type='virtio-serial' controller='0' bus='0' port='1'/> > </channel> > <input type='tablet' bus='usb'> > <alias name='input0'/> > <address type='usb' bus='0' port='1'/> > </input> > <input type='mouse' bus='ps2'> > <alias name='input1'/> > </input> > <input type='keyboard' bus='ps2'> > <alias name='input2'/> > </input> > <graphics type='vnc' port='5901' autoport='no' listen='0.0.0.0'> > <listen type='address' address='0.0.0.0'/> > </graphics> > <video> > <model type='cirrus' vram='16384' heads='1' primary='yes'/> > <alias name='video0'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> > </video> > <memballoon model='none'/> > </devices> > <seclabel type='dynamic' model='dac' relabel='yes'> > <label>+0:+100</label> > <imagelabel>+0:+100</imagelabel> > </seclabel> ></domain>

1 0

[libvirt-users] Mount URL as cdrom/iso KVM/QEMU
by Inception Hosting 10 Aug '18

10 Aug '18

Hi Folks, According to the examples in http://libvirt.org/formatdomain.html#elementsDisks it should be possible to mount an iso /url as a cdrom, the example given is: </disk> <disk type='network' device='cdrom'> <driver name='qemu' type='raw'/> <source protocol="http" name="url_path"> <host name="hostname" port="80"/> </source> <target dev='hde' bus='ide' tray='open'/> <readonly/> </disk> I am unable to get this to work at all and wondered if I am just missing something obvious: Boot failed: Could not read from CDROM (code 0003) Actual XML in use snippet: <disk type='network' device='cdrom'> <driver name='qemu' type='raw'/> <source protocol="http" name="/debian-cd/current/amd64/iso-cd/debian-9.5.0-amd64-netinst.iso"> <host name="mirror.bytemark.co.uk" port="80"/> </source> <target dev='hdb' bus='ide' tray='open'/> <readonly/> </disk> libvirtd (libvirt) 3.9.0 QEMU emulator version 2.10.0 Is anyone able to offer any assistance or tips? I have tried putting the complete path including fqdn in the url_path as well, with and without http:// Many Thanks. Anthony.

3 8

[libvirt-users] Windows Guest I/O performance issues (already using virtio)
by Matt Schumacher 09 Aug '18

09 Aug '18

List, I have a number of Windows 2016 servers I am deploying, but I’m having some I/O performance issues. I have done all of the obvious things like virtio drivers, but am finding there is more performance to be found with hyper-v extensions, how we virtualize the hardware clock, and iothreads. I’m using ZVOLs to back the VM, and I’m using 4k block sizes, which seems to offer the best 4k random read/write performance (mail and database workloads), but maybe I’m missing something at this layer too. Questions: 1. Does my VM config look reasonable for the latest releases of windows? Are there features I should be using that will help performance? 2. Why does the hypervclock timer make so much performance difference in windows VMs? 3. Does my virtualized CPU model make sense? I defined Haswell-noTSX-IBRS and libvirt added the features. 4. Which kernel branch offers the best stability and performance? 5. Are there performance gains in using UEFI booting the windows guest and defining “<blockio logical_block_size='4096' physical_block_size='4096'/>”? Perhaps better block size consistency through to the zvol? Here is my setup: 48 core Haswell CPU 192G Ram Linux 4.14.61 or 4.9.114 (testing both) ZFS file system on optane SSD drive or ZFS file system on dumb HBA with 8 spindles of 15k disks (testing both) 4k block size zvol for virtual machines 32G arc cache Here is my VM: <domain type='kvm' id='12'> <name>testvm</name> <memory unit='KiB'>33554432</memory> <currentMemory unit='KiB'>33554432</currentMemory> <vcpu placement='static'>12</vcpu> <iothreads>1</iothreads> <os> <type arch='x86_64' machine='pc-i440fx-2.12'>hvm</type> <boot dev='cdrom'/> <boot dev='hd'/> </os> <features> <acpi/> <hyperv> <relaxed state='on'/> <vapic state='on'/> <spinlocks state='on' retries='8191'/> <vpindex state='on'/> <runtime state='on'/> <synic state='on'/> <reset state='on'/> <vendor_id state='on' value='KVM Hv'/> </hyperv> </features> <cpu mode='custom' match='exact' check='full'> <model fallback='forbid'>Haswell-noTSX-IBRS</model> <topology sockets='1' cores='6' threads='2'/> <feature policy='require' name='vme'/> <feature policy='require' name='f16c'/> <feature policy='require' name='rdrand'/> <feature policy='require' name='hypervisor'/> <feature policy='require' name='arat'/> <feature policy='disable' name='spec-ctrl'/> <feature policy='require' name='xsaveopt'/> <feature policy='require' name='abm'/> </cpu> <clock offset='localtime'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='yes'/> <timer name='hypervclock' present='yes'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/bin/qemu-system-x86_64</emulator> <disk type='block' device='disk'> <driver name='qemu' type='raw' cache='none' io='native' ioeventfd='on' iothread='1'/> <source dev='/dev/zvol/datastore/vm/testvm-vda'/> <backingStore/> <target dev='vda' bus='virtio'/> <alias name='virtio-disk0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu'/> <target dev='hdc' bus='ide'/> <readonly/> <alias name='ide0-1-0'/> <address type='drive' controller='0' bus='1' target='0' unit='0'/> </disk> <controller type='ide' index='0'> <alias name='ide'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='usb' index='0' model='piix3-uhci'> <alias name='usb'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> </controller> <controller type='pci' index='0' model='pci-root'> <alias name='pci.0'/> </controller> <controller type='virtio-serial' index='0'> <alias name='virtio-serial0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </controller> <interface type='bridge'> <source bridge='lan'/> <target dev='vnet0'/> <model type='virtio'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> <channel type='unix'> <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-12-testvm/org.qemu.guest_agent.0'/> <target type='virtio' name='org.qemu.guest_agent.0' state='connected'/> <alias name='channel0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='tablet' bus='usb'> <alias name='input0'/> <address type='usb' bus='0' port='1'/> </input> <input type='mouse' bus='ps2'> <alias name='input1'/> </input> <input type='keyboard' bus='ps2'> <alias name='input2'/> </input> <graphics type='vnc' port='5901' autoport='no' listen='0.0.0.0'> <listen type='address' address='0.0.0.0'/> </graphics> <video> <model type='cirrus' vram='16384' heads='1' primary='yes'/> <alias name='video0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </video> <memballoon model='none'/> </devices> <seclabel type='dynamic' model='dac' relabel='yes'> <label>+0:+100</label> <imagelabel>+0:+100</imagelabel> </seclabel> </domain>

2 1