
Hi everybody, since long time, I'm using Xen on CentOS with XL, currently latest CentOS 7 with Xen 4.15 from the CentOS-Xen-Project. For several VMs I have to use SR-IOV, to lower the CPU-usage on Dom0 on the host. CentOS 7 comes to an end, Xen is not supported by RHEL nor RockyLinux anymore, so unfortunately I have to switch to KVM. First step will be now, converting all the scripts for managing and running VMs, that they run with the additional libvirt-layer. Mostly everything is working, but I do not get a network interface in the VM, when I start it with "virsh start ..." instead of "xl create ...". First of all, is there documentation how to configure the VMs in the dom-definition for XEN (all docs I found are KVM-related)? The converted xl-config does not do the job: virsh -c xen:/// domxml-from-native --format xen-xl vm > vm.xml These are some none working examples I tried out: ... <interface type='hostdev' managed='yes'> <mac address='02:16:32:10:20:30'/> <driver name='xen'/> <source> <address type='pci' domain='0x0000' bus='0x81' slot='0x02' function='0x6'/> </source> <vlan> <tag id='11'/> </vlan> </interface> ... ... <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='xen'/> <source> <address domain='0x0000' bus='0x81' slot='0x02' function='0x6'/> </source> <address type='pci' domain='0x0000' bus='0x81' slot='0x02' function='0x6'/> </hostdev> ... Result is always the same, the VM does not find any interface to configure, when starting as libvirt: # dmesg | egrep -i "net|eth" ... [ 4.523173] iavf: Intel(R) Ethernet Adaptive Virtual Function Network Driver - version 4.4.2.1 # lspci | egrep -i "net|eth" 00:00.6 Ethernet controller: Intel Corporation Ethernet Virtual Function 700 Series (rev 02) # lspci -vmmks 00:00.6 Slot: 00:00.6 Class: Ethernet controller Vendor: Intel Corporation Device: Ethernet Virtual Function 700 Series SVendor: Intel Corporation SDevice: Device 0000 Rev: 02 Module: i40evf Module: iavf NUMANode: 0 # lsmod | egrep -i "iavf|i40" iavf 135168 0 auxiliary 16384 1 iavf ptp 20480 1 iavf # ifconfig eth0 eth0: error fetching interface information: Device not found The same VM after starting with XL: # dmesg | egrep -i "net|eth" ... [ 4.742038] iavf: Intel(R) Ethernet Adaptive Virtual Function Network Driver - version 4.4.2.1 [ 40.578461] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready [ 40.642868] iavf 0000:00:00.6 eth0: NIC Link is Up Speed is 10 Gbps Full Duplex [ 40.644015] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready # lspci | egrep -i "net|eth" 00:00.6 Ethernet controller: Intel Corporation Ethernet Virtual Function 700 Series (rev 02) # lspci -vmmks 00:00.6 Slot: 00:00.6 Class: Ethernet controller Vendor: Intel Corporation Device: Ethernet Virtual Function 700 Series SVendor: Intel Corporation SDevice: Device 0000 Rev: 02 Driver: iavf Module: i40evf Module: iavf NUMANode: 0 # lsmod | egrep -i "iavf|i40" iavf 135168 0 auxiliary 16384 1 iavf ptp 20480 1 iavf # ifconfig eth0 eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 10.20.30.40 netmask 255.255.255.0 broadcast 10.20.30.1 ... ether 02:16:32:10:20:30 txqueuelen 1000 (Ethernet) I expect, that all the SR-IOV stuff is configured correctly, because the VMs runs in XL-mode without any problems. Thanks a lot for any ideas, -- kind regards, Thorolf

On 3/21/23 11:05, nospam@godawa.de wrote:
Hi everybody,
since long time, I'm using Xen on CentOS with XL, currently latest CentOS 7 with Xen 4.15 from the CentOS-Xen-Project. For several VMs I have to use SR-IOV, to lower the CPU-usage on Dom0 on the host.
CentOS 7 comes to an end, Xen is not supported by RHEL nor RockyLinux anymore, so unfortunately I have to switch to KVM.
First step will be now, converting all the scripts for managing and running VMs, that they run with the additional libvirt-layer.
What is the libvirt version?
Mostly everything is working, but I do not get a network interface in the VM, when I start it with "virsh start ..." instead of "xl create ...".
First of all, is there documentation how to configure the VMs in the dom-definition for XEN (all docs I found are KVM-related)?
I know it's not complete, but https://libvirt.org/formatdomain.html makes an attempt to identify hypervisor support for the various configuration settings.
The converted xl-config does not do the job:
virsh -c xen:/// domxml-from-native --format xen-xl vm > vm.xml
These are some none working examples I tried out:
... <interface type='hostdev' managed='yes'> <mac address='02:16:32:10:20:30'/> <driver name='xen'/> <source> <address type='pci' domain='0x0000' bus='0x81' slot='0x02' function='0x6'/> </source> <vlan> <tag id='11'/> </vlan> </interface> ...
... <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='xen'/> <source> <address domain='0x0000' bus='0x81' slot='0x02' function='0x6'/> </source> <address type='pci' domain='0x0000' bus='0x81' slot='0x02' function='0x6'/> </hostdev> ...
I've used both configurations successfully. Has the sriov virtual function been unbound from the native driver and bound to xen-pciback? You can check which driver is in use with something like ls /sys/bus/pci/devices/0000\:81\:02.6/driver If bound to xen-pciback, it vf should also appear in the output of 'xl pci-assignable-list'. Regards, Jim

Jim Fehlig schrieb:
What is the libvirt version?
It's the "latest and greatest" I get from this source: [root@xen1 ~]# virsh --version 6.6.0 [root@xen1 ~]# libvirtd --version libvirtd (libvirt) 6.6.0 [root@xengfs1f ~]# yum list | grep libvirt libvirt.x86_64 1:6.6.0-6.xen415.el7 @centos-virt-xen-415 libvirt-bash-completion.x86_64 1:6.6.0-6.xen415.el7 @centos-virt-xen-415 libvirt-client.x86_64 1:6.6.0-6.xen415.el7 @centos-virt-xen-415 libvirt-daemon.x86_64 1:6.6.0-6.xen415.el7 @centos-virt-xen-415
I know it's not complete, but https://libvirt.org/formatdomain.html makes an attempt to identify hypervisor support for the various configuration settings.
I will "recheck" it, but I don't see anything new, that might be missing.
I've used both configurations successfully. Has the sriov virtual function been unbound from the native driver and bound to xen-pciback?
Should be, otherwise it would not work with XL?
You can check which driver is in use with something like
ls /sys/bus/pci/devices/0000\:81\:02.6/driver
I can't see the exact definition, but again, same output when starting with xl or virsh: [root@xen1 ~]# ls /sys/bus/pci/devices/0000\:81\:02.6/driver 0000:81:02.0 0000:81:02.7 0000:81:03.6 0000:81:04.5 0000:81:05.4 0000:81:06.5 0000:81:07.4 0000:81:08.3 0000:81:09.2 module slots 0000:81:02.1 0000:81:03.0 0000:81:03.7 0000:81:04.6 0000:81:05.5 0000:81:06.6 0000:81:07.5 0000:81:08.4 0000:81:09.3 new_id uevent 0000:81:02.2 0000:81:03.1 0000:81:04.0 0000:81:04.7 0000:81:06.0 0000:81:06.7 0000:81:07.6 0000:81:08.5 0000:81:09.4 new_slot unbind 0000:81:02.3 0000:81:03.2 0000:81:04.1 0000:81:05.0 0000:81:06.1 0000:81:07.0 0000:81:07.7 0000:81:08.6 0000:81:09.5 permissive 0000:81:02.4 0000:81:03.3 0000:81:04.2 0000:81:05.1 0000:81:06.2 0000:81:07.1 0000:81:08.0 0000:81:08.7 bind quirks 0000:81:02.5 0000:81:03.4 0000:81:04.3 0000:81:05.2 0000:81:06.3 0000:81:07.2 0000:81:08.1 0000:81:09.0 irq_handlers remove_id 0000:81:02.6 0000:81:03.5 0000:81:04.4 0000:81:05.3 0000:81:06.4 0000:81:07.3 0000:81:08.2 0000:81:09.1 irq_handler_state remove_slot [root@xen1 ~]# cat /sys/bus/pci/devices/0000\:81\:02.6/device 0x154c [root@xen1 ~]# cat /sys/bus/pci/devices/0000\:81\:02.6/vendor 0x8086 [root@xen1 ~]# cat /sys/bus/pci/devices/0000\:81\:02.6/subsystem_vendor 0x8086 [root@xen1 ~]# ls /sys/bus/pci/devices/0000\:81\:02.6/driver/module/drivers/ pci:pciback xen-backend:xen-pciback [root@xen1 ~]# ls /sys/bus/pci/devices/0000\:81\:02.6/driver/0000:81:02.6 broken_parity_status consistent_dma_mask_bits dma_mask_bits enable local_cpus numa_node reset resource0_wc subsystem uevent class d3cold_allowed driver irq modalias physfn resource resource3 subsystem_device vendor config device driver_override local_cpulist msi_bus power resource0 resource3_wc subsystem_vendor
If bound to xen-pciback, it vf should also appear in the output of 'xl pci-assignable-list'.
Before starting the VM, it's there: [root@xengfs1f ~]# xl pci-assignable-list | grep "0000:81:02.6" 0000:81:02.6 [root@xen1 ~]# [root@xen1 ~]# virsh start vm-test1 Domain vm-test1 gestartet [root@xen1 ~]# xl pci-assignable-list | grep "0000:81:02.6" [root@xen1 ~]# After starting it's "gone", but I assume that it's correct, because it's the same behaviour when starting the with xl create :-) Thanks a lot, -- kind regards, Thorolf

On 3/22/23 04:23, nospam@godawa.de wrote:
Jim Fehlig schrieb:
What is the libvirt version?
It's the "latest and greatest" I get from this source:
[root@xen1 ~]# virsh --version 6.6.0
[root@xen1 ~]# libvirtd --version libvirtd (libvirt) 6.6.0
[root@xengfs1f ~]# yum list | grep libvirt libvirt.x86_64 1:6.6.0-6.xen415.el7 @centos-virt-xen-415 libvirt-bash-completion.x86_64 1:6.6.0-6.xen415.el7 @centos-virt-xen-415 libvirt-client.x86_64 1:6.6.0-6.xen415.el7 @centos-virt-xen-415 libvirt-daemon.x86_64 1:6.6.0-6.xen415.el7 @centos-virt-xen-415
That's an old libvirt, but there hasn't been a lot of changes to the PCI passthrough code. One notable change that came with libvirt 6.8.0 https://gitlab.com/libvirt/libvirt/-/commit/9d15647dcb96831c93ac8c1d67c47265... However, that wouldn't be needed unless you were using the 'permissive' setting in your xl config.
I know it's not complete, but https://libvirt.org/formatdomain.html makes an attempt to identify hypervisor support for the various configuration settings.
I will "recheck" it, but I don't see anything new, that might be missing.
I've used both configurations successfully. Has the sriov virtual function been unbound from the native driver and bound to xen-pciback?
Should be, otherwise it would not work with XL?
You can check which driver is in use with something like
ls /sys/bus/pci/devices/0000\:81\:02.6/driver
I can't see the exact definition, but again, same output when starting with xl or virsh:
[root@xen1 ~]# ls /sys/bus/pci/devices/0000\:81\:02.6/driver 0000:81:02.0 0000:81:02.7 0000:81:03.6 0000:81:04.5 0000:81:05.4 0000:81:06.5 0000:81:07.4 0000:81:08.3 0000:81:09.2 module slots 0000:81:02.1 0000:81:03.0 0000:81:03.7 0000:81:04.6 0000:81:05.5 0000:81:06.6 0000:81:07.5 0000:81:08.4 0000:81:09.3 new_id uevent 0000:81:02.2 0000:81:03.1 0000:81:04.0 0000:81:04.7 0000:81:06.0 0000:81:06.7 0000:81:07.6 0000:81:08.5 0000:81:09.4 new_slot unbind 0000:81:02.3 0000:81:03.2 0000:81:04.1 0000:81:05.0 0000:81:06.1 0000:81:07.0 0000:81:07.7 0000:81:08.6 0000:81:09.5 permissive 0000:81:02.4 0000:81:03.3 0000:81:04.2 0000:81:05.1 0000:81:06.2 0000:81:07.1 0000:81:08.0 0000:81:08.7 bind quirks 0000:81:02.5 0000:81:03.4 0000:81:04.3 0000:81:05.2 0000:81:06.3 0000:81:07.2 0000:81:08.1 0000:81:09.0 irq_handlers remove_id 0000:81:02.6 0000:81:03.5 0000:81:04.4 0000:81:05.3 0000:81:06.4 0000:81:07.3 0000:81:08.2 0000:81:09.1 irq_handler_state remove_slot
[root@xen1 ~]# cat /sys/bus/pci/devices/0000\:81\:02.6/device 0x154c [root@xen1 ~]# cat /sys/bus/pci/devices/0000\:81\:02.6/vendor 0x8086 [root@xen1 ~]# cat /sys/bus/pci/devices/0000\:81\:02.6/subsystem_vendor 0x8086
[root@xen1 ~]# ls /sys/bus/pci/devices/0000\:81\:02.6/driver/module/drivers/ pci:pciback xen-backend:xen-pciback
[root@xen1 ~]# ls /sys/bus/pci/devices/0000\:81\:02.6/driver/0000:81:02.6 broken_parity_status consistent_dma_mask_bits dma_mask_bits enable local_cpus numa_node reset resource0_wc subsystem uevent class d3cold_allowed driver irq modalias physfn resource resource3 subsystem_device vendor config device driver_override local_cpulist msi_bus power resource0 resource3_wc subsystem_vendor
If bound to xen-pciback, it vf should also appear in the output of 'xl pci-assignable-list'.
Before starting the VM, it's there:
[root@xengfs1f ~]# xl pci-assignable-list | grep "0000:81:02.6" 0000:81:02.6
Ah, xen-pciback is already bound to the device, so no need for managed='yes' in your device config. The 'managed' attribute tells libvirt whether or not to manage attaching/detaching drivers to/from the device. If you've done that elsewhere, set managed='no', or simply drop the attribute since 'no' is the default. It's explained in the 'pci' attribute of the hostdev element https://libvirt.org/formatdomain.html#usb-pci-scsi-devices But I wouldn't expect that minor config optimization to fix your issue. I tried starting a VM with managed='yes' and the device already assigned to xen-pciback, but it still worked for me. The sriov vf was visible and usable in the VM. Are there any errors from xen-pciback in dom0, or any hints in the output of 'xl dmesg'? Also, it might be worth comparing the relevant nodes in the output of xenstore-ls between VM started with xl and libvirt. Regards, Jim

Hi Jim, thank you very much for your answers! Unfortunately I was busy with some other things, so couldn't look at it earlier.
That's an old libvirt, but there hasn't been a lot of changes to the PCI passthrough code. One notable change that came with libvirt 6.8.0
https://gitlab.com/libvirt/libvirt/-/commit/9d15647dcb96831c93ac8c1d67c47265...
However, that wouldn't be needed unless you were using the 'permissive' setting in your xl config.
I see, current version is somewhere at 9.x. But I'm not sure, if it's possible to upgade the servers to a new version of libvirt, I assume, that I also would have to upgrade Xen to the latest version.
I know it's not complete, but https://libvirt.org/formatdomain.html makes an attempt to identify hypervisor support for the various configuration settings.
I found this: "if you are using a version of libvirt older than 0.9.11, you should use standard <hostdev> to assign the device to the guest instead of <interface type='hostdev'/>." I think that 6.6 is newer than 0.9.11? Or do I have 0.6.6 in real?
Ah, xen-pciback is already bound to the device, so no need for managed='yes' in your device config. The 'managed' attribute tells libvirt whether or not to manage attaching/detaching drivers to/from the device. If you've done that elsewhere, set managed='no', or simply drop the attribute since 'no' is the default. It's explained in the 'pci' attribute of the hostdev element
I set this to "no", as you expected it didn't changed anything.
Are there any errors from xen-pciback in dom0, or any hints in the output of 'xl dmesg'? Also, it might be worth comparing the relevant nodes in the output of xenstore-ls between VM started with xl and libvirt.
I already had compared some informations between XL and libvirt, see my very first message. Unfortunately there are no errors in the logs, it's just not working. Any hints how to get further informations that might help debugging? Or could it just be a old bug in this version of libvirt? Thanks a lot, -- kind regards, Thorolf

On 3/24/23 09:22, nospam@godawa.de wrote:
Hi Jim,
thank you very much for your answers!
Unfortunately I was busy with some other things, so couldn't look at it earlier.
That's an old libvirt, but there hasn't been a lot of changes to the PCI passthrough code. One notable change that came with libvirt 6.8.0
https://gitlab.com/libvirt/libvirt/-/commit/9d15647dcb96831c93ac8c1d67c47265...
However, that wouldn't be needed unless you were using the 'permissive' setting in your xl config.
I see, current version is somewhere at 9.x. But I'm not sure, if it's possible to upgade the servers to a new version of libvirt, I assume, that I also would have to upgrade Xen to the latest version.
I know it's not complete, but https://libvirt.org/formatdomain.html makes an attempt to identify hypervisor support for the various configuration settings.
I found this:
"if you are using a version of libvirt older than 0.9.11, you should use standard <hostdev> to assign the device to the guest instead of <interface type='hostdev'/>."
I think that 6.6 is newer than 0.9.11? Or do I have 0.6.6 in real?
You have 6.6.0, which is much newer than 0.9.11 :-).
Ah, xen-pciback is already bound to the device, so no need for managed='yes' in your device config. The 'managed' attribute tells libvirt whether or not to manage attaching/detaching drivers to/from the device. If you've done that elsewhere, set managed='no', or simply drop the attribute since 'no' is the default. It's explained in the 'pci' attribute of the hostdev element
I set this to "no", as you expected it didn't changed anything.
Are there any errors from xen-pciback in dom0, or any hints in the output of 'xl dmesg'? Also, it might be worth comparing the relevant nodes in the output of xenstore-ls between VM started with xl and libvirt.
I already had compared some informations between XL and libvirt, see my very first message. Unfortunately there are no errors in the logs, it's just not working.
I may have missed it, but I don't see that you checked the output of xenstore-ls and 'xl list --long <vm-name>'. Can you compare the output of those commands when the VM is started with xl and with libvirt? Regards, Jim
participants (2)
-
Jim Fehlig
-
nospam@godawa.de