12:05 p.m.
Hi everybody,
since long time, I'm using Xen on CentOS with XL, currently latest
CentOS 7 with Xen 4.15 from the CentOS-Xen-Project. For several VMs I
have to use SR-IOV, to lower the CPU-usage on Dom0 on the host.
CentOS 7 comes to an end, Xen is not supported by RHEL nor RockyLinux
anymore, so unfortunately I have to switch to KVM.
First step will be now, converting all the scripts for managing and
running VMs, that they run with the additional libvirt-layer.
Mostly everything is working, but I do not get a network interface in
the VM, when I start it with "virsh start ..." instead of "xl create
...".
First of all, is there documentation how to configure the VMs in the
dom-definition for XEN (all docs I found are KVM-related)?
The converted xl-config does not do the job:
virsh -c xen:/// domxml-from-native --format xen-xl vm > vm.xml
These are some none working examples I tried out:
...
<interface type='hostdev' managed='yes'>
<mac address='02:16:32:10:20:30'/>
<driver name='xen'/>
<source>
<address type='pci' domain='0x0000' bus='0x81'
slot='0x02'
function='0x6'/>
</source>
<vlan>
<tag id='11'/>
</vlan>
</interface>
...
...
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='xen'/>
<source>
<address domain='0x0000' bus='0x81' slot='0x02'
function='0x6'/>
</source>
<address type='pci' domain='0x0000' bus='0x81'
slot='0x02'
function='0x6'/>
</hostdev>
...
Result is always the same, the VM does not find any interface to
configure, when starting as libvirt:
# dmesg | egrep -i "net|eth"
...
[ 4.523173] iavf: Intel(R) Ethernet Adaptive Virtual Function Network
Driver - version 4.4.2.1
# lspci | egrep -i "net|eth"
00:00.6 Ethernet controller: Intel Corporation Ethernet Virtual Function
700 Series (rev 02)
# lspci -vmmks 00:00.6
Slot: 00:00.6
Class: Ethernet controller
Vendor: Intel Corporation
Device: Ethernet Virtual Function 700 Series
SVendor: Intel Corporation
SDevice: Device 0000
Rev: 02
Module: i40evf
Module: iavf
NUMANode: 0
# lsmod | egrep -i "iavf|i40"
iavf 135168 0
auxiliary 16384 1 iavf
ptp 20480 1 iavf
# ifconfig eth0
eth0: error fetching interface information: Device not found
The same VM after starting with XL:
# dmesg | egrep -i "net|eth"
...
[ 4.742038] iavf: Intel(R) Ethernet Adaptive Virtual Function Network
Driver - version 4.4.2.1
[ 40.578461] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[ 40.642868] iavf 0000:00:00.6 eth0: NIC Link is Up Speed is 10 Gbps
Full Duplex
[ 40.644015] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
# lspci | egrep -i "net|eth"
00:00.6 Ethernet controller: Intel Corporation Ethernet Virtual Function
700 Series (rev 02)
# lspci -vmmks 00:00.6
Slot: 00:00.6
Class: Ethernet controller
Vendor: Intel Corporation
Device: Ethernet Virtual Function 700 Series
SVendor: Intel Corporation
SDevice: Device 0000
Rev: 02
Driver: iavf
Module: i40evf
Module: iavf
NUMANode: 0
# lsmod | egrep -i "iavf|i40"
iavf 135168 0
auxiliary 16384 1 iavf
ptp 20480 1 iavf
# ifconfig eth0
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.20.30.40 netmask 255.255.255.0 broadcast 10.20.30.1
...
ether 02:16:32:10:20:30 txqueuelen 1000 (Ethernet)
I expect, that all the SR-IOV stuff is configured correctly, because the
VMs runs in XL-mode without any problems.
Thanks a lot for any ideas,
--
kind regards,
Thorolf
8:35 p.m.
On 3/21/23 11:05, nospam(a)godawa.de wrote:
Hi everybody,
since long time, I'm using Xen on CentOS with XL, currently latest CentOS 7 with
Xen 4.15 from the CentOS-Xen-Project. For several VMs I have to use SR-IOV, to
lower the CPU-usage on Dom0 on the host.
CentOS 7 comes to an end, Xen is not supported by RHEL nor RockyLinux anymore,
so unfortunately I have to switch to KVM.
First step will be now, converting all the scripts for managing and running VMs,
that they run with the additional libvirt-layer.
What is the libvirt version?
Mostly everything is working, but I do not get a network interface in
the VM,
when I start it with "virsh start ..." instead of "xl create ...".
First of all, is there documentation how to configure the VMs in the
dom-definition for XEN (all docs I found are KVM-related)?
I know it's not complete, but https://libvirt.org/formatdomain.html makes an
attempt to identify hypervisor support for the various configuration settings.
The converted xl-config does not do the job:
virsh -c xen:/// domxml-from-native --format xen-xl vm > vm.xml
These are some none working examples I tried out:
...
<interface type='hostdev' managed='yes'>
<mac address='02:16:32:10:20:30'/>
<driver name='xen'/>
<source>
<address type='pci' domain='0x0000' bus='0x81'
slot='0x02'
function='0x6'/>
</source>
<vlan>
<tag id='11'/>
</vlan>
</interface>
...
...
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='xen'/>
<source>
<address domain='0x0000' bus='0x81' slot='0x02'
function='0x6'/>
</source>
<address type='pci' domain='0x0000' bus='0x81'
slot='0x02' function='0x6'/>
</hostdev>
...
I've used both configurations successfully. Has the sriov virtual function been
unbound from the native driver and bound to xen-pciback? You can check which
driver is in use with something like
ls /sys/bus/pci/devices/0000\:81\:02.6/driver
If bound to xen-pciback, it vf should also appear in the output of 'xl
pci-assignable-list'.
Regards,
Jim
5:23 a.m.
Jim Fehlig schrieb:
What is the libvirt version?
It's the "latest and greatest" I get from this source:
[root@xen1 ~]# virsh --version
6.6.0
[root@xen1 ~]# libvirtd --version
libvirtd (libvirt) 6.6.0
[root@xengfs1f ~]# yum list | grep libvirt
libvirt.x86_64 1:6.6.0-6.xen415.el7
@centos-virt-xen-415
libvirt-bash-completion.x86_64 1:6.6.0-6.xen415.el7
@centos-virt-xen-415
libvirt-client.x86_64 1:6.6.0-6.xen415.el7
@centos-virt-xen-415
libvirt-daemon.x86_64 1:6.6.0-6.xen415.el7
@centos-virt-xen-415
I know it's not complete, but
https://libvirt.org/formatdomain.html
makes an attempt to identify hypervisor support for the various
configuration settings.
I will "recheck" it, but I don't see anything new, that might be missing.
I've used both configurations successfully. Has the sriov virtual
function been unbound from the native driver and bound to xen-pciback?
Should be, otherwise it would not work with XL?
You can check which driver is in use with something like
ls /sys/bus/pci/devices/0000\:81\:02.6/driver
I can't see the exact definition, but again, same output when starting
with xl or virsh:
[root@xen1 ~]# ls /sys/bus/pci/devices/0000\:81\:02.6/driver
0000:81:02.0 0000:81:02.7 0000:81:03.6 0000:81:04.5 0000:81:05.4
0000:81:06.5 0000:81:07.4 0000:81:08.3 0000:81:09.2 module
slots
0000:81:02.1 0000:81:03.0 0000:81:03.7 0000:81:04.6 0000:81:05.5
0000:81:06.6 0000:81:07.5 0000:81:08.4 0000:81:09.3 new_id
uevent
0000:81:02.2 0000:81:03.1 0000:81:04.0 0000:81:04.7 0000:81:06.0
0000:81:06.7 0000:81:07.6 0000:81:08.5 0000:81:09.4 new_slot
unbind
0000:81:02.3 0000:81:03.2 0000:81:04.1 0000:81:05.0 0000:81:06.1
0000:81:07.0 0000:81:07.7 0000:81:08.6 0000:81:09.5 permissive
0000:81:02.4 0000:81:03.3 0000:81:04.2 0000:81:05.1 0000:81:06.2
0000:81:07.1 0000:81:08.0 0000:81:08.7 bind quirks
0000:81:02.5 0000:81:03.4 0000:81:04.3 0000:81:05.2 0000:81:06.3
0000:81:07.2 0000:81:08.1 0000:81:09.0 irq_handlers remove_id
0000:81:02.6 0000:81:03.5 0000:81:04.4 0000:81:05.3 0000:81:06.4
0000:81:07.3 0000:81:08.2 0000:81:09.1 irq_handler_state remove_slot
[root@xen1 ~]# cat /sys/bus/pci/devices/0000\:81\:02.6/device
0x154c
[root@xen1 ~]# cat /sys/bus/pci/devices/0000\:81\:02.6/vendor
0x8086
[root@xen1 ~]# cat /sys/bus/pci/devices/0000\:81\:02.6/subsystem_vendor
0x8086
[root@xen1 ~]# ls /sys/bus/pci/devices/0000\:81\:02.6/driver/module/drivers/
pci:pciback xen-backend:xen-pciback
[root@xen1 ~]# ls /sys/bus/pci/devices/0000\:81\:02.6/driver/0000:81:02.6
broken_parity_status consistent_dma_mask_bits dma_mask_bits enable
local_cpus numa_node reset resource0_wc subsystem
uevent
class d3cold_allowed driver irq
modalias physfn resource resource3 subsystem_device
vendor
config device driver_override
local_cpulist msi_bus power resource0 resource3_wc
subsystem_vendor
If bound to xen-pciback, it vf should also appear in the output of
'xl
pci-assignable-list'.
Before starting the VM, it's there:
[root@xengfs1f ~]# xl pci-assignable-list | grep "0000:81:02.6"
0000:81:02.6
[root@xen1 ~]#
[root@xen1 ~]# virsh start vm-test1
Domain vm-test1 gestartet
[root@xen1 ~]# xl pci-assignable-list | grep "0000:81:02.6"
[root@xen1 ~]#
After starting it's "gone", but I assume that it's correct, because
it's
the same behaviour when starting the with xl create :-)
Thanks a lot,
--
kind regards,
Thorolf
3:50 p.m.
On 3/22/23 04:23, nospam(a)godawa.de wrote:
Jim Fehlig schrieb:
>
> What is the libvirt version?
It's the "latest and greatest" I get from this source:
[root@xen1 ~]# virsh --version
6.6.0
[root@xen1 ~]# libvirtd --version
libvirtd (libvirt) 6.6.0
[root@xengfs1f ~]# yum list | grep libvirt
libvirt.x86_64 1:6.6.0-6.xen415.el7 @centos-virt-xen-415
libvirt-bash-completion.x86_64 1:6.6.0-6.xen415.el7 @centos-virt-xen-415
libvirt-client.x86_64 1:6.6.0-6.xen415.el7 @centos-virt-xen-415
libvirt-daemon.x86_64 1:6.6.0-6.xen415.el7 @centos-virt-xen-415
That's an old libvirt, but there hasn't been a lot of changes to the PCI
passthrough code. One notable change that came with libvirt 6.8.0
https://gitlab.com/libvirt/libvirt/-/commit/9d15647dcb96831c93ac8c1d67c47...
However, that wouldn't be needed unless you were using the 'permissive'
setting
in your xl config.
> I know it's not complete, but
https://libvirt.org/formatdomain.html makes an
> attempt to identify hypervisor support for the various configuration settings.
I will "recheck" it, but I don't see anything new, that might be missing.
> I've used both configurations successfully. Has the sriov virtual function
> been unbound from the native driver and bound to xen-pciback?
Should be, otherwise it would not work with XL?
> You can check which driver is in use with something like
>
> ls /sys/bus/pci/devices/0000\:81\:02.6/driver
I can't see the exact definition, but again, same output when starting with xl
or virsh:
[root@xen1 ~]# ls /sys/bus/pci/devices/0000\:81\:02.6/driver
0000:81:02.0 0000:81:02.7 0000:81:03.6 0000:81:04.5 0000:81:05.4
0000:81:06.5 0000:81:07.4 0000:81:08.3 0000:81:09.2 module slots
0000:81:02.1 0000:81:03.0 0000:81:03.7 0000:81:04.6 0000:81:05.5
0000:81:06.6 0000:81:07.5 0000:81:08.4 0000:81:09.3 new_id uevent
0000:81:02.2 0000:81:03.1 0000:81:04.0 0000:81:04.7 0000:81:06.0
0000:81:06.7 0000:81:07.6 0000:81:08.5 0000:81:09.4 new_slot unbind
0000:81:02.3 0000:81:03.2 0000:81:04.1 0000:81:05.0 0000:81:06.1
0000:81:07.0 0000:81:07.7 0000:81:08.6 0000:81:09.5 permissive
0000:81:02.4 0000:81:03.3 0000:81:04.2 0000:81:05.1 0000:81:06.2
0000:81:07.1 0000:81:08.0 0000:81:08.7 bind quirks
0000:81:02.5 0000:81:03.4 0000:81:04.3 0000:81:05.2 0000:81:06.3
0000:81:07.2 0000:81:08.1 0000:81:09.0 irq_handlers remove_id
0000:81:02.6 0000:81:03.5 0000:81:04.4 0000:81:05.3 0000:81:06.4
0000:81:07.3 0000:81:08.2 0000:81:09.1 irq_handler_state remove_slot
[root@xen1 ~]# cat /sys/bus/pci/devices/0000\:81\:02.6/device
0x154c
[root@xen1 ~]# cat /sys/bus/pci/devices/0000\:81\:02.6/vendor
0x8086
[root@xen1 ~]# cat /sys/bus/pci/devices/0000\:81\:02.6/subsystem_vendor
0x8086
[root@xen1 ~]# ls /sys/bus/pci/devices/0000\:81\:02.6/driver/module/drivers/
pci:pciback xen-backend:xen-pciback
[root@xen1 ~]# ls /sys/bus/pci/devices/0000\:81\:02.6/driver/0000:81:02.6
broken_parity_status consistent_dma_mask_bits dma_mask_bits enable
local_cpus numa_node reset resource0_wc subsystem uevent
class d3cold_allowed driver irq
modalias physfn resource resource3 subsystem_device vendor
config device driver_override local_cpulist
msi_bus power resource0 resource3_wc subsystem_vendor
> If bound to xen-pciback, it vf should also appear in the output of 'xl
> pci-assignable-list'.
Before starting the VM, it's there:
[root@xengfs1f ~]# xl pci-assignable-list | grep "0000:81:02.6"
0000:81:02.6
Ah, xen-pciback is already bound to the device, so no need for managed='yes' in
your device config. The 'managed' attribute tells libvirt whether or not to
manage attaching/detaching drivers to/from the device. If you've done that
elsewhere, set managed='no', or simply drop the attribute since 'no' is
the
default. It's explained in the 'pci' attribute of the hostdev element
https://libvirt.org/formatdomain.html#usb-pci-scsi-devices
But I wouldn't expect that minor config optimization to fix your issue. I tried
starting a VM with managed='yes' and the device already assigned to xen-pciback,
but it still worked for me. The sriov vf was visible and usable in the VM.
Are there any errors from xen-pciback in dom0, or any hints in the output of 'xl
dmesg'? Also, it might be worth comparing the relevant nodes in the output of
xenstore-ls between VM started with xl and libvirt.
Regards,
Jim
10:22 a.m.
Hi Jim,
thank you very much for your answers!
Unfortunately I was busy with some other things, so couldn't look at it
earlier.
That's an old libvirt, but there hasn't been a lot of changes
to the PCI
passthrough code. One notable change that came with libvirt 6.8.0
https://gitlab.com/libvirt/libvirt/-/commit/9d15647dcb96831c93ac8c1d67c47...
However, that wouldn't be needed unless you were using the 'permissive'
setting in your xl config.
I see, current version is somewhere at 9.x. But I'm not sure, if it's
possible to upgade the servers to a new version of libvirt, I assume,
that I also would have to upgrade Xen to the latest version.
>> I know it's not complete, but
https://libvirt.org/formatdomain.html
>> makes an attempt to identify hypervisor support for the various
>> configuration settings.
I found this:
"if you are using a version of libvirt older than 0.9.11, you should use
standard <hostdev> to assign the device to the guest instead of
<interface type='hostdev'/>."
I think that 6.6 is newer than 0.9.11? Or do I have 0.6.6 in real?
Ah, xen-pciback is already bound to the device, so no need for
managed='yes' in your device config. The 'managed' attribute tells
libvirt whether or not to manage attaching/detaching drivers to/from the
device. If you've done that elsewhere, set managed='no', or simply drop
the attribute since 'no' is the default. It's explained in the 'pci'
attribute of the hostdev element
https://libvirt.org/formatdomain.html#usb-pci-scsi-devices
I set this to "no", as you expected it didn't changed anything.
Are there any errors from xen-pciback in dom0, or any hints in the
output of 'xl dmesg'? Also, it might be worth comparing the relevant
nodes in the output of xenstore-ls between VM started with xl and libvirt.
I already had compared some informations between XL and libvirt, see my
very first message. Unfortunately there are no errors in the logs, it's
just not working.
Any hints how to get further informations that might help debugging? Or
could it just be a old bug in this version of libvirt?
Thanks a lot,
--
kind regards,
Thorolf
9:44 p.m.
On 3/24/23 09:22, nospam(a)godawa.de wrote:
Hi Jim,
thank you very much for your answers!
Unfortunately I was busy with some other things, so couldn't look at it earlier.
> That's an old libvirt, but there hasn't been a lot of changes to the PCI
> passthrough code. One notable change that came with libvirt 6.8.0
>
> https://gitlab.com/libvirt/libvirt/-/commit/9d15647dcb96831c93ac8c1d67c47...
>
> However, that wouldn't be needed unless you were using the 'permissive'
> setting in your xl config.
I see, current version is somewhere at 9.x. But I'm not sure, if it's possible
to upgade the servers to a new version of libvirt, I assume, that I also would
have to upgrade Xen to the latest version.
>>> I know it's not complete, but https://libvirt.org/formatdomain.html makes
an
>>> attempt to identify hypervisor support for the various configuration
settings.
I found this:
"if you are using a version of libvirt older than 0.9.11, you should use
standard <hostdev> to assign the device to the guest instead of <interface
type='hostdev'/>."
I think that 6.6 is newer than 0.9.11? Or do I have 0.6.6 in real?
You have 6.6.0, which is much newer than 0.9.11 :-).
> Ah, xen-pciback is already bound to the device, so no need for managed='yes'
> in your device config. The 'managed' attribute tells libvirt whether or not
to
> manage attaching/detaching drivers to/from the device. If you've done that
> elsewhere, set managed='no', or simply drop the attribute since 'no'
is the
> default. It's explained in the 'pci' attribute of the hostdev element
>
> https://libvirt.org/formatdomain.html#usb-pci-scsi-devices
I set this to "no", as you expected it didn't changed anything.
> Are there any errors from xen-pciback in dom0, or any hints in the output of
> 'xl dmesg'? Also, it might be worth comparing the relevant nodes in the
output
> of xenstore-ls between VM started with xl and libvirt.
I already had compared some informations between XL and libvirt, see my very
first message. Unfortunately there are no errors in the logs, it's just not
working.
I may have missed it, but I don't see that you checked the output of xenstore-ls
and 'xl list --long <vm-name>'. Can you compare the output of those commands
when the VM is started with xl and with libvirt?
Regards,
Jim
598
days inactive
612
days old
5 comments
2 participants
participants (2)
-
Jim Fehlig -
nospam＠godawa.de