[libvirt-users] sr-iov on Intel 82576 and rhel 7 - would not work
hi everybody a windows kvm guest would not start, process gets killed with: Out of memory: Kill process 21984 (qemu-kvm) score 44 or sacrifice child I really don't know where/what I might be missing, config seems fine, everything looks ok - I only am not sure, do I need to first stub a SR-IOV device like regular passthrough? I'm trying sr-iov, having one NIC left to the host and the second give to a pool. Then I have in a guest: <interface type='network'> <mac address='52:54:00:99:df:43'/> <source network='passthrough-pool'/> <model type='rtl8139'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </interface> and the net: <network> <name>passthrough-pool</name> <uuid>dd5b4aa3-7c46-4874-a17f-6c0582738bd5</uuid> <forward mode='hostdev' managed='yes'> <pf dev='enp2s0f1'/> </forward> </network> ---- 02:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01) Subsystem: Super Micro Computer Inc Device 10c9 Flags: bus master, fast devsel, latency 0, IRQ 40 Memory at dfce0000 (32-bit, non-prefetchable) [size=128K] Memory at dfcc0000 (32-bit, non-prefetchable) [size=128K] I/O ports at a800 [size=32] Memory at dfc9c000 (32-bit, non-prefetchable) [size=16K] Expansion ROM at dfca0000 [disabled] [size=128K] Capabilities: [40] Power Management version 3 Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+ Capabilities: [70] MSI-X: Enable+ Count=10 Masked- Capabilities: [a0] Express Endpoint, MSI 00 Capabilities: [100] Advanced Error Reporting Capabilities: [140] Device Serial Number 00-25-90-ff-ff-72-bb-e2 Capabilities: [150] Alternative Routing-ID Interpretation (ARI) Capabilities: [160] Single Root I/O Virtualization (SR-IOV) Kernel driver in use: igb 02:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01) Subsystem: Super Micro Computer Inc Device 10c9 Flags: bus master, fast devsel, latency 0, IRQ 41 Memory at dfc20000 (32-bit, non-prefetchable) [size=128K] Memory at dfc00000 (32-bit, non-prefetchable) [size=128K] I/O ports at a400 [size=32] Memory at dfbdc000 (32-bit, non-prefetchable) [size=16K] Expansion ROM at dfbe0000 [disabled] [size=128K] Capabilities: [40] Power Management version 3 Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+ Capabilities: [70] MSI-X: Enable+ Count=10 Masked- Capabilities: [a0] Express Endpoint, MSI 00 Capabilities: [100] Advanced Error Reporting Capabilities: [140] Device Serial Number 00-25-90-ff-ff-72-bb-e2 Capabilities: [150] Alternative Routing-ID Interpretation (ARI) Capabilities: [160] Single Root I/O Virtualization (SR-IOV) Kernel driver in use: igb
On Fri, Sep 19, 2014 at 03:35:07PM +0100, lejeczek wrote:
hi everybody
a windows kvm guest would not start, process gets killed with:
Out of memory: Kill process 21984 (qemu-kvm) score 44 or sacrifice child
Not paying attention to the rest of the mail, I'd just say that kernel had to kill something due to not having enough memory. What are the memory statistics when starting the guest? What's your overcommit configuration?
I really don't know where/what I might be missing, config seems fine, everything looks ok - I only am not sure, do I need to first stub a SR-IOV device like regular passthrough?
I'm trying sr-iov, having one NIC left to the host and the second give to a pool. Then I have
in a guest:
<interface type='network'> <mac address='52:54:00:99:df:43'/> <source network='passthrough-pool'/> <model type='rtl8139'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </interface>
and the net:
<network> <name>passthrough-pool</name> <uuid>dd5b4aa3-7c46-4874-a17f-6c0582738bd5</uuid> <forward mode='hostdev' managed='yes'> <pf dev='enp2s0f1'/> </forward> </network>
---- 02:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01) Subsystem: Super Micro Computer Inc Device 10c9 Flags: bus master, fast devsel, latency 0, IRQ 40 Memory at dfce0000 (32-bit, non-prefetchable) [size=128K] Memory at dfcc0000 (32-bit, non-prefetchable) [size=128K] I/O ports at a800 [size=32] Memory at dfc9c000 (32-bit, non-prefetchable) [size=16K] Expansion ROM at dfca0000 [disabled] [size=128K] Capabilities: [40] Power Management version 3 Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+ Capabilities: [70] MSI-X: Enable+ Count=10 Masked- Capabilities: [a0] Express Endpoint, MSI 00 Capabilities: [100] Advanced Error Reporting Capabilities: [140] Device Serial Number 00-25-90-ff-ff-72-bb-e2 Capabilities: [150] Alternative Routing-ID Interpretation (ARI) Capabilities: [160] Single Root I/O Virtualization (SR-IOV) Kernel driver in use: igb 02:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01) Subsystem: Super Micro Computer Inc Device 10c9 Flags: bus master, fast devsel, latency 0, IRQ 41 Memory at dfc20000 (32-bit, non-prefetchable) [size=128K] Memory at dfc00000 (32-bit, non-prefetchable) [size=128K] I/O ports at a400 [size=32] Memory at dfbdc000 (32-bit, non-prefetchable) [size=16K] Expansion ROM at dfbe0000 [disabled] [size=128K] Capabilities: [40] Power Management version 3 Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+ Capabilities: [70] MSI-X: Enable+ Count=10 Masked- Capabilities: [a0] Express Endpoint, MSI 00 Capabilities: [100] Advanced Error Reporting Capabilities: [140] Device Serial Number 00-25-90-ff-ff-72-bb-e2 Capabilities: [150] Alternative Routing-ID Interpretation (ARI) Capabilities: [160] Single Root I/O Virtualization (SR-IOV) Kernel driver in use: igb
_______________________________________________ libvirt-users mailing list libvirt-users@redhat.com https://www.redhat.com/mailman/listinfo/libvirt-users
On 09/19/2014 10:35 AM, lejeczek wrote:
hi everybody
a windows kvm guest would not start, process gets killed with:
Out of memory: Kill process 21984 (qemu-kvm) score 44 or sacrifice child
I really don't know where/what I might be missing, config seems fine, everything looks ok - I only am not sure, do I need to first stub a SR-IOV device like regular passthrough?
I'm trying sr-iov, having one NIC left to the host and the second give to a pool.
I'm not aware of any way that error message might be related to device passthrough. Your config looks fine, as long as enp2s0f1 is the SRIOV card's PF (Physical Function) and the output of ip link show dev enp2s0f1 shows a list of VFs. In the case that you have no VFs created for that device, yoou can make sure this happens by adding the following to /etc/modprobe.d/local.conf: options igb max_vfs=7 BTW, unless you're concerned about the amount of bandwidth available on a single port of your switch, you can use the PF for host networking, and still use the VFs of that same PF for passthrough assignment to guests.
Then I have
in a guest:
<interface type='network'> <mac address='52:54:00:99:df:43'/> <source network='passthrough-pool'/> <model type='rtl8139'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </interface>
and the net:
<network> <name>passthrough-pool</name> <uuid>dd5b4aa3-7c46-4874-a17f-6c0582738bd5</uuid> <forward mode='hostdev' managed='yes'> <pf dev='enp2s0f1'/> </forward> </network>
---- 02:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01) Subsystem: Super Micro Computer Inc Device 10c9 Flags: bus master, fast devsel, latency 0, IRQ 40 Memory at dfce0000 (32-bit, non-prefetchable) [size=128K] Memory at dfcc0000 (32-bit, non-prefetchable) [size=128K] I/O ports at a800 [size=32] Memory at dfc9c000 (32-bit, non-prefetchable) [size=16K] Expansion ROM at dfca0000 [disabled] [size=128K] Capabilities: [40] Power Management version 3 Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+ Capabilities: [70] MSI-X: Enable+ Count=10 Masked- Capabilities: [a0] Express Endpoint, MSI 00 Capabilities: [100] Advanced Error Reporting Capabilities: [140] Device Serial Number 00-25-90-ff-ff-72-bb-e2 Capabilities: [150] Alternative Routing-ID Interpretation (ARI) Capabilities: [160] Single Root I/O Virtualization (SR-IOV) Kernel driver in use: igb 02:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01) Subsystem: Super Micro Computer Inc Device 10c9 Flags: bus master, fast devsel, latency 0, IRQ 41 Memory at dfc20000 (32-bit, non-prefetchable) [size=128K] Memory at dfc00000 (32-bit, non-prefetchable) [size=128K] I/O ports at a400 [size=32] Memory at dfbdc000 (32-bit, non-prefetchable) [size=16K] Expansion ROM at dfbe0000 [disabled] [size=128K] Capabilities: [40] Power Management version 3 Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+ Capabilities: [70] MSI-X: Enable+ Count=10 Masked- Capabilities: [a0] Express Endpoint, MSI 00 Capabilities: [100] Advanced Error Reporting Capabilities: [140] Device Serial Number 00-25-90-ff-ff-72-bb-e2 Capabilities: [150] Alternative Routing-ID Interpretation (ARI) Capabilities: [160] Single Root I/O Virtualization (SR-IOV) Kernel driver in use: igb
_______________________________________________ libvirt-users mailing list libvirt-users@redhat.com https://www.redhat.com/mailman/listinfo/libvirt-users
yes, I've tried both PFs' VFs - no difference if I only take SR-IOV out and use virtio that guest works perfectly fine. Hardware is a Supermicro H8DG6/H8DGi + Opterons 6366 HE and an Intel 82576 Gigabit Network Connection (rev 01). It seems very weird, I think if I am not missing anything config-wise then either the hardware platform as a whole or/and libvirt-qemu-kernel is buggy. any suggestion on how to troubleshoot it? many thanks ps. would you know how to make sure abrtd auto-reports to bugzilla in a non-gui target? On 20/09/14 22:59, Laine Stump wrote:
hi everybody
a windows kvm guest would not start, process gets killed with:
Out of memory: Kill process 21984 (qemu-kvm) score 44 or sacrifice child
I really don't know where/what I might be missing, config seems fine, everything looks ok - I only am not sure, do I need to first stub a SR-IOV device like regular passthrough?
I'm trying sr-iov, having one NIC left to the host and the second give to a pool. I'm not aware of any way that error message might be related to device
On 09/19/2014 10:35 AM, lejeczek wrote: passthrough.
Your config looks fine, as long as enp2s0f1 is the SRIOV card's PF (Physical Function) and the output of ip link show dev enp2s0f1 shows a list of VFs. In the case that you have no VFs created for that device, yoou can make sure this happens by adding the following to /etc/modprobe.d/local.conf:
options igb max_vfs=7
BTW, unless you're concerned about the amount of bandwidth available on a single port of your switch, you can use the PF for host networking, and still use the VFs of that same PF for passthrough assignment to guests.
Then I have
in a guest:
<interface type='network'> <mac address='52:54:00:99:df:43'/> <source network='passthrough-pool'/> <model type='rtl8139'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </interface>
and the net:
<network> <name>passthrough-pool</name> <uuid>dd5b4aa3-7c46-4874-a17f-6c0582738bd5</uuid> <forward mode='hostdev' managed='yes'> <pf dev='enp2s0f1'/> </forward> </network>
---- 02:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01) Subsystem: Super Micro Computer Inc Device 10c9 Flags: bus master, fast devsel, latency 0, IRQ 40 Memory at dfce0000 (32-bit, non-prefetchable) [size=128K] Memory at dfcc0000 (32-bit, non-prefetchable) [size=128K] I/O ports at a800 [size=32] Memory at dfc9c000 (32-bit, non-prefetchable) [size=16K] Expansion ROM at dfca0000 [disabled] [size=128K] Capabilities: [40] Power Management version 3 Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+ Capabilities: [70] MSI-X: Enable+ Count=10 Masked- Capabilities: [a0] Express Endpoint, MSI 00 Capabilities: [100] Advanced Error Reporting Capabilities: [140] Device Serial Number 00-25-90-ff-ff-72-bb-e2 Capabilities: [150] Alternative Routing-ID Interpretation (ARI) Capabilities: [160] Single Root I/O Virtualization (SR-IOV) Kernel driver in use: igb 02:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01) Subsystem: Super Micro Computer Inc Device 10c9 Flags: bus master, fast devsel, latency 0, IRQ 41 Memory at dfc20000 (32-bit, non-prefetchable) [size=128K] Memory at dfc00000 (32-bit, non-prefetchable) [size=128K] I/O ports at a400 [size=32] Memory at dfbdc000 (32-bit, non-prefetchable) [size=16K] Expansion ROM at dfbe0000 [disabled] [size=128K] Capabilities: [40] Power Management version 3 Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+ Capabilities: [70] MSI-X: Enable+ Count=10 Masked- Capabilities: [a0] Express Endpoint, MSI 00 Capabilities: [100] Advanced Error Reporting Capabilities: [140] Device Serial Number 00-25-90-ff-ff-72-bb-e2 Capabilities: [150] Alternative Routing-ID Interpretation (ARI) Capabilities: [160] Single Root I/O Virtualization (SR-IOV) Kernel driver in use: igb
_______________________________________________ libvirt-users mailing list libvirt-users@redhat.com https://www.redhat.com/mailman/listinfo/libvirt-users
participants (3)
-
Laine Stump -
lejeczek -
Martin Kletzander