| Subject: Re: [libvirt-users] automatically detaching PCI host devices from guest for SRIOV usage.
|
Dear Alex ,
|
| Thanks Once again for responding. I tried to replicate your steps. in our case readlink always reveals that the pci device is being
| handled by the pci-stub driver (as in our case ixgbevf is absent in the host)
|
| Below is an annotaded sequence of events.
|
| Check status :
# readlink -f /sys/bus/pci/devices/0000\:04\:10.0/driver
/sys/bus/pci/drivers/pci-stub
|
| Content of the nic.xml:
|
root@debian-squeeze:/usr/local/libvirt/etc/libvirt/qemu# cat /root/nic.xml
<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address domain='0x0000' bus='0x04' slot='0x10' function='0x0'/>
</source>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</hostdev>
# readlink -f /sys/bus/pci/devices/0000\:04\:10.0/driver
/sys/bus/pci/drivers/pci-stub
|
| corrent connectivity is OK
|
# ping 192.168.101.10
PING 192.168.101.10 (192.168.101.10) 56(84) bytes of data.
64 bytes from 192.168.101.10: icmp_req=1 ttl=64 time=0.491 ms
64 bytes from 192.168.101.10: icmp_req=2 ttl=64 time=0.384 ms
64 bytes from 192.168.101.10: icmp_req=3 ttl=64 time=0.310 ms
^C
--- 192.168.101.10 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1998ms
rtt min/avg/max/mdev = 0.310/0.395/0.491/0.074 ms
# readlink -f /sys/bus/pci/devices/0000\:04\:10.0/driver
/sys/bus/pci/drivers/pci-stub
|
| Detach and reattach and check connectivity in running guest.
| note: the guest has pci hotplug support , the ethernet interfaces
| are configured to allow hotplug and IP is statically assigned.
|
# virsh detach-device new-installed /root/nic.xml
Device detached successfully
# readlink -f /sys/bus/pci/devices/0000\:04\:10.0/driver
/sys/bus/pci/drivers/pci-stub
root@debian-squeeze:/usr/local/libvirt/etc/libvirt/qemu# ping 192.168.101.10
PING 192.168.101.10 (192.168.101.10) 56(84) bytes of data.
From 192.168.100.6 icmp_seq=9 Destination Host Unreachable
From 192.168.100.6 icmp_seq=10 Destination Host Unreachable
From 192.168.100.6 icmp_seq=11 Destination Host Unreachable
^C
--- 192.168.101.10 ping statistics ---
14 packets transmitted, 0 received, +3 errors, 100% packet loss, time 13096ms
pipe 3
# virsh attach-device new-installed /root/nic.xml
Device attached successfully
# ping 192.168.101.10
PING 192.168.101.10 (192.168.101.10) 56(84) bytes of data.
64 bytes from 192.168.101.10: icmp_req=1 ttl=64 time=0.507 ms
64 bytes from 192.168.101.10: icmp_req=2 ttl=64 time=0.388 ms
64 bytes from 192.168.101.10: icmp_req=3 ttl=64 time=0.314 ms
64 bytes from 192.168.101.10: icmp_req=4 ttl=64 time=0.315 ms
64 bytes from 192.168.101.10: icmp_req=5 ttl=64 time=0.335 ms
^C
--- 192.168.101.10 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 3998ms
rtt min/avg/max/mdev = 0.314/0.371/0.507/0.076 ms
# readlink -f /sys/bus/pci/devices/0000\:04\:10.0/driver
/sys/bus/pci/drivers/pci-stub
| Above proves that detach-attach cycle was successful.
|
# virsh detach-device new-installed /root/nic.xml
Device detached successfully
|
| Please note that guest is being shutdown with nic detached.
|
|
# virsh shutdown new-installed
Domain new-installed is being shutdown
| Guest being started again :
|
| # virsh start new-installed
Domain new-installed started
# ping 192.168.101.10
PING 192.168.101.10 (192.168.101.10) 56(84) bytes of data.
64 bytes from 192.168.101.10: icmp_req=1 ttl=64 time=0.583 ms
64 bytes from 192.168.101.10: icmp_req=2 ttl=64 time=0.402 ms
64 bytes from 192.168.101.10: icmp_req=3 ttl=64 time=0.308 ms
64 bytes from 192.168.101.10: icmp_req=4 ttl=64 time=0.316 ms
64 bytes from 192.168.101.10: icmp_req=5 ttl=64 time=0.318 ms
64 bytes from 192.168.101.10: icmp_req=6 ttl=64 time=0.247 ms
^C
--- 192.168.101.10 ping statistics ---
6 packets transmitted, 6 received, 0% packet loss, time 4997ms
rtt min/avg/max/mdev = 0.247/0.362/0.583/0.109 ms
| Please note on starting the guest the PCI device came up
| and connectivity was established. The device was also defined
| inside the guest XML.
|
|
| Now shutting down the Guest without detaching.
|
| # virsh shutdown new-installed
Domain new-installed is being shutdown
# virsh start new-installed
Domain new-installed started
|
|
# ping 192.168.101.10
PING 192.168.101.10 (192.168.101.10) 56(84) bytes of data.
From 192.168.100.6 icmp_seq=9 Destination Host Unreachable
From 192.168.100.6 icmp_seq=10 Destination Host Unreachable
From 192.168.100.6 icmp_seq=11 Destination Host Unreachable
^C
--- 192.168.101.10 ping statistics ---
14 packets transmitted, 0 received, +3 errors, 100% packet loss, time 13095ms
pipe 3
CONNECTION IS LOST
|
| We are using latest libvirtd (0.9.13) , lastest ixgbe/ixgbevf (from sf) , lastest stable kernel (3.5.1)
|
| Regards,
| Rajesh Kumar Mallah
| Tel. 91-11-46710500 (Ext:303)
| Cell +919811255597
| Website: http://www.tradeindia.com/
| \|/ \|/ \|/ stop printing \|/ \|/ \|/
| | | | start planting | | |
|
| | ----- "Alex Jia" <ajia@redhat.com> wrote:
| | From: "Alex Jia" <ajia@redhat.com>
| | To: "Rajesh Kumar Mallah" <mallah@tradeindia.com>
| | Cc: libvirt-users@redhat.com
| | Sent: Tuesday, August 21, 2012 9:34:24 AM
| | Subject: Re: [libvirt-users] automatically detaching PCI host devices from guest for SRIOV usage.
| |
| |
On 08/21/2012 11:22 AM, Rajesh Kumar Mallah wrote:
| |
| |
Dear Alex / List ,
| |
| |
Thanks for the response.
| |
| |
The problem is regarding detaching from the guest not host.
| |
| |
I see, if you attach a PCI device into guest with 'managed'
mode(managed='yes') then the device is automatically detached from
guest and returned to host after guest exiting, for example.
| |
| |
# lspci | grep Eth
| |
00:19.0 Ethernet controller: Intel Corporation 82566DM-2 Gigabit
Network Connection (rev 02)
| |
| |
# readlink /sys/bus/pci/devices/0000\:00\:19.0/driver/ -f
| |
/sys/bus/pci/drivers/e1000e
| |
| |
# virsh start foo
| |
Domain foo started
| |
| |
# ping -c 3 xx.xx.xx.xx
| |
PING xx.xx.xx.xx (xx.xx.xx.xx) 56(84) bytes of data.
| |
64 bytes from nx-in-f160.1e100.net (xx.xx.xx.xx): icmp_seq=1 ttl=49
time=87.8 ms
| |
64 bytes from nx-in-f160.1e100.net (xx.xx.xx.xx): icmp_seq=2 ttl=49
time=85.2 ms
| |
64 bytes from nx-in-f160.1e100.net (xx.xx.xx.xx): icmp_seq=3 ttl=49
time=91.8 ms
| |
--- xx.xx.xx.xx ping statistics ---
| |
3 packets transmitted, 3 received, 0% packet loss, time 2093ms
| |
rtt min/avg/max/mdev = 85.240/88.327/91.881/2.731 ms
| |
| |
# cat hostdev.xml
| |
<hostdev mode='subsystem' type='pci' managed='yes'>
| |
<source>
| |
<address domain='0x0000' bus='0x00' slot='0x19'
function='0x0'/>
| |
</source>
| |
</hostdev>
| |
| |
# virsh attach-device foo hostdev.xml
| |
Device attached successfully
| |
| |
# readlink /sys/bus/pci/devices/0000\:00\:19.0/driver/ -f
| |
/sys/bus/pci/drivers/pci-stub
| |
| |
# ping -c 3 xx.xx.xx.xx
| |
connect: Network is unreachable
| |
| |
# virsh destroy foo
| |
Domain foo destroyed
| |
| |
# readlink /sys/bus/pci/devices/0000\:00\:19.0/driver/ -f
| |
/sys/bus/pci/drivers/e1000e
| |
| |
# ping -c 3 xx.xx.xx.xx
| |
PING xx.xx.xx.xx (xx.xx.xx.xx) 56(84) bytes of data.
| |
64 bytes from xx.xx.xx.xx: icmp_seq=1 ttl=49 time=79.7 ms
| |
64 bytes from xx.xx.xx.xx: icmp_seq=2 ttl=49 time=83.2 ms
| |
64 bytes from xx.xx.xx.xx: icmp_seq=3 ttl=49 time=84.5 ms
| |
| |
--- xx.xx.xx.xx ping statistics ---
| |
3 packets transmitted, 3 received, 0% packet loss, time 2087ms
| |
rtt min/avg/max/mdev = 79.714/82.480/84.513/2.040 ms
| |
| |
Notes, need to wait for local network recovery again then check
network connectivity.
| |
| |
If your network still is disconnect, it may be a bug IMHO, please
show your libvirt
| |
and qemu-kvm version, or my example is not what you want, please let
me know.
| |
| |
Thanks,
| |
Alex
| |
| |
| | In our case since ixgbevf driver is NOT present in the
host hence the virtual
| |
interfaces never really attach themselves to the host.
| |
| |
Regards,
| |
Rajesh Kumar Mallah
| |
Tel. 91-11-46710500 (Ext:303)
| |
Cell +919811255597
| |
Website: http://www.tradeindia.com/
| |
\|/ \|/ \|/ stop printing \|/ \|/ \|/
| |
| | | start planting | | |
| |
| |
| |
----- "Alex Jia"
<ajia@redhat.com> wrote:
| |
| From: "Alex Jia"
<ajia@redhat.com>| |
| To: "Rajesh Kumar Mallah"
<mallah@tradeindia.com>| |
| Cc:
libvirt-users@redhat.com| |
| Sent: Tuesday, August 21, 2012 8:31:21 AM
| |
| Subject: Re: [libvirt-users] automatically detaching PCI host
devices from guest for SRIOV usage.
| |
|
| |
| On 08/20/2012 08:34 PM, Rajesh Kumar Mallah wrote:
Dear List ,
We are using intel 82599EB based SRIOV capable 10GbE cards
on the host machine.The VFs are being assigned to the guest
in pass through mode and are providing the network connectivity
as expected.
However when we shutdown the guest the connectivity does not
comes back via the VFs on the guest in next boot unless
the VFs are detached from the *guest* *before* initiating a
shutdown of the guests.
Dear Rajesh,
| |
|
| |
| Please see the following comments.
| |
|
| |
|
(
the detach is explicitly done via virsh command
/usr/local/libvirt/bin/virsh detach-device guest01 ~/nic.xml
nic.xml contained
<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address domain='0x0000' bus='0x04' slot='0x10' function='0x0'/>
</source>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</hostdev>
)
The question is anyone else facing similar problem and
if so what solution was used?
Secondly , is there anyway to automate the detaching of the
VFs from the guest when the guest is put in the shutdown
sequence ? (or for that matter destroyed).
| |
| When
managed is
"yes"
for a PCI device, it is automatically detached from the host
before being passed on to the guest, and automatically
reattached
to the host after the
guest exits such as shutdown a
guest.
| |
|
| |
| Regards,
| |
| Alex
| |
|
| |
|
Regards,
Rajesh Kumar Mallah
Tel. 91-11-46710500 (Ext:303)
Cell +919811255597
Website: http://www.tradeindia.com/
\|/ \|/ \|/ stop printing \|/ \|/ \|/
| | | start planting | | |
_______________________________________________
libvirt-users mailing list
libvirt-users@redhat.com
https://www.redhat.com/mailman/listinfo/libvirt-users
| |
|
| |