live migration of SR-IOV vm
I have a vm using an sr-iov NIC that I'm testing live migration on (Debian
12, OS packages).
Per the documentation, I have the sr-iov link set as transient with a
pointer to the persistent virtio link:
<interface type='network'>
<mac address='52:54:00:a1:e0:38'/>
<source network='sr-iov-intel-10G-1'/>
<vlan>
<tag id='400'/>
</vlan>
<model type='virtio'/>
<teaming type='transient' persistent='ua-sr-iov-backup'/>
<address type='pci' domain='0x0000' bus='0x01'
slot='0x00'
function='0x0'/>
</interface>
and a persistent virto link down by default:
<interface type='direct'>
<mac address='52:54:00:a1:e0:38'/>
<source dev='eno5np0.400' mode='bridge'/>
<model type='virtio'/>
<teaming type='persistent'/>
<link state='down'/>
<alias name='ua-sr-iov-backup'/>
<address type='pci' domain='0x0000' bus='0x07'
slot='0x00'
function='0x0'/>
</interface>
The failover driver finds this in the vm:
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state
UP mode DEFAULT group default qlen 100
0
3: eth1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel
master eth0 state DOWN mode DEFAULT group
default qlen 1000
10: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master
eth0 state UP mode DEFAULT group default
qlen 1000
and the network works fine. However, during migration, the sr-iov
interface is removed, but the link on the virtio interface is *not*
brought up:
2: eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue
state DOWN mode DEFAULT group default qlen
1000
3: eth1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel
master eth0 state DOWN mode DEFAULT group
default qlen 1000
resulting in no network for part of the migration.
Once the box finished migrating, the replacement sv-iov link is plugged
back in, and all is well once again:
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state
UP mode DEFAULT group default qlen 1000
3: eth1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel
master eth0 state DOWN mode DEFAULT group default qlen 1000
11: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master
eth0 state UP mode DEFAULT group default qlen 1000
My understanding was that virsh migrate was supposed to automatically
bring up the virtio link when the sr-iov link is removed? Or do I need
to explicitly bring it up myself before the migration and take it down
after?
If I bring up the link manually before the migration, there are a few
packets lost at that time, but then none are lost during the migration
when the sr-iov link is pulled, or after the migration when I shut that
link down again. Ideally no packets would be lost :), but I realize
that's unlikely in practice...
Thanks...