Re: [libvirt] [Qemu-devel] [RFC 0/2] Attempt to implement the standby feature for assigned network devices
10:18 a.m.
Quoting Sameeh Jubran (2018-10-25 13:01:10)
On Thu, Oct 25, 2018 at 5:06 PM Sameeh Jubran
<sameeh(a)daynix.com> wrote:
>
> From: Sameeh Jubran <sjubran(a)redhat.com>
>
> Hi all,
>
> Background:
>
> There has been a few attempts to implement the standby feature for vfio
> assigned devices which aims to enable the migration of such devices. This
> is another attempt.
>
> The series implements an infrastructure for hiding devices from the bus
> upon boot. What it does is the following:
>
> * In the first patch the infrastructure for hiding the device is added
> for the qbus and qdev APIs. A "hidden" boolean is added to the device
> state and it is set based on a callback to the standby device which
> registers itself for handling the assessment: "should the primary device
> be hidden?" by cross validating the ids of the devices.
>
> * In the second patch the virtio-net uses the API to hide the vfio
> device and unhides it when the feature is acked.
>
> Disclaimers:
>
> * I have only scratch tested this and from qemu side, it seems to be
> working.
> * This is an RFC so it lacks some proper error handling in few cases
> and proper resource freeing. I wanted to get some feedback first
> before it is finalized.
>
> Command line example:
>
> /home/sameeh/Builds/failover/qemu/x86_64-softmmu/qemu-system-x86_64 \
> -netdev
tap,id=hostnet0,script=world_bridge_standalone.sh,downscript=no,ifname=cc1_71 \
> -netdev
tap,vhost=on,id=hostnet1,script=world_bridge_standalone.sh,downscript=no,ifname=cc1_72,queues=4
\
> -device
virtio-net,host_mtu=1500,netdev=hostnet1,id=cc1_72,vectors=10,mq=on,primary=cc1_71 \
> -device e1000,netdev=hostnet0,id=cc1_71,standby=cc1_72 \
>
> Migration support:
>
> Pre migration or during setup phase of the migration we should send an
> unplug request to the guest to unplug the primary device. I haven't had
> the chance to implement that part yet but should do soon. Do you know
> what's the best approach to do so? I wanted to have a callback to the
> virtio-net device which tries to send an unplug request to the guest and
> if succeeds then the migration continues. It needs to handle the case where
> the migration fails and then it has to replug the primary device back.
I think that the "add_migration_state_change_notifier" API call can be used
from within the virtio-net device to achieve this, what do you think?
I think it would be good to hear from the libvirt folks (on Cc:) on this as
having QEMU unplug a device without libvirt's involvement seems like it
could cause issues. Personally I think it seems cleaner to just have QEMU
handle the 'hidden' aspects of the device and leave it to QMP/libvirt to do
the unplug beforehand. On the libvirt side I could imagine adding an option
like virsh migrate --switch-to-standby-networking or something along
that line to do it automatically (if we decide doing it automatically is
even needed on that end).
>
> The following terms are used as interchangeable:
> standby - virtio-net
> primary - vfio-device - physical device - assigned device
>
> Please share your thoughts and suggestions,
> Thanks!
>
> Sameeh Jubran (2):
> qdev/qbus: Add hidden device support
> virtio-net: Implement VIRTIO_NET_F_STANDBY feature
>
> hw/core/qdev.c | 48 +++++++++++++++++++++++++---
> hw/net/virtio-net.c | 25 +++++++++++++++
> hw/pci/pci.c | 1 +
> include/hw/pci/pci.h | 2 ++
> include/hw/qdev-core.h | 11 ++++++-
> include/hw/virtio/virtio-net.h | 5 +++
> qdev-monitor.c | 58 ++++++++++++++++++++++++++++++++--
> 7 files changed, 142 insertions(+), 8 deletions(-)
>
> --
> 2.17.0
>
--
Respectfully,
Sameeh Jubran
Linkedin
Software Engineer @ Daynix.
11:09 a.m.
New subject: [libvirt] [Qemu-devel] [RFC 0/2] Attempt to implement the standby feature for assigned network devices
On Wed, Dec 05, 2018 at 10:18:29 -0600, Michael Roth wrote:
Quoting Sameeh Jubran (2018-10-25 13:01:10)
> On Thu, Oct 25, 2018 at 5:06 PM Sameeh Jubran <sameeh(a)daynix.com> wrote:
> > From: Sameeh Jubran <sjubran(a)redhat.com>
> > Migration support:
> >
> > Pre migration or during setup phase of the migration we should send an
> > unplug request to the guest to unplug the primary device. I haven't had
> > the chance to implement that part yet but should do soon. Do you know
> > what's the best approach to do so? I wanted to have a callback to the
> > virtio-net device which tries to send an unplug request to the guest and
> > if succeeds then the migration continues. It needs to handle the case where
> > the migration fails and then it has to replug the primary device back.
> I think that the "add_migration_state_change_notifier" API call can be
used
> from within the virtio-net device to achieve this, what do you think?
I think it would be good to hear from the libvirt folks (on Cc:) on this as
having QEMU unplug a device without libvirt's involvement seems like it
could cause issues. Personally I think it seems cleaner to just have QEMU
handle the 'hidden' aspects of the device and leave it to QMP/libvirt to do
the unplug beforehand. On the libvirt side I could imagine adding an option
like virsh migrate --switch-to-standby-networking or something along
that line to do it automatically (if we decide doing it automatically is
even needed on that end).
I remember talking about this approach some time ago.
In general the migration itself is a very complex process which has too
many places where it can fail. The same applies to device hotunplug.
This series proposes to merge those two together into an even more
complex behemoth.
Few scenarios which don't have clear solution come into my mind:
- Since unplug request time is actually unbounded. The guest OS may
arbitrarily reject it or execute it at any later time, migration may get
stuck in a halfway state without any clear rollback or failure scenario.
- After migration, device hotplug may fail for whatever reason, leaving
networking crippled and again no clear single-case rollback scenario.
Then there's stuff which requires libvirt/management cooperation
- picking of the network device on destination
- making sure that the device is present etc.
From managements point of view, bundling all this together is really not
a good idea since it creates a very big matrix of failure scenarios. In
general even libvirt will prefer that upper layer management drives this
externally, since any rolback scenario will result in a policy decision
of what to do in certain cases, and what timeouts to pick.
11:22 a.m.
New subject: [libvirt] [Qemu-devel] [RFC 0/2] Attempt to implement the standby feature for assigned network devices
On Wed, Dec 05, 2018 at 06:09:16PM +0100, Peter Krempa wrote:
From managements point of view, bundling all this together is really
not
a good idea since it creates a very big matrix of failure scenarios.
I think this is clear. This is why we are doing it in QEMU where we can
actually do all the rollbacks transparently.
In
general even libvirt will prefer that upper layer management drives this
externally, since any rolback scenario will result in a policy decision
of what to do in certain cases, and what timeouts to pick.
Architectural ugliness of implementing what is from users perspective a
mechanism and not a policy aside, experience teaches that this isn't
going to happen. People have been talking about the idea of doing
this at the upper layers for years.
--
MST
11:26 a.m.
New subject: [libvirt] [Qemu-devel] [RFC 0/2] Attempt to implement the standby feature for assigned network devices
On Wed, Dec 05, 2018 at 12:22:18PM -0500, Michael S. Tsirkin wrote:
On Wed, Dec 05, 2018 at 06:09:16PM +0100, Peter Krempa wrote:
> From managements point of view, bundling all this together is really not
> a good idea since it creates a very big matrix of failure scenarios.
I think this is clear. This is why we are doing it in QEMU where we can
actually do all the rollbacks transparently.
> In
> general even libvirt will prefer that upper layer management drives this
> externally, since any rolback scenario will result in a policy decision
> of what to do in certain cases, and what timeouts to pick.
Architectural ugliness of implementing what is from users perspective a
mechanism and not a policy aside, experience teaches that this isn't
going to happen. People have been talking about the idea of doing
this at the upper layers for years.
The ability to unplugg+replug VFIO devices either side of migration
has existed in OpenStack for a long time. They also have metadata
that can be exposed to the guest to allow it to describe which pairs
of (emulated,vfio) devices should be used together.
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
11:43 a.m.
New subject: [libvirt] [Qemu-devel] [RFC 0/2] Attempt to implement the standby feature for assigned network devices
On Wed, Dec 05, 2018 at 06:09:16PM +0100, Peter Krempa wrote:
On Wed, Dec 05, 2018 at 10:18:29 -0600, Michael Roth wrote:
> Quoting Sameeh Jubran (2018-10-25 13:01:10)
> > On Thu, Oct 25, 2018 at 5:06 PM Sameeh Jubran <sameeh(a)daynix.com> wrote:
> > > From: Sameeh Jubran <sjubran(a)redhat.com>
> > > Migration support:
> > >
> > > Pre migration or during setup phase of the migration we should send an
> > > unplug request to the guest to unplug the primary device. I haven't
had
> > > the chance to implement that part yet but should do soon. Do you know
> > > what's the best approach to do so? I wanted to have a callback to the
> > > virtio-net device which tries to send an unplug request to the guest and
> > > if succeeds then the migration continues. It needs to handle the case
where
> > > the migration fails and then it has to replug the primary device back.
> > I think that the "add_migration_state_change_notifier" API call can
be used
> > from within the virtio-net device to achieve this, what do you think?
>
> I think it would be good to hear from the libvirt folks (on Cc:) on this as
> having QEMU unplug a device without libvirt's involvement seems like it
> could cause issues. Personally I think it seems cleaner to just have QEMU
> handle the 'hidden' aspects of the device and leave it to QMP/libvirt to do
> the unplug beforehand. On the libvirt side I could imagine adding an option
> like virsh migrate --switch-to-standby-networking or something along
> that line to do it automatically (if we decide doing it automatically is
> even needed on that end).
I remember talking about this approach some time ago.
In general the migration itself is a very complex process which has too
many places where it can fail. The same applies to device hotunplug.
This series proposes to merge those two together into an even more
complex behemoth.
Few scenarios which don't have clear solution come into my mind:
- Since unplug request time is actually unbounded. The guest OS may
arbitrarily reject it or execute it at any later time, migration may get
stuck in a halfway state without any clear rollback or failure scenario.
IMHO this is the really big deal. Doing this inside QEMU can arbitrarily
delay the start of migration, but this is opaque to mgmt apps because it
all becomes hidden behind the migrate command. It is common for mgmt apps
to serialize migration operations, otherwise they compete for limited
network bandwidth making it less likely that any will complete. If we're
waiting for a guest OS to do the unplug though, we don't want to be
stopping other migrations from being started in the mean time. Having
the unplugs done from the mgmt app explicitly gives it the flexibility
to decide how to order and serialize things to suit its needs.
- After migration, device hotplug may fail for whatever reason,
leaving
networking crippled and again no clear single-case rollback scenario.
I'd say s/crippled/degraded/. Anyway depending on the the reason that
triggered the migration, you may not even want to rollback to the source
host, despite the VFIO hotplug failing on the target.
If the original host was being evacuated in order to upgrade it to the latest
security patches, or due to hardware problems, it can be preferrable to just
let the VM start running on the target host with just emulated NICs only and
worry about getting working VFIO later.
Then there's stuff which requires libvirt/management cooperation
- picking of the network device on destination
- making sure that the device is present etc.
From managements point of view, bundling all this together is really not
a good idea since it creates a very big matrix of failure scenarios. In
general even libvirt will prefer that upper layer management drives this
externally, since any rolback scenario will result in a policy decision
of what to do in certain cases, and what timeouts to pick.
Indeed, leaving these policies decisions to the mgmt app has been a
better approach in general over time, as the view of what's the best
way to approach a problem has changed over time.
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
2182
days inactive
2182
days old
4 comments
4 participants
participants (4)
-
Daniel P. Berrangé -
Michael Roth -
Michael S. Tsirkin -
Peter Krempa