Re: [libvirt] [PATCH v4 5/6] vz: support misc migration options

On Wed, Sep 02, 2015 at 03:09:26PM +0300, Nikolay Shirokovskiy wrote: > From: nshirokovskiy(a)virtuozzo.com <nshirokovskiy(a)virtuozzo.com&gt; > > Migration API has a lot of options. This patch intention is to provide > support for those options that can be trivially supported and give > estimation for other options support in this commit message. > > I. Supported. > > 1. VIR_MIGRATE_COMPRESSED. Means 'use compression when migration domain > memory'. It is supported but quite uncommon way: vz migration demands that this > option should be set. This is due to vz is hardcoded to moving VMs memory using > compression. So anyone who wants to migrate vz domain should set this option > thus declaring it knows it uses compression. > > Why bother? May be just support this option and ignore if it is not set or > don't support at all as we can't change behaviour in this aspect. Well I > believe that this option is, first, inherent to hypervisor implementation as > we have a task of moving domain memory to different place and, second, we have > a tradeoff here between cpu and network resources and some managment should > choose the stratery via this option. If we choose ignoring or unsupporting > implementation than this option has a too vague meaning. Let's go into more > detail. > > First if we ignore situation where option is not set than we put user into > fallacy that vz hypervisor don't use compression and thus have lower cpu > usage. Second approach is to not support the option. The main reason not to > follow this way is that 'not supported and not set' is indistinguishable from > 'supported and not set' and thus again fool the user. I think it is reasonable to say that the absence of 'VIR_MIGRATE_COMPRESSED' does not mean compression should be turned off.

IOW, if VZ does compression by default, that's fine. You don't need to raise an error if VIR_MIGRATE_COMPRESSED is not passed. > 2. VIR_MIGRATE_LIVE. Means 'reduce domain downtime by suspending it as lately > as possible' which technically means 'migrate as much domain memory as possible > before suspending'. Supported in same manner as VIR_MIGRATE_COMPRESSED as both > vz VMs and CTs are always migrated via live scheme. > > One may be fooled by vz sdk flags of migration api: PVMT_HOT_MIGRATION(aka > live) and PVMT_WARM_MIGRATION(aka normal). Current implementation ignore these > flags and always use live migration. Actually that's not quite right interpretation. VIR_MIGRATE_LIVE means that the guest CPUs should remain running throughout migration. ie, if VIR_MIGRATE_LIVE is *not* provided by the caller, then the guest CPUs /must/ be paused while migration takes place. ie the

migration is non-live / coma migration. If VZ can't do this automatically as part of its migration API, you can fake it by just pausing the guest before hand & unpausing after.

Or if you don't wish to support coma-migration, you could raise an error if VIR_MIGRATE_LIVE is not specified by caller. > 3. VIR_MIGRATE_PERSIST_DEST, VIR_MIGRATE_UNDEFINE_SOURCE. This two comes > together. Vz domain are always persistent so we have to demand that options > VIR_MIGRATE_PERSIST_DEST and VIR_MIGRATE_UNDEFINE_SOURCE are both set (and > this is done just by unsupporting it). This is a tricky one. It would be nice if we could have just "done the right thing" by default with the migrate API, since the default behaviour is kind of stupid honestly, but we have to live with backwards compatible API semantics now :( To avoid forcing people to always specify these flags, could you "undo" by VZ does by default ? ie, if VIR_MIGRATE_PERSIST_DEST is *not* passed, then once migration has completed, you could use the domainMigrateFinish step to delete the config that VZ just created.

Similarly if VIR_MIGRATE_UNDEFINE_SOURCE is *not* passed, then once migration completed, you could use the domainMigrateConfirm step to re-create the config on the src that VZ just deleted.

> 4. VIR_MIGRATE_PAUSED. Means 'don't resume domain on destination'. This is > trivially supported as we have a corresponding option in vz migration. > > 5. VIR_MIGRATE_OFFLINE. Means 'migrate only XML definition of a domain'. It is > a forcing option that is it is ignored if domain is running and must be set > to migrate stopped domain. Vz implemenation follows this unformal definition > with one exception: non-shared disks will be migrated too. This desicion is on > par with VIR_MIGRATE_NON_SHARED_DISK condideration(see last part of this > notes). Ok > All that said the minimal command to migrate vz domain looks like next: > > migrate $DOMAIN $DESTINATION --compressed --live --persistent --undefinesource Pretty much all apps should be passing those regardless. eg OpenStack should definitely pass all those flags by default. virsh is kind of a dumb tool so it forces users to decide exactly what they want. > > Not good. Say if you want to just migrate a domain without further > details you will get error messages until you add these options to > command line. I think there is a lack of notion 'default' behaviour > in all these aspects. If we have it we could just issue: > > migrate $DOMAIN $DESTINATION I think we can get to that, if you're able to take the approachs I suggest above to workaround/override VZ defaults. > For vz this would give default compression for example, for qemu - default > no-compression. Then we could have flags --compressed and -no-compressed > and for vz the latter would give unsupported error. > > II. Unsupported. > > 1. VIR_MIGRATE_UNSAFE. Vz disks are always have 'cache=none' set (this > is not reflected in current version of vz driver and will be fixed > soon). So we need not to support this option. Ok, that's fine. > 2. VIR_MIGRATE_CHANGE_PROTECTION. Unsupported as we have no appopriate > support from vz sdk. Although we have locks they are advisory and > cant help us. Also fine. > 3. VIR_MIGRATE_TUNNELLED. Means 'use libvirtd to libvirtd connection > to pass hypervisor migration traffic'. Unsupported as not among > vz hypervisor usecases. That's fine, particularly if VZ provides encryption for its migration data stream anyway. The tunnnelled mode is just a hack to deal with fact that QEMU migration stream is clear text on the wire :-( > 4. Direct migration. Which is exposed via *toURI* interface with > VIR_MIGRATE_PEER2PEER flag unset. Means 'migrate without using > libvirtd on the other side'. To support it we should add authN > means to vz driver as mentioned in 'backbone patch' which looks > ugly. I don't think you need to worry about supporting that. You can support the traditional migrate those ie the non-ToURI migrate method without P2P set. > > 5. VIR_MIGRATE_ABORT_ON_ERROR, VIR_MIGRATE_AUTO_CONVERGE, > VIR_MIGRATE_RDMA_PIN_ALL, VIR_MIGRATE_NON_SHARED_INC, > VIR_MIGRATE_PARAM_DEST_XML, VIR_MIGRATE_PARAM_BANDWIDTH, > VIR_MIGRATE_PARAM_GRAPHICS_URI, VIR_MIGRATE_PARAM_LISTEN_ADDRESS, > VIR_MIGRATE_PARAM_MIGRATE_DISKS. > Without further discussion. They are just not usecases of vz hypevisor. Also fine. > > III. Undecided and thus unsupported. > > 6. VIR_MIGRATE_NON_SHARED_DISK. The meaning of this option is not clear to me. > Look, if qemu domain has a non-shared disk than it will refuse to migrate. But > after you specify this option it will refuse too. You need to create image file > for the disk on the destination side. Only after that you can migrate. Yep, kind of silly QEMU semantics here. > Unexpectedly existence of this file is enough to migrate without option too. In > this case you will get a domain on the destination with disk image unrelated to > source one and this is in case of live migration! Looks like a bug. Ok, imagine If the file exists on the dst, it is assumed the file is exposed via NFS or some other sensible shared filesystem. Libvirt probably ought to actually validate that the FS is truely shared and raise an error if not. > this is fixed so that migration of non-shared disk is only possible with > actual coping disk to destination. What we get from this option? We get > that you have to specify this option if you want to migrate a domain with > non-shared disk like some forcing option. May be it is a good approach > but it is incompatible with vz. Vz don't demand any user awareness of > migration of non-shared disks. And this case incompatibility can not > be easily resolved as for 'compressed' option as this option depends on > classifying of shared/non-shared for disks which is done inside vz. > vz: implement misc migration options So are you saying that if the disk is not on shared filesystem, then VZ will just "do the right thing" and copy the disk data across to the host ? If so, then I agree we can deal with this the same way as the compressed flag. ie just ignore it and let vz do the right thing automatically and not require the user to explicitly pass the SHARED flag.

Regards, Daniel