[libvirt] QEMU migration with non-shared storage

Hello, I am trying to understand libvirt's logic for checking whether migration of a VM is safe, and how it determines which disks should be mirrored by QEMU. My particular use case involves VMs that may have disks backed onto LVM or onto Ceph RBD, or both. As far as I can tell, the qemuMigrationIsSafe check is there to ensure that all disks are readonly, or have cache=none, or their backends can guarantee cache coherence. As far as I can tell, however, QEMU flushes *all* block devices when it pauses a VM's CPUs (just before the final part of migration, for instance), so I'm wondering why this check is needed. Is there any possible situation for the source VM to be paused, for its block devices to be flushed, and yet the destination VM can't see all completed writes? Why is RBD is handled specially in this function? The current logic is that an RBD-backed disk is safe to be migrated even if it's got caching enabled, but I'm not sure how RBD is different from other backends in this regard. If VIR_MIGRATE_NON_SHARED_DISK or _INC is specified, should these safety checks be relaxed? It seems to me that if any non-shared disk is going to be *explicitly* copied from the source to the destination VM, then cache coherence in the backend is irrelevant. At the moment, the set of non-shared block devices copied by VIR_MIGRATE_NON_SHARED_* differs depending on whether NBD is being used in the migration: - If NBD can't be used (e.g. with a tunnelled migration), then QEMU will copy *all* non-readonly block devices; - If NBD is being used, then QEMU will only mirror "shareable", "readonly" or "sourceless" disks. A problem arises with RBD disks that have caching enabled. According to qemuMigrationIsSafe, these disks are "safe" to be migrated. However in either the NBD or the non-NBD case, the RBD disk will be copied. This is clearly not desirable. If RBD is a special case in qemuMigrationIsSafe, does it also need to be a special case when configuring the NBD server? Or, if an NBD server is not going to be used, should the migration be considered "unsafe" if an RBD disk is present? I'd very much appreciate some help in understanding all of this. At the moment, I think my only option is to run RBD without caching at all. However, not only does that result in very poor performance, it also doesn't seem to match the qemuMigrationIsSafe check. Regards, Michael