Anthony Liguori <anthony@codemonkey.ws> wrote:
On 11/14/2011 04:16 AM, Daniel P. Berrange wrote:
On Sat, Nov 12, 2011 at 12:25:34PM +0200, Avi Kivity wrote:
On 11/11/2011 12:15 PM, Kevin Wolf wrote:
Am 10.11.2011 22:30, schrieb Anthony Liguori:
Live migration with qcow2 or any other image format is just not going to work right now even with proper clustered storage. I think doing a block level flush cache interface and letting block devices decide how to do it is the best approach.
I would really prefer reusing the existing open/close code. It means less (duplicated) code, is existing code that is well tested and doesn't make migration much of a special case.
If you want to avoid reopening the file on the OS level, we can reopen only the topmost layer (i.e. the format, but not the protocol) for now and in 1.1 we can use bdrv_reopen().
Intuitively I dislike _reopen style interfaces. If the second open yields different results from the first, does it invalidate any computations in between?
What's wrong with just delaying the open?
If you delay the 'open' until the mgmt app issues 'cont', then you loose the ability to rollback to the source host upon open failure for most deployed versions of libvirt. We only fairly recently switched to a five stage migration handshake to cope with rollback when 'cont' fails.
Delayed open isn't a panacea. With the series I sent, we should be able to migration with a qcow2 file on coherent shared storage.
There are two other cases that we care about: migration with nfs cache!=none and direct attached storage with cache!=none
Whether the open is deferred matters less with NFS than if the open happens after the close on the source. To fix NFS cache!=none, we would have to do a bdrv_close() before sending the last byte of migration data and make sure that we bdrv_open() after receiving the last byte of migration data.
The problem with this IMHO is it creates a large window where noone has the file open and you're critically vulnerable to losing your VM.
Red Hat NFS guru told that fsync() on source + open() after that on target is enough. But anyways, it still depends of nothing else having the file opened on target.
I'm much more in favor of a smarter caching policy. If we can fcntl() our way to O_DIRECT on NFS, that would be fairly interesting. I'm not sure if this is supported today but it's something we could look into adding in the kernel. That way we could force NFS to O_DIRECT during migration which would solve this problem robustly.
We would need O_DIRECT on target during migration, I agree than that would work.
Deferred open doesn't help with direct attached storage. There simple is no guarantee that there isn't data in the page cache.
Yeap, I asked the clustered filesystem people how they fixed the problem, because clustered filesystem have this problem, right. After lots of arm twisting, I got the ioctl(BLKFLSBUF,...), but that only works: - on linux - on some block devices So, we are back to square 1.
Again, I think defaulting DAS to cache=none|directsync is what makes the most sense here.
I think it is the only sane solution. Otherwise, we need to write the equivalent of a lock manager, to know _who_ has the storage, and distributed lock managers are a mess :-(
We can even add a migration blocker for DAS with cache=on. If we can do dynamic toggling of the cache setting, then that's pretty friendly at the end of the day.
That could fix the problem also. At the moment that we start migration, we do an fsync() + switch to O_DIRECT for all filesystems. As you said, time for implementing fcntl(O_DIRECT). Later, Juan.