Re: [libvirt] Use flock() instead of fcntl()

On Fri, Jul 26, 2013 at 11:35:32AM +0100, Daniel P. Berrange wrote: > On Fri, Jul 26, 2013 at 12:31:35PM +0200, David Weber wrote: > > Am Freitag, 26. Juli 2013, 10:14:59 schrieb Daniel P. Berrange: > > > On Fri, Jul 26, 2013 at 10:44:24AM +0200, David Weber wrote: > > > > > Looking again at flock() I see it cannot support locking of > > > > > ranges, only > > > > > the entire file. This makes it unsuitable as a replacement for > > > > > libvirt's > > > > > use of fcntl() I'm afraid. I can only sugggest you configure OCFS2 > > > > > so > > > > > that it supports fcntl(), or setup virtlockd to use separate > > > > > indirect > > > > > leases on a diffrent shared filesystem, or perhaps try sanlock > > > > > instead > > > > > which doesn't require any special filesystem support. > > > > > > > > It's true that flock() doesn't support locking of ranges but I can't > > > > see > > > > how this is necessary. > > > > > > The code may not currently use ranges, but that doesn't mean it'll > > > stay > > > that way. By adding support for flock() we're preventing us from > > > making > > > use of this feature in the future, and I don't want to see that. > > > > Just curious, what would be a possible feature which would require > > range > > based locking? > > > > I would really like to see flock() support in virtlockd because all > > other > > solutions have major drawbacks for me. > > Currently we use locks to protect the content of disk images. > > During startup/shutdown, however, libvirt also makes changes to the > metadata of images by setting SELinux labels, uid/gid ownership and > potentially ACLs. Currently we've delibrately crippled some of our > code during shutdown since it isn't safe in the face of multiple > libvirt's running. We need to introduce locking of file metadata > to protect this code. The metadata locks, however, must not conflict > with the content lock. Thus the reason why we only lock a single > byte (range 0-1) for content locks, is that we want to be able to > additional locks (range 1-2 or similar) for the metadata locks > on the same files.

I've been wondering over the weekend if there is any viable alternative strategy we could take which could allow us to use flock(), without badly compromising our option for metadata locking. Given that metadata locks would be only held for very short periods of time, I think it could be reasonable to say we don't need to do locks on the individual disks files. It would be sufficient to do locks on the directory containing the disk file, if we only have flock() available. This would mean we wouldn't be blocked by the inability to lock byte ranges.

I'm also wondering if there is a way to detect when fcntl() is not available for OCFS2 ? With the current virtlockd code, what is the behaviour when it tries to lock a file with fcntl() ? Does the fcntl lock attempt succeed, but only provide protection scoped to the single host, or do we get a hard errno from fcntl() on OCFS (eg EINVAL or something ? If we can detect broken fcntl() on OCFS, then we should not need to have a global config parameter - we would be able to automatically use fcntl() by default ,and fallback to flock() on OCFS2 deployments which aren't using cluster technology to enable fcntl().

Daniel