On Mon, Aug 11, 2014 at 04:30:19PM -0400, John Ferlan wrote: > Currently the safezero() function uses build conditionals to choose either > the posix_fallocate() or mmap() with a fallback to safewrite() in order to > preallocate a file. > > This patch will modify the logic in order to allow fallbacks in the > event that posix_fallocate() or the ftruncate()and mmap() doesn't work > properly. The fallback will be to use the slow safewrite of zero filled > buffers to the file. Have you actually encountered failing of posix_fallocate() in the real world ? It is supposed to automatically fallback to the equivalent of writing zeros if the filesystem / kernel does not support it, so we should not have todo runtime fallback ourselves. The existance of fallback is the main distinction between the posix_fallocate() and fallocate() system calls.

On Fri, Aug 22, 2014 at 10:56:47AM -0400, John Ferlan wrote: > > > On 08/22/2014 10:46 AM, Daniel P. Berrange wrote: > > On Mon, Aug 11, 2014 at 04:30:19PM -0400, John Ferlan wrote: > >> Currently the safezero() function uses build conditionals to choose either > >> the posix_fallocate() or mmap() with a fallback to safewrite() in order to > >> preallocate a file. > >> > >> This patch will modify the logic in order to allow fallbacks in the > >> event that posix_fallocate() or the ftruncate()and mmap() doesn't work > >> properly. The fallback will be to use the slow safewrite of zero filled > >> buffers to the file. > > > > Have you actually encountered failing of posix_fallocate() in the > > real world ? It is supposed to automatically fallback to the > > equivalent of writing zeros if the filesystem / kernel does not > > support it, so we should not have todo runtime fallback ourselves. > > The existance of fallback is the main distinction between the > > posix_fallocate() and fallocate() system calls. > > > > It wasn't so much as a "failure" as "unexpected results" - the key being > that the resulting created (or resized) file was not sized as expected. > > For an NFS target the results are not what was expected. I've left some > history in the prior set of patches with the following probably having > the most details: > > http://www.redhat.com/archives/libvir-list/2014-August/msg00367.html So, IIUC, the bug happens when the rsize mount option to NFS is not 4k. strace'ing libvirtd on an NFS volume in this case shows: open("/var/lib/libvirt/images/lettuce/foo", O_RDWR|O_CREAT|O_EXCL, 0600) = 24 fstat(24, {st_mode=S_IFREG|0600, st_size=0, ...}) = 0 ftruncate(24, 1073741824) = 0 fallocate(24, 0, 0, 1073741824) = -1 EOPNOTSUPP (Operation not supported) fallocate(24, 0, 0, 1073741824) = -1 EOPNOTSUPP (Operation not supported) fstat(24, {st_mode=S_IFREG|0600, st_size=1073741824, ...}) = 0 fstatfs(24, {f_type="NFS_SUPER_MAGIC", f_bsize=1048576, f_blocks=118342, f_bfree=71002, f_bavail=65632, f_files=7678560, f_ffree=5495931, f_fsid={0, 0}, f_namelen=255, f_frsize=1048576}) = 0 pread(24, "\0", 1, 1048575) = 1 pwrite(24, "\0", 1, 1048575) = 1 pread(24, "\0", 1, 2097151) = 1 pwrite(24, "\0", 1, 2097151) = 1 pread(24, "\0", 1, 3145727) = 1 So we can see glibc here trying fallocate() and then falling back to writing zeros. Since the volume does not come out at the right size this seems to show a bug in glibc. So I think we really ought to report that bug to glibc to be fixed there rather than working around it in libvirt, as there are many more applications besides libvirt that will be impacted by this bug.

On 08/21/2014 06:51 PM, John Ferlan wrote: > >

> Hmm.. Perhaps a better way to do this would be to modify safezero() to > add a 4th boolean parameter "resize" and make the "safezero_mmap()" be > the false side and the check/call to SYS_fallocate() be the true side. > That way all the logic resides in virfile.c How about 'fallback_to_mmap' instead of resize? Or even simpler, we don't really need a different set of fallbacks for resize than the other uses. It seems the patch adding the preallocation to FileResize should have used safezero instead of reimplementing it. (I also wonder if there are systems without posix_fallocate, but having the syscall...) ACK to safezero implementation with all four methods: int safezero(int fd, off_t offset, off_t len) { if (virFileFdPosixFallocate(fd, offset, len) == 0) return 0; if (safezero_sys_fallocate(fd, offset, len) == 0) return 0; if (safezero_mmap(fd, offset, len) == 0) return 0; return safezero_slow(fd, offset, len); }

Changes over v1 - different tact as more research discovers that the posix_fallocate() does not perform the correct pre-allocation of space for an NFS backed file/disk, some details of the findings can be found: http://www.redhat.com/archives/libvir-list/2014-August/msg00367.html The first patch in this series will refactor the safezero to allow for more fallback than the current code. Initially implemented as a series of three 'safezero()' functions in commit id 'c29d0929' using build conditionals to determine which of the 3 is being used. The refactored code will have one function that will make a series of calls rather the just failing on whatever is built into the system. The first is a global virFileFdPosixFallocate() with the build conditional controlling only whether the function runs or will just return -1. It will also be re-used by the Volume Resize code in a future patch. If on creation the virFileFdPosixFallocate() fails, attempts will be made to then try the mmap() method (if it exists), and then fall back to the slow, old, safewrite of zero filled buffers. The mmap function will follow the format of the posix_fallocate insomuch as if HAVE_MMAP is defined, then that will be attempted or a -1 will be returned immediately. The major change of this patch over the prior code is the fallback. One other minor change is if the virFileFdPosixFallocate() tries to call posix_fallocate() and fails, then a VIR_WARN will be used to indicate something went wrong. Previously, the code would return errno and eventually that would filter back to a caller which would use the errno in a sys error message. This I think only is odd when it comes to the resize path. The second patch looks to resolve the first issue discovered by: https://bugzilla.redhat.com/show_bug.cgi?id=1077068 It does this by checking the returned size/len of the file in the posix_fallocate() call and if it does not match a VIR_WARN will be posted and a failure returned allowing either the mmap or write of zero filled buffers to the file. The third patch changes the virStorageFileResize() logic to follow the safezero() logic with respect to build conditionals. Also, rather than reinvent the wheel, it will reuse the virFileFdPosixFallocate(). The new static API resizeSysFallocate() will return -1 if the build conditionals are not met. A current "downside" is by calling virFileFdPosixFallocate() twice (once directly and the other indirectly through safezero), the VIR_WARN will be displayed twice, but that should only affect someone trying to debug things. NOTE: Investigation and testing while creating this series also showed it's possible to modify the NFS Pool Start code to add a "-o wsize=4096" to the MOUNT command in order to force smaller write's. While that did fix the symptoms, the fix just didn't "feel right". The block size of the "source" export was 4096 bytes, while the block size of the "target" file was 1048576 bytes (current NFSv4 wsize default value when not specified). Whether those play into what posix_fallocate() does I am not sure, but if it does and that's fixed, then the way these patches are coded up - it won't matter. Compared to perhaps needing to revisit or document heavily how to set some new field in the pool XML source - this just seems more right. John Ferlan (3): virfile: Refactor safezero, introduce virFileFdPosixFallocate virfile: Check returned size from virFileFdPosixFallocate virstoragefile: Refactor virStorageFileResize src/libvirt_private.syms | 1 + src/util/virfile.c | 58 ++++++++++++++++++++++++++++++++++++++--------- src/util/virfile.h | 2 ++ src/util/virstoragefile.c | 52 +++++++++++++++++++++++++++--------------- 4 files changed, 84 insertions(+), 29 deletions(-)