Daniel P. Berrange wrote: > QEMU defaults to allowing the host OS to cache all disk I/O. THis has a > couple of problems > > - It is a waste of memory because the guest already caches I/O ops > - It is unsafe on host OS crash - all unflushed guest I/O will be > lost, and there's no ordering guarentees, so metadata updates could > be flushe to disk, while the journal updates were not. Say goodbye > to your filesystem. > - It makes benchmarking more or less impossible / worthless because > what the benchmark things are disk writes just sit around in memory > so guest disk performance appears to exceed host diskperformance. > > This patch disables caching on all QEMU guests. NB, Xen has long done this > for both PV & HVM guests - QEMU only gained this ability when -drive was > introduced, and sadly kept the default to unsafe cache=on settings. I'm for this in general, but I'm a little worried about the "performance regression" aspect of this. People are going to upgrade to 0.4.7 (or whatever), and suddenly find that their KVM guests perform much more slowly. This is better in the end for their data, but we might hear large complaints about it.

Might it be a better idea to make the default "cache=off", but provide a toggle in the domain XML to turn it back to "cache=on" for the people who really want it and know what they are doing?

Daniel P. Berrange wrote: >On Wed, Oct 08, 2008 at 01:15:46PM +0200, Chris Lalancette wrote: >>Daniel P. Berrange wrote: >>>QEMU defaults to allowing the host OS to cache all disk I/O. THis has a >>>couple of problems >>> >>> - It is a waste of memory because the guest already caches I/O ops >>> - It is unsafe on host OS crash - all unflushed guest I/O will be >>> lost, and there's no ordering guarentees, so metadata updates could >>> be flushe to disk, while the journal updates were not. Say goodbye >>> to your filesystem. >>> - It makes benchmarking more or less impossible / worthless because >>> what the benchmark things are disk writes just sit around in memory >>> so guest disk performance appears to exceed host diskperformance. >>> >>>This patch disables caching on all QEMU guests. NB, Xen has long done >>>this >>>for both PV & HVM guests - QEMU only gained this ability when -drive was >>>introduced, and sadly kept the default to unsafe cache=on settings. >>I'm for this in general, but I'm a little worried about the "performance >>regression" aspect of this. People are going to upgrade to 0.4.7 (or >>whatever), >>and suddenly find that their KVM guests perform much more slowly. This is >>better in the end for their data, but we might hear large complaints >>about it. > >Yes & no. They will find their guests perform more consistently. With the >current system their guests will perform very erratically depending on >memory & I/O pressure on the host. If the host I/O cache is empty & has >no I/O load, current guests will be "fast", They will perform marginally better than if cache=off. This is the Linux host knows more about the underlying hardware than the guest and is able to do smarter read-ahead. When using cache=off, the host cannot perform any sort of read-ahead. >but if host I/O cache is full >and they do something which requires more host memory (eg start up another >guest), then all existing guests get their I/O performance trashed as the >I/O cache has to be flushed out, and future I/O is unable to be cached. This is not accurate. Dirty pages in the host page cache are not reclaimable until they're written to disk. If you're in a seriously low memory situation, they the thing allocating memory is going to sleep until the data is written to disk. If an existing guest is trying to do I/O, then what things will degenerate to is basically cache=off since the guest must wait for other pending IO to complete >Xen went through this same change and there were not any serious >complaints, particularly when explained that previous system had >zero data integrity guarentees. The current system merely provides an >illusion of performance - any attempt to show that performance has >decreased is impossible because any attempt to run benchmarks with >existing caching just results in meaningless garbage. > >https://bugzilla.redhat.com/show_bug.cgi?id=444047 I can't see this bug, but a quick grep of ioemu in xen-unstable for O_DIRECT reveals that they are not in fact using O_DIRECT.

Daniel P. Berrange wrote: >On Wed, Oct 08, 2008 at 11:06:27AM -0500, Anthony Liguori wrote: > >Sorry, it was mistakenly private - fixed now. > >Xen does use O_DIRECT for paravirt driver case - blktap is using the combo >of AIO+O_DIRECT. You have to use O_DIRECT with linux-aio. And blktap is well known to have terrible performance. Most serious users use blkback/blkfront and blkback does not avoid the host page cache. It maintains data integrity by passing through barriers from the guest to the host. You can approximate this in userspace by using fdatasync. The issue the bug addresses, iozone performs better than native, can be addressed in the following way: 1) For IDE, you have to disable write-caching in the guest. This should force an fdatasync in the host. 2) For virtio-blk, we need to implement barrier support. This is what blkfront/blkback do. 3) For SCSI, we should support ordered queuing which would result in an fdatasync when barriers are injected.

Anthony Liguori wrote: > Daniel P. Berrange wrote: > >> On Wed, Oct 08, 2008 at 11:06:27AM -0500, Anthony Liguori wrote: >> Sorry, it was mistakenly private - fixed now. >> Xen does use O_DIRECT for paravirt driver case - blktap is using the >> combo >> of AIO+O_DIRECT. >> > You have to use O_DIRECT with linux-aio. And blktap is well known to > have terrible performance. Most serious users use blkback/blkfront and > blkback does not avoid the host page cache. It maintains data integrity > by passing through barriers from the guest to the host. You can > approximate this in userspace by using fdatasync. > This is not accurate (at least for HVM guests using PV drivers on Xen 3.2). blkback does indeed bypass the host page cache completely. It's I/O behavior is akin to O_DIRECT.

I/O is dma'd directly to/from guest pages without involving any dom0 buffering. blkback barrier support only enforces write ordering of the blkback I/O stream(s). It does nothing to synchronize data in the host page cache. Data written through blkback will modify the storage "underneath" any data in the host page cache (w/o flushing the page cache). Subsequent access to the page cache by qemu-dm will access stale data. In our own Xen product we must explicitly flush the host page cache backing store data at qemu-dm start up, to guarantee proper data access. It is not safe to access the same backing object with both qemu-dm and blkback simultaneously. > The issue the bug addresses, iozone performs better than native, can be > addressed in the following way: > > 1) For IDE, you have to disable write-caching in the guest. This should > force an fdatasync in the host. > 2) For virtio-blk, we need to implement barrier support. This is what > blkfront/blkback do. > I don't think this is enough. Barrier semantics are local to a particular I/O stream. There would be no reason for the barrier to affect the host page cache (unless the I/Os are buffered by the cache).

On Wed, Oct 08, 2008 at 10:51:16AM -0500, Anthony Liguori wrote: > A great deal of virtualization users are doing some form of homogeneous > consolidation. If they have a good set of management tools or > sophisticated storage, then their guests will be sharing base images or > something like that. Caching in the host will result in major > performance improvements because otherwise, the same data will be > fetched multiple times. > NB, this has no impact on caching of backing files - QEMU masks out the O_DIRECT flag when opening the backing file

- so in a shared master image scenario, all reads for the shared file will still be cached, only write5Cs to the cow file are impacted. Daniel

On Wed, Oct 08, 2008 at 11:53:14AM -0500, Anthony Liguori wrote: > Daniel P. Berrange wrote: > >> On Wed, Oct 08, 2008 at 10:51:16AM -0500, Anthony Liguori wrote: >> >> >>> A great deal of virtualization users are doing some form of homogeneous >>> consolidation. If they have a good set of management tools or >>> sophisticated storage, then their guests will be sharing base images or >>> something like that. Caching in the host will result in major >>> performance improvements because otherwise, the same data will be >>> fetched multiple times. >>> >>> >> NB, this has no impact on caching of backing files - QEMU masks out >> the O_DIRECT flag when opening the backing file >> > It doesn't mask out O_DIRECT, it just doesn't pass any flags to the > backing file when it opens it. IMHO, this is a bug. > Perhaps I'm interpreting the wrong bit of code, but I was looking at QEMU's block.c in the bdrv_open2() function. The last thing it does is this, which masks out all flags except for the open mode: if (bs->backing_file[0] != '\0') { if (bdrv_open(bs->backing_hd, backing_filename, flags & (BDRV_O_RDONLY | BDRV_O_RDWR)) < 0) goto fail; }

Daniel

Daniel P. Berrange wrote: > - It is unsafe on host OS crash - all unflushed guest I/O will be > lost, and there's no ordering guarentees, so metadata updates could > be flushe to disk, while the journal updates were not. Say goodbye > to your filesystem. This has nothing to do with cache=off. The IDE device defaults to write-back caching. As such, IDE makes no guarantee that when a data write completes, it's actually completed on disk. This only comes into play when write-back is disabled. I'm perfectly happy to accept a patch that adds explicit sync's when write-back is disabled.

For SCSI, an unordered queue is advertised. Again, everything depends on whether or not write-back caching is enabled or not. Again, perfectly happy to take patches here.

More importantly, the most common journaled filesystem, ext3, does not enable write barriers by default (even for journal updates). This is how it ship in Red Hat distros.

So there is no greater risk of corrupting a journal in QEMU than there is on bare metal.

Mark McLoughlin wrote: And let's take a step back too. If people are really concerned about this point, let's introduce a sync=on option that opens the image with O_SYNC. This will effectively make the cache write-through without the baggage associated with O_DIRECT.

While I object to libvirt always setting cache=off, I think sync=on for IDE and SCSI may be reasonable (you don't want it for virtio-blk once we implement proper barriers with fdatasync I think). Regards, Anthony Liguori > Cheers, > Mark. > >

* Anthony Liguori (anthony(a)codemonkey.ws) wrote: > Mark McLoughlin wrote: > >> This is the bit I really don't buy - we're equating qemu caching to IDE >> write-back caching and saying the risk of corruption is the same in both >> cases. >> > Yes. > I'm with Mark here.

I've been persuaded. Relying on the host's integrity for guest data integrity is not a good idea by default. I don't think we should use cache=off to address this though. I've sent a patch and started a thread on qemu-devel. Let's continue the conversation there.

On Wed, Oct 08, 2008 at 10:51:16AM -0500, Anthony Liguori wrote: > Daniel P. Berrange wrote: >> - It is unsafe on host OS crash - all unflushed guest I/O will be >> lost, and there's no ordering guarentees, so metadata updates could >> be flushe to disk, while the journal updates were not. Say goodbye >> to your filesystem. > This has nothing to do with cache=off. The IDE device defaults to > write-back caching. As such, IDE makes no guarantee that when a data > write completes, it's actually completed on disk. This only comes into > play when write-back is disabled. I'm perfectly happy to accept a patch > that adds explicit sync's when write-back is disabled. > > For SCSI, an unordered queue is advertised. Again, everything depends > on whether or not write-back caching is enabled or not. Again, > perfectly happy to take patches here. > > More importantly, the most common journaled filesystem, ext3, does not > enable write barriers by default (even for journal updates). This is > how it ship in Red Hat distros. So there is no greater risk of > corrupting a journal in QEMU than there is on bare metal. Interesting discussion, I'm wondering about the non-local storage effect though, if the Node is caching writes, how can we ensure a coherent view on remote storage for example when migrating a domain ? Maybe migration is easy to fix because qemu is aware and can issue a sync, but as we start adding cloning APIs to libvirt, we could face the issue if issuing an LVM snapshot operation on the guest storage while the Node still cache some of the data. The more layers of caching the harder it is to have a predictable behaviour, no ?