Random write IOPS will drop dramatically if qcow2 l2 cache could not cover the whole disk. This patch gives libvirt user a chance to adjust the qcow2 cache configuration. Three new qcow2 driver parameters are added. They are l2-cache-size, refcount-cache-size and cache-clean-interval. The following are from qcow2-cache.txt. The amount of virtual disk that can be mapped by the L2 and refcount caches (in bytes) is: disk_size = l2_cache_size * cluster_size / 8 disk_size = refcount_cache_size * cluster_size * 8 / refcount_bits The parameter "cache-clean-interval" defines an interval (in seconds). All cache entries that haven't been accessed during that interval are removed from memory. Signed-off-by: Liu Qing <liuqing(a)huayun.com&gt; --- a) add new element qcow2 as drive's child, if any qcow2 specific configuration exists. b) merge the three test cases to one. c) update docs. cache size only support bytes. d) move the parameters checking to virDomainDiskDefParseValidate docs/formatdomain.html.in | 43 ++++++++++ docs/schemas/domaincommon.rng | 35 ++++++++ src/conf/domain_conf.c | 95 +++++++++++++++++++++- src/qemu/qemu_command.c | 6 ++ src/util/virstoragefile.c | 3 + src/util/virstoragefile.h | 6 ++ .../qemuxml2argv-disk-drive-qcow2-cache.xml | 43 ++++++++++ .../qemuxml2xmlout-disk-drive-qcow2-cache.xml | 43 ++++++++++ tests/qemuxml2xmltest.c | 1 + 9 files changed, 274 insertions(+), 1 deletion(-) create mode 100644 tests/qemuxml2argvdata/qemuxml2argv-disk-drive-qcow2-cache.xml create mode 100644 tests/qemuxml2xmloutdata/qemuxml2xmlout-disk-drive-qcow2-cache.xml diff --git a/docs/formatdomain.html.in b/docs/formatdomain.html.in index 8ca7637..245d5c4 100644 --- a/docs/formatdomain.html.in +++ b/docs/formatdomain.html.in @@ -3036,6 +3036,49 @@ set. (<span class="since">Since 3.5.0</span>) </li> </ul> + The <code>drive</code> element may contain a qcow2 sub element, which + allows to specifying further details related to qcow2 driver type. + <span class="since">Since 3.8.0</span> + <ul> + <li> + The optional <code>l2_cache_size</code> attribute controls how much + memory will be consumed by qcow2 l2 table cache in bytes. This + option is only valid when the driver type is qcow2. The default + size is 2097152. The amount of virtual disk that can be mapped by + the L2 caches (in bytes) is: + disk_size = l2_cache_size * cluster_size / 8 + <span class='since'>Since 3.8.0</span> + + <b>In general you should leave this option alone, unless you + are very certain you know what you are doing.</b> + </li> + <li> + The optional <code>refcount_cache_size</code> attribute controls + how much memory will be consumed by qcow2 reference count table + cache in bytes. This option is only valid when the driver type is + qcow2. The default size is 262144. The amount of virtual disk that + can be mapped by the refcount caches (in bytes) is: + disk_size = refcount_cache_size * cluster_size * 8 / refcount_bits + <span class='since'>Since 3.8.0</span> + + <b>In general you should leave this option alone, unless you + are very certain you know what you are doing.</b> + </li> + <li> + The optional <code>cache_clean_interval</code> attribute defines + an interval (in seconds). All cache entries that haven't been + accessed during that interval are removed from memory. This option + is only valid when the driver type is qcow2. The default + value is 0, which disables this feature. If the interval is too + short, it will cause frequent cache write back and load, which + impact performance. If the interval is too long, unused cache + will still consume memory until it's been written back to disk. + <span class='since'>Since 3.8.0</span> + + <b>In general you should leave this option alone, unless you + are very certain you know what you are doing.</b> + </li> + </ul> </dd> <dt><code>backenddomain</code></dt> <dd>The optional <code>backenddomain</code> element allows specifying a diff --git a/docs/schemas/domaincommon.rng b/docs/schemas/domaincommon.rng index 06c5a91..ec44e58 100644 --- a/docs/schemas/domaincommon.rng +++ b/docs/schemas/domaincommon.rng @@ -1762,6 +1762,23 @@ </element> </define> <!-- + Parameters for qcow2 driver + --> + <define name="qcow2Driver"> + <element name="qcow2"> + <optional> + <ref name="qcow2_l2_cache_size"/> + </optional> + <optional> + <ref name="qcow2_refcount_cache_size"/> + </optional> + <optional> + <ref name="qcow2_cache_clean_interval"/> + </optional> + </element> + </define> + + <!-- Disk may use a special driver for access. --> <define name="diskDriver"> @@ -1800,6 +1817,9 @@ <ref name="detect_zeroes"/> </optional> <ref name="virtioOptions"/> + <zeroOrMore> + <ref name="qcow2Driver"/> + </zeroOrMore> <empty/> </element> </define> @@ -1895,6 +1915,21 @@ </choice> </attribute> </define> + <define name="qcow2_l2_cache_size"> + <attribute name='l2_cache_size'> + <ref name="unsignedInt"/> + </attribute> + </define> + <define name="qcow2_refcount_cache_size"> + <attribute name='refcount_cache_size'> + <ref name="unsignedInt"/> + </attribute> + </define> + <define name="qcow2_cache_clean_interval"> + <attribute name='cache_clean_interval'> + <ref name="unsignedInt"/> + </attribute> + </define> <define name="controller"> <element name="controller"> <optional> diff --git a/src/conf/domain_conf.c b/src/conf/domain_conf.c index 2fc1fc3..c7d4b9b 100644 --- a/src/conf/domain_conf.c +++ b/src/conf/domain_conf.c @@ -5673,6 +5673,28 @@ virDomainDeviceLoadparmIsValid(const char *loadparm) static void +virDoaminQcow2CacheOptionsFormat(virBufferPtr buf, + virDomainDiskDefPtr def) +{ + virBuffer qcow2Buff = VIR_BUFFER_INITIALIZER; + if (def->src->l2_cache_size > 0) + virBufferAsprintf(&qcow2Buff, " l2_cache_size='%llu'", + def->src->l2_cache_size); + if (def->src->refcount_cache_size > 0) + virBufferAsprintf(&qcow2Buff, " refcount_cache_size='%llu'", + def->src->refcount_cache_size); + if (def->src->cache_clean_interval > 0) + virBufferAsprintf(&qcow2Buff, " cache_clean_interval='%llu'", + def->src->cache_clean_interval); + + if (virBufferUse(&qcow2Buff)) { + virBufferAddLit(buf, "<qcow2"); + virBufferAddBuffer(buf, &qcow2Buff); + virBufferAddLit(buf, "/>\n"); + } +} + +static void virDomainVirtioOptionsFormat(virBufferPtr buf, virDomainVirtioOptionsPtr virtio) { @@ -8511,15 +8533,69 @@ virDomainDiskDefParseValidate(const virDomainDiskDef *def) } } + if (def->src->format != VIR_STORAGE_FILE_QCOW2 && + (def->src->l2_cache_size > 0 || def->src->refcount_cache_size > 0 || + def->src->cache_clean_interval > 0)) { + virReportError(VIR_ERR_XML_ERROR, + _("Setting l2_cache_size, refcount_cache_size, " + "cache_clean_interval is not allowed for types " + "other than QCOW2")); + return -1; + } + return 0; } static int +virDomainDiskDefQcow2ParseXML(virDomainDiskDefPtr def, + xmlNodePtr cur) +{ + char *tmp = NULL; + int ret = -1; + + if ((tmp = virXMLPropString(cur, "l2_cache_size")) && + (virStrToLong_ullp(tmp, NULL, 10, &def->src->l2_cache_size) < 0)) { + virReportError(VIR_ERR_XML_ERROR, + _("Invalid l2_cache_size attribute in disk " + "driver element: %s"), tmp); + goto cleanup; + } + VIR_FREE(tmp); + + if ((tmp = virXMLPropString(cur, "refcount_cache_size")) && + (virStrToLong_ullp(tmp, NULL, 10, &def->src->refcount_cache_size) < 0)) { + virReportError(VIR_ERR_XML_ERROR, + _("Invalid refcount_cache_size attribute in disk " + "driver element: %s"), tmp); + goto cleanup; + } + VIR_FREE(tmp); + + if ((tmp = virXMLPropString(cur, "cache_clean_interval")) && + (virStrToLong_ullp(tmp, NULL, 10, &def->src->cache_clean_interval) < 0)) { + virReportError(VIR_ERR_XML_ERROR, + _("Invalid cache_clean_interval attribute in " + "disk driver element: %s"), tmp); + goto cleanup; + } + VIR_FREE(tmp); + + ret = 0; + + cleanup: + VIR_FREE(tmp); + + return ret; +} + + +static int virDomainDiskDefDriverParseXML(virDomainDiskDefPtr def, xmlNodePtr cur) { char *tmp = NULL; + xmlNodePtr child; int ret = -1; def->src->driverName = virXMLPropString(cur, "name"); @@ -8622,6 +8698,12 @@ virDomainDiskDefDriverParseXML(virDomainDiskDefPtr def, } VIR_FREE(tmp); + for (child = cur->children; child != NULL; child = child->next) { + if (virXMLNodeNameEqual(child, "qcow2") && + virDomainDiskDefQcow2ParseXML(def, child) < 0) { + goto cleanup; + } + } ret = 0; cleanup: @@ -21908,7 +21990,18 @@ virDomainDiskDefFormat(virBufferPtr buf, if (virBufferUse(&driverBuf)) { virBufferAddLit(buf, "<driver"); virBufferAddBuffer(buf, &driverBuf); - virBufferAddLit(buf, "/>\n"); + + if (def->src->l2_cache_size == 0 && + def->src->refcount_cache_size == 0 && + def->src->cache_clean_interval == 0) { + virBufferAddLit(buf, "/>\n"); + } else { + virBufferAddLit(buf, ">\n"); + virBufferAdjustIndent(buf, 2); + virDoaminQcow2CacheOptionsFormat(buf, def); + virBufferAdjustIndent(buf, -2); + virBufferAddLit(buf, "</driver>\n"); + } } if (def->src->auth) { diff --git a/src/qemu/qemu_command.c b/src/qemu/qemu_command.c index ed8cb6e..391aaba 100644 --- a/src/qemu/qemu_command.c +++ b/src/qemu/qemu_command.c @@ -1433,6 +1433,12 @@ qemuBuildDriveSourceStr(virDomainDiskDefPtr disk, qemuformat = "luks"; virBufferAsprintf(buf, "format=%s,", qemuformat); } + if (disk->src->l2_cache_size > 0) + virBufferAsprintf(buf, "l2-cache-size=%llu,", disk->src->l2_cache_size); + if (disk->src->refcount_cache_size > 0) + virBufferAsprintf(buf, "refcount-cache-size=%llu,", disk->src->refcount_cache_size); + if (disk->src->cache_clean_interval > 0) + virBufferAsprintf(buf, "cache-clean-interval=%llu,", disk->src->cache_clean_interval); ret = 0; diff --git a/src/util/virstoragefile.c b/src/util/virstoragefile.c index fbc8245..c685331 100644 --- a/src/util/virstoragefile.c +++ b/src/util/virstoragefile.c @@ -2038,6 +2038,9 @@ virStorageSourceCopy(const virStorageSource *src, ret->physical = src->physical; ret->readonly = src->readonly; ret->shared = src->shared; + ret->l2_cache_size = src->l2_cache_size; + ret->refcount_cache_size = src->refcount_cache_size; + ret->cache_clean_interval = src->cache_clean_interval; /* storage driver metadata are not copied */ ret->drv = NULL; diff --git a/src/util/virstoragefile.h b/src/util/virstoragefile.h index 6c388b1..9b5a5f3 100644 --- a/src/util/virstoragefile.h +++ b/src/util/virstoragefile.h @@ -280,6 +280,12 @@ struct _virStorageSource { /* metadata that allows identifying given storage source */ char *nodeformat; /* name of the format handler object */ char *nodestorage; /* name of the storage object */ + + unsigned long long l2_cache_size; /* qcow2 l2 cache size */ + /* qcow2 reference count table cache size */ + unsigned long long refcount_cache_size; + /* clean unused cache entries interval */ + unsigned long long cache_clean_interval; }; diff --git a/tests/qemuxml2argvdata/qemuxml2argv-disk-drive-qcow2-cache.xml b/tests/qemuxml2argvdata/qemuxml2argv-disk-drive-qcow2-cache.xml new file mode 100644 index 0000000..3f464db --- /dev/null +++ b/tests/qemuxml2argvdata/qemuxml2argv-disk-drive-qcow2-cache.xml @@ -0,0 +1,43 @@ +<domain type='qemu'> + <name>QEMUGuest1</name> + <uuid>c7a5fdbd-edaf-9455-926a-d65c16db1809</uuid> + <memory unit='KiB'>219136</memory> + <currentMemory unit='KiB'>219136</currentMemory> + <vcpu placement='static'>1</vcpu> + <os> + <type arch='i686' machine='pc'>hvm</type> + <boot dev='hd'/> + </os> + <clock offset='utc'/> + <on_poweroff>destroy</on_poweroff> + <on_reboot>restart</on_reboot> + <on_crash>destroy</on_crash> + <devices> + <emulator>/usr/bin/qemu-system-i686</emulator> + <disk type='block' device='disk'> + <driver name='qemu' type='qcow2' cache='none'> + <qcow2 l2_cache_size='2097152' refcount_cache_size='524288' cache_clean_interval='900'/> + </driver> + <source dev='/dev/HostVG/QEMUGuest1'/> + <target dev='hda' bus='ide'/> + <address type='drive' controller='0' bus='0' target='0' unit='0'/> + </disk> + <disk type='block' device='cdrom'> + <driver name='qemu' type='raw'/> + <source dev='/dev/HostVG/QEMUGuest2'/> + <target dev='hdc' bus='ide'/> + <readonly/> + <address type='drive' controller='0' bus='1' target='0' unit='0'/> + </disk> + <controller type='usb' index='0'> + <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> + </controller> + <controller type='ide' index='0'> + <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> + </controller> + <controller type='pci' index='0' model='pci-root'/> + <input type='mouse' bus='ps2'/> + <input type='keyboard' bus='ps2'/> + <memballoon model='none'/> + </devices> +</domain> diff --git a/tests/qemuxml2xmloutdata/qemuxml2xmlout-disk-drive-qcow2-cache.xml b/tests/qemuxml2xmloutdata/qemuxml2xmlout-disk-drive-qcow2-cache.xml new file mode 100644 index 0000000..3f464db --- /dev/null +++ b/tests/qemuxml2xmloutdata/qemuxml2xmlout-disk-drive-qcow2-cache.xml @@ -0,0 +1,43 @@ +<domain type='qemu'> + <name>QEMUGuest1</name> + <uuid>c7a5fdbd-edaf-9455-926a-d65c16db1809</uuid> + <memory unit='KiB'>219136</memory> + <currentMemory unit='KiB'>219136</currentMemory> + <vcpu placement='static'>1</vcpu> + <os> + <type arch='i686' machine='pc'>hvm</type> + <boot dev='hd'/> + </os> + <clock offset='utc'/> + <on_poweroff>destroy</on_poweroff> + <on_reboot>restart</on_reboot> + <on_crash>destroy</on_crash> + <devices> + <emulator>/usr/bin/qemu-system-i686</emulator> + <disk type='block' device='disk'> + <driver name='qemu' type='qcow2' cache='none'> + <qcow2 l2_cache_size='2097152' refcount_cache_size='524288' cache_clean_interval='900'/> + </driver> + <source dev='/dev/HostVG/QEMUGuest1'/> + <target dev='hda' bus='ide'/> + <address type='drive' controller='0' bus='0' target='0' unit='0'/> + </disk> + <disk type='block' device='cdrom'> + <driver name='qemu' type='raw'/> + <source dev='/dev/HostVG/QEMUGuest2'/> + <target dev='hdc' bus='ide'/> + <readonly/> + <address type='drive' controller='0' bus='1' target='0' unit='0'/> + </disk> + <controller type='usb' index='0'> + <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> + </controller> + <controller type='ide' index='0'> + <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> + </controller> + <controller type='pci' index='0' model='pci-root'/> + <input type='mouse' bus='ps2'/> + <input type='keyboard' bus='ps2'/> + <memballoon model='none'/> + </devices> +</domain> diff --git a/tests/qemuxml2xmltest.c b/tests/qemuxml2xmltest.c index 97ff36c..1ffb831 100644 --- a/tests/qemuxml2xmltest.c +++ b/tests/qemuxml2xmltest.c @@ -461,6 +461,7 @@ mymain(void) DO_TEST("disk-drive-cache-v2-none", NONE); DO_TEST("disk-drive-cache-directsync", NONE); DO_TEST("disk-drive-cache-unsafe", NONE); + DO_TEST("disk-drive-qcow2-cache", NONE); DO_TEST("disk-drive-network-nbd", NONE); DO_TEST("disk-drive-network-nbd-export", NONE); DO_TEST("disk-drive-network-nbd-ipv6", NONE); -- 1.8.3.1

On Tue, Sep 12, 2017 at 05:21:53PM -0400, John Ferlan wrote: I am thinking add a web link but afraid of 404 error when the link is not accessable. A indirect link like docs/qcow2-cache.txt is a good idea. The cache attribute in driver element is how qemu will deals with page cache of qcow2 file on the host. When opening a qcow2 file on host, the oflag will be dertimined by cache. L2 and refcount tables are metadatas of qcow2. So cache='xxx' determines how the guest data will be cached, and l2/refcount table cache controls the metadata cache. A point to source document is fine. I have done all the modifications in v5. If any missing please let me know. Thanks. Qing

On 09/07/2017 02:09 AM, Liu Qing wrote: The format of the data has every 5th element counted, so this needs before the following: /* 270 */ Ugh! I see the first 3 are necessary because of qemu changes similar to the +gluster debug setting, yuck. At least for that one, the argument appears to have changed name as well... Seems qemu 2.8 is the magic inflection point, though. Similar formatting comment add before the following group: /* 270 */ Similar to patch1, I don't mind making the changes here. Especially since I already know I have some conflicts queued up in other patches on list that I'm trying to get reviewed/ACK'd. Secondarily, perhaps others may have comments about the new format and I'd rather see them addressed now rather than after pushing... John