- Devel - libvirt List Archives

[libvirt] [PATCH 0/2] qemu: Properly report port numbers in spice events
by Peter Krempa 30 Jun '15

30 Jun '15

Peter Krempa (2): qemu: event: Properly handle spice events qemu: event: Clean up VNC monitor handling src/qemu/qemu_monitor_json.c | 110 ++++++++++++++++++++++++++++++++++--------- 1 file changed, 87 insertions(+), 23 deletions(-) -- 2.4.1

2 5

[libvirt] [PATCH] qemu: Close the agent connection only on agent channel events
by Peter Krempa 30 Jun '15

30 Jun '15

processSerialChangedEvent processes events for all channels. Commit 2af51483 broke all agent interaction if a channel other than the agent closes since it did not check that the event actually originated from the guest agent channel. Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=1236924 Fixes up: https://bugzilla.redhat.com/show_bug.cgi?id=890648 --- src/qemu/qemu_driver.c | 18 ++++++++++++++---- 1 file changed, 14 insertions(+), 4 deletions(-) diff --git a/src/qemu/qemu_driver.c b/src/qemu/qemu_driver.c index 2b530c8..353be31 100644 --- a/src/qemu/qemu_driver.c +++ b/src/qemu/qemu_driver.c @@ -4448,10 +4448,20 @@ processSerialChangedEvent(virQEMUDriverPtr driver, if (newstate == VIR_DOMAIN_CHR_DEVICE_STATE_DISCONNECTED && virDomainObjIsActive(vm) && priv->agent) { - /* Close agent monitor early, so that other threads - * waiting for the agent to reply can finish and our - * job we acquire below can succeed. */ - qemuAgentNotifyClose(priv->agent); + /* peek into the domain definition to find the channel */ + if (virDomainDefFindDevice(vm->def, devAlias, &dev, true) == 0 && + dev.type == VIR_DOMAIN_DEVICE_CHR && + dev.data.chr->deviceType == VIR_DOMAIN_CHR_DEVICE_TYPE_CHANNEL && + dev.data.chr->targetType == VIR_DOMAIN_CHR_CHANNEL_TARGET_TYPE_VIRTIO && + STREQ_NULLABLE(dev.data.chr->target.name, "org.qemu.guest_agent.0")) + /* Close agent monitor early, so that other threads + * waiting for the agent to reply can finish and our + * job we acquire below can succeed. */ + qemuAgentNotifyClose(priv->agent); + + /* now discard the data, since it may possibly change once we unlock + * while entering the job */ + memset(&dev, 0, sizeof(dev)); } if (qemuDomainObjBeginJob(driver, vm, QEMU_JOB_MODIFY) < 0) -- 2.4.1

2 2

[libvirt] [PATCH] vz: fix SDK event dispatching
by nshirokovskiy＠virtuozzo.com 30 Jun '15

30 Jun '15

From: Nikolay Shirokovskiy <nshirokovskiy(a)virtuozzo.com> Current version of SDK event dispatcing is incorrect. For most VM events (add, delete etc) issuer type is PIE_DISPATCHER. Actually analyzing issuer type doesn't have any benifints so this patch get rid of it. All dispatching is done only on event type. Signed-off-by: Nikolay Shirokovskiy <nshirokovskiy(a)virtuozzo.com> --- src/vz/vz_sdk.c | 58 +++++++++++++++--------------------------------------- 1 files changed, 16 insertions(+), 42 deletions(-) diff --git a/src/vz/vz_sdk.c b/src/vz/vz_sdk.c index 98f7a57..2ca74c4 100644 --- a/src/vz/vz_sdk.c +++ b/src/vz/vz_sdk.c @@ -1697,21 +1697,33 @@ prlsdkHandlePerfEvent(vzConnPtr privconn, return PRL_ERR_SUCCESS; } -static void -prlsdkHandleVmEvent(vzConnPtr privconn, PRL_HANDLE prlEvent) +static PRL_RESULT +prlsdkEventsHandler(PRL_HANDLE prlEvent, PRL_VOID_PTR opaque) { + vzConnPtr privconn = opaque; PRL_RESULT pret = PRL_ERR_FAILURE; + PRL_HANDLE_TYPE handleType; char uuidstr[VIR_UUID_STRING_BUFLEN + 2]; unsigned char uuid[VIR_UUID_BUFLEN]; PRL_UINT32 bufsize = ARRAY_CARDINALITY(uuidstr); PRL_EVENT_TYPE prlEventType; - pret = PrlEvent_GetType(prlEvent, &prlEventType); + pret = PrlHandle_GetType(prlEvent, &handleType); prlsdkCheckRetGoto(pret, cleanup); + /* Currently, there is no need to handle anything but events */ + if (handleType != PHT_EVENT) + goto cleanup; + + if (privconn == NULL) + goto cleanup; + pret = PrlEvent_GetIssuerId(prlEvent, uuidstr, &bufsize); prlsdkCheckRetGoto(pret, cleanup); + pret = PrlEvent_GetType(prlEvent, &prlEventType); + prlsdkCheckRetGoto(pret, cleanup); + if (prlsdkUUIDParse(uuidstr, uuid) < 0) goto cleanup; @@ -1736,44 +1748,7 @@ prlsdkHandleVmEvent(vzConnPtr privconn, PRL_HANDLE prlEvent) prlEvent = PRL_INVALID_HANDLE; break; default: - virReportError(VIR_ERR_INTERNAL_ERROR, - _("Can't handle event of type %d"), prlEventType); - } - - cleanup: - PrlHandle_Free(prlEvent); - return; -} - -static PRL_RESULT -prlsdkEventsHandler(PRL_HANDLE prlEvent, PRL_VOID_PTR opaque) -{ - vzConnPtr privconn = opaque; - PRL_RESULT pret = PRL_ERR_FAILURE; - PRL_HANDLE_TYPE handleType; - PRL_EVENT_ISSUER_TYPE prlIssuerType = PIE_UNKNOWN; - - pret = PrlHandle_GetType(prlEvent, &handleType); - prlsdkCheckRetGoto(pret, cleanup); - - /* Currently, there is no need to handle anything but events */ - if (handleType != PHT_EVENT) - goto cleanup; - - if (privconn == NULL) - goto cleanup; - - PrlEvent_GetIssuerType(prlEvent, &prlIssuerType); - prlsdkCheckRetGoto(pret, cleanup); - - switch (prlIssuerType) { - case PIE_VIRTUAL_MACHINE: - prlsdkHandleVmEvent(privconn, prlEvent); - // above function takes own of event - prlEvent = PRL_INVALID_HANDLE; - break; - default: - VIR_DEBUG("Skipping event of issuer type %d", prlIssuerType); + VIR_DEBUG("Skipping event of type %d", prlEventType); } cleanup: @@ -1781,7 +1756,6 @@ prlsdkEventsHandler(PRL_HANDLE prlEvent, PRL_VOID_PTR opaque) return PRL_ERR_SUCCESS; } - int prlsdkSubscribeToPCSEvents(vzConnPtr privconn) { PRL_RESULT pret = PRL_ERR_UNINITIALIZED; -- 1.7.1

2 2

[libvirt] [PATCH] Fix typo incomaptible -> incompatible
by Andrea Bolognani 30 Jun '15

30 Jun '15

--- src/cpu/cpu_x86.c | 4 ++-- src/libvirt-domain-snapshot.c | 2 +- src/qemu/qemu_driver.c | 8 ++++---- 3 files changed, 7 insertions(+), 7 deletions(-) diff --git a/src/cpu/cpu_x86.c b/src/cpu/cpu_x86.c index 2a14705..f5f7697 100644 --- a/src/cpu/cpu_x86.c +++ b/src/cpu/cpu_x86.c @@ -1507,14 +1507,14 @@ x86Compute(virCPUDefPtr host, static virCPUCompareResult x86Compare(virCPUDefPtr host, virCPUDefPtr cpu, - bool failIncomaptible) + bool failIncompatible) { virCPUCompareResult ret; char *message = NULL; ret = x86Compute(host, cpu, NULL, &message); - if (failIncomaptible && ret == VIR_CPU_COMPARE_INCOMPATIBLE) { + if (failIncompatible && ret == VIR_CPU_COMPARE_INCOMPATIBLE) { ret = VIR_CPU_COMPARE_ERROR; if (message) { virReportError(VIR_ERR_CPU_INCOMPATIBLE, "%s", message); diff --git a/src/libvirt-domain-snapshot.c b/src/libvirt-domain-snapshot.c index 4625e03..b7c566f 100644 --- a/src/libvirt-domain-snapshot.c +++ b/src/libvirt-domain-snapshot.c @@ -1048,7 +1048,7 @@ virDomainSnapshotHasMetadata(virDomainSnapshotPtr snapshot, * new hypervisor instance rather than reusing the existing hypervisor * (since this would terminate all connections to the domain, such as * such as VNC or Spice graphics) - this condition arises from active - * snapshots that are provably ABI incomaptible, as well as from + * snapshots that are provably ABI incompatible, as well as from * inactive snapshots with a @flags request to start the domain after * the revert. * diff --git a/src/qemu/qemu_driver.c b/src/qemu/qemu_driver.c index 2b530c8..bdf9b15 100644 --- a/src/qemu/qemu_driver.c +++ b/src/qemu/qemu_driver.c @@ -12948,7 +12948,7 @@ qemuConnectCompareCPU(virConnectPtr conn, virQEMUDriverPtr driver = conn->privateData; int ret = VIR_CPU_COMPARE_ERROR; virCapsPtr caps = NULL; - bool failIncomaptible; + bool failIncompatible; virCheckFlags(VIR_CONNECT_COMPARE_CPU_FAIL_INCOMPATIBLE, VIR_CPU_COMPARE_ERROR); @@ -12956,14 +12956,14 @@ qemuConnectCompareCPU(virConnectPtr conn, if (virConnectCompareCPUEnsureACL(conn) < 0) goto cleanup; - failIncomaptible = !!(flags & VIR_CONNECT_COMPARE_CPU_FAIL_INCOMPATIBLE); + failIncompatible = !!(flags & VIR_CONNECT_COMPARE_CPU_FAIL_INCOMPATIBLE); if (!(caps = virQEMUDriverGetCapabilities(driver, false))) goto cleanup; if (!caps->host.cpu || !caps->host.cpu->model) { - if (failIncomaptible) { + if (failIncompatible) { virReportError(VIR_ERR_CPU_INCOMPATIBLE, "%s", _("cannot get host CPU capabilities")); } else { @@ -12971,7 +12971,7 @@ qemuConnectCompareCPU(virConnectPtr conn, ret = VIR_CPU_COMPARE_INCOMPATIBLE; } } else { - ret = cpuCompareXML(caps->host.cpu, xmlDesc, failIncomaptible); + ret = cpuCompareXML(caps->host.cpu, xmlDesc, failIncompatible); } cleanup: -- 2.4.3

2 1

[libvirt] [PATCH] qemu: Fix assignment of the default spicevmc channel name
by Jiri Denemark 30 Jun '15

30 Jun '15

Make sure we only assign the default spicevmc channel name to spicevmc virtio channels. Caused by commits 3269ee65 and 1133ee2b, which moved the assignment from XML parsing code to QEMU but failed to keep the logic. https://bugzilla.redhat.com/show_bug.cgi?id=1179680 Signed-off-by: Jiri Denemark <jdenemar(a)redhat.com> --- src/qemu/qemu_command.c | 4 ++- .../qemuxml2argv-channel-virtio-default.args | 16 ++++++++++ .../qemuxml2argv-channel-virtio-default.xml | 36 ++++++++++++++++++++++ tests/qemuxml2argvtest.c | 3 ++ 4 files changed, 58 insertions(+), 1 deletion(-) create mode 100644 tests/qemuxml2argvdata/qemuxml2argv-channel-virtio-default.args create mode 100644 tests/qemuxml2argvdata/qemuxml2argv-channel-virtio-default.xml diff --git a/src/qemu/qemu_command.c b/src/qemu/qemu_command.c index 2209cf6..9b06a49 100644 --- a/src/qemu/qemu_command.c +++ b/src/qemu/qemu_command.c @@ -6427,7 +6427,9 @@ qemuBuildVirtioSerialPortDevStr(virDomainDefPtr def, virQEMUCapsGet(qemuCaps, QEMU_CAPS_DEVICE_SPICEVMC))) { virBufferAsprintf(&buf, ",chardev=char%s,id=%s", dev->info.alias, dev->info.alias); - if (dev->deviceType == VIR_DOMAIN_CHR_DEVICE_TYPE_CHANNEL) { + if (dev->deviceType == VIR_DOMAIN_CHR_DEVICE_TYPE_CHANNEL && + (dev->source.type == VIR_DOMAIN_CHR_TYPE_SPICEVMC || + dev->target.name)) { virBufferAsprintf(&buf, ",name=%s", dev->target.name ? dev->target.name : "com.redhat.spice.0"); } diff --git a/tests/qemuxml2argvdata/qemuxml2argv-channel-virtio-default.args b/tests/qemuxml2argvdata/qemuxml2argv-channel-virtio-default.args new file mode 100644 index 0000000..7908445 --- /dev/null +++ b/tests/qemuxml2argvdata/qemuxml2argv-channel-virtio-default.args @@ -0,0 +1,16 @@ +LC_ALL=C PATH=/bin HOME=/home/test USER=test LOGNAME=test QEMU_AUDIO_DRV=none \ +/usr/bin/qemu -S -M pc -m 214 -smp 1 -nographic -nodefconfig -nodefaults \ +-chardev socket,id=charmonitor,path=/tmp/test-monitor,server,nowait \ +-mon chardev=charmonitor,id=monitor,mode=readline \ +-no-acpi \ +-boot c \ +-device virtio-serial-pci,id=virtio-serial1,bus=pci.0,addr=0xa \ +-device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x3 \ +-usb \ +-hda /dev/HostVG/QEMUGuest1 \ +-chardev pty,id=charchannel0 \ +-device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0 \ +-chardev spicevmc,id=charchannel1,name=vdagent \ +-device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,\ +name=com.redhat.spice.0 \ +-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4 diff --git a/tests/qemuxml2argvdata/qemuxml2argv-channel-virtio-default.xml b/tests/qemuxml2argvdata/qemuxml2argv-channel-virtio-default.xml new file mode 100644 index 0000000..74baded --- /dev/null +++ b/tests/qemuxml2argvdata/qemuxml2argv-channel-virtio-default.xml @@ -0,0 +1,36 @@ +<domain type='qemu'> + <name>QEMUGuest1</name> + <uuid>c7a5fdbd-edaf-9455-926a-d65c16db1809</uuid> + <memory unit='KiB'>219100</memory> + <currentMemory unit='KiB'>219100</currentMemory> + <vcpu placement='static' cpuset='1-4,8-20,525'>1</vcpu> + <os> + <type arch='i686' machine='pc'>hvm</type> + <boot dev='hd'/> + </os> + <clock offset='utc'/> + <on_poweroff>destroy</on_poweroff> + <on_reboot>restart</on_reboot> + <on_crash>destroy</on_crash> + <devices> + <emulator>/usr/bin/qemu</emulator> + <disk type='block' device='disk'> + <source dev='/dev/HostVG/QEMUGuest1'/> + <target dev='hda' bus='ide'/> + <address type='drive' controller='0' bus='0' target='0' unit='0'/> + </disk> + <controller type='usb' index='0'/> + <controller type='ide' index='0'/> + <controller type='virtio-serial' index='1'> + <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/> + </controller> + <controller type='pci' index='0' model='pci-root'/> + <channel type='pty'> + <target type='virtio'/> + </channel> + <channel type="spicevmc"> + <target type="virtio"/> + </channel> + <memballoon model='virtio'/> + </devices> +</domain> diff --git a/tests/qemuxml2argvtest.c b/tests/qemuxml2argvtest.c index a212d30..4154601 100644 --- a/tests/qemuxml2argvtest.c +++ b/tests/qemuxml2argvtest.c @@ -1110,6 +1110,9 @@ mymain(void) DO_TEST("channel-spicevmc-old", QEMU_CAPS_DEVICE, QEMU_CAPS_NODEFCONFIG, QEMU_CAPS_SPICE, QEMU_CAPS_DEVICE_SPICEVMC); + DO_TEST("channel-virtio-default", + QEMU_CAPS_DEVICE, QEMU_CAPS_CHARDEV, QEMU_CAPS_NODEFCONFIG, + QEMU_CAPS_SPICE, QEMU_CAPS_CHARDEV_SPICEVMC); DO_TEST("smartcard-host", QEMU_CAPS_CHARDEV, QEMU_CAPS_DEVICE, -- 2.4.5

2 2

[libvirt] [PATCH] qemuxml2argv: Remove CPU models from unrelated tests
by Jiri Denemark 30 Jun '15

30 Jun '15

Proper Haswell CPU model handling is tested in several qemuxml2argv-cpu-* which are run in a special environment. Let's remove the CPU model from other tests to make them less fragile. Signed-off-by: Jiri Denemark <jdenemar(a)redhat.com> --- .../qemuxml2argv-hugepages-numa.args | 1 - .../qemuxml2argv-hugepages-numa.xml | 3 +-- .../qemuxml2argv-interface-server.xml | 25 +--------------------- 3 files changed, 2 insertions(+), 27 deletions(-) diff --git a/tests/qemuxml2argvdata/qemuxml2argv-hugepages-numa.args b/tests/qemuxml2argvdata/qemuxml2argv-hugepages-numa.args index b697942..37511b1 100644 --- a/tests/qemuxml2argvdata/qemuxml2argv-hugepages-numa.args +++ b/tests/qemuxml2argvdata/qemuxml2argv-hugepages-numa.args @@ -2,7 +2,6 @@ LC_ALL=C PATH=/bin HOME=/home/test USER=test LOGNAME=test QEMU_AUDIO_DRV=spice \ /usr/bin/qemu-system-x86_64 \ -S \ -M pc-i440fx-2.3 \ --cpu Haswell \ -m size=1048576k,slots=16,maxmem=1099511627776k \ -smp 2 \ -object memory-backend-file,id=ram-node0,prealloc=yes,\ diff --git a/tests/qemuxml2argvdata/qemuxml2argv-hugepages-numa.xml b/tests/qemuxml2argvdata/qemuxml2argv-hugepages-numa.xml index 8c1f19c..8cda5c6 100644 --- a/tests/qemuxml2argvdata/qemuxml2argv-hugepages-numa.xml +++ b/tests/qemuxml2argvdata/qemuxml2argv-hugepages-numa.xml @@ -17,8 +17,7 @@ <apic/> <pae/> </features> - <cpu mode='custom' match='exact'> - <model fallback='allow'>Haswell</model> + <cpu> <numa> <cell id='0' cpus='0-1' memory='1048576' unit='KiB'/> </numa> diff --git a/tests/qemuxml2argvdata/qemuxml2argv-interface-server.xml b/tests/qemuxml2argvdata/qemuxml2argv-interface-server.xml index 9edf773..a92aff4 100644 --- a/tests/qemuxml2argvdata/qemuxml2argv-interface-server.xml +++ b/tests/qemuxml2argvdata/qemuxml2argv-interface-server.xml @@ -19,30 +19,7 @@ <apic/> <pae/> </features> - <cpu mode='custom' match='exact'> - <model fallback='allow'>Haswell</model> - <vendor>Intel</vendor> - <feature policy='require' name='tm2'/> - <feature policy='require' name='est'/> - <feature policy='require' name='vmx'/> - <feature policy='require' name='osxsave'/> - <feature policy='require' name='smx'/> - <feature policy='require' name='ss'/> - <feature policy='require' name='ds'/> - <feature policy='require' name='vme'/> - <feature policy='require' name='dtes64'/> - <feature policy='require' name='abm'/> - <feature policy='require' name='ht'/> - <feature policy='require' name='acpi'/> - <feature policy='require' name='pbe'/> - <feature policy='require' name='tm'/> - <feature policy='require' name='pdcm'/> - <feature policy='require' name='pdpe1gb'/> - <feature policy='require' name='ds_cpl'/> - <feature policy='require' name='rdrand'/> - <feature policy='require' name='f16c'/> - <feature policy='require' name='xtpr'/> - <feature policy='require' name='monitor'/> + <cpu> <numa> <cell id='0' cpus='0' memory='1048576' unit='KiB'/> <cell id='1' cpus='1' memory='1048576' unit='KiB'/> -- 2.4.5

2 2

[libvirt] Libvirt - updates backing file permission of qcow2 image on launching instance
by Agrawal, Ankit 30 Jun '15

30 Jun '15

Hi All, I am trying to fix a race condition between imagebackend and imagecache in nova (openstack) while creating an instance using qcow2 image backend. On creating a qcow2 image libvirt stores the base/backing file path reference while copying the image at instance disk info. While spawning the instance nova usage this base file reference as a backing file to launch instance. When base file is created in nova image cache directory its owner is set to openstack (a nova user) but on launching the instance from guest.launch, libvirt updates the base/backing file owner from openstack to libvirt-qemu which causes permission error if we try to use that base file later. Can someone please help me to understand why libvirt is updating the owner of base file stored in image cache directory, is it a bug in libvirt or the expected behaviour? Thanks & Regards, Ankit Agrawal ______________________________________________________________________ Disclaimer: This email and any attachments are sent in strictest confidence for the sole use of the addressee and may contain legally privileged, confidential, and proprietary data. If you are not the intended recipient, please advise the sender by replying promptly to this email and then delete and destroy this email and any attachments without any further use, copying or forwarding.

1 0

[libvirt] [PATCH] storage: Set correct vol->type at VolCreate
by John Ferlan 30 Jun '15

30 Jun '15

https://bugzilla.redhat.com/show_bug.cgi?id=1227664 If the requested format type for the new entry in the file system pool is a 'dir', then be sure to set the vol->type correctly as would be done when the pool is refreshed. Signed-off-by: John Ferlan <jferlan(a)redhat.com> --- src/storage/storage_backend_fs.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/src/storage/storage_backend_fs.c b/src/storage/storage_backend_fs.c index b751687..0eebac3 100644 --- a/src/storage/storage_backend_fs.c +++ b/src/storage/storage_backend_fs.c @@ -1051,7 +1051,10 @@ virStorageBackendFileSystemVolCreate(virConnectPtr conn ATTRIBUTE_UNUSED, virStorageVolDefPtr vol) { - vol->type = VIR_STORAGE_VOL_FILE; + if (vol->target.format == VIR_STORAGE_FILE_DIR) + vol->type = VIR_STORAGE_VOL_DIR; + else + vol->type = VIR_STORAGE_VOL_FILE; VIR_FREE(vol->target.path); if (virAsprintf(&vol->target.path, "%s/%s", -- 2.1.0

2 1

[libvirt] [PATCH] libvirt/hooks: Static and dynamic hugepage hooks
by Alex Williamson 29 Jun '15

29 Jun '15

This patch provides scripts for hugepage allocation, as well as a bit of infrastructure and common hook config file that I hope may some day be enabled by default in libvirt. For now, we place the files in /usr/share and ask users to install the config file and copy or link the scripts, more like "contrib" scripts for now. Two methods of hugepage allocation are provided, static and dynamic. The static mechanism allocates pages at libvirt daemon startup and releases them at shutdown. It allows full size, locality, and policy configuration. For instance, if I want to allocate a set of 2M pages exclusively on host NUMA node 1, it can do that, along with plenty more. This is especially useful for 1G hugepages on x86, since they can now be allocated dynamically, but become impractical to allocate due to memory fragmentation as the host runs. Systems dedicated to hosting VMs are also likely to prefer static allocation. Static allocation requires explicit XML entries in the hook config file to be activated. The dynamic method allocates hugepages only around the instantiation of the VM. This is enabled by adding an entry for the domain in the config file and configuring the domain normally for hugepages. The dynamic hugepage script is activated via the QEMU domain prepare hook, reads the domain XML and allocates hugepages as necessary. On domain shutdown, hugepages are freed via the release hook. This model is more appropriate for systems that are not dedicated VM hosts and guests that use hugepage sizes and quantities are are likely to be dynamically allocated as the VM is started. In addition to the documentation provided within each script, a README file is provided with overal instructions and summaries of the individual scripts. Signed-off-by: Alex Williamson <alex.williamson(a)redhat.com> --- Makefile.am | 2 configure.ac | 3 hooks/Makefile.am | 27 ++++ hooks/README | 47 +++++++ hooks/daemon | 33 +++++ hooks/daemon.d/Makefile.am | 21 +++ hooks/daemon.d/static-hugepages.sh | 100 +++++++++++++++ hooks/functions.sh | 180 +++++++++++++++++++++++++++ hooks/libvirt-hook-config.xml | 38 ++++++ hooks/qemu | 40 ++++++ hooks/qemu.d/Makefile.am | 21 +++ hooks/qemu.d/dynamic-hugepages.sh | 238 ++++++++++++++++++++++++++++++++++++ libvirt.spec.in | 13 ++ 13 files changed, 761 insertions(+), 2 deletions(-) create mode 100644 hooks/Makefile.am create mode 100644 hooks/README create mode 100755 hooks/daemon create mode 100644 hooks/daemon.d/Makefile.am create mode 100755 hooks/daemon.d/static-hugepages.sh create mode 100755 hooks/functions.sh create mode 100644 hooks/libvirt-hook-config.xml create mode 100755 hooks/qemu create mode 100644 hooks/qemu.d/Makefile.am create mode 100755 hooks/qemu.d/dynamic-hugepages.sh diff --git a/Makefile.am b/Makefile.am index 9796069..e4a709a 100644 --- a/Makefile.am +++ b/Makefile.am @@ -24,7 +24,7 @@ SUBDIRS = . gnulib/lib include src daemon tools docs gnulib/tests \ examples/dominfo examples/domsuspend examples/apparmor \ examples/xml/nwfilter examples/openauth examples/systemtap \ tools/wireshark examples/dommigrate \ - examples/lxcconvert examples/domtop + examples/lxcconvert examples/domtop hooks ACLOCAL_AMFLAGS = -I m4 diff --git a/configure.ac b/configure.ac index 93f9e38..b0dc035 100644 --- a/configure.ac +++ b/configure.ac @@ -2812,7 +2812,8 @@ AC_CONFIG_FILES([\ examples/xml/nwfilter/Makefile \ examples/lxcconvert/Makefile \ tools/wireshark/Makefile \ - tools/wireshark/src/Makefile]) + tools/wireshark/src/Makefile \ + hooks/Makefile hooks/daemon.d/Makefile hooks/qemu.d/Makefile]) AC_OUTPUT AC_MSG_NOTICE([]) diff --git a/hooks/Makefile.am b/hooks/Makefile.am new file mode 100644 index 0000000..1fe6076 --- /dev/null +++ b/hooks/Makefile.am @@ -0,0 +1,27 @@ +## Copyright (C) 2015 Red Hat, Inc. +## +## This library is free software; you can redistribute it and/or +## modify it under the terms of the GNU Lesser General Public +## License as published by the Free Software Foundation; either +## version 2.1 of the License, or (at your option) any later version. +## +## This library is distributed in the hope that it will be useful, +## but WITHOUT ANY WARRANTY; without even the implied warranty of +## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +## Lesser General Public License for more details. +## +## You should have received a copy of the GNU Lesser General Public +## License along with this library. If not, see +## <http://www.gnu.org/licenses/>. + +SUBDIRS = daemon.d qemu.d + +hooksdir = $(pkgdatadir)/hooks +hooks_DATA = \ + daemon \ + functions.sh \ + libvirt-hook-config.xml \ + qemu \ + README + +EXTRA_DIST = $(hooks_DATA) diff --git a/hooks/README b/hooks/README new file mode 100644 index 0000000..8114c16 --- /dev/null +++ b/hooks/README @@ -0,0 +1,47 @@ +libivrt contributed and example hook scripts + +libvirt provides several static hooks for use around daemon startup and +shutdown as well as various points around domain and network lifecycle. +See https://www.libvirt.org/hooks.html for formal documentation. + +Summary of scripts available here: + +- daemon: + Supports sub-scripts in daemon.d directory +- qemu: + Supports sub-scripts in qemu.d directory +- daemon.d/static-hugepages.sh: + Supports hugepage allocations around libvirt daemon init + (Requires: xmllint numactl) +- qemu.d/dynamic-hugepages.sh: + Supports hugepage allocations around QEMU domain instantiation + (Requires: xmllint numactl) +- functions.sh: + Shared utility functions for scripts +- libvirt-hook-config.xml: + Base config file for scripts + +Please consult each script for further details. + +To use: + +1) Copy libvirt-hook-config.xml to /etc/sysconfig/ and modify as desired + +2) If only a single sub-script per hook type is needed, it may be copied + directly to /etc/libvirt/hooks, Ex: + cp qemu.d/dynamic-hugepages.sh /etc/libvirt/hooks/qemu + + If multiple sub-scripts per hook type are needed, or to support additional + sub-scripts later, use the stub qemu and daemon scripts here, create + qemu.d and daemon.d sub-directories under /etc/libvirt/hooks/ and copy + the desired scripts to the sub-directories, Ex: + cp qemu /etc/libvirt/hooks/ + mkdir /etc/libvirt/hooks/qemu.d + cp qemu.d/dynamic-hugepages.sh /etc/libvirt/hooks/qemu.d/ + + To easily get all of the scripts, it's possible to simply soft link + the entire directory, Ex: + ln -s /usr/share/libvirt/hooks /etc/libvirt/hooks + +3) Restart the libvirt daemon to re-read hook scripts: + systemctl restart libvirtd.service diff --git a/hooks/daemon b/hooks/daemon new file mode 100755 index 0000000..7b8947d --- /dev/null +++ b/hooks/daemon @@ -0,0 +1,33 @@ +#!/bin/sh + +# Copyright (C) 2015 Red Hat, Inc. +# +# This library is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 2.1 of the License, or (at your option) any later version. +# +# This library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this library. If not, see +# <http://www.gnu.org/licenses/>. + +# This script is intended to implement a libvirt /etc/libvirt/hooks/daemon +# script and extend daemon execution out to all executable scripts in the +# daemon.d subdirectory. To use, copy to /etc/libvirt/hooks/ and place +# any sub-scripts in a daemon.d subdirectory. + +DIR="/etc/libvirt/hooks/daemon.d/" + +if [ -d $DIR ]; then + for SCRIPT in $(find $DIR -type f -executable); do + $SCRIPT $@ + if [ $? -ne 0 ]; then + exit 1 + fi + done +fi diff --git a/hooks/daemon.d/Makefile.am b/hooks/daemon.d/Makefile.am new file mode 100644 index 0000000..ad9d6cb --- /dev/null +++ b/hooks/daemon.d/Makefile.am @@ -0,0 +1,21 @@ +## Copyright (C) 2015 Red Hat, Inc. +## +## This library is free software; you can redistribute it and/or +## modify it under the terms of the GNU Lesser General Public +## License as published by the Free Software Foundation; either +## version 2.1 of the License, or (at your option) any later version. +## +## This library is distributed in the hope that it will be useful, +## but WITHOUT ANY WARRANTY; without even the implied warranty of +## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +## Lesser General Public License for more details. +## +## You should have received a copy of the GNU Lesser General Public +## License along with this library. If not, see +## <http://www.gnu.org/licenses/>. + +daemonddir = $(pkgdatadir)/hooks/daemon.d +daemond_DATA = \ + static-hugepages.sh + +EXTRA_DIST = $(daemond_DATA) diff --git a/hooks/daemon.d/static-hugepages.sh b/hooks/daemon.d/static-hugepages.sh new file mode 100755 index 0000000..081209d --- /dev/null +++ b/hooks/daemon.d/static-hugepages.sh @@ -0,0 +1,100 @@ +#!/bin/sh + +# Copyright (C) 2015 Red Hat, Inc. +# +# This library is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 2.1 of the License, or (at your option) any later version. +# +# This library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this library. If not, see +# <http://www.gnu.org/licenses/>. + +# This script makes use of the libvirt hook interface for the libvirt daemon +# to allocate hugepages for domains during libvirt startup, releasing pages +# at libvirt shutdown. Hugepages are allocated statically, so long as the +# libvirt daemon is running. This mechanism is predominantly useful for +# hosts where the domains use very large hugepages or the majority of host +# memory, making dynamic allocation unreliable. The config file entry to +# enable this hook includes some number of hugepages elements of the form: +# +# <config> +# <daemon> +# <static-hugepages> +# <hugepages size="2048" unit="k" mode="strict" nodeset="0-1">1024</hugepages> +# <hugepages size="1" unit="G" mode="preferred" nodeset="2">4</hugepages> +# </static-hugepages> +# </daemon> +# </config> +# +# The above example allocates 1024, 2M pages (2G total), strictly across nodes +# 0-1 and 4, 1G pages (4G) total with a preferred node policy of node 2. Like +# the numatune elements in libvirt, the @mode may be one of "strict", +# "interleave", or "preferred", matching the libnuma policy. The default mode +# is strict. @size and @unit specify the size of hugepage, using the standard +# libvirt unit values. 1KiB is the default unit size. @nodeset makes use of +# the standard libvirt range support, allowing sets, ranges, and exclusions. +# NB, mode="preferred" requires a singleton nodeset. + +CONFIG="/etc/sysconfig/libvirt-hook-config.xml" + +source /usr/share/libvirt/hooks/functions.sh + +# Utilities required to parse our config file and set NUMA policy +which xmllint > /dev/null 2>&1 +if [ $? -ne 0 ]; then + exit 0 +fi + +which numactl > /dev/null 2>&1 +if [ $? -ne 0 ]; then + exit 0 +fi + +if [ ! -e $CONFIG ]; then + exit 0 +fi + +case "$2" in + start | shutdown) + ;; + *) + exit 0 + ;; +esac + +XMLLINT="xmllint --xpath" + +BASE="/config/daemon/static-hugepages/hugepages" + +for i in $(seq 1 $($XMLLINT "count($BASE)" $CONFIG 2>/dev/null)); do + SIZE=$($XMLLINT "string($BASE[$i]/@size)" $CONFIG 2>/dev/null) + UNIT=$($XMLLINT "string($BASE[$i]/@unit)" $CONFIG 2>/dev/null) + if [ -z "$UNIT" ]; then + UNIT=KiB + fi + COUNT=$($XMLLINT "$BASE[$i]/text()" $CONFIG 2>/dev/null) + + POLICY=$($XMLLINT "string($BASE[$i]/@mode)" $CONFIG 2>/dev/null) + NODESET=$($XMLLINT "string($BASE[$i]/@nodeset)" $CONFIG 2>/dev/null) + if [ -n "$NODESET" ] && [ -z "$POLICY" ]; then + POLICY=strict + fi + + case "$2" in + start) + addhugepages $SIZE $UNIT $COUNT $NODESET $POLICY + ;; + shutdown) + addhugepages $SIZE $UNIT -$COUNT $NODESET $POLICY + ;; + esac +done + +exit 0 diff --git a/hooks/functions.sh b/hooks/functions.sh new file mode 100755 index 0000000..07f52db --- /dev/null +++ b/hooks/functions.sh @@ -0,0 +1,180 @@ +#!/bin/sh + +# Copyright (C) 2015 Red Hat, Inc. +# +# This library is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 2.1 of the License, or (at your option) any later version. +# +# This library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this library. If not, see +# <http://www.gnu.org/licenses/>. + +# parserange($RANGE) +# +# This takes a libvirt defined range, of the form "1-7,^4,15-17" and +# parses it into an explicit list "1 2 3 5 6 7 15 16 17". The negations +# are only allowed for single numbers, not ranges because that's what +# libvirt does. Ordering is also important, "1-7,^4" is not the same as +# "^4,1-7". We assume a well formed range since we get it from libvirt. +function parserange { + declare -a BITMAP=() + CUR=0 + + while [ $CUR -lt ${#1} ]; do + unset NEGATE + START=0 + END=0 + + if [ ${1:$CUR:1} == ',' ]; then + CUR=$(( $CUR + 1 )) + continue + fi + + if [ ${1:$CUR:1} == '^' ]; then + CUR=$(( $CUR + 1 )) + NEGATE=1 + fi + + while [ $CUR -lt ${#1} ]; do + case ${1:$CUR:1} in + [0-9]) + START=$(( ( $START * 10 ) + ${1:$CUR:1} )) + CUR=$(( $CUR + 1 )) + ;; + *) + break + ;; + esac + done + + if [ $CUR -eq ${#1} ] || [ ${1:$CUR:1} == ',' ]; then + if [ -n "$NEGATE" ]; then + unset BITMAP[$START] + else + BITMAP[$START]=$START + fi + elif [ ${1:$CUR:1} == '-' ]; then + CUR=$(( $CUR + 1 )) + while [ $CUR -lt ${#1} ]; do + case ${1:$CUR:1} in + [0-9]) + END=$(( ( $END * 10 ) + ${1:$CUR:1} )) + CUR=$(( $CUR + 1 )) + ;; + *) + break + ;; + esac + done + + for i in $(seq $START $END); do + BITMAP[$i]=$i + done + fi + done + + echo ${BITMAP[@]} +} + +# parsesize($SIZE [$UNIT]) +# +# This takes a $SIZE and optional $UNIT and returns bytes. If a unit is not +# provided, 1KiB is assumed. +function parsesize { + MULTIPLIER=1024 + if [ -n "$2" ]; then + case "$2" in + b) + MULTIPLIER=1 + ;; + KB) + MULTIPLIER=1000 + ;; + KiB | k | kB) + MULTIPLIER=1024 + ;; + MB) + MULTIPLIER=1000000 + ;; + MiB | M) + MULTIPLIER=$(( 1024 * 1024 )) + ;; + GB) + MULTIPLIER=1000000000 + ;; + GiB | G) + MULTIPLIER=$(( 1024 * 1024 * 1024 )) + ;; + *) + echo "$0: Invalid hugepage unit: $2" >&2 + exit 1 + esac + fi + + echo $(( ( $1 * $MULTIPLIER ) )) +} + +# addhugepages($SIZE $UNIT $COUNT [$NODES [$POLICY]]) +# +# Given a hugepage size, as defined by $SIZE and $UNIT, attempt to allocate +# $COUNT, with an optional $NODES NUMA node range and $POLICY. This uses +# the raw sysfs hugepages interface rather than `virsh allocpages` in order +# to avoid issues with calling libvirt from within a hook function. +function addhugepages { + SYSFSNODE="/sys/kernel/mm/hugepages/hugepages-" + + HUGE=$(parsesize $1 $2) + SYSFSNODE+=$(( $HUGE / 1024 )) + SYSFSNODE+="kB" + + if [ ! -d $SYSFSNODE ]; then + echo "$0: Hugepage size $(( $HUGE / 1024 ))kB not supported by kernel" >&2 + exit 1 + fi + + NR="nr_hugepages" + PREFIX="" + if [ -n "$4" ]; then + NR+="_mempolicy" + + NODES=$(parserange "$4") + NODES=$(echo $NODES | sed -e 's/ /,/g') + + if [ -z "$5" ]; then + PREFIX="numactl -m $NODES" + else + case "$5" in + strict) + PREFIX="numactl -m $NODES" + ;; + interleave) + PREFIX="numactl -i $NODES" + ;; + preferred) + if [ $(expr index "$NODES" ",") -ne 0 ]; then + echo "$0: \"preferred\" memory policy only supports single node" >&2 + exit 1 + fi + PREFIX="numactl --preferred $NODES" + ;; + esac + fi + fi + + # NB: count ($3) can be negative! + NEW=$(( $(cat $SYSFSNODE/$NR) + $3 )) + if [ $NEW -lt 0 ]; then + NEW=0 + fi + $PREFIX echo $NEW > $SYSFSNODE/$NR + + # TBD: should we check for sufficient free pages or go on blindly? + # For now, the latter. +} diff --git a/hooks/libvirt-hook-config.xml b/hooks/libvirt-hook-config.xml new file mode 100644 index 0000000..1d8bdce --- /dev/null +++ b/hooks/libvirt-hook-config.xml @@ -0,0 +1,38 @@ + + +<config> + <daemon> + <static-hugepages> +  + </static-hugepages> + </daemon> + <qemu> + <dynamic-hugepages> +  + </dynamic-hugepages> + </qemu> +</config> diff --git a/hooks/qemu b/hooks/qemu new file mode 100755 index 0000000..2652831 --- /dev/null +++ b/hooks/qemu @@ -0,0 +1,40 @@ +#!/bin/sh + +# Copyright (C) 2015 Red Hat, Inc. +# +# This library is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 2.1 of the License, or (at your option) any later version. +# +# This library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this library. If not, see +# <http://www.gnu.org/licenses/>. + +# This script is intended to implement a libvirt /etc/libvirt/hooks/qemu +# script and extend qemu execution out to all executable scripts in the +# qemu.d subdirectory. To use, copy to /etc/libvirt/hooks/ and place +# any sub-scripts in a qemu.d subdirectory. + +DIR="/etc/libvirt/hooks/qemu.d/" + +# Domain xml is passed as stdin to the script, supply the same for sub-scripts +XML=$(mktemp) +cat - > $XML + +if [ -d $DIR ]; then + for SCRIPT in $(find $DIR -type f -executable); do + $SCRIPT $@ < $XML + if [ $? -ne 0 ]; then + rm $XML + exit 1 + fi + done +fi + +rm $XML diff --git a/hooks/qemu.d/Makefile.am b/hooks/qemu.d/Makefile.am new file mode 100644 index 0000000..64fe64d --- /dev/null +++ b/hooks/qemu.d/Makefile.am @@ -0,0 +1,21 @@ +## Copyright (C) 2015 Red Hat, Inc. +## +## This library is free software; you can redistribute it and/or +## modify it under the terms of the GNU Lesser General Public +## License as published by the Free Software Foundation; either +## version 2.1 of the License, or (at your option) any later version. +## +## This library is distributed in the hope that it will be useful, +## but WITHOUT ANY WARRANTY; without even the implied warranty of +## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +## Lesser General Public License for more details. +## +## You should have received a copy of the GNU Lesser General Public +## License along with this library. If not, see +## <http://www.gnu.org/licenses/>. + +qemuddir = $(pkgdatadir)/hooks/qemu.d +qemud_DATA = \ + dynamic-hugepages.sh + +EXTRA_DIST = $(qemud_DATA) diff --git a/hooks/qemu.d/dynamic-hugepages.sh b/hooks/qemu.d/dynamic-hugepages.sh new file mode 100755 index 0000000..438c0c9 --- /dev/null +++ b/hooks/qemu.d/dynamic-hugepages.sh @@ -0,0 +1,238 @@ +#!/bin/sh + +# Copyright (C) 2015 Red Hat, Inc. +# +# This library is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 2.1 of the License, or (at your option) any later version. +# +# This library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this library. If not, see +# <http://www.gnu.org/licenses/>. + +# This script makes use of the libvirt hook interface for qemu in order to +# attempt to allocate hugepages for a domain around instantiation. We will +# only try to allocate hugepages for VMs both configured for hugepages and +# listed in our config file as enabled for dynamic hugepages. The config file +# entry is simply a list of domain names enabled for dynamic hugepages, ex: +# +# <config> +# <qemu> +# <dynamic-hugepages> +# <domain name="domain1"/> +# <domain name="domain2"/> +# </dynamic-hugepages> +# </qemu> +# </config> +# +# The host system will often be able to accomodate hugepage requests so long +# memory isn't too fragmented. Domains making use of a significant portion of +# host memory or very large hugepages may be better served using static +# hugepage hooks. + +CONFIG="/etc/sysconfig/libvirt-hook-config.xml" + +source /usr/share/libvirt/hooks/functions.sh + +# Utilities required to parse our config file and set NUMA policy +which xmllint > /dev/null 2>&1 +if [ $? -ne 0 ]; then + exit 0 +fi + +which numactl > /dev/null 2>&1 +if [ $? -ne 0 ]; then + exit 0 +fi + +if [ ! -e $CONFIG ]; then + exit 0 +fi + +case "$2" in + prepare | release) + ;; + *) + exit 0 + ;; +esac + +XMLLINT="xmllint --xpath" + +BASE="/config/qemu/dynamic-hugepages/domain" + +# Is this domain enabled in the config file? +$XMLLINT "$BASE[@name=\"$1\"]/@name" $CONFIG > /dev/null 2>&1 +if [ $? -ne 0 ]; then + exit 0 +fi + +# Domain XML is provided via stdin, save it somewhere +XML=$(mktemp) +cat - > $XML + +# Abort if the domain isn't configure for hugepages +$XMLLINT "/domain/memoryBacking/hugepages" $XML > /dev/null 2>&1 +if [ $? -ne 0 ]; then + rm $XML + exit 0 +fi + +# The easy non-NUMA guest case first. If no page size is specified, read the +# default hugepage size from /proc/meminfo. NB, our size parsing needs to +# handle "kB" in addition to the libvirt defined units for this. If page size +# is specified, there can be only one, so parse @size, @unit, and @nodeset and +# do the allocation/deallocation. + +$XMLLINT "/domain/cpu/numa" $XML > /dev/null 2>&1 +if [ $? -ne 0 ]; then + BASE="/domain/memoryBacking/hugepages/page" + $XMLLINT "$BASE/@size" $XML > /dev/null 2>&1 + if [ $? -ne 0 ]; then + # grep Hugepagesize /proc/meminfo + # Hugepagesize: 2048 kB + HPAGESIZE=$(grep Hugepagesize /proc/meminfo | awk '{print $2}') + HPAGEUNIT=$(grep Hugepagesize /proc/meminfo | awk '{print $3}') + else + HPAGESIZE=$($XMLLINT "string($BASE/@size)" $XML 2>/dev/null) + HPAGEUNIT=$($XMLLINT "string($BASE/@unit)" $XML 2>/dev/null) + HPAGENODE=$($XMLLINT "string($BASE/@nodeset)" $XML 2>/dev/null) + fi + + + MEMORY=$($XMLLINT "/domain/memory/text()" $XML 2>/dev/null) + UNIT=$($XMLLINT "string(/domain/memory/@unit)" $XML 2> /dev/null) + MEMORY=$(parsesize $MEMORY $UNIT) + HSIZE=$(parsesize $HPAGESIZE $HPAGEUNIT) + COUNT=$(( $MEMORY / $HSIZE )) + case "$2" in + prepare) + addhugepages $HPAGESIZE $HPAGEUNIT $COUNT $HPAGENODE + ;; + release) + addhugepages $HPAGESIZE $HPAGEUNIT -$COUNT $HPAGENODE + ;; + esac + + rm $XML + exit 0 +fi + +# The harder case, get the guest NUMA id and node memory size from the cpu/numa +# tags. Cross reference this to numatune options to get the host nodeset for +# the memory. Then, figure out which page sizes are allowed on which host +# nodes. The libvirt xml is not well specified here, a numatune nodeset can +# span host nodes, which are not required to use the same page size. We can +# also still have hugepages that are not associated with a node. This code +# assumes that lack of a nodeset matches anything; we don't look for the "best" +# match and that the numatune nodeset must be fully contained within a host +# hugepages size set, or else we skip it. +for ID in $($XMLLINT "/domain/cpu/numa/cell/@id" $XML 2>/dev/null); do + BASE="/domain/cpu/numa/cell" + MEMORY=$($XMLLINT "string($BASE[@$ID]/@memory)" $XML 2>/dev/null) + UNIT=$($XMLLINT "string($BASE[@$ID]/@unit)" $XML 2>/dev/null) + if [ -z "$UNIT" ]; then + UNIT=KiB + fi + CELLID=$($XMLLINT "string($BASE[@$ID]/@id)" $XML 2>/dev/null) + CELLID=$(echo cellid=\"$CELLID\") + + unset POLICY + unset CELLNODE + + # First choice, use a memnode matching this cellid + BASE="/domain/numatune/memnode" + $XMLLINT "$BASE" $XML > /dev/null 2>&1 + if [ $? -eq 0 ]; then + $XMLLINT "$BASE[@$CELLID]" $XML > /dev/null 2>&1 + if [ $? -eq 0 ]; then + POLICY=$($XMLLINT "string($BASE[@$CELLID]/@mode)" $XML 2>/dev/null) + CELLNODE=$($XMLLINT "string($BASE[@$CELLID]/@nodeset)" $XML 2>/dev/null) + if [ -n "$CELLNODE" ] && [ -z "$POLICY" ]; then + POLICY=strict + fi + fi + fi + + # Second choice, use the memory element as the default (if exists) + if [ -z "$POLICY" ]; then + BASE="/domain/numatune/memory" + $XMLLINT "$BASE" $XML > /dev/null 2>&1 + if [ $? -eq 0 ]; then + POLICY=$($XMLLINT "string($BASE/@mode)" $XML 2>/dev/null) + CELLNODE=$($XMLLINT "string($BASE/@nodeset)" $XML 2>/dev/null) + if [ -n "$CELLNODE" ] && [ -z "$POLICY" ]; then + POLICY=strict + fi + fi + fi + + FOUND=0 + BASE="/domain/memoryBacking/hugepages/page" + $XMLLINT "$BASE/@size" $XML > /dev/null 2>&1 + if [ $? -ne 0 ]; then + HPAGESIZE=$(grep Hugepagesize /proc/meminfo | awk '{print $2}') + HPAGEUNIT=$(grep Hugepagesize /proc/meminfo | awk '{print $3}') + FOUND=1 + else + for SIZE in $($XMLLINT "$BASE/@size" $XML 2>/dev/null); do + HPAGESIZE=$($XMLLINT "string($BASE[@$SIZE]/@size)" $XML 2>/dev/null) + HPAGEUNIT=$($XMLLINT "string($BASE[@$SIZE]/@unit)" $XML 2>/dev/null) + HPAGENODE=$($XMLLINT "string($BASE[@$SIZE]/@nodeset)" $XML 2>/dev/null) + + # If the host hugepage size has no node association, we assume it's + # global. If the guest node has no hostnode association take the first + # available hugepage entry. + if [ -z "$HPAGENODE" ] || [ -z "$CELLNODE" ]; then + FOUND=1 + break + fi + + # Create an arry for the nodeset for the hugepages + declare -a NODEMAP=() + for i in $(parserange $HPAGENODE); do + NODEMAP[$i]=$i + done + + FAIL=0 + # Verify that each node of the numatune nodeset is accounted for + for i in $(parserange $CELLNODE); do + if [ -z "${NODEMAP[$i]}" ]; then + FAIL=1 + break + fi + done + if [ $FAIL -ne 0 ]; then + continue + fi + + FOUND=1 + break + done + fi + + if [ $FOUND -eq 0 ]; then + continue + fi + + MEMORY=$(parsesize $MEMORY $UNIT) + HSIZE=$(parsesize $HPAGESIZE $HPAGEUNIT) + COUNT=$(( $MEMORY / $HSIZE )) + case "$2" in + prepare) + addhugepages $HPAGESIZE $HPAGEUNIT $COUNT $CELLNODE $POLICY + ;; + release) + addhugepages $HPAGESIZE $HPAGEUNIT -$COUNT $CELLNODE $POLICY + ;; + esac +done + +rm $XML +exit 0 diff --git a/libvirt.spec.in b/libvirt.spec.in index dcd174a..ac9f07b 100644 --- a/libvirt.spec.in +++ b/libvirt.spec.in @@ -2248,6 +2248,9 @@ exit 0 %dir %{_datadir}/libvirt/ %dir %{_datadir}/libvirt/schemas/ +%dir %{_datadir}/libvirt/hooks/ +%dir %{_datadir}/libvirt/hooks/daemon.d/ +%dir %{_datadir}/libvirt/hooks/qemu.d/ %{_datadir}/libvirt/schemas/basictypes.rng %{_datadir}/libvirt/schemas/capability.rng @@ -2268,6 +2271,16 @@ exit 0 %{_datadir}/libvirt/cpu_map.xml %{_datadir}/libvirt/libvirtLogo.png +%{_datadir}/libvirt/hooks/README +%{_datadir}/libvirt/hooks/libvirt-hook-config.xml +%{_datadir}/libvirt/hooks/functions.sh +%attr(0755, root, root) %{_datadir}/libvirt/hooks/daemon +%attr(0755, root, root) %{_datadir}/libvirt/hooks/qemu + +%attr(0755, root, root) %{_datadir}/libvirt/hooks/daemon.d/static-hugepages.sh + +%attr(0755, root, root) %{_datadir}/libvirt/hooks/qemu.d/dynamic-hugepages.sh + %if %{with_systemd} %{_unitdir}/libvirt-guests.service %else

1 1

[libvirt] RFC: add Intel CMT feature support
by Ren, Qiaowei 29 Jun '15

29 Jun '15

Hi All, Some Intel processor families (e.g. the Intel Xeon processor E5 v3 family) introduced some PQos (Platform Qos) features to monitor or control shared resource. - CMT (Cache Monitoring Technology): measure the usage of cache by applications running on the platform. - CAT (Cache Allocation Technology): enable an OS or Hypervisor/VMM to specify the amount of cache space into which an application can fill. - MBM (Memory Bandwidth Monitoring): build on the CMT infrastructure to allow monitoring of bandwidth from one level of the cache hierarchy to the next - in this case focusing on the L3 cache, which is typically backed directly by system memory. As a result of this implementation, memory bandwidth can be monitored. For more information, see "Intel 64 and IA-32 Architectures Software Developer's Manual". Among these PQos features, currently CMT patches has been merged into Linux kernel mainline. So this patch series proposed in this mail will focus on CMT feature support in libvirt. At the library API layer, I plan on adding cache related field into virDomainInfo and virNodeInfo: Add "cache" member into virNodeInfo to get total size of cache in one node. struct virNodeInfo { ... unsigned int cores; /* number of cores per socket, total number of processors in case of unusual NUMA topology*/ unsigned int threads; /* number of threads per core, 1 in case of unusual numa topology */ + unsigned int cache; /* cache size in bytes */ }; Add "cacheOcc" member into virDomainInfo to get cache occupancy for one VM. struct virDomainInfo { ... unsigned short nrVirtCpu; /* the number of virtual CPUs for the domain */ unsigned long long cpuTime; /* the CPU time used in nanoseconds */ + unsigned long long cacheOcc; /* the cache occupancy in Bytes */ }; With this change, those applications like OpenStack based on libvirt can collect cache usage for metering and monitoring (e.g. any VM using more than X% of cache). Because CMT implementation in Linux kernel is based on perf mechanism and there is no nice and public API library for perf, I have to wrapper system call for perf in libvirt to use CMT feature, and enable/disable perf event for CMT when VM is created/destroyed. Any thoughts on this plan before I start submitting code patches? Do you think whether I need add several new APIs and XML element for CMT feature (or more PQos features)? Thanks, Qiaowei

2 2