March 2017 - Devel - libvirt List Archives

[libvirt] [RFC PATCH v2 REBASE 00/18] Introduce vGPU mdev framework to libvirt
by Erik Skultety 21 Mar '17

21 Mar '17

since the original v2 [1]: - resolved a few merge conflicts caused by @9d92f533 which refactored out some duplicate code which eventually lead to dropping patch 14/18 from the original series due to being unnecessary - rebased onto fresh HEAD [1] https://www.redhat.com/archives/libvir-list/2017-February/msg00739.html Erik Skultety (18): util: Introduce new module virmdev conf: Introduce new hostdev device type mdev conf: Introduce new address type mdev conf: Update XML parser, formatter, and RNG schema to support mdev conf: Introduce virDomainHostdevDefPostParse conf: Add post parse code for mdevs to virDomainHostdevDefPostParse security: dac: Enable labeling of vfio mediated devices security: selinux: Enable labeling of vfio mediated devices conf: Enable cold-plug of a mediated device qemu: Assign PCI addresses for mediated devices as well hostdev: Maintain a driver list of active mediated devices hostdev: Introduce a reattach method for mediated devices qemu: cgroup: Adjust cgroups' logic to allow mediated devices qemu: Bump the memory locking limit for mdevs as well qemu: Format mdevs on qemu command line test: Add some test cases for our test suite regarding the mdevs docs: Document the new hostdev and address type 'mdev' news: Update the NEWS.xml about the new mdev feature docs/formatdomain.html.in | 48 ++- docs/news.xml | 9 + docs/schemas/domaincommon.rng | 26 ++ po/POTFILES.in | 1 + src/Makefile.am | 1 + src/conf/device_conf.h | 1 + src/conf/domain_conf.c | 203 ++++++++++-- src/conf/domain_conf.h | 9 + src/libvirt_private.syms | 20 ++ src/qemu/qemu_command.c | 49 +++ src/qemu/qemu_command.h | 5 + src/qemu/qemu_domain.c | 23 +- src/qemu/qemu_domain.h | 1 + src/qemu/qemu_domain_address.c | 16 +- src/qemu/qemu_hostdev.c | 37 +++ src/qemu/qemu_hostdev.h | 8 + src/qemu/qemu_hotplug.c | 2 + src/security/security_apparmor.c | 3 + src/security/security_dac.c | 55 ++++ src/security/security_selinux.c | 54 ++++ src/util/virhostdev.c | 229 ++++++++++++- src/util/virhostdev.h | 16 + src/util/virmdev.c | 358 +++++++++++++++++++++ src/util/virmdev.h | 93 ++++++ tests/domaincapsschemadata/full.xml | 1 + .../qemuxml2argv-hostdev-mdev-unmanaged.args | 25 ++ .../qemuxml2argv-hostdev-mdev-unmanaged.xml | 37 +++ tests/qemuxml2argvtest.c | 6 + .../qemuxml2xmlout-hostdev-mdev-unmanaged.xml | 40 +++ tests/qemuxml2xmltest.c | 1 + 30 files changed, 1333 insertions(+), 44 deletions(-) create mode 100644 src/util/virmdev.c create mode 100644 src/util/virmdev.h create mode 100644 tests/qemuxml2argvdata/qemuxml2argv-hostdev-mdev-unmanaged.args create mode 100644 tests/qemuxml2argvdata/qemuxml2argv-hostdev-mdev-unmanaged.xml create mode 100644 tests/qemuxml2xmloutdata/qemuxml2xmlout-hostdev-mdev-unmanaged.xml -- 2.10.2

8 45

[libvirt] [PATCH 00/11] storage: modularize the storage driver backends
by Peter Krempa 21 Mar '17

21 Mar '17

Split up the storage driver backends into loadable modules so that binary distributions don't have to compromise on shipping the storage driver with all backends which may pull in too many dependencies. Peter Krempa (11): configure: Fix configure output for RBD storage backend tests: storagepoolxml2xml: Remove compile conditionals tests: drivermodule: Drop unused macro arguments driver: Split/refactor driver module loading daemon: Refactor connection driver module loading storage: backend: Refactor registration of the backend drivers storage: Turn driver backends into (static) modules storage: Turn storage backends into dynamic modules tests: drivermodule: Make sure that all compiled storage backends can be loaded spec: Modularize the storage driver news: Mention storage driver split daemon/libvirtd.c | 136 +++++++++------------- docs/news.xml | 10 ++ libvirt.spec.in | 185 +++++++++++++++++++++++++----- m4/virt-storage-rbd.m4 | 2 +- src/Makefile.am | 199 ++++++++++++++++++++++++++++++--- src/driver.c | 136 ++++++++++++++-------- src/driver.h | 6 +- src/libvirt_driver_modules.syms | 1 + src/storage/storage_backend.c | 151 +++++++++++++++++++------ src/storage/storage_backend.h | 5 + src/storage/storage_backend_disk.c | 7 ++ src/storage/storage_backend_disk.h | 4 +- src/storage/storage_backend_fs.c | 27 +++++ src/storage/storage_backend_fs.h | 11 +- src/storage/storage_backend_gluster.c | 13 ++- src/storage/storage_backend_gluster.h | 5 +- src/storage/storage_backend_iscsi.c | 7 ++ src/storage/storage_backend_iscsi.h | 4 +- src/storage/storage_backend_logical.c | 7 ++ src/storage/storage_backend_logical.h | 4 +- src/storage/storage_backend_mpath.c | 8 ++ src/storage/storage_backend_mpath.h | 4 +- src/storage/storage_backend_rbd.c | 7 ++ src/storage/storage_backend_rbd.h | 4 +- src/storage/storage_backend_scsi.c | 7 ++ src/storage/storage_backend_scsi.h | 4 +- src/storage/storage_backend_sheepdog.c | 7 ++ src/storage/storage_backend_sheepdog.h | 4 +- src/storage/storage_backend_vstorage.c | 7 ++ src/storage/storage_backend_vstorage.h | 4 +- src/storage/storage_backend_zfs.c | 7 ++ src/storage/storage_backend_zfs.h | 4 +- src/storage/storage_driver.c | 19 +++- src/storage/storage_driver.h | 1 + tests/Makefile.am | 4 +- tests/storagepoolxml2xmltest.c | 6 - tests/virdrivermoduletest.c | 52 +++++---- tests/virstoragetest.c | 4 + 38 files changed, 788 insertions(+), 285 deletions(-) -- 2.11.0

3 25

[libvirt] [RFC] finegrained disk driver options control
by Denis V. Lunev 20 Mar '17

20 Mar '17

Hello, All! There is a problem in the current libvirt implementation. domain.xml allows to specify only basic set of options, especially in the case of QEMU, when there are really a lot of tweaks in format drivers. Most likely these options will never be supported in a good way in libvirt as recognizable entities. Right now in order to debug libvirt QEMU VM in production I am using very strange approach: - disk section of domain XML is removed - exact command line options to start the disk are specified at the end of domain.xml whithin <qemu:commandline> as described by Stefan http://blog.vmsplice.net/2011/04/how-to-pass-qemu-command-line-options.html The problem is that when debug is finished and viable combinations of options is found I can not drop VM in such state in the production. This is the pain and problem. For example, I have spend 3 days with the VM of one customer which blames us for slow IO in the guest. I have found very good combination of non-standard options which increases disk performance 5 times (not 5%). Currently I can not put this combination in the production as libvirt does not see the disk. I propose to do very simple thing, may be I am not the first one here, but it would be nice to allow to pass arbitrary option to the QEMU command line. This could be done in a very generic way if we will allow to specify additional options inside <driver> section like this: <disk type='file' device='disk'> <driver name='qemu' type='qcow2' cache='none' io='native' iothread='1'> <option name='l2-cache-size' value='64M/> <option name='cache-clean-interval' value='32'/> </driver> <source file='/var/lib/libvirt/images/rhel7.qcow2'/> <target dev='sda' bus='scsi'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> and so on. The meaning (at least for QEMU) is quite simple - these options will just be added to the end of the -drive command line. The meaning for other drivers should be the same and I think that there are ways to pass generic options in them. Any opinion? Den

4 19

[libvirt] [PATCH] qemu: Don't try to update undefined guest CPU
by Jiri Denemark 20 Mar '17

20 Mar '17

Calling virCPUUpdateLive on a domain with no guest CPU configuration does not make sense. Especially when doing so would crash libvirtd. Signed-off-by: Jiri Denemark <jdenemar(a)redhat.com> --- src/qemu/qemu_process.c | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/src/qemu/qemu_process.c b/src/qemu/qemu_process.c index 780f9587a..ec0e36d2e 100644 --- a/src/qemu/qemu_process.c +++ b/src/qemu/qemu_process.c @@ -3815,11 +3815,6 @@ qemuProcessVerifyCPUFeatures(virDomainDefPtr def, { int rc; - if (!def->cpu || - (def->cpu->mode == VIR_CPU_MODE_CUSTOM && - !def->cpu->model)) - return 0; - rc = virCPUCheckFeature(def->os.arch, def->cpu, "invtsc"); if (rc < 0) { @@ -3870,6 +3865,13 @@ qemuProcessUpdateLiveGuestCPU(virQEMUDriverPtr driver, qemuProcessVerifyHypervFeatures(def, cpu) < 0) goto cleanup; + if (!def->cpu || + (def->cpu->mode == VIR_CPU_MODE_CUSTOM && + !def->cpu->model)) { + ret = 0; + goto cleanup; + } + if (qemuProcessVerifyCPUFeatures(def, cpu) < 0) goto cleanup; -- 2.12.0

2 1

Re: [libvirt] [PATCH 2/2] qemu: Validate the domain after marking the current domain as transient
by Daniel P. Berrange 20 Mar '17

20 Mar '17

On Wed, Mar 08, 2017 at 10:32:32AM +0100, Marc Hartmayer wrote: > On Mon, Feb 27, 2017 at 11:20 AM +0100, "Daniel P. Berrange" <berrange(a)redhat.com> wrote: > > On Mon, Feb 27, 2017 at 10:59:39AM +0100, Marc Hartmayer wrote: > >> On Thu, Feb 23, 2017 at 05:36 PM +0100, "Daniel P. Berrange" <berrange(a)redhat.com> wrote: > >> > On Thu, Feb 23, 2017 at 05:22:44PM +0100, Marc Hartmayer wrote: > >> >> On Thu, Feb 23, 2017 at 03:33 PM +0100, Michal Privoznik <mprivozn(a)redhat.com> wrote: > >> >> > On 02/23/2017 10:44 AM, Marc Hartmayer wrote: > >> >> >> Validate the domain that actually will be started. It's semantically > >> >> >> more clear and also it can detect failures that may have happened in > >> >> >> virDomainObjSetDefTransient(). > >> >> >> > >> >> >> Signed-off-by: Marc Hartmayer <mhartmay(a)linux.vnet.ibm.com> > >> >> >> Reviewed-by: Bjoern Walk <bwalk(a)linux.vnet.ibm.com> > >> >> >> Reviewed-by: Boris Fiuczynski <fiuczy(a)linux.vnet.ibm.com> > >> >> >> --- > >> >> >> src/qemu/qemu_process.c | 6 +++--- > >> >> >> 1 file changed, 3 insertions(+), 3 deletions(-) > >> >> >> > >> >> >> diff --git a/src/qemu/qemu_process.c b/src/qemu/qemu_process.c > >> >> >> index a57d136..bd3a8b8 100644 > >> >> >> --- a/src/qemu/qemu_process.c > >> >> >> +++ b/src/qemu/qemu_process.c > >> >> >> @@ -4746,9 +4746,6 @@ qemuProcessInit(virQEMUDriverPtr driver, > >> >> >> vm->def->os.machine))) > >> >> >> goto cleanup; > >> >> >> > >> >> >> - if (qemuProcessStartValidate(driver, vm, priv->qemuCaps, caps, flags) < 0) > >> >> >> - goto cleanup; > >> >> >> - > >> >> >> /* Do this upfront, so any part of the startup process can add > >> >> >> * runtime state to vm->def that won't be persisted. This let's us > >> >> >> * report implicit runtime defaults in the XML, like vnc listen/socket > >> >> >> @@ -4757,6 +4754,9 @@ qemuProcessInit(virQEMUDriverPtr driver, > >> >> >> if (virDomainObjSetDefTransient(caps, driver->xmlopt, vm) < 0) > >> >> >> goto cleanup; > >> >> >> > >> >> >> + if (qemuProcessStartValidate(driver, vm, priv->qemuCaps, caps, flags) < 0) > >> >> >> + goto cleanup; > >> >> >> + > >> >> > > >> >> > This needs to be goto stop for the reasons described in the previous e-mail. > >> >> > > >> >> >> if (flags & VIR_QEMU_PROCESS_START_PRETEND) { > >> >> >> if (qemuDomainSetPrivatePaths(driver, vm) < 0) > >> >> >> goto cleanup; > >> >> >> > >> >> > > >> >> > Honestly, I like what we have now better. I mean, SetDefTransient() is > >> >> > very unlikely to fail. It's just doing a copy of domain definition (in a > >> >> > very stupid way, but lets save that for a different discussion). > >> >> > Basically, it will fail on OOM only (which you will not get on a Linux > >> >> > system, unless you really try). > >> >> > >> >> It's semantically more clear (at least for me) and for example it > >> >> enables us to change some parts of the transient domain before > >> >> validation (affect the transient domain only, not the persistent). > >> > > >> > What are you planning to change in the config before validation ? > >> > > >> > >> I'm implementing a feature which allows to select the boot device at > >> domain start time (something like 'virsh start --with-bootdevice sda > >> domain1'). The main reason why we want this is because the s390 > >> architecture boots only from the first device specified in the boot > >> order. > > > > There's no need to changes in the QEMU driver todo this. You can just > > query the current XML config, change the boot device in it, and then > > run virDomainCreateXML to launch the guest with the changed config, > > or virDomainDefineXML again to make the changed boot device permanent. > > > > First of all, I hope I understand you right :) (you can tell me as soon > as you have read the following text) > > I've followed your advice and tried to add this functionality without > any changes in the QEMU/hypervisor. For this I've created a new function > in the remote driver which will call the needed functions (get current > XML config for the domain, manipulate the XML config, and then use > virDomainCreateXML for starting). > > To get the current XML config is straightforward as you've mentioned - > just use 'virDomanGetXMLDesc(...)'. > > But how can I change the boot device? > 1) I could use libxml2 for manipulating the XML string which we will get > with calling 'virDomainGetXMLDesc(..)' but this is error-prone and just > duplicate work as we have to parse the XML string and the information of > it. Also, if there are any additions in the future for boot devices > there will be no 'compile time reminder' that you have to edit this > function. Yes, this is the way it is normally done. There is a tradeoff here between wanting to be reminded at compile time but seeing build breakage, and libvirt wanting to be able to extend its config info without breaking apps. The libvirt view is that the latter is more important. > or > 2) Parse the XML string into our internal representation (virDomainDef), > adjust the boot orders and create a new XML string out of this internal > representation (reverse to the way how virDomainDefCopy(...) works). But > there is the huge problem: > > We need 'virCapsPtr caps' and 'virDomainXMLOptionPtr xmlopt' for > virDomainDefFormat(...) and virDomainDefParseString(...) but we're on > the remote driver layer and therefore we've no direct access to this. We > could add a function to the hypervisor driver interface for getting > these but this is just awkward. The virDomainDef are private data structures just for internal libvirt usage. Even though virsh is in the libvirt source tree, it should be treated as if it were a standalone application and thus only use the public APIs. > So my question to you is: do you have a better idea for this? > > Many thanks in advance. > > PS. I hope it's okay for you that I'm writing to you only for the moment > (as it's somehow off topic). If not I'll send it to the mailing list :) It isn't off-topic, so I'm copying back in the list. Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://entangle-photo.org -o- http://search.cpan.org/~danberr/ :|

2 1

[libvirt] [PATCH 0/3] log|lock daemon: Don't spam logs with IO error messages after client disconnects
by Peter Krempa 18 Mar '17

18 Mar '17

See patch 3 for explanation Peter Krempa (3): rpc: socket: Add possibility to suppress errors on read hangup rpc: serverclient: Add option to suppress errors on EOF (log|lock)daemon: Don't spam logs with IO error messages after client disconnects src/locking/lock_daemon.c | 3 +++ src/logging/log_daemon.c | 3 +++ src/rpc/virnetserverclient.c | 13 +++++++++++++ src/rpc/virnetserverclient.h | 2 ++ src/rpc/virnetsocket.c | 33 +++++++++++++++++++++++++++------ src/rpc/virnetsocket.h | 2 ++ 6 files changed, 50 insertions(+), 6 deletions(-) -- 2.12.0

2 8

[libvirt] [REPOST PATCH 0/4] Split out storage object into its own module
by John Ferlan 18 Mar '17

18 Mar '17

Repost after merging with latest changes of: http://www.redhat.com/archives/libvir-list/2017-March/msg00305.html Continuing down the pile of drivers from my RFC for making a common pool object - we're now at storage... For reference see patch 3 of: http://www.redhat.com/archives/libvir-list/2017-February/msg00519.html This series works through the storage conf adjustments. This pile is fairly straightforward. The virStorageVol* API's make it a bit more interesting, but like other series we split out anything referencing virStoragePoolObj and go from there. John Ferlan (4): conf: Introduce virstorageobj conf: Adjust coding style for storage conf sources conf: Alter coding style of storage conf function prototypes conf: Use consistent function name prefixes for virstorageobj po/POTFILES.in | 1 + src/Makefile.am | 3 +- src/conf/storage_conf.c | 975 ++--------------------------------------- src/conf/storage_conf.h | 174 ++------ src/conf/virstorageobj.c | 997 ++++++++++++++++++++++++++++++++++++++++++ src/conf/virstorageobj.h | 156 +++++++ src/libvirt_private.syms | 34 +- src/storage/storage_backend.h | 2 +- src/storage/storage_driver.c | 24 +- src/storage/storage_driver.h | 2 +- src/storage/storage_util.h | 1 - src/test/test_driver.c | 1 + 12 files changed, 1263 insertions(+), 1107 deletions(-) create mode 100644 src/conf/virstorageobj.c create mode 100644 src/conf/virstorageobj.h -- 2.9.3

2 5

[libvirt] [PATCH 0/4] Split out storage object into its own module
by John Ferlan 17 Mar '17

17 Mar '17

Continuing down the pile of drivers from my RFC for making a common pool object - we're now at storage... For reference see patch 3 of: http://www.redhat.com/archives/libvir-list/2017-February/msg00519.html This series works through the storage conf adjustments. This pile is fairly straightforward. The virStorageVol* API's make it a bit more interesting, but like other series we split out anything referencing virStoragePoolObj and go from there. John Ferlan (4): conf: Introduce virstorageobj conf: Adjust coding style for storage conf sources conf: Alter coding style of storage conf function prototypes conf: Use consistent function name prefixes for virstorageobj po/POTFILES.in | 1 + src/Makefile.am | 3 +- src/conf/storage_conf.c | 965 ++--------------------------------------- src/conf/storage_conf.h | 174 ++------ src/conf/virstorageobj.c | 986 ++++++++++++++++++++++++++++++++++++++++++ src/conf/virstorageobj.h | 156 +++++++ src/libvirt_private.syms | 34 +- src/storage/storage_backend.h | 2 +- src/storage/storage_driver.c | 24 +- src/storage/storage_driver.h | 2 +- src/storage/storage_util.h | 1 - src/test/test_driver.c | 1 + 12 files changed, 1255 insertions(+), 1094 deletions(-) create mode 100644 src/conf/virstorageobj.c create mode 100644 src/conf/virstorageobj.h -- 2.9.3

2 6

[libvirt] [PATCH v2 0/4] qemu: Add support for generic PCIe Root Ports
by Andrea Bolognani 17 Mar '17

17 Mar '17

Blurby McBlurbface. Andrea Bolognani (4): qemu: Add support for generic PCIe Root Ports qemu: Use generic PCIe Root Ports by default when available tests: Test generic PCIe Root Ports news: Document support for generic PCIe Root Ports docs/news.xml | 10 ++++++ docs/schemas/domaincommon.rng | 1 + src/conf/domain_conf.c | 4 ++- src/conf/domain_conf.h | 1 + src/qemu/qemu_capabilities.c | 2 ++ src/qemu/qemu_capabilities.h | 1 + src/qemu/qemu_command.c | 18 ++++++++-- src/qemu/qemu_domain_address.c | 13 +++++-- tests/qemucapabilitiesdata/caps_2.9.0.x86_64.xml | 1 + .../qemuxml2argv-pcie-root-port-model-generic.args | 22 ++++++++++++ .../qemuxml2argv-pcie-root-port-model-generic.xml | 22 ++++++++++++ .../qemuxml2argv-pcie-root-port-model-ioh3420.args | 21 ++++++++++++ .../qemuxml2argv-pcie-root-port-model-ioh3420.xml | 19 ++++++++++ tests/qemuxml2argvtest.c | 14 ++++++++ ...qemuxml2xmlout-pcie-root-port-model-generic.xml | 40 ++++++++++++++++++++++ ...qemuxml2xmlout-pcie-root-port-model-ioh3420.xml | 35 +++++++++++++++++++ tests/qemuxml2xmltest.c | 14 ++++++++ 17 files changed, 231 insertions(+), 7 deletions(-) create mode 100644 tests/qemuxml2argvdata/qemuxml2argv-pcie-root-port-model-generic.args create mode 100644 tests/qemuxml2argvdata/qemuxml2argv-pcie-root-port-model-generic.xml create mode 100644 tests/qemuxml2argvdata/qemuxml2argv-pcie-root-port-model-ioh3420.args create mode 100644 tests/qemuxml2argvdata/qemuxml2argv-pcie-root-port-model-ioh3420.xml create mode 100644 tests/qemuxml2xmloutdata/qemuxml2xmlout-pcie-root-port-model-generic.xml create mode 100644 tests/qemuxml2xmloutdata/qemuxml2xmlout-pcie-root-port-model-ioh3420.xml -- 2.7.4

2 10

[libvirt] [PATCH v1 0/2] libvirt-python: virEventAsyncIOImpl (was: libvirtaio)
by Wojtek Porczyk 17 Mar '17

17 Mar '17

Hello, As promised, this is the event loop implementation which dispatches the callbacks via asyncio's event loop. There are two patches: the first one fixes the bug around freeing opaque objects [1][2], and the second one is the actual implementation. Compared to standalone version I did earlier, this has public names more similar in style to other functions in the module. Also some comments are corrected, and packaging code is lightly touched. [1] https://www.redhat.com/archives/libvir-list/2017-January/msg00863.html [2] https://bugzilla.redhat.com/show_bug.cgi?id=1433028 Wojtek Porczyk (2): Allow for ff callbacks to be called by custom event implementations Add asyncio event loop implementation libvirt-override.c | 36 ++--- libvirt-override.py | 45 ++++++ libvirt-python.spec.in | 1 + libvirtaio.py | 398 +++++++++++++++++++++++++++++++++++++++++++++++++ sanitytest.py | 5 +- setup.py | 12 ++ 6 files changed, 471 insertions(+), 26 deletions(-) create mode 100644 libvirtaio.py -- pozdrawiam / best regards _.-._ Wojtek Porczyk .-^' '^-. Invisible Things Lab |'-.-^-.-'| | | | | I do not fear computers, | '-.-' | I fear lack of them. '-._ : ,-' -- Isaac Asimov `^-^-_>

2 3