- Devel - libvirt List Archives

[libvirt] [PATCH v5 00/24] network: refactor to decouple virt drivers from network driver
by Daniel P. Berrangé 23 May '19

23 May '19

An update to v1: https://www.redhat.com/archives/libvir-list/2018-December/msg00681.html v2: https://www.redhat.com/archives/libvir-list/2019-February/msg01581.html v3: https://www.redhat.com/archives/libvir-list/2019-March/msg01259.html v4: https://www.redhat.com/archives/libvir-list/2019-April/msg01186.html Currently the network driver registers a set of callbacks with the virt driver in order to handle allocating/releasing network ports associated with guest NICs. This series introduces a virNetworkPortPtr object and associated XML that describes a network port. The virt drivers now call public APIs associated with this new object to create/delete ports for guest NICs. Changed in v5: - Separately track ports with type=network vs type=bridge to deal with fallout after reverting patches which merged them - Add RNG schema and XML format docs - Change to use classID attr on <bandwidth> isntead of of separate <class id="xx"/> element - Drop obsolete driver deps from RPM spec - Other misc bug fixes found during testing Changed in v4: - Merged the ACKd patches which didn't depend on other un-acked parts - Improve bandwidth error messages - Ensure we set floor sum to zero when starting network - Misc fixes for previous review comments NB, I have not added missing docs for the new XML doc format yet. This is work in progress. Changed in v3: - Remove unused API symbol - Fix dist of test data files Changed in v2: - Fix many bugs related to upgrades with running VMs - Convert over bandwidth controls to the new APIs - Handle reconnecting VIFs to bridges during startup - Much much more that I can't remember Daniel P. Berrangé (24): conf: allow bandwidth parsing / formatting to include class ID conf: introduce virNetworkPortDefPtr struct and XML support network: make networkLogAllocation independent of domain conf conf: add APIs to convert virDomainNetDef to virNetworkPortDef network: convert networkAllocateActualDevice to virNetworkPortDef network: convert networkNotifyActualDevice to virNetworkPortDef network: convert networkReleaseActualDevice to virNetworkPortDef network: convert hook script to take a network port XML network: remove the virDomainNetBandwidthChangeAllowed callback network: introduce networkAllocatePort network: introduce networkNotifyPort network: introduce networkReleasePort network: introduce networkUpdatePortBandwidth network: add public APIs for network port object access: add permissions for network port objects remote: add support for new network port APIs virsh: add support for network port APIs conf: support recording ports against virNetworkObjPtr network: add implementation of network port APIs lxc, libxl: notify network driver of NICs during reconnect lxc, libxl: save domain status after reconnect conf: record a portid against the domain conf conf: switch over to use network port APIs for virt drivers rpm: remove dependancy from qemu to network/storage drivers docs/docs.html.in | 1 + docs/formatdomain.html.in | 8 + docs/formatnetworkport.html.in | 212 +++ docs/hooks.html.in | 24 +- docs/schemas/domaincommon.rng | 5 + docs/schemas/networkport.rng | 165 +++ include/libvirt/libvirt-network.h | 122 ++ include/libvirt/virterror.h | 3 + libvirt.spec.in | 3 - src/access/genpolkit.pl | 2 +- src/access/viraccessdriver.h | 6 + src/access/viraccessdrivernop.c | 11 + src/access/viraccessdriverpolkit.c | 26 + src/access/viraccessdriverstack.c | 25 + src/access/viraccessmanager.c | 16 + src/access/viraccessmanager.h | 6 + src/access/viraccessperm.c | 6 + src/access/viraccessperm.h | 44 + src/conf/Makefile.inc.am | 2 + src/conf/domain_conf.c | 531 +++++++- src/conf/domain_conf.h | 49 +- src/conf/netdev_bandwidth_conf.c | 30 +- src/conf/netdev_bandwidth_conf.h | 2 + src/conf/network_conf.c | 8 +- src/conf/virnetworkobj.c | 303 +++++ src/conf/virnetworkobj.h | 34 + src/conf/virnetworkportdef.c | 509 +++++++ src/conf/virnetworkportdef.h | 113 ++ src/datatypes.c | 60 + src/datatypes.h | 41 + src/driver-network.h | 41 + src/libvirt-network.c | 444 +++++++ src/libvirt_private.syms | 23 +- src/libvirt_public.syms | 15 + src/libxl/libxl_driver.c | 33 + src/lxc/lxc_process.c | 35 + src/network/bridge_driver.c | 1182 ++++++++++------- src/qemu/qemu_driver.c | 8 +- src/remote/remote_daemon_dispatch.c | 73 + src/remote/remote_driver.c | 69 + src/remote/remote_protocol.x | 124 +- src/remote_protocol-structs | 69 + src/rpc/gendispatch.pl | 18 +- src/util/virerror.c | 9 + src/util/virhook.c | 4 +- src/util/virhook.h | 4 +- tests/Makefile.am | 7 + .../net-virtio-network-portgroup.xml | 6 +- tests/virnetdevbandwidthtest.c | 1 + .../plug-bridge-mactbl.xml | 9 + .../virnetworkportxml2xmldata/plug-bridge.xml | 15 + .../virnetworkportxml2xmldata/plug-direct.xml | 12 + .../plug-hostdev-pci.xml | 12 + .../plug-network.xml | 16 + tests/virnetworkportxml2xmldata/plug-none.xml | 8 + tests/virnetworkportxml2xmltest.c | 104 ++ tests/virschematest.c | 1 + tools/virsh-completer.c | 50 + tools/virsh-completer.h | 4 + tools/virsh-network.c | 399 +++++- tools/virsh-network.h | 5 + 61 files changed, 4506 insertions(+), 661 deletions(-) create mode 100644 docs/formatnetworkport.html.in create mode 100644 docs/schemas/networkport.rng create mode 100644 src/conf/virnetworkportdef.c create mode 100644 src/conf/virnetworkportdef.h create mode 100644 tests/virnetworkportxml2xmldata/plug-bridge-mactbl.xml create mode 100644 tests/virnetworkportxml2xmldata/plug-bridge.xml create mode 100644 tests/virnetworkportxml2xmldata/plug-direct.xml create mode 100644 tests/virnetworkportxml2xmldata/plug-hostdev-pci.xml create mode 100644 tests/virnetworkportxml2xmldata/plug-network.xml create mode 100644 tests/virnetworkportxml2xmldata/plug-none.xml create mode 100644 tests/virnetworkportxml2xmltest.c -- 2.21.0

2 36

[libvirt] [PATCH] virDomainDefPostParse: use DOMAIN_DEVICE_ITERATE_MISSING_INFO
by Ján Tomko 23 May '19

23 May '19

Apart from virDomainDefValidate, virDomainDefPostParse is another place where operating on info-less devices makes sense. Signed-off-by: Ján Tomko <jtomko(a)redhat.com> --- src/conf/domain_conf.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/conf/domain_conf.c b/src/conf/domain_conf.c index 926c2139b7..006920e954 100644 --- a/src/conf/domain_conf.c +++ b/src/conf/domain_conf.c @@ -5803,7 +5803,8 @@ virDomainDefPostParse(virDomainDefPtr def, /* iterate the devices */ ret = virDomainDeviceInfoIterateInternal(def, virDomainDefPostParseDeviceIterator, - DOMAIN_DEVICE_ITERATE_ALL_CONSOLES, + DOMAIN_DEVICE_ITERATE_ALL_CONSOLES | + DOMAIN_DEVICE_ITERATE_MISSING_INFO, &data); if (virDomainDefPostParseCheckFailure(def, parseFlags, ret) < 0) -- 2.19.2

2 1

[libvirt] [PATCH 0/2] Introduce virDomainDeviceIterate
by Ján Tomko 23 May '19

23 May '19

A function for iterating over all devices [0] instead of just the ones with DeviceInfo. [0] Terms and conditions apply. Leases might be included as well. Ján Tomko (2): Introduce virDomainDeviceIterate Introduce DOMAIN_DEVICE_ITERATE_MISSING_INFO src/conf/domain_conf.c | 47 ++++++++++++++++++++++++++++++------------ src/conf/domain_conf.h | 3 +++ 2 files changed, 37 insertions(+), 13 deletions(-) -- 2.19.2

2 5

[libvirt] [PATCH] docs: Change the 'Launch Security' section id to "launchSecurity"
by Erik Skultety 23 May '19

23 May '19

Although there's currently only support for SEV, it's likely other solutions will appear, so we should not refer to the documentation section simply with 'sev'. Signed-off-by: Erik Skultety <eskultet(a)redhat.com> --- docs/formatdomain.html.in | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/formatdomain.html.in b/docs/formatdomain.html.in index e1da878fcc..c843b1d667 100644 --- a/docs/formatdomain.html.in +++ b/docs/formatdomain.html.in @@ -8924,7 +8924,7 @@ qemu-kvm -net nic,model=? /dev/null <p>Note: DEA/TDEA is synonymous with DES/TDES.</p> - <h3><a id="sev">Launch Security</a></h3> + <h3><a id="launchSecurity">Launch Security</a></h3> <p> The contents of the <code><launchSecurity type='sev'></code> element -- 2.20.1

1 1

[libvirt] [PATCH] misc: Drop useless checks from *Dispose() functions
by Michal Privoznik 23 May '19

23 May '19

Due to the way that our virObjectUnref() is written it's not possible that a NULL is passed into *Dispose() function. However, some functions check for that regardless. Signed-off-by: Michal Privoznik <mprivozn(a)redhat.com> --- src/conf/domain_conf.c | 3 --- src/conf/virstorageobj.c | 9 --------- src/rpc/virnetsshsession.c | 3 --- src/util/virhostdev.c | 3 --- 4 files changed, 18 deletions(-) diff --git a/src/conf/domain_conf.c b/src/conf/domain_conf.c index a3a514136b..732d37329f 100644 --- a/src/conf/domain_conf.c +++ b/src/conf/domain_conf.c @@ -2602,9 +2602,6 @@ virDomainChrSourceDefDispose(void *obj) virDomainChrSourceDefPtr def = obj; size_t i; - if (!def) - return; - virDomainChrSourceDefClear(def); virObjectUnref(def->privateData); diff --git a/src/conf/virstorageobj.c b/src/conf/virstorageobj.c index cdd9863eb8..31b5af8e9e 100644 --- a/src/conf/virstorageobj.c +++ b/src/conf/virstorageobj.c @@ -156,9 +156,6 @@ virStorageVolObjDispose(void *opaque) { virStorageVolObjPtr obj = opaque; - if (!obj) - return; - virStorageVolDefFree(obj->voldef); } @@ -190,9 +187,6 @@ virStorageVolObjListDispose(void *opaque) { virStorageVolObjListPtr vols = opaque; - if (!vols) - return; - virHashFree(vols->objsKey); virHashFree(vols->objsName); virHashFree(vols->objsPath); @@ -362,9 +356,6 @@ virStoragePoolObjDispose(void *opaque) { virStoragePoolObjPtr obj = opaque; - if (!obj) - return; - virStoragePoolObjClearVols(obj); virObjectUnref(obj->volumes); diff --git a/src/rpc/virnetsshsession.c b/src/rpc/virnetsshsession.c index 324b35b675..04b720014d 100644 --- a/src/rpc/virnetsshsession.c +++ b/src/rpc/virnetsshsession.c @@ -135,9 +135,6 @@ virNetSSHSessionDispose(void *obj) virNetSSHSessionPtr sess = obj; VIR_DEBUG("sess=0x%p", sess); - if (!sess) - return; - if (sess->channel) { libssh2_channel_send_eof(sess->channel); libssh2_channel_close(sess->channel); diff --git a/src/util/virhostdev.c b/src/util/virhostdev.c index 19ae001971..fe176f35e4 100644 --- a/src/util/virhostdev.c +++ b/src/util/virhostdev.c @@ -131,9 +131,6 @@ virHostdevManagerDispose(void *obj) { virHostdevManagerPtr hostdevMgr = obj; - if (!hostdevMgr) - return; - virObjectUnref(hostdevMgr->activePCIHostdevs); virObjectUnref(hostdevMgr->inactivePCIHostdevs); virObjectUnref(hostdevMgr->activeUSBHostdevs); -- 2.21.0

2 1

[libvirt] [PATCH 0/3] network: Delay creating private chains until starting network
by Daniel P. Berrangé 22 May '19

22 May '19

This is an series to delay creating libvirt firewall chains if nothing needs them during startup. Daniel P. Berrangé (3): network: pull global chain init into separate method network: add more debugging of firewall chain creation network: delay global firewall setup if no networks are running src/network/bridge_driver.c | 2 +- src/network/bridge_driver_linux.c | 99 +++++++++++++++++++++---- src/network/bridge_driver_nop.c | 3 +- src/network/bridge_driver_platform.h | 2 +- tests/networkxml2firewalldata/base.args | 34 +++++++++ tests/networkxml2firewalltest.c | 36 +++++++-- 6 files changed, 152 insertions(+), 24 deletions(-) create mode 100644 tests/networkxml2firewalldata/base.args -- 2.21.0

3 7

[libvirt] [PATCH 0/8] Refactor Intel IOMMU support
by Andrea Bolognani 22 May '19

22 May '19

Another prerequisite for vSMMUv3 support. Well, technically I could go ahead and implement that without refactoring Intel IOMMU support first, but I'd rather not add more validation code to qemu_command. Andrea Bolognani (8): tests: Simplify Intel IOMMU testing tests: Use DO_TEST_CAPS_*() for Intel IOMMU tests: Add negative test for Intel IOMMU conf: Allow NULL for virDomainDeviceInfoCallback conf: Drop DOMAIN_DEVICE_ITERATE_GRAPHICS conf: Add IOMMU support to virDomainDeviceInfoIterate() qemu: Introduce qemuDomainDeviceDefValidateIOMMU() qemu: Tweak Intel IOMMU command line generation src/bhyve/bhyve_device.c | 4 ++ src/conf/domain_addr.c | 9 +++ src/conf/domain_conf.c | 59 ++++++++++++---- src/libxl/libxl_driver.c | 3 + src/qemu/qemu_command.c | 69 +++---------------- src/qemu/qemu_domain.c | 66 +++++++++++++++++- src/qemu/qemu_domain_address.c | 36 +++++++++- src/qemu/qemu_hotplug.c | 3 + ...tel-iommu-caching-mode.x86_64-latest.args} | 15 ++-- .../intel-iommu-caching-mode.xml | 23 +------ ...tel-iommu-device-iotlb.x86_64-latest.args} | 22 +++--- .../intel-iommu-device-iotlb.xml | 1 + ...rgs => intel-iommu-eim.x86_64-latest.args} | 15 ++-- tests/qemuxml2argvdata/intel-iommu-eim.xml | 1 + ... => intel-iommu-machine.x86_64-2.6.0.args} | 9 ++- .../qemuxml2argvdata/intel-iommu-machine.xml | 3 +- ...hine.xml => intel-iommu-wrong-machine.xml} | 8 +-- ...ne.args => intel-iommu.x86_64-latest.args} | 18 +++-- tests/qemuxml2argvdata/intel-iommu.xml | 1 + tests/qemuxml2argvtest.c | 34 ++------- ...ntel-iommu-caching-mode.x86_64-latest.xml} | 0 ...ntel-iommu-device-iotlb.x86_64-latest.xml} | 0 ....xml => intel-iommu-eim.x86_64-latest.xml} | 0 ...l => intel-iommu-machine.x86_64-2.6.0.xml} | 0 ...ommu.xml => intel-iommu.x86_64-latest.xml} | 0 tests/qemuxml2xmltest.c | 15 ++-- 26 files changed, 242 insertions(+), 172 deletions(-) rename tests/qemuxml2argvdata/{intel-iommu-device-iotlb.args => intel-iommu-caching-mode.x86_64-latest.args} (61%) rename tests/qemuxml2argvdata/{intel-iommu-caching-mode.args => intel-iommu-device-iotlb.x86_64-latest.args} (53%) rename tests/qemuxml2argvdata/{intel-iommu-eim.args => intel-iommu-eim.x86_64-latest.args} (62%) rename tests/qemuxml2argvdata/{intel-iommu.args => intel-iommu-machine.x86_64-2.6.0.args} (73%) copy tests/qemuxml2argvdata/{intel-iommu-machine.xml => intel-iommu-wrong-machine.xml} (71%) rename tests/qemuxml2argvdata/{intel-iommu-machine.args => intel-iommu.x86_64-latest.args} (55%) rename tests/qemuxml2xmloutdata/{intel-iommu-caching-mode.xml => intel-iommu-caching-mode.x86_64-latest.xml} (100%) rename tests/qemuxml2xmloutdata/{intel-iommu-device-iotlb.xml => intel-iommu-device-iotlb.x86_64-latest.xml} (100%) rename tests/qemuxml2xmloutdata/{intel-iommu-eim.xml => intel-iommu-eim.x86_64-latest.xml} (100%) rename tests/qemuxml2xmloutdata/{intel-iommu-machine.xml => intel-iommu-machine.x86_64-2.6.0.xml} (100%) rename tests/qemuxml2xmloutdata/{intel-iommu.xml => intel-iommu.x86_64-latest.xml} (100%) -- 2.21.0

4 26

[libvirt] (Dropping) OOM Handling in libvirt
by Daniel P. Berrangé 22 May '19

22 May '19

This is a long mail about ENOMEM (OOM) handling in libvirt. The executive summary is that it is not worth the maint cost because: - this code will almost never run on Linux hosts - if it does run it will likely have bad behaviour silently dropping data or crashing the process - apps using libvirt often do so via a non-C language that aborts/exits the app on OOM regardless, or use other C libraries that abort - we can build a system that is more reliable when OOM happens by not catching OOM, instead simply letting apps exit, restart and carry on where they left off The long answer follows... The background ============== Since the first commit libvirt has attempted to handle out of memory (OOM) errors in the same way it does for any other problem. Namely, a virError will be raised, and the method will jump to its cleanup/error label. The error bubbles up the stack and in theory someone or something will catch this and do something sensible/useful. This has long been considered "best practice" for most C libraries, especially those used for system services. This mail makes the argument that it is in fact /not/ "best practice" to try to handle OOM. OOM handling is very difficult to get correct because it is something that developers almost never encounter and thus the code paths are rarely run. We designed our memory allocation APIs such that we get compile time errors if code forgets to check the return value for failure. This is good as it eliminates an entire class of bugs. Our error handling goto label pattern tries to align OOM handling with other general error handling which is more commonly tested. We have code in the allocators which lets us run unit tests simulating the failure of any allocation that is made during the test. Executing this is extraordinarily time consuming as some of our unit tests have many 1000's or even 10's of 1000's of allocations. The running time to simulate OOM for each allocation is O(N^2) which does not scale well. As a result we've probably only run these OOM tests a handful of times over the years. The tests show we generally do remarkly well at OOM handling, but none the less we have *always* found bugs in our handling where we fail to report the OOM (as in, continue to run but with missing data), or we crash when cleaning up from OOM. Our unit tests don't cover anywhere near all of our codebase though. We have great coverage of XML parsing/formatting, and sporadic coverage of everything else. IOW, despite our clever API designs & coding patterns designed to improve our OOM handling, we can not have confidence that we will actually operate correctly in an OOM scenario. Of course this applies to all error conditions that may arise, but OOM should be considered special. With other error conditions from syscalls or API calls, the effects are largely isolated to the site of the call. With OOM, the OOM condition may well persist and so during cleanup we will hit further OOM problems. We may well fail to even allocate the memory needed for raise a virErrorPtr. All threads can see the OOM concurrently so the effect spreads across the whole process very quickly. Who benefits from OOM handling ============================== Lets pretend for a minute though that our OOM handling is perfect and instead ask who/what is benefitting from it ? Libvirt's own processes. aka libvirtd, virtlockd, virtlogd ---------------------------------------------------------- For libvirtd we have essentially zero confidence that it will handle OOM at all. It is running all virt driver code over which we have massively inadequate unit testing cover to have any confidence that OOM will be well handled. Even if OOM is handled in a worker thread, every iteration of the event loop does an allocation to hold the poll FD array, so we can easily see the event loop hitting OOM which will make the entire process shutdown, or worse, hang during cleanup as worker threads block waiting for the event loop todo something despite not being running anymore. We already expect that bugs will cause libvirtd to crash and so have designed the drivers to be robust such that it can restart and attempt to carry on as normal afterwards. So arguably it would be fine to handle OOM by simply doing an abort and let systemd restart the daemon. For virtlockd and virtlogd we again have little confidence that they will handle OOM correctly. They are, however, more critical processes that we badly need to stay running at all times. We go to great effort to make it possible to re-exec on upgrades keeping state open. For virtlogd we could potentially change the way we deal with stdout/err for QEMU. Instead of using an anonymous pipe, we could create a named fifo on disk for each QEMU process. stdout/err would be connected to one end, and virtlogd to the other end. This would enable us to have virtlogd restarted and re-open the stdout/err for QEMU. This would mean we no longer need the re-exec logic either, which is good as that's another thing that's rarely tested. This would make abort on OOM acceptable. For virtlockd we have the hardest problem. It fundamentally has to keep open the FDs in order to maintain active locks. If it does crash / abort, then all open locks are lost. It would have to manually re-acquire locks when it starts up again, provided it had a record of which locks it was supposed to have. In theory nothing bad should happen in this window where virtlockd is dead, as if libvirtd tries to start another VM it will trigger auto-start of virtlockd via its systemd socket unit, which could then cause it to reacquire previous locks it had. IOW, it should be possible to make virtlockd robust enough that doing abort on OOM is tolerable. We really need this robustness regardless, because virtlockd can of course already crash due to code bugs. If we make it robust to all crashes, then by implication it will be robust enough for aborton OOM. Clients of libvirt. aka oVirt, OpenStack, KubeVirt, virt-manager, cockpit ------------------------------------------------------------------------- Clients of libvirt can consume it via a number of different APIs, all eventually calling into libvirt.so - Core C API aka libvirt.so Catches and propagates OOM to callers. virsh just reports it as it would any other error. In single shot mode it will be exiting regardles with error code. In interactive mode, in theory it can carry on with processing the next command. In practice its CLI parsing will probably then hit OOM anyway causing it to exit. virt-viewer links to GTK and this already has abort-on-OOM behaviour via GLib. So in all liklihood, even if it catches the OOM from libvirt, it will none the less abort on OOM in a GTK/GLib call. libvirt-dbus is written using glib, so again even if it catches the OOM from libvirt it will none the less abort on OOM in a GLib call. - Python binding Python bindings will raise a MemoryError exception which propagates back up to the python code. In theory the app can catch this and take some behaviour. This mostly only works though if the cause was a single huge allocation, such that most other "normal" sized allocations continue working. In a true OOM scenario where arbitrary allocs start failing, the python interpretor will fail too. - Perl binding It is hard to find clear info about Perl behaviour during ENOMEM. The libvirt bindings assume the Perl allocation functions won't ever fail, so if they do we're going to reference a NULL pointer. If normal Perl code gets OOM, the interpretor will raise an error. In theory this can be caught, but in practice few, if any, apps will try so the process will likely quit. - Go binding Errors from libvirt are all turned into Golang errors and propagated to the caller. If the Go runtime gets an OOM from an allocation it will panic the process which can't be caught, so it will exit with stack trace. - Java binding The JVM tends to allocate a fixed size heap for its own use. So Java code can see OOM exceptions even if the OS has plenty of memory. If the OS does run out of memory, assuming the JVM heap was already fully allocated it shouldn't immediately be affect, but might suffer collatoral damage. But in theory libvirt OOM could get turned into a Java exception that an app can catch & handle nicely. There are other bindings, but the above captures the most important usage of libvirt. The complication of Linux ========================= Note that all of the above discussion had the implicit assumption that malloc will actually return ENOMEM when memory is exhausted. On Linux at least this is not the case in a default install. Linux tends to enable memory overcommit causing the kernel to satisfy malloc requests even if it exceeds available RAM + Swap. The allocated memory won't even be paged in to physical RAM until the page is written to. If there is insufficient to make a page resident in RAM, then the OOM killer is set free. Some poor victim will get killed off. It is possible to disable RAM overcommit and also disable or deprioritize the OOM killer. This might make it possible to actually see ENOMEM from an allocation, but few if any developers or applications run in this scenario so it should be considered untested in reality. With cgroups it is possible to RAM and swap usage limits. In theory you can disable the OOM killer per-cgroup and this will cause ENOMEM to the app. In my testing with a trivial C program that simply mallocs a massive array and them memsets it, it is hard to get ENOMEM to happen reliably. Sometimes the app ends up just hanging when testing this. Other operating systems have different behaviour and so are more likely to really report ENOMEM to application code, but even if you can get ENOMEM on other OS the earlier points illustrate that its not worth worrying about. The conclusion ============== To repeat the top of this document, attempts to handle OOM are not worth the maint cost they incur: - this code will almost never run on Linux hosts - if it does run it will likely have bad behaviour silently dropping data or crashing the process - apps using libvirt often do so via a non-C language that aborts/exits the app on OOM regardless, or use other C libraries that abort - we can build a system that is more reliable when OOM happens by not catching OOM, instead simply letting apps exit, restart and carry on where they left off. The proposal is thus to change all libvirt code (ie both client and server side), to abort on OOM. This will allow us to simplify the control flow in many methods eliminating ~1500 checks for memory failure, their associated goto jumps, and many of the cleanup/error labels. This makes the code easier to follow & maintain and opens up new avenues for improving libvirt's future development. The main blocking prequisite to making the change is to address the needs for reliable restart of virtlockd and virtlogd daemons. As a point of reference libguestfs has had abort-on-oom behaviour forever and no one has complained about this behaviour. Apps using libvirt often also use libguestfs in the same process. The implementation ================== Assuming a decision to abort on OOM, libvirt can nwo follow QEMU's lead and make use of the GLib library. The initial impl would thus be to just link to GLib and switch VIR_ALLOC related APIs to use g_new/g_malloc & friends internally. Over time code can be changed to use g_new/g_malloc directly thus removing the error handling incrementally. Use of GLib brings a number of other significant opportunities / advantages to ongoing development - Access to many standard data structures (hash tables, linked lists, queues, growable arrays, growable strings). - Access to platform portability APIs for threads, event loops, file I/O, filename parsing and more. - Access to GObject to replace our own virObject clone with something that is far more feature rich. This is especially true of its callback facility via signal handlers. - Access to GIO to replace much of our socket I/O portability wrappers, and DBus client APIs. - Opens the possibility of calling to code in a non-C language via the GObject introspection bindings for GObject & other helper APIs. - Ability to drop use of gnulib once we use GLib for all portability problems we currently rely on gnulib for, once a critical set of functionality is ported to the GLib APIs. Most important is the sockets / event loop stuff due to Windows portability. - Ability to drop use of autoconf/automake in favour of a more maintainable option like meson/ninja, once we eliminate use of gnulib. Ultimately the big picture benefit is that we can spend less time working on low level generic code related to platform portability, data structures, or system services. Note this is not claiming that all GLib's APIs are better than stuff we have implemented in libvirt already. The belief is that even if some of the GLib APIs are worse, it is still a benefit to the project to use them, as it will reduce the code we maintain ourselves. This in turns lets us spend more time working on high level code directly related to virtualization features and be more productive when doing so. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|

8 17

[libvirt] Why virsh domxml-to-native qemu-argv cut VN name
by Tal Attaly 21 May '19

21 May '19

Hi, I think there is a bug when running virsh domxml-to-native qemu-argv with VM with long name, it getting cut - .. -name guest=l-dev-vrt-140-022-RH-7.0-Tal,debug-threads=on -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain--1-l-dev-vrt-140-022-RH/master-key.aes -machine pc-i440fx-2.12,accel=kvm,usb=off,dump-guest-core=off -m 8192 -realtime mlock=off -smp 2,sockets=2,cores=1,threads=1 -uuid cb48f75c-40f6-432d-964b-906ee0bdcf48 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain--1-l-dev-vrt-140-022-RH/monitor.sock,server,nowait .. -name is ok, but -object and -chardev are cut, which cause QEMU to file on - qemu-system-x86_64: -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/do main--1-l-dev-vrt-140-022-RH/monitor.sock,server,nowait: Failed to bind socket to /var/lib/libvirt/qemu/doma in--1-l-dev-vrt-140-022-RH/monitor.sock: No such file or directory Libvirt version is 3.9.0, here is the full domain XML - <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> <name>l-dev-vrt-140-022-RH-7.0-Tal</name> <uuid>cb48f75c-40f6-432d-964b-906ee0bdcf48</uuid> <memory unit='KiB'>8388608</memory> <currentMemory unit='KiB'>4194304</currentMemory> <vcpu placement='static'>2</vcpu> <os> <type arch='x86_64' machine='pc-i440fx-2.12'>hvm</type> <boot dev='hd'/> </os> <features> <acpi/> <apic/> <pae/> </features> <clock offset='utc'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>x86_64-softmmu/qemu-system-x86_64</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2' cache='none'/> <source file='/images/l-dev-vrt-140-022/l-dev-vrt-140-022.img'/> <target dev='hda' bus='ide'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <controller type='usb' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> </controller> <controller type='ide' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='pci' index='0' model='pci-root'/> <interface type='bridge'> <mac address='00:50:56:18:8c:16'/> <source bridge='br0:'/> <model type='e1000'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> <serial type='pty'> <target type='isa-serial' port='0'> <model name='isa-serial'/> </target> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <graphics type='vnc' port='-1' autoport='yes'> <listen type='address'/> </graphics> <video> <model type='cirrus' vram='16384' heads='1' primary='yes'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </video> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </memballoon> </devices> </domain> Thanks, Tal

2 1

[libvirt] [PATCH 0/3] docs: hacking: Discourage some patterns to help code readability
by Peter Krempa 21 May '19

21 May '19

Peter Krempa (3): docs: hacking: Document few practices for creating error messages docs: hacking: Add good practices for shortening conditional expressions docs: hacking: Discourage use of the ternary operator and ban it's abuse docs/hacking.html.in | 61 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 61 insertions(+) -- 2.21.0

3 7