February 2024 - Users - libvirt List Archives

trustGuestRxFilters broken after upgrade to Debian 12
by Paul B. Henson 01 Nov '24

01 Nov '24

We've been running Debian 11 for a while, using sr-iov: <network> <name>sr-iov-intel-10G-1</name> <uuid>6bdaa4c8-e720-4ea0-9a50-91cb7f2c83b1</uuid> <forward mode='hostdev' managed='yes'> <pf dev='eth2'/> </forward> </network> and allocating vf's from the pool: <interface type='network' trustGuestRxFilters='yes'> <mac address='52:54:00:08:da:5b'/> <source network='sr-iov-intel-10G-1'/> <vlan> <tag id='50'/> </vlan> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> After upgrading to Debian 12, when I try to start any vm which uses the trustGuestRxFilters option, it fails to start with the message: error: internal error: unable to execute QEMU command 'query-rx-filter': invalid net client name: hostdev0 If I remove the option, it starts fine (but of course is broken functionality wise as the option wasn't there just for fun :) ). Any thoughts on what's going on here? The Debian 12 versions are: libvirt-daemon/stable,now 9.0.0-4 qemu-system-x86/stable,now 1:7.2+dfsg-7+deb12u3 I see Debian 12 backports has version 8.1.2+ds-1~bpo12+1 of qemu, but no newer versions of libvirt. I haven't tried the backports version to see if that resolves the problem. Thanks much...

5 7

all domains paused, maybe logging might be helpfull
by Lennart Fricke 21 Mar '24

21 Mar '24

Hello, I just hit the situation that all domains on a host where paused due to missing space. It took me some time to figure out that there was no space left for the images on the host. I learned that 'virsh domstate --reason $GUEST' and 'domblkerror $GUEST' could have helped me. But the logs are silent about the problem. Would it be possible to show these problems in logs or is there other documentation than the reference to find out how to troubleshoot those issues? Thank you Lennart

2 3

Info regarding AMX support and libvirt implications
by Gianluca Cecchi 08 Mar '24

08 Mar '24

Hello, I'm trying to use AMX in my virtual machines. More info on AMX: https://www.intel.com/content/www/us/en/products/docs/accelerator-engines/a… My system in test is currently SLES 15 SP5. I'm also verifying in parallel with Suse (especially regarding the backported features in their 5.14 based kernel), but in the meantime I would like to understand implication, if any, of libvirt in the certification loop I have to analyse. From what I see we have in upstream: . support in the KVM kernel module since 5.17 . support of cpu model SapphireRapids, the first offering AMX as an ISA extension, in QEMU since 7.0 Is there any dependance to check on libvirt too? When I run virsh cpu-models x86_64 Is libvirt sw stack querying qemu directly? Or the kvm kernel module? Or any internal "database" file? From man page it is not clear to me what "known" means: " cpu-models Syntax: cpu-models arch Print the list of CPU models known by libvirt for the specified architecture. Whether a specific hypervisor is able to create a domain which uses any of the printed CPU models is a separate question which can be answered by looking at the domain capabilities XML returned by domcapabilities command. Moreover, for some architectures libvirt does not know any CPU models and the usable CPU models are only limited by the hypervisor. This command will print that all CPU models are accepted for these architectures and the actual list of supported CPU models can be checked in the domain capabilities XML. " In SLES 15 SP5 with: qemu-7.1.0-150500.49.9.2.x86_64 kernel-default-5.14.21-150500.55.49.1.x86_64 libvirtd-*-9.0.0-150500.6.11.1.x86_64 I get # virsh cpu-models x86_64 ... Cascadelake-Server Cascadelake-Server-noTSX Icelake-Client Icelake-Client-noTSX Icelake-Server Icelake-Server-noTSX Cooperlake Snowridge athlon phenom Opteron_G1 Opteron_G2 ... # virsh domcapabilities | grep -i sapphirerapid # In fedora39 with qemu-8.1.3-4.fc39.x86_64 kernel-6.7.5-200.fc39.x86_64 libvirt-*-9.7.0-2.fc39.x86_64 I get # virsh cpu-models x86_64 ... Cascadelake-Server Cascadelake-Server-noTSX Icelake-Client Icelake-Client-noTSX Icelake-Server Icelake-Server-noTSX Cooperlake Snowridge SapphireRapids athlon phenom Opteron_G1 Opteron_G2 ... # virsh domcapabilities | grep -i sapphirerapids <model usable='no' vendor='Intel'>SapphireRapids</model> # because I'm running on a client system without AMX support Thanks in advance, Gianluca

2 3

How to monitor domains in regards steal time and other important metrics (VIR_DOMAIN_STATS_VCPU) ?
by Christian Rohmann 02 Mar '24

02 Mar '24

Hey libvirt-users, first allow me to give a little background. We monitor performance metrics of OpenStack Nova VMs using libvirt as hypervisor. We used to run the libvirt prometheus exporter written by zhangjianweibj [1]. This exporter, compared to the one from kumina / tinkoff ([2]) makes use of the DigitalOcean go-libvirt [3], but that should not make much of a difference for my questions. Since the development of that exporter seems to have stalled and we wanted to rework and contribute new features to it, we created a fork [4]. After working trough the various ideas we had and applying them to the code, we proposed the prometheus-community to adopt the exporter [5] to ensure it is maintained and to serve as a reference exporter even. Now to my actual question ... Libvirt exposes per VCPU stats for domains via [6]. I'd like to be able to export those via the exporter. One important metric to me would be things like the steal time (vcpu.<num>.delay), to determine is domains are starting to get cut short or even starve on cpu time. Apparently those metrics are / cannot be expose anymore since the switch to CGroupsV2? Reading [7] or [8] others seem to have run into this. Is this actually still the case, even for more recent kernels? If so, I am wondering if there is an issue being tracked to implement this functionality? How is the steal time reported to the guest if the hypervisor is unable to export this info? Then there are other approaches like vmtop by Digital Ocean [9], which does use info and metrics available via /proc to determine steal time and other vcpu based metrics. So it seems the required data is somewhat available from the kernel? Last but not least I'd like your opinion on what other key metrics are important to monitoring on hypervisors and their guests? Regards Christian [1] https://github.com/zhangjianweibj/prometheus-libvirt-exporter [2] https://github.com/Tinkoff/libvirt-exporter [3] https://github.com/digitalocean/go-libvirt [4] https://github.com/inovex/prometheus-libvirt-exporter [5] https://github.com/prometheus-community/community/issues/50 [6] https://libvirt.org/html/libvirt-libvirt-domain.html#VIR_DOMAIN_STATS_VCPU [7] https://bugzilla.redhat.com/show_bug.cgi?id=2015763 [8] https://bugzilla.redhat.com/show_bug.cgi?id=1796543 [9] https://github.com/digitalocean/vmtop/

3 7

non-root bridge set-up on Fedora 39 aarch64
by Chuck Lever 28 Feb '24

28 Feb '24

Hello- I'm somewhat new to the libvirt world, and I've encountered a problem that needs better troubleshooting skills than I have. I've searched Google/Ecosia and stackoverflow without finding a solution. I set up libvirt on an x86_64 system without a problem, but on my new aarch64 / Fedora 39 system, virsh doesn't seem to want to start virbr0 when run from my own user account: cel@boudin:~/kdevops$ virsh net-start default error: Failed to start network default error: error creating bridge interface virbr0: Operation not permitted cel@boudin:~/kdevops$ cat /etc/qemu/bridge.conf allow virbr0 cel@boudin:~/kdevops$ Where can I look next? -- Chuck Lever

3 12

Re: restarting libvirtd with sr-iov
by Paul B. Henson 28 Feb '24

28 Feb '24

To reply to myself, I see that the sr-iov pool is initialized by networkCreateInterfacePool in network/bridge_driver.c, and it looks like ports are allocated by networkAllocatePort. The latter looks for a device with 0 connections as defined by netdef->forward.ifs[i].connections, and later bumps that count if the device is successfully allocated. So it seems the answer to my question is that libvirt does maintain this state in memory only, and does not try to re-create it if restarted. As such I don't think there's any way to recover from my situation currently short of shutting down everything :(. networkCreateInterfacePool iterates over all the vf's while configuring the pool. Would there by any way for it to check to see if a vf is already is use while doing so, and initialize connections to 1 so it won't be used until the running vm releases it, or at least generate a warning that the vf is in use and *not* add it to the pool? Thanks...

1 0

restarting libvirtd with sr-iov
by Paul B. Henson 28 Feb '24

28 Feb '24

We're running libvirt under Debian 12, package version 9.0.0-4. Earlier today I made a configuration change and restarted libvirtd. I've done this for years and never had a problem, after restarting it shows all the active storage pools, networks, and virtual machines and worked fine. However, I guess this is the first time I've done it on a system with an sr-iov network pool. After restarting, I was unable to initialize any virtual machines, as it would try to reallocate vf's that were in use by existing machines already running, resulting in an error from qemu. I spent a fair amount of time trying to recover from this, ideally with some way to make libvirt scan existing vm's and update the sr-iov pool in use status, or even some manual way to tell it which ones were in use. Unfortunately, I couldn't find anything and ended up having to shut down all the vm's and then restart them to fix it. Where is the sr-iov pool state stored? Is it just in an in-memory data structure that goes away when libvirt restarts? libvirt doesn't inventory existing vm's and figure out what's in use at startup if that's the case? Is there any way to recover from this situation short of the nuclear "shut down and restart everything" option? Thanks much...

1 0

Set up networking so the VM Guest uses LAN DHCP?
by Jeffrey Walton 27 Feb '24

27 Feb '24

Hi Everyone, I'm having trouble understanding what I need to do so my VM guests use the DHCP server on my LAN. I've read <https://wiki.libvirt.org/VirtualNetworking.html#virtual-networking>, but I don't see the use case covered. There is a section on dns-dhcp, but it looks like some sort of libvirt-internal setup so guests get their networking params from libvirt, and not my DHCP server. (I am looking for something similar to VirtualBox and Bridged networking. VBox does what I want when I select a bridged adapter). How do I set up networking so the VM Guest uses LAN DHCP? Thanks in advance.

3 5

Windows VM shutting down reason=crashed
by Jürgen Echter 27 Feb '24

27 Feb '24

Hello, i have a few Windows Server VM's running and one of them is randomly shutting itself down. In the log file i have the following line: 2024-02-26 06:53:13.286+0000: shutting down, reason=crashed What would be a good approach to figure out what the reason is? libvirt version 9.8.0 Gentoo linux with kernel 6.6.13 Thanks for some hints Juergen

2 1

add nvdimm and set it as a dax device
by Pierre Clouzet 14 Feb '24

14 Feb '24

Hello, I'm trying to add an nvdimm device on my vm and configure it to a dax mode. Directly with qemu, I was: sudo ndctl disable-namespace namespace0.0 sudo ndctl create-namespace -m devdax sudo daxctl reconfigure-device -m system-ram all --force However, when running with virt-manager, I get the following error message when I run sudo ndctl create-namespace -m devdax Error: create namespace: region0 align setting is 0x1000000 size 0x1dde0000 is misaligned. Here are the xml infos for the nvdimm device: <memory model="nvdimm" access="shared"> <source> <path>/mnt/scratch/pclouzet/libvirt/dax0.0</path> <alignsize unit="KiB">2048</alignsize> <pmem/> </source> <target> <size unit="KiB">488282</size> <node>0</node> <label> <size unit="KiB">128</size> </label> </target> <address type="dimm" slot="0"/> </memory> </devices> Is there an additional command I missed to set it as a dax device? Thanks, Pierre

1 1