- Users - libvirt List Archives

Statically Defining a Network
by richard.schmitt＠starlab.io 13 Jun '24

13 Jun '24

We install libvirt in a AlmaLinux 9 Kickstart file. The version of libvirt we are using is 10.0.0. What we'd like to do is to statically define a network in the kickstart file without having to run `virsh net-define`. In other words, we had hoped that by installing a <network>.xml file in /usr/share/libvirt/networks, that that network would be recognized by libvirt the first time the system is booted. This does not appear to be the case. Is there a way to statically define libvirt networks without needing to run 'virsh net-define'? Thanks

3 3

Re: Guests: Unable to get IP address
by Michal Prívozník 11 Jun '24

11 Jun '24

Please keep the list on CC so that others can benefit from the conversation too. On 5/21/24 14:34, Arun Mani J wrote: > Thanks for the reply! > > I recreated the NAT network using the guide from https://wiki.libvirt.org/Networking.html. > > The configuration now reads as: > <network> > <name>default</name> > <uuid>33376669-6292-44c9-b26b-9d2ee69826a7</uuid> > <forward mode="nat"> > <nat> > <port start="1024" end="65535"/> > </nat> > </forward> > <bridge name="virbr0" stp="on" delay="0"/> > <mac address="52:54:00:78:76:0f"/> > <ip address="192.168.122.1" netmask="255.255.255.0"> > <dhcp> > <range start="192.168.122.2" end="192.168.122.254"/> > </dhcp> > </ip> > </network> > > There is a dhcp section but still I'm not able to get my guest working. > > I have dnsmasq-base package installed in my host. However I'm not sure how to see if that is running. Like I opened Systems Monitor and can't find any process with that name. > > So may be that's the problem? Well, have you restarted the network? You need the 'virsh net-destroy default && virsh net-start default' combo to restart it. After that you should see dnsmasq process, e.g. via 'ps axf | grep dnsmasq'. Michal

2 8

TLS live migrations and cgroups (cpuset)
by Giovanni Tirloni 30 May '24

30 May '24

Hello, I have got a situation where a live migration (TLS) is failing because the virtual machine has the following configuration: <vcpu placement='static' cpuset='0-1,3,5,7-47,49,51,53,55-95'>2</vcpu> This is in place because the host is shared with some other application with strict requirements and so the `machine.slice` was limited to these CPUs. During a live migration, libvirtd (running as root) seems to spawn a new rpc-libvirtd process with the same uid/gid defined in qemu.conf (non-root). It then fails to write to `cpuset.cpus`: virNetClientProgramDispatchError:172 : Unable to write to '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d80\x2dinstance\x2d0000007c.scope/libvirt/emulator/cpuset.cpus': Permission denied Here are the running processes during the live migration: UID PID GID PPID CMD root 2639801 0 2639361 /usr/sbin/libvirtd --listen 12345 3033179 109 2638878 /usr/sbin/libvirtd --listen 12345 3033179 109 2638878 [rpc-libvirtd] (processes with uid=12345 are spawned during the migration. gid=109 is `kvm`) Changing group membership and file permissions has no effect because the file/directory permissions in machine.slice are not inherited inside the machine scope (a workaround was setting regular files as 664 and making the non-root UID of rpc-libvirtd be a member of the root group). I did not try to change the user/group in qemu.conf for fear of breaking other parts of the system or sabotaging security (e.g. by setting it to root/root for qemu-system-x86_64). The systemd scope for the machine has "Delegate=yes" so systemd expects the manager process to fully own the cgroups configuration. So looking at libvirtd, I cannot find a configuration setting that would either change the UID/GID of that rpc-libvirtd process or how to tweak the cgroups file permissions. Any insights? Environment: * Ubuntu 22.04 x86_64 * systemd 249.11 * libvirt 8.0.0

1 0

How can I create a VM in virt-manager that can access the internet through the host but can't even see the host or other devices on the LAN?
by imthenachoman＠gmail.com 22 May '24

22 May '24

I want to be able to create multiple VMs for testing purposes -- questionable websites, files that could be infected with a virus, etc. I want to make sure these VMs should be able to access the internet (through the host) but they should not be able to access the host or any other device on the LAN. I tried `NAT` mode but that let's me access the host and other devices on the LAN. I tried `Isolated` mode but that doesn't give me internet access. I don't want to run a second VM and route traffic through it. I want to avoid creating FW rules on the host or VMs because I'll be creating and destroying VMs all the time. And I think I need to use some kind of NAT because I don't want my router to know/see the VMs as clients. What do I need to do?

7 10

How to live migrate with copy-storage-inc and zfs snapshot replication?
by Dan Vatca 21 May '24

21 May '24

Hello everybody, I am trying to understand how I can create a setup with two libvirt + kvm + zfs systems, in which I can: 1. Create a domain on sys-a, with storage space on a zfs filesystem. This filesystem stores the qcow2 image, any qcow2 disk snapshots, memory dumps and domain XML definition. 2. Take consistent ZFS snapshots every 15 minutes, and sync them to sys-b. The snapshots will be replicated using syncoid. These will be using scripts to quiesce the VM. 3. Given that most of the disk is synced (at most 15 minutes ago), what are the things I need to ensure (libvirt snapshots, memory snapshot, etc), so that I can do a live migration while copying just the incremental changes since the last snapshot I already transferred? I was trying to use virsh migrate —copy-storage-inc, however I could not find any working example that would copy just the incremental changes. I could find the theory of the snapshot backing chain, however —copy-storage-inc complains that it cannot replicate while there are snapshots present. (error: Requested operation is not valid: cannot migrate domain with 1 snapshots). I this scenario possible? Dan Vatca

2 1

Guests: Unable to get IP address
by Arun Mani J 21 May '24

21 May '24

Hey all. I'm exploring virt-manager, libvirt etc. I downloaded Debian 12 KDE ISO and when launched using virt-manager, it does not seem to connect to network. nmcli says it has failed to get an IP address. However, if I run the same ISO inside GNOME Boxes, then it is able to connect to network. I checked the XML configuration of both and found this: ```xml <interface type="user"> <mac address="52:54:00:ea:f9:69"/> <model type="virtio"/> <address type="pci" domain="0x0000" bus="0x01" slot="0x00" function="0x0"/> </interface> ``` However, the one created via virt-manager has this: ```xml <interface type="network"> <mac address="52:54:00:5d:c7:9e"/> <source network="default"/> <model type="virtio"/> <address type="pci" domain="0x0000" bus="0x01" slot="0x00" function="0x0"/> </interface> ``` What I'm not able to understand is, why does the user mode networking work fine but not network mode? My ultimate goal is to set up routed configuration, so one can directly SSH into the guests. Packages: - virt-manager - 4.1.0 (Debian) - libvirt0 - 10.3.0-3 (Debian) - GNOME Boxes - 46.0 (Flatpak) Please let me know if you need any other information. It is a pretty much clean installation with no extra configuration, so I don't think I messed up anything. Thank you! Arun Mani J

2 1

stats : active memory / disk latency
by Peter Quiring 21 May '24

21 May '24

Two stats that VMWare offers that I can't seem to find in libvirt is "active" memory and disk latency. Any idea how I could extract these stats? libvirt offers memory stats such as maximum, current, unused, usable, etc. but these are all 100% or 0% of what the VM has available. Looking for an "active" value. /proc/memstat has similar stats within a Linux VM but need to get from libvirt. Disk latency is also sometimes critical information. Thanks, -- Peter Quiring

2 1

qemu guest agent with openbsd
by Christian Haumesser 13 May '24

13 May '24

Hello, I have an OpenBSD 7.5 guest running on Debian bookworm with libvirt (9.0) and qemu (7.2). I'd like to use the qemu guest agent in this guest, but I can't seem to figure out how to craft the libvirt xml to expose the serial port in a way that OpenBSD and libvirt can use it together (or if it’s even possible). Per the libvirt docs <https://wiki.libvirt.org/Qemu_guest_agent.html>, I’m currently using: > <channel type='unix'> > <source mode='bind'/> > <target type='virtio' name='org.qemu.guest_agent.0'/> > </channel> But OpenBSD doesn’t directly support the virtio console driver. Consistent with this 2020 thread <https://undeadly.org/cgi?action=article;sid=20200514073852> from the OpenBSD ports making list, I see the following message in my guest’s dmesg output: > virtio5 at pci0 dev 10 function 0 "Qumranet Virtio Console" rev 0x00 > virtio5: no matching child driver; not configured Thanks to this libvir-list thread <https://listman.redhat.com/archives/libvir-list/2015-October/120250.html> from 2015, I’ve realized that I can manually expose an ISA serial console that will allow me to connect to the guest agent using native qemu tooling; but there doesn’t seem to be a way to create a channel that libvirt can communicate over without virtio console support in the guest. The thread discusses a couple of approaches to resolving this issue with code changes, but it seems like the discussion stalled out before anything happened. Am I missing something or is it still impossible? Thanks!

2 2

jfKVM - new app using libvirt
by Peter Quiring 06 May '24

06 May '24

jfKVM is a new web-based VM management system based on libvirt. jfkvm.sf.net https://pquiring.github.io/javaforce/projects/jfkvm/docs/help.html Thanks, -- Peter Quiring

1 1

What's wrong with internal VM snapshots?
by Marius Vollmer 06 May '24

06 May '24

Hi! I am trying to improve the support for VM snapshots in the Cockpit web console, and I am afraid I have questions... We have been asked to prefer the "external" over the "internal" snapshot format, at least on RHEL. I haven't yet figured out why, and consequently I am struggling with deciding how hard the Cockpit UI should push people towards external snapshots. So, what's wrong with internal snapshots? I heard they are "unreliable", but how so in detail? Does the data structure inside the qcow2 files get corrupted easily? Do they behave poorly when the snapshot process runs out of disk in the middle? That sort of thing would help me a lot to figure out what Cockpit should be doing on platforms other than RHEL. And how well (or how soon) can external snapshots be expected to work? I have severely messed up my libvirt state a couple of times while playing around with them, and my confidence in them right now isn't great. :-) Are you surprised by this? Or are external snapshots not yet considered ready? (Most recent example from my experiments: Deleting a full system snapshot of a paused machine fails with "internal error: unable to execute QEMU command 'block-commit': Block node is read-only". Reverting to it works and after that it can also be deleted, all while the VM is paused.) Thanks!

2 5

Error Cannot acquire state change lock from remoteDispatchDomainMigratePrepare3Params during live migration of domains
by Christian Rohmann 25 Apr '24

25 Apr '24

Hallo libvirt-users! we observe lock-ups / timeouts with in prometheus-libvirt-exporter (https://github.com/inovex/prometheus-libvirt-exporter) when libvirt is live-migrating domains: > Timed out during operation: cannot acquire state change lock (held by > monitor=remoteDispatchDomainMigratePrepare3Params) All of the source code can be found at: https://github.com/inovex/prometheus-libvirt-exporter/blob/master/pkg/expor…. Basically the error happens when DomainMemoryStats or other operational domain info is queried via the libvirt socket. 1) We are actually using the read-only socket at '/var/run/libvirt/libvirt-sock-ro', so there should not be any locking required. Is there any way to not run into lock contention, like running a request with some "nolock" indication? 2) This being reported as timeout waiting for the lock, what is the timeout and would waiting a bit longer help? Or is the lock active during the whole time a domain live migration is running? 3) Is this in any way related to the type of migration? Tunneled vs. native (https://libvirt.org/migration.html)? 4) Is there any indication that we could use to skip those domains (or certain queries)? The same issue was actually previously reported for another implementation of a Prometheus exporter (https://github.com/kumina/libvirt_exporter/issues/33) Currently the exporter locks up or throws the mentioned timeout errors during the the migration of 200 domains, 5 at a time. It would be awesome to find a way to make this work as smooth as possible, even during live migrations! I am thankful for any insights into how the libvirt socket, the various calls, the locking mechanisms or live migration modes work! Regards Christian

5 10

How to configure the default connection for virsh
by Nir Soffer 18 Apr '24

18 Apr '24

We are using minikube vms, which require adding the user to to libvirt group[1]. We use `virsh -c qemu:///system` for debugging these vms or related libvirt networks. Using virsh without specifying the '-c' argument is a common mistake that leads to trouble, for example creating the libvirt default network incorrectly. I wonder if it is possible to configure virsh so it uses -c qemu:///system by default? We know that this is not great, but we don't know a better way to use minikube with the kvm2 driver. The minikube vms are started for creating a development environment and during CI, and they cannot require a password or any complicated setup. [1] https://github.com/kubernetes/minikube/issues/3467 Nir

2 2

Linking virtual sound cards in two virtual machines?
by Lars Kellogg-Stedman 15 Apr '24

15 Apr '24

I would like to create a virtual audio cable between two libvirt guests such that sound output from guest A goes to sound input on guest B, and vice versa. I figure the solution to this is using something like the pulseaudio's "virtual sink", but I'm not entirely sure what the syntax would look like (fwiw, I'm on Fedora 39, which means libvirt 9.7.0, which means this has to be through libvirt's pulseaudio driver rather than the pipewire driver, since the latter isn't available in 9.7.0). I started by creating a virtual sink (and yes, I realize that's only half the process): pactl load-module module-null-sink sink_name=vaudio0 And adding the following snippet to the domain xml for guest "node1.virt": <audio id='1' type='pulseaudio'> <input name='vaudio0' streamName='node1-in' latency='100'/> <output name='vaudio0' streamName='node1-out' latency='100'/> </audio> But trying to start this guest results in: error: Failed to start domain 'node1.virt' error: internal error: process exited while connecting to monitor: audio: Could not init `pa' audio driver There doesn't appear to be any additional information in /var/log/libvirt/qemu/node1.virt.log. I'm wondering if someone has an example of what this might look like? Thanks, -- Lars Kellogg-Stedman <lars(a)redhat.com>

1 0

'nconnect' nfs mount option support
by Abhisar Sinha 15 Apr '24

15 Apr '24

Hi, Is there a way to specify the 'nconnect' nfs mount opt value in the XML file used by 'virsh create-pool' command? If not, how hard would it be to implement this functionality? Please advise. Thanks, Abhisar

2 1

TLS Configuration with libvirt and Perfect Forward Secrecy
by Tommy.J.Mairo 09 Apr '24

09 Apr '24

Dear libvirt Users, I'm seeking guidance on configuring TLS with libvirt to achieve perfect forward secrecy (PFS). While I followed the tutorial on setting up TLS with libvirt from the project wiki [1], I encountered an error when bringing up the legacy monolithic libvirt daemon. The error message, "Certificate /etc/pki/libvirt/servercert.pem usage does not permit key encipherment," appeared even though I declared the "encryption_key" field in the certtool template file when generating the server certificate. Moreover, when I inspected the generated certificate with certtool, I discovered that it only had a "Digital signature" entry under X509v3 key usage. The deficiency in "Key Encipherment" usage could result from the key type specifications when generating the private key with the command "certtool --generate-privkey --key-type=ed25519 --curve=Ed25519 --sec-param=ultra > /etc/pki/libvirt/private/serverkey.pem". I prefer using elliptic-curve cryptography, specifically curve25519, over RSA, which the tutorial employs. Although a digital signature-only certificate can still perform Diffie–Hellman exchanges with ECDSA, it cannot encrypt client-generated pre-master secrets. Nonetheless, a TLS session can still be established without pre-master secrets, avoiding session key leakage even if the decryption key is compromised in the future. This fulfills the PFS requirement, and I believe that a certificate without "Key Encipherment" usage contributes to the system's overall security. Based on my understanding, I am seeking any insights or suggestions on whether there is a way to suppress the "certificate usage does not permit key encipherment" error and allow the legacy monolithic libvirt daemon to proceed. Thanks, Tommy.J.Mairo [1] https://wiki.libvirt.org/TLSSetup.html

1 0

all domains paused, maybe logging might be helpfull
by Lennart Fricke 21 Mar '24

21 Mar '24

Hello, I just hit the situation that all domains on a host where paused due to missing space. It took me some time to figure out that there was no space left for the images on the host. I learned that 'virsh domstate --reason $GUEST' and 'domblkerror $GUEST' could have helped me. But the logs are silent about the problem. Would it be possible to show these problems in logs or is there other documentation than the reference to find out how to troubleshoot those issues? Thank you Lennart

2 3

Re: PCI Hotplug to a VM does not work in Alma
by Michal Prívozník 20 Mar '24

20 Mar '24

On 3/19/24 22:51, Chanda Mendon (cmendon) via Users wrote: > Hi > > We have a chassis with a peripheral PCI device installed. We have a > hypervisor running on the chassis where we have deployed a VM which can > use the PCI device once it is attached. > > When the PCI device is powered on or off we need to do a hotplug in/out > using virsh commands. Even though the virsh commands for hotplug is > executed successfully, the VM sees the PCI inside the VM for one or 2 > secs. What do you think is the issue? > [root ~]# uname -a > > Linux 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Mar 15 05:32:38 UTC 2024 > x86_64 x86_64 x86_64 GNU/Linux > > [root ~]# virsh --version > > 8.0.0 > > [root ~]# rpm -qa | grep libvirt > > libvirt-daemon-driver-nwfilter-8.0.0-5.el8.x86_64 > > python3-libvirt-7.8.0-1.el8.x86_64 > > libvirt-daemon-driver-storage-logical-8.0.0-5.el8.x86_64 > > libvirt-libs-8.0.0-5.el8.x86_64 > > libvirt-daemon-config-nwfilter-8.0.0-5.el8.x86_64 > > libvirt-daemon-driver-storage-gluster-8.0.0-5.el8.x86_64 > > libvirt-8.0.0-5.el8.x86_64 > > libvirt-daemon-8.0.0-5.el8.x86_64 > > libvirt-daemon-driver-nodedev-8.0.0-5.el8.x86_64 > > libvirt-daemon-config-network-8.0.0-5.el8.x86_64 > > libvirt-daemon-driver-storage-iscsi-8.0.0-5.el8.x86_64 > > libvirt-daemon-driver-storage-rbd-8.0.0-5.el8.x86_64 > > libvirt-daemon-driver-network-8.0.0-5.el8.x86_64 > > libvirt-daemon-driver-secret-8.0.0-5.el8.x86_64 > > python2-libvirt-python-5.10.0-1.el8.x86_64 > > libvirt-daemon-driver-qemu-8.0.0-5.el8.x86_64 > > libvirt-daemon-driver-storage-core-8.0.0-5.el8.x86_64 > > libvirt-daemon-driver-storage-iscsi-direct-8.0.0-5.el8.x86_64 > > libvirt-daemon-driver-storage-scsi-8.0.0-5.el8.x86_64 > > libvirt-client-8.0.0-5.el8.x86_64 > > libvirt-daemon-driver-storage-disk-8.0.0-5.el8.x86_64 > > libvirt-daemon-driver-storage-8.0.0-5.el8.x86_64 > > libvirt-daemon-driver-interface-8.0.0-5.el8.x86_64 > > libvirt-daemon-driver-storage-mpath-8.0.0-5.el8.x86_64 > > libvirt-daemon-kvm-8.0.0-5.el8.x86_64 > > [root ~]# rpm -qa | grep qemu > > qemu-kvm-ui-opengl-6.2.0-11.el8.x86_64 > > qemu-kvm-6.2.0-11.el8.x86_64 > > qemu-img-6.2.0-11.el8.x86_64 > > qemu-kvm-block-iscsi-6.2.0-11.el8.x86_64 > > ipxe-roms-qemu-20200823-7.git4bd064de.el8.noarch > > qemu-kvm-block-gluster-6.2.0-11.el8.x86_64 > > qemu-kvm-block-rbd-6.2.0-11.el8.x86_64 > > qemu-kvm-block-curl-6.2.0-11.el8.x86_64 > > qemu-kvm-core-6.2.0-11.el8.x86_64 > > qemu-kvm-hw-usbredir-6.2.0-11.el8.x86_64 > > libvirt-daemon-driver-qemu-8.0.0-5.el8.x86_64 > > qemu-kvm-ui-spice-6.2.0-11.el8.x86_64 > > qemu-kvm-docs-6.2.0-11.el8.x86_64 > > qemu-kvm-block-ssh-6.2.0-11.el8.x86_64 > > qemu-kvm-common-6.2.0-11.el8.x86_64 > > > > [root ~]# *virsh nodedev-dettach** **pci_0000_04_00_0* > > [12013.987821] pci_probe_reset_slot: call pci_slot_reset with probe=1 > > [12014.063669] pci_slot_reset (printk info): reset hotplug slot. > > [12014.134164] pci_reset_hotplug_slot(printk INFO): calling reset_slot > (probe = 1) > > [Mar19 21:09] pci_probe_reset_slot: call pci_slot_reset with probe=1 > > [ +0.075848] pci_slot_reset (printk info): reset hotplug slot. > > [ +0.070495] pci_reset_hotplug_slot(printk INFO): calling reset_slot > (probe = 1) > > *Device pci_0000_04_00_0 detached*** > > > > [root ~]# *virsh attach-device ROUTER8 /opt/us/bin/mrvl.xml* > > [12024.217540] pci_probe_reset_slot: call pci_slot_reset with probe=1 > > [ +10.083376] pci_probe_reset_slot: call pci_slot_reset with > probe=1[12024.293548] pci_slot_reset (printk info): reset hotplug slot. The fact that the device resets so often might suggest a problem with the device itself. But since QEMU is seeing the device (even if only for a brief period), I think libvirt's out of the picture. Perhaps QEMU folks might have a better answer. Michal

1 0

PCI Hotplug to a VM does not work in Alma
by Chanda Mendon (cmendon) 19 Mar '24

19 Mar '24

Hi We have a chassis with a peripheral PCI device installed. We have a hypervisor running on the chassis where we have deployed a VM which can use the PCI device once it is attached. When the PCI device is powered on or off we need to do a hotplug in/out using virsh commands. Even though the virsh commands for hotplug is executed successfully, the VM sees the PCI inside the VM for one or 2 secs. What do you think is the issue? [root ~]# uname -a Linux 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Mar 15 05:32:38 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux [root ~]# virsh --version 8.0.0 [root ~]# rpm -qa | grep libvirt libvirt-daemon-driver-nwfilter-8.0.0-5.el8.x86_64 python3-libvirt-7.8.0-1.el8.x86_64 libvirt-daemon-driver-storage-logical-8.0.0-5.el8.x86_64 libvirt-libs-8.0.0-5.el8.x86_64 libvirt-daemon-config-nwfilter-8.0.0-5.el8.x86_64 libvirt-daemon-driver-storage-gluster-8.0.0-5.el8.x86_64 libvirt-8.0.0-5.el8.x86_64 libvirt-daemon-8.0.0-5.el8.x86_64 libvirt-daemon-driver-nodedev-8.0.0-5.el8.x86_64 libvirt-daemon-config-network-8.0.0-5.el8.x86_64 libvirt-daemon-driver-storage-iscsi-8.0.0-5.el8.x86_64 libvirt-daemon-driver-storage-rbd-8.0.0-5.el8.x86_64 libvirt-daemon-driver-network-8.0.0-5.el8.x86_64 libvirt-daemon-driver-secret-8.0.0-5.el8.x86_64 python2-libvirt-python-5.10.0-1.el8.x86_64 libvirt-daemon-driver-qemu-8.0.0-5.el8.x86_64 libvirt-daemon-driver-storage-core-8.0.0-5.el8.x86_64 libvirt-daemon-driver-storage-iscsi-direct-8.0.0-5.el8.x86_64 libvirt-daemon-driver-storage-scsi-8.0.0-5.el8.x86_64 libvirt-client-8.0.0-5.el8.x86_64 libvirt-daemon-driver-storage-disk-8.0.0-5.el8.x86_64 libvirt-daemon-driver-storage-8.0.0-5.el8.x86_64 libvirt-daemon-driver-interface-8.0.0-5.el8.x86_64 libvirt-daemon-driver-storage-mpath-8.0.0-5.el8.x86_64 libvirt-daemon-kvm-8.0.0-5.el8.x86_64 [root ~]# rpm -qa | grep qemu qemu-kvm-ui-opengl-6.2.0-11.el8.x86_64 qemu-kvm-6.2.0-11.el8.x86_64 qemu-img-6.2.0-11.el8.x86_64 qemu-kvm-block-iscsi-6.2.0-11.el8.x86_64 ipxe-roms-qemu-20200823-7.git4bd064de.el8.noarch qemu-kvm-block-gluster-6.2.0-11.el8.x86_64 qemu-kvm-block-rbd-6.2.0-11.el8.x86_64 qemu-kvm-block-curl-6.2.0-11.el8.x86_64 qemu-kvm-core-6.2.0-11.el8.x86_64 qemu-kvm-hw-usbredir-6.2.0-11.el8.x86_64 libvirt-daemon-driver-qemu-8.0.0-5.el8.x86_64 qemu-kvm-ui-spice-6.2.0-11.el8.x86_64 qemu-kvm-docs-6.2.0-11.el8.x86_64 qemu-kvm-block-ssh-6.2.0-11.el8.x86_64 qemu-kvm-common-6.2.0-11.el8.x86_64 [root ~]# virsh nodedev-dettach pci_0000_04_00_0 [12013.987821] pci_probe_reset_slot: call pci_slot_reset with probe=1 [12014.063669] pci_slot_reset (printk info): reset hotplug slot. [12014.134164] pci_reset_hotplug_slot(printk INFO): calling reset_slot (probe = 1) [Mar19 21:09] pci_probe_reset_slot: call pci_slot_reset with probe=1 [ +0.075848] pci_slot_reset (printk info): reset hotplug slot. [ +0.070495] pci_reset_hotplug_slot(printk INFO): calling reset_slot (probe = 1) Device pci_0000_04_00_0 detached [root ~]# virsh attach-device ROUTER8 /opt/us/bin/mrvl.xml [12024.217540] pci_probe_reset_slot: call pci_slot_reset with probe=1 [ +10.083376] pci_probe_reset_slot: call pci_slot_reset with probe=1[12024.293548] pci_slot_reset (printk info): reset hotplug slot. [12024.434828] pci_reset_hotplug_slot(printk INFO): calling reset_slot (probe = 1) [ +0.076008] pci_slot_reset (printk info): reset hotplug slot. [ +0.141280] pci_reset_hotplug_slot(printk INFO): calling reset_slot (probe = 1)4.526807] pci_probe_reset_slot: call pci_slot_reset with probe=1 [12024.749367] pci_slot_reset (printk info): reset hotplug slot. [12024.821327] pci_reset_hotplug_slot(printk INFO): calling reset_slot (probe = 1) [ +0.091979] pci_probe_reset_slot: call pci_slot_reset with probe=1[12024.911065] pci_probe_reset_slot: call pci_slot_reset with probe=1 [ +0.222560] pci_slot_reset (printk info): reset hotplug slot. [ +0.071960] pci_reset_hotplug_slot(printk INFO): calling reset_slot (probe = 1) 5.057584] pci_slot_reset (printk info): reset hotplug slot. [ +0.089738] pci_probe_reset_slot: call pci_slot_reset with probe=1 [12025.278585] pci_reset_hotplug_slot(printk INFO): calling reset_slot (probe = 1) [ +0.146519] pci_slot_reset (printk info): reset hotplug slot. [ +0.221001] pci_reset_hotplug_slot(printk INFO): calling reset_slot (probe = 1) 5.441574] pci_probe_reset_slot: call pci_slot_reset with probe=1 [ +0.162989] pci_probe_reset_slot: call pci_slot_reset with probe=1[12025.665968] pci_slot_reset (printk info): reset hotplug slot. [12025.806751] pci_reset_hotplug_slot(printk INFO): calling reset_slot (probe = 1) [12025.898060] __pci_reset_slot (printk info): Reset slot (not hotplug), probe = 1. [ +0.224394] pci_slot_reset (printk info): reset hotplug slot. [ +0.140783] pci_reset_hotplug_slot(printk INFO): calling reset_slot (probe = 1)5.988477] pci_slot_reset (printk info): reset hotplug slot. [ +0.091309] __pci_reset_slot (printk info): Reset slot (not hotplug), probe = 1. 12026.205654] pci_reset_hotplug_slot(printk INFO): calling reset_slot (probe = 1) [12026.383424] __pci_reset_slot (printk info): pci_slot_trylock is non-zero, so reset hotplug slot. [ +0.090417] pci_slot_reset (printk info): reset hotplug slot. [ +0.217177] pci_reset_hotplug_slot(printk INFO): calling reset_slot (probe = 1)6.490406] pci_reset_hotplug_slot(printk INFO): calling reset_slot (probe = 0) [ +0.177770] __pci_reset_slot (printk info): pci_slot_trylock is non-zero, so reset hotplug slot. 2026.726579] pciehp_reset_slot: SLOTCTRL 58 write cmd 0 [ +0.106982] pci_reset_hotplug_slot(printk INFO): calling reset_slot (probe = 0) [ +0.236167] pcieport 0000:00:0e.0: pciehp: pciehp_reset_slot: SLOTCTRL 58 write cmd 0 [ +0.000006] pciehp_reset_slot: SLOTCTRL 58 write cmd 0 [12027.744480] pending interrupts 0x0108 from Slot Status [ +1.017890] pcieport 0000:00:0e.0: pciehp: pending interrupts 0x0108 from Slot Status [ +0.000011] pending interrupts 0x0108 from Slot Status [12027.936469] pciehp_reset_slot: SLOTCTRL 58 write cmd 1008 [ +0.191983] pcieport 0000:00:0e.0: pciehp: pciehp_reset_slot: SLOTCTRL 58 write cmd 100812028.027182] pcieport 0000:00:0e.0: pciehp: Slot(4): Link Down [12028.190357] Slot(4): Link Down [ +0.000006] pciehp_reset_slot: SLOTCTRL 58 write cmd 1008[12028.230605] pcieport 0000:00:0e.0: pciehp: Slot(4): Card not present [12028.369819] Slot(4): Card not present Device attached successfully [12028.415302] pciehp_unconfigure_device: domain:bus:dev = 0000:04:00 [ +0.090713] pcieport 0000:00:0e.0: pciehp: Slot(4): Link Down[12028.524356] vfio-pci 0000:04:00.0: Relaying device request to user (#0) [ +0.163175] Slot(4): Link Down [ +0.040248] pcieport 0000:00:0e.0: pciehp: Slot(4): Card not present [ +0.139214] Slot(4): Card not present [ +0.045479] pcieport 0000:00:0e.0: pciehp: pciehp_unconfigure_device: domain:bus:dev = 0000:04:00 [ +0.000004] pciehp_unconfigure_device: domain:bus:dev = 0000:04:00 [ +0.109054] vfio-pci 0000:04:00.0: Relaying device request to user (#0) 9119] pci_probe_reset_slot: call pci_slot_reset with probe=1 [root@nfvis ~]# [12029.151422] pci_slot_reset (printk info): reset hotplug slot. [12029.237806] pci_reset_hotplug_slot(printk INFO): calling reset_slot (probe = 1) [ +0.334763] pci_probe_reset_slot: call pci_slot_reset with probe=1[12029.327572] pci_probe_reset_slot: call pci_slot_reset with probe=1 [ +0.292303] pci_slot_reset (printk info): reset hotplug slot. [ +0.086384] pci_r12029.474447] pci_slot_reset (printk info): reset hotplug slot. eset_hotplug_slot(printk INFO): calling reset_slot (probe = 1) [12029.632533] pci_reset_hotplug_slot(printk INFO): calling reset_slot (probe = 1) [ +0.089766] pci_probe_reset_slot: call pci_slot_reset with probe=1[12029.788423] __pci_reset_slot (printk info): Reset slot (not hotplug), probe = 1. [ +0.146875] pci_slot_reset (printk info): reset hotplug slot. [12029.950782] pci_slot_reset (printk info): reset hotplug slot. [ +0.158086] pci_reset_hotplug_slot(printk INFO): calling reset_slot (probe = 1) [ +0.155890] __pci_reset_slot (printk info): Reset slot (not hotplug), probe = 1. .089787] pci_reset_hotplug_slot(printk INFO): calling reset_slot (probe = 1) [12030.346708] __pci_reset_slot (printk info): pci_slot_trylock = 0, so don't reset hotplug slot. [ +0.162359] pci_slot_reset (printk info): reset hotplug slot.[12030.451555] vfio-pci 0000:04:00.0: can't change power state from D0 to D3hot (config space inaccessible) [ +0.139005] pci_reset_hotplug_slot(printk INFO): calling reset_slot (probe = 1)12030.632974] pci 0000:04:00.0: Removing from iommu group 58 [ +0.256921] __pci_reset_slot (printk info): pci_slot_trylock = 0, so don't reset hotplug slot. [ +0.104847] vfio-pci 0000:04:00.0: can't change power state from D0 to D3hot (config space inaccessible) 5279] pciehp_check_link_active: lnk_status = 7011 [ +0.181419] pci 0000:04:00.0: Removing from iommu group 58 [12031.056747] pcieport 0000:00:0e.0: pciehp: Slot(4): Card present [12031.194871] Slot(4): Card present [ +0.152301] pcieport 0000:00:0e.0: pciehp: pciehp_check_link_active: lnk_status = 701112031.236173] pcieport 0000:00:0e.0: pciehp: Slot(4): Link Up [12031.395181] Slot(4): Link Up [ +0.000004] pciehp_check_link_active: lnk_status = 7011 [ +0.271468] pcieport 0000:00:0e.0: pciehp: Slot(4): Card present [ +0.138124] Slot(4): Card present [ +0.041302] pcieport 0000:00:0e.0: pciehp: Slot(4): Link Up [ +0.159008] Slot(4): Link Up 31.560448] read_dev_vendor_id: read config_dword worked. bus = 4, dev_vendor id = 0xE61E11AB [12031.803909] read_dev_vendor_id: device found, return true. bus = 4, dev_vendor id = 0xE61E11AB [12031.909784] pciehp_check_link_status: lnk_status = 7011 [ +0.165267] read_dev_vendor_id: read config_dword worked. bus = 4, dev_vendor id = 0xE61E11AB2031.974018] read_dev_vendor_id: read config_dword worked. bus = 4, dev_vendor id = 0xE61E11AB [12032.176762] read_dev_vendor_id: device found, return true. bus = 4, dev_vendor id = 0xE61E11AB [ +0.243461] read_dev_vendor_id: device found, return true. bus = 4, dev_vendor id = 0xE61E11AB [ +0.105871] pcieport 0000:00:0e.0: pciehp: pciehp_check_link_status: lnk_status = 7011282681] pci 0000:04:00.0: [11ab:e61e] type 00 class 0x020000 [ +0.000004] pciehp_check_link_status: lnk_status = 7011 [12032.545840] pci 0000:04:00.0: reg 0x10: [mem 0x7b44000000-0x7b440fffff 64bit pref] [12032.701716] pci 0000:04:00.0: reg 0x18: [mem 0x7b40000000-0x7b43ffffff 64bit pref] [ +0.064234] read_dev_vendor_id: read config_dword worked. bus = 4, dev_vendor id = 0xE61E11AB2032.794062] pci 0000:04:00.0: reg 0x20: [mem 0x7b30000000-0x7b3fffffff 64bit pref] [12032.984416] pci 0000:04:00.0: supports D1 D2 [ +0.202744] read_dev_vendor_id: device found, return true. bus = 4, dev_vendor id = 0xE61E11AB2033.039350] pci 0000:04:00.0: Adding to iommu group 58 [12033.201376] pci 0000:04:00.0: BAR 4: assigned [mem 0x7b30000000-0x7b3fffffff 64bit pref] [12033.299989] pci 0000:04:00.0: BAR 2: assigned [mem 0x7b40000000-0x7b43ffffff 64bit pref] [ +0.105919] pci 0000:04:00.0: [11ab:e61e] type 00 class 0x020000[12033.400664] pci 0000:04:00.0: BAR 0: assigned [mem 0x7b44000000-0x7b440fffff 64bit pref] [ +0.263159] pci 0000:04:00.0: reg 0x10: [mem 0x7b44000000-0x7b440fffff 64bit pref]12033.568019] pcieport 0000:00:0e.0: PCI bridge to [bus 04-06] [12033.725967] pcieport 0000:00:0e.0: bridge window [io 0xd000-0xdfff] [ +0.155876] pci 0000:04:00.0: reg 0x18: [mem 0x7b40000000-0x7b43ffffff 64bit pref]12033.805813] pcieport 0000:00:0e.0: bridge window [mem 0xdc000000-0xdcffffff] [12033.982533] pcieport 0000:00:0e.0: bridge window [mem 0x7b30000000-0x7b6fffffff 64bit pref] [ +0.092346] pci 0000:04:00.0: reg 0x20: [mem 0x7b30000000-0x7b3fffffff 64bit pref]12034.086495] pci-stub 0000:04:00.0: claimed by stub [ +0.190354] pci 0000:04:00.0: supports D1 D2 [ +0.054934] pci 0000:04:00.0: Adding to iommu group 58 [ +0.162026] pci 0000:04:00.0: BAR 4: assigned [mem 0x7b30000000-0x7b3fffffff 64bit pref] [ +0.098613] pci 0000:04:00.0: BAR 2: assigned [mem 0x7b40000000-0x7b43ffffff 64bit pref] [ +0.100675] pci 0000:04:00.0: BAR 0: assigned [mem 0x7b44000000-0x7b440fffff 64bit pref] [ +0.167355] pcieport 0000:00:0e.0: PCI bridge to [bus 04-06] [ +0.157948] pcieport 0000:00:0e.0: bridge window [io 0xd000-0xdfff] [ +0.079846] pcieport 0000:00:0e.0: bridge window [mem 0xdc000000-0xdcffffff] [ +0.176720] pcieport 0000:00:0e.0: bridge window [mem 0x7b30000000-0x7b6fffffff 64bit pref] [ +0.103962] pci-stub 0000:04:00.0: claimed by stub Thank you for replying in advance, -Chanda

1 0

Automate VM migration
by aheath1992＠gmail.com 19 Mar '24

19 Mar '24

Are their any tools that can automatically migrate VMs from one host to another, like in VMware, proxmox, and oVirt?

4 5

Info regarding AMX support and libvirt implications
by Gianluca Cecchi 08 Mar '24

08 Mar '24

Hello, I'm trying to use AMX in my virtual machines. More info on AMX: https://www.intel.com/content/www/us/en/products/docs/accelerator-engines/a… My system in test is currently SLES 15 SP5. I'm also verifying in parallel with Suse (especially regarding the backported features in their 5.14 based kernel), but in the meantime I would like to understand implication, if any, of libvirt in the certification loop I have to analyse. From what I see we have in upstream: . support in the KVM kernel module since 5.17 . support of cpu model SapphireRapids, the first offering AMX as an ISA extension, in QEMU since 7.0 Is there any dependance to check on libvirt too? When I run virsh cpu-models x86_64 Is libvirt sw stack querying qemu directly? Or the kvm kernel module? Or any internal "database" file? From man page it is not clear to me what "known" means: " cpu-models Syntax: cpu-models arch Print the list of CPU models known by libvirt for the specified architecture. Whether a specific hypervisor is able to create a domain which uses any of the printed CPU models is a separate question which can be answered by looking at the domain capabilities XML returned by domcapabilities command. Moreover, for some architectures libvirt does not know any CPU models and the usable CPU models are only limited by the hypervisor. This command will print that all CPU models are accepted for these architectures and the actual list of supported CPU models can be checked in the domain capabilities XML. " In SLES 15 SP5 with: qemu-7.1.0-150500.49.9.2.x86_64 kernel-default-5.14.21-150500.55.49.1.x86_64 libvirtd-*-9.0.0-150500.6.11.1.x86_64 I get # virsh cpu-models x86_64 ... Cascadelake-Server Cascadelake-Server-noTSX Icelake-Client Icelake-Client-noTSX Icelake-Server Icelake-Server-noTSX Cooperlake Snowridge athlon phenom Opteron_G1 Opteron_G2 ... # virsh domcapabilities | grep -i sapphirerapid # In fedora39 with qemu-8.1.3-4.fc39.x86_64 kernel-6.7.5-200.fc39.x86_64 libvirt-*-9.7.0-2.fc39.x86_64 I get # virsh cpu-models x86_64 ... Cascadelake-Server Cascadelake-Server-noTSX Icelake-Client Icelake-Client-noTSX Icelake-Server Icelake-Server-noTSX Cooperlake Snowridge SapphireRapids athlon phenom Opteron_G1 Opteron_G2 ... # virsh domcapabilities | grep -i sapphirerapids <model usable='no' vendor='Intel'>SapphireRapids</model> # because I'm running on a client system without AMX support Thanks in advance, Gianluca

2 3

Raspberry Pi 5 suggestion
by K otgc 07 Mar '24

07 Mar '24

Hello, the libvirt documentation <https://libvirt.org/apps.html> has nothing on installing libvirt to Raspberry Pi 5, so the documentation says to email <https://libvirt.org/contact.html> any suggestions. Cheers, Ken.

3 3

virsh undefine --nvram
by Chuck Lever 06 Mar '24

06 Mar '24

Hello - The virsh(1) man page says: > --nvram and --keep-nvram specify accordingly to delete or keep nvram (/domain/os/nvram/) file. However on my Fedora 39 system with libvirt 9.7.0, using "virsh undefine --nvram" option appears to leave the NVRAM backing file: cel@boudin:~$ ls ~/.config/libvirt/qemu/nvram guestfs-02a4oblo9vg7888d_VARS.qcow2 guestfs-7hmiolqcqwz5b0qs_VARS.qcow2 guestfs-el7lqymnwgzpnr38_VARS.qcow2 guestfs-jp0tfneh205xjrh8_VARS.qcow2 guestfs-mhqbp4gwmqsuu9w3_VARS.qcow2 guestfs-sat7x2kgpg9opjds_VARS.qcow2 guestfs-vhhi476rsq7vh83z_VARS.qcow2 guestfs-177mrq9xrb5589wl_VARS.qcow2 guestfs-88p8heyq1j0sy8bk_VARS.qcow2 guestfs-gxul3j9e1h3po19o_VARS.qcow2 guestfs-k0jf8msw4kh3yuff_VARS.qcow2 guestfs-oakcwpbozezvids8_VARS.qcow2 guestfs-sinmkv97r4k4dzcd_VARS.qcow2 guestfs-vj5cu9hgwcv0f18u_VARS.qcow2 guestfs-6jnim67ceme3h369_VARS.qcow2 guestfs-89g707uzdttxb9hv_VARS.qcow2 guestfs-h6vzt9zqxhoxx0hj_VARS.qcow2 guestfs-k4hw4gkcq8u0eebj_VARS.qcow2 guestfs-pljksgeblxgyf0pw_VARS.qcow2 guestfs-t8bnntykpzm3qvh2_VARS.qcow2 guestfs-ynbxqtj01sn4ovwy_VARS.qcow2 guestfs-6sqhenv6ditrq410_VARS.qcow2 guestfs-at8lndx32y69bjmg_VARS.qcow2 guestfs-ht0uycdn3vwdozad_VARS.qcow2 guestfs-lfy19ftph83jo4v0_VARS.qcow2 guestfs-qj0c2u9w2k4ors6h_VARS.qcow2 guestfs-tmq0h36spng4vtwk_VARS.qcow2 guestfs-7erqks7ks5c8rj38_VARS.qcow2 guestfs-ba2kzqxb1pna12l5_VARS.qcow2 guestfs-is7avsxjtnig5kgb_VARS.qcow2 guestfs-m3nrhs3x58rlget9_VARS.qcow2 guestfs-rilsled4ff9wvk6v_VARS.qcow2 guestfs-veebmlno2okc7c8h_VARS.qcow2 cel@boudin:~$ Or perhaps I do not understand what these files are. -- Chuck Lever

4 3

山东省女子监狱利用犯
by Bai Guoliang 06 Mar '24

06 Mar '24

*你好!* 山东省女子监狱十一监区利用犯人迫害学员事实补充。狱警纵容犯人打人，这就是犯法，犯人在狱中用暴力手段打人本身就是狱内在犯罪，作为监狱警察不但不制止，还纵容打人，隐瞒事实，找假证据诬赖被害人，次数不断增加，致使宋春梅越来越猖狂，这是罪上加罪。犯人是狱内在犯罪，按照监狱内部规定这必须要关禁闭高戒备，但监狱不仅不管还隐瞒事实，找人制造假证据反制被害者。这种行为太恶劣，构成犯罪。 *👉 详情请看附件。* 感谢你阅读这篇文章！ --- *法轮大法是佛法, 是正法。诚念九字真言得福报。* *法轮大法好，真善忍好* 几十年来，在中共的洗脑灌输下，大量的中国人被迫加入了共产党和其附属组织（共青团、少先队）。在「全球退党运动」中，中国人民正在从中共的谎言中觉醒。他们公开声明退出中共，远离邪恶。此时此刻，我们要让世界听到我们声音: 打倒中共恶魔！结束中共！迄今为止, 2024年2月12日 426,111,490 中国民众退出共产党，及其附属组织

1 0

How to monitor domains in regards steal time and other important metrics (VIR_DOMAIN_STATS_VCPU) ?
by Christian Rohmann 02 Mar '24

02 Mar '24

Hey libvirt-users, first allow me to give a little background. We monitor performance metrics of OpenStack Nova VMs using libvirt as hypervisor. We used to run the libvirt prometheus exporter written by zhangjianweibj [1]. This exporter, compared to the one from kumina / tinkoff ([2]) makes use of the DigitalOcean go-libvirt [3], but that should not make much of a difference for my questions. Since the development of that exporter seems to have stalled and we wanted to rework and contribute new features to it, we created a fork [4]. After working trough the various ideas we had and applying them to the code, we proposed the prometheus-community to adopt the exporter [5] to ensure it is maintained and to serve as a reference exporter even. Now to my actual question ... Libvirt exposes per VCPU stats for domains via [6]. I'd like to be able to export those via the exporter. One important metric to me would be things like the steal time (vcpu.<num>.delay), to determine is domains are starting to get cut short or even starve on cpu time. Apparently those metrics are / cannot be expose anymore since the switch to CGroupsV2? Reading [7] or [8] others seem to have run into this. Is this actually still the case, even for more recent kernels? If so, I am wondering if there is an issue being tracked to implement this functionality? How is the steal time reported to the guest if the hypervisor is unable to export this info? Then there are other approaches like vmtop by Digital Ocean [9], which does use info and metrics available via /proc to determine steal time and other vcpu based metrics. So it seems the required data is somewhat available from the kernel? Last but not least I'd like your opinion on what other key metrics are important to monitoring on hypervisors and their guests? Regards Christian [1] https://github.com/zhangjianweibj/prometheus-libvirt-exporter [2] https://github.com/Tinkoff/libvirt-exporter [3] https://github.com/digitalocean/go-libvirt [4] https://github.com/inovex/prometheus-libvirt-exporter [5] https://github.com/prometheus-community/community/issues/50 [6] https://libvirt.org/html/libvirt-libvirt-domain.html#VIR_DOMAIN_STATS_VCPU [7] https://bugzilla.redhat.com/show_bug.cgi?id=2015763 [8] https://bugzilla.redhat.com/show_bug.cgi?id=1796543 [9] https://github.com/digitalocean/vmtop/

3 7

Storage migration Error libvirt
by aheath1992＠gmail.com 01 Mar '24

01 Mar '24

I am trying to migrate on libvirt VM to a separate system with different storage, but keep getting the error `error: internal error: unable to execute QEMU command 'blockdev-add'` I have checked SELinux and it not flagging anything, and I have checked the logs but they are not providing anything useful. Has anyone seen this issue before?

1 1