- Users - libvirt List Archives

RBD pool not starting "An error occurred, but the cause is unknown"
by Stuart Longland VK4MSL 28 Jan '25

28 Jan '25

Hi all, I have an issue getting a RBD pool going on a newly deployed compute node. The storage back-end is a Ceph storage cluster running Ceph 14 (Nautilus… yes I know this is old, an update to 18 is planned soon). I have an existing node, running Debian 10 (again, updating this is planned, but I'd like to deploy new nodes to migrate the instances to whilst this node is updated), which runs about a dozen VMs with disks on this back-end. I've loaded a new machine (a MSI Cubi 5 mini PC) up with AlpineLinux 3.21. Boot disk is a 240GB SATA SSD, and there's a 1TB nVME for local VM storage. My intent is to allow VMs to mount RBDs for back-up purposes. The machine has two Ethernet interfaces (a 2.5Gbps and a 1Gbps link), one will be the "front-end" used by the VMs, the other will be a "back-end" link to talk to Ceph and administer the host. - OpenVSwitch 2.17.11 is deployed with two bridges - libvirtd 10.9.0 installed - a LVM pool called 'data' has been created on nVME - Ceph 19.2.0 is installed (libvirtd is linked to this version of librbd) - /etc/ceph has been cloned from my existing working compute node I have two RBD pools; 'one' and 'ha'. 'one' has most of my virtual machine images in it (it is from a former OpenNebula install), 'ha' has core router root disk images in it ('ha' for high availability; it has stronger replication settings than 'one' to guarantee better reliability). I've created a `libvirt` user in Ceph, and on the intended node, this works: > ~ # rbd --id libvirt ls -p one | head > mastodon-vda > mastodon-vdb > mastodon-vdd > mastodon-vde > one-14 > one-15 > one-19 > one-20 > one-22 > one-23 > ~ # rbd --id libvirt ls -p ha | head > core-router-obsd75-vda > core-router-obsd76-vda I can also access RBD images just fine: > ~ # rbd --id libvirt map one/shares-vda > /dev/rbd0 > ~ # fdisk -l /dev/rbd0 > Disk /dev/rbd0: 20 GB, 21474836480 bytes, 41943040 sectors > 2610 cylinders, 255 heads, 63 sectors/track > Units: sectors of 1 * 512 = 512 bytes > > Device Boot StartCHS EndCHS StartLBA EndLBA Sectors Size Id Type > /dev/rbd0p1 * 2,0,33 611,8,56 2048 616447 614400 300M 83 Linux > /dev/rbd0p2 611,8,57 1023,15,63 616448 2584575 1968128 961M 82 Linux swap > /dev/rbd0p3 1023,15,63 1023,15,63 2584576 41943039 39358464 18.7G 83 Linux > ~ # rbd unmap one/shares-vda This is registered in libvirtd: > ~ # virsh secret-list > UUID Usage > -------------------------------------------------------------------- > c14a16b5-bba5-473a-ae9b-53a9a6b0a4e3 ceph client.libvirt secret > ~ # virsh secret-dumpxml c14a16b5-bba5-473a-ae9b-53a9a6b0a4e3 > <secret ephemeral='no' private='no'> > <uuid>c14a16b5-bba5-473a-ae9b-53a9a6b0a4e3</uuid> > <usage type='ceph'> > <name>client.libvirt secret</name> > </usage> > </secret> I have defined four pools, 'temp', 'local', 'ha-images' and 'opennebula-images': > ~ # virsh pool-list --all > Name State Autostart > ------------------------------------------- > default active yes > ha-images active yes > local active yes > opennebula-images inactive yes > temp active yes 'ha-images' works just fine, this is its config: > ~ # virsh pool-dumpxml ha-images > <pool type='rbd'> > <name>ha-images</name> > <uuid>6beab982-52b3-495b-a4a7-ab7ebb522ef5</uuid> > <capacity unit='bytes'>20003977953280</capacity> > <allocation unit='bytes'>159339114496</allocation> > <available unit='bytes'>13142248669184</available> > <source> > <host name='172.31.252.1' port='6789'/> > <host name='172.31.252.2' port='6789'/> > <host name='172.31.252.5' port='6789'/> > <host name='172.31.252.6' port='6789'/> > <host name='172.31.252.7' port='6789'/> > <host name='172.31.252.8' port='6789'/> > <host name='172.31.252.9' port='6789'/> > <host name='172.31.252.10' port='6789'/> > <name>ha</name> > <auth type='ceph' username='libvirt'> > <secret uuid='c14a16b5-bba5-473a-ae9b-53a9a6b0a4e3'/> > </auth> > </source> > </pool> 'opennebula-images' does not, its config: > ~ # virsh pool-dumpxml opennebula-images > <pool type='rbd'> > <name>opennebula-images</name> > <uuid>fcaa2fa8-f0d2-4919-9168-756a9f4ad7ee</uuid> > <capacity unit='bytes'>20003977953280</capacity> > <allocation unit='bytes'>5454371495936</allocation> > <available unit='bytes'>13142254759936</available> > <source> > <host name='172.31.252.1' port='6789'/> > <host name='172.31.252.2' port='6789'/> > <host name='172.31.252.5' port='6789'/> > <host name='172.31.252.6' port='6789'/> > <host name='172.31.252.7' port='6789'/> > <host name='172.31.252.8' port='6789'/> > <host name='172.31.252.9' port='6789'/> > <host name='172.31.252.10' port='6789'/> > <name>one</name> > <auth type='ceph' username='libvirt'> > <secret uuid='c14a16b5-bba5-473a-ae9b-53a9a6b0a4e3'/> > </auth> > </source> > </pool> It's not obvious what the differences are. `name`, `uuid`, `allocation`, `available` and `source/name` are expected to be different, everything else 100% matches. I've tried removing and zeroing out the `capacity`, `allocation` and `available` tags to no effect. > ~ # virsh pool-dumpxml ha-images > /tmp/ha-images.xml > ~ # virsh pool-dumpxml opennebula-images > /tmp/opennebula-images.xml > ~ # diff -u /tmp/ha-images.xml /tmp/opennebula-images.xml > --- /tmp/ha-images.xml > +++ /tmp/opennebula-images.xml > @@ -1,9 +1,9 @@ > <pool type='rbd'> > - <name>ha-images</name> > - <uuid>6beab982-52b3-495b-a4a7-ab7ebb522ef5</uuid> > + <name>opennebula-images</name> > + <uuid>fcaa2fa8-f0d2-4919-9168-756a9f4ad7ee</uuid> > <capacity unit='bytes'>20003977953280</capacity> > - <allocation unit='bytes'>159339114496</allocation> > - <available unit='bytes'>13142248669184</available> > + <allocation unit='bytes'>5454371495936</allocation> > + <available unit='bytes'>13142254759936</available> > <source> > <host name='172.31.252.1' port='6789'/> > <host name='172.31.252.2' port='6789'/> > @@ -13,7 +13,7 @@ > <host name='172.31.252.8' port='6789'/> > <host name='172.31.252.9' port='6789'/> > <host name='172.31.252.10' port='6789'/> > - <name>ha</name> > + <name>one</name> > <auth type='ceph' username='libvirt'> > <secret uuid='c14a16b5-bba5-473a-ae9b-53a9a6b0a4e3'/> > </auth> > ~ # diff -y /tmp/ha-images.xml /tmp/opennebula-images.xml When I start this errant pool, I get this: > ~ # virsh pool-start opennebula-images > error: Failed to start pool opennebula-images > error: An error occurred, but the cause is unknown If I crank debugging up in `libvirtd` (via the not-recommended `log_level` and directing all output to a file), I see it successfully connects to the pool for about 15 seconds, lists the sizes of about a dozen disk images, then seemingly gives up and disconnects. > 2025-01-19 05:16:55.176+0000: 3609: info : vir_object_finalize:319 : OBJECT_DISPOSE: obj=0x7f975fc816a0 > 2025-01-19 05:16:55.177+0000: 3609: info : virObjectUnref:378 : OBJECT_UNREF: obj=0x7f975fc816a0 > 2025-01-19 05:16:55.183+0000: 3609: debug : virStorageBackendRBDRefreshPool:693 : Utilization of RBD pool one: (kb: 19535134720 kb_ > avail: 12800438616 num_bytes: 5489355030528) > 2025-01-19 05:16:55.988+0000: 3609: debug : volStorageBackendRBDRefreshVolInfo:569 : Refreshed RBD image one/mastodon-vda (capacity > : 21474836480 allocation: 21474836480 obj_size: 4194304 num_objs: 5120) > 2025-01-19 05:16:55.993+0000: 3609: info : virObjectNew:256 : OBJECT_NEW: obj=0x7f975fa6bba0 classname=virStorageVolObj > 2025-01-19 05:16:55.993+0000: 3609: info : virObjectRef:400 : OBJECT_REF: obj=0x7f975fa6bba0 > 2025-01-19 05:16:55.993+0000: 3609: info : virObjectRef:400 : OBJECT_REF: obj=0x7f975fa6bba0 > 2025-01-19 05:16:55.993+0000: 3609: info : virObjectRef:400 : OBJECT_REF: obj=0x7f975fa6bba0 > 2025-01-19 05:16:55.993+0000: 3609: info : virObjectUnref:378 : OBJECT_UNREF: obj=0x7f975fa6bba0 > 2025-01-19 05:16:56.011+0000: 3609: debug : volStorageBackendRBDRefreshVolInfo:569 : Refreshed RBD image one/mastodon-vdb (capacity > : 536870912000 allocation: 536870912000 obj_size: 4194304 num_objs: 128000) …snip… > 2025-01-19 05:17:03.756+0000: 3609: debug : volStorageBackendRBDRefreshVolInfo:569 : Refreshed RBD image one/wsmail-vdb (capacity: > 21474836480 allocation: 21474836480 obj_size: 4194304 num_objs: 5120) > 2025-01-19 05:17:03.758+0000: 3609: info : virObjectNew:256 : OBJECT_NEW: obj=0x7f975f9cf250 classname=virStorageVolObj > 2025-01-19 05:17:03.758+0000: 3609: info : virObjectRef:400 : OBJECT_REF: obj=0x7f975f9cf250 > 2025-01-19 05:17:03.758+0000: 3609: info : virObjectRef:400 : OBJECT_REF: obj=0x7f975f9cf250 > 2025-01-19 05:17:03.758+0000: 3609: info : virObjectRef:400 : OBJECT_REF: obj=0x7f975f9cf250 > 2025-01-19 05:17:03.758+0000: 3609: info : virObjectUnref:378 : OBJECT_UNREF: obj=0x7f975f9cf250 > 2025-01-19 05:17:03.777+0000: 3609: debug : volStorageBackendRBDRefreshVolInfo:569 : Refreshed RBD image one/sjl-router-obsd76-vda > (capacity: 34359738368 allocation: 34359738368 obj_size: 4194304 num_objs: 8192) > 2025-01-19 05:17:03.778+0000: 3609: debug : virStorageBackendRBDCloseRADOSConn:369 : Closing RADOS IoCTX > 2025-01-19 05:17:03.778+0000: 3609: debug : virStorageBackendRBDCloseRADOSConn:374 : Closing RADOS connection > 2025-01-19 05:17:03.783+0000: 3609: debug : virStorageBackendRBDCloseRADOSConn:378 : RADOS connection existed for 15 seconds > 2025-01-19 05:17:03.783+0000: 3609: info : virObjectUnref:378 : OBJECT_UNREF: obj=0x7f975f9cf2b0 > 2025-01-19 05:17:03.783+0000: 3609: info : virObjectUnref:378 : OBJECT_UNREF: obj=0x7f975ef90ac0 > 2025-01-19 05:17:03.783+0000: 3609: info : virObjectUnref:378 : OBJECT_UNREF: obj=0x7f975fa6bd20 > 2025-01-19 05:17:03.783+0000: 3609: info : virObjectUnref:378 : OBJECT_UNREF: obj=0x7f975fa6ecc0 …snip… > 2025-01-19 05:17:03.785+0000: 3609: info : vir_object_finalize:319 : OBJECT_DISPOSE: obj=0x7f975f9cee90 > 2025-01-19 05:17:03.785+0000: 3609: info : virObjectUnref:378 : OBJECT_UNREF: obj=0x7f975f9cee90 > 2025-01-19 05:17:03.785+0000: 3609: info : vir_object_finalize:319 : OBJECT_DISPOSE: obj=0x7f975fa6e960 > 2025-01-19 05:17:03.785+0000: 3609: info : virObjectUnref:378 : OBJECT_UNREF: obj=0x7f975fa6e960 > 2025-01-19 05:17:03.785+0000: 3609: error : storageDriverAutostartCallback:213 : internal error: Failed to autostart storage pool 'opennebula-images': no error > 2025-01-19 05:17:03.785+0000: 3609: info : virObjectUnref:378 : OBJECT_UNREF: obj=0x7f975fcd0490 > 2025-01-19 05:17:03.785+0000: 3609: info : virObjectRef:400 : OBJECT_REF: obj=0x7f975fcd27c0 > 2025-01-19 05:17:03.785+0000: 3609: info : virObjectUnref:378 : OBJECT_UNREF: obj=0x7f975fcd27c0 > 2025-01-19 05:17:03.786+0000: 3609: info : virObjectRef:400 : OBJECT_REF: obj=0x7f975fcd06d0 > 2025-01-19 05:17:03.786+0000: 3609: info : virObjectUnref:378 : OBJECT_UNREF: obj=0x7f975fcd06d0 > 2025-01-19 05:17:03.786+0000: 3609: info : virObjectRef:400 : OBJECT_REF: obj=0x7f975fcd0130 If there's no cause for the error, it should not fail. If it fails, there should be a cause listed, there's no excuse for it being "unknown" -- just because Microsoft's OSes make up error codes that its own help system can't explain is no excuse for the open-source world to follow their example. I'd happily provide more information, if someone can provide guidance on how to locate it. -- Stuart Longland (aka Redhatter, VK4MSL) I haven't lost my mind... ...it's backed up on a tape somewhere.

2 2

Authoritative info on backup-begin versus snapshots/other state capture
by camccuk＠yahoo.com 17 Jan '25

17 Jan '25

Hello all Apologies for the basic nature of the question, but having recently started working with libvirt - and virtualisation in general - I find there is a lot of out-of-date and sometimes contradictory material out there across blogs, articles, stackoverflow, the usual sources... I thought I might be able to get definitive answers here. For the record, I assume libvirt.org is authoritative but while there is a lot of material there, its structure is not always clear to me. Also the lack of dates on any pages leaves some room for doubt. I am wondering if there is a recent, reliable summary of the various approaches and current best practices for backing up VMs that covers snapshots both internal and external, approaches that use backup-begin and third-party approaches which simply stop the VM and copy off files. If there is not such a summary, can anyone confirm my reading of https://libvirt.org/kbase/domainstatecapture.html that a simple backup-begin <domain-name> will: - pause the VM and quiesce the disk (in which case is qemu agent a requirement on the guest?) - generate a date-suffixed disk-only copy of a VMs disks alongside the originals wherever that storage is - not generate any backing image chains or metadata that needs to be retained Furthermore, is it then possible to restore to that point by stopping a VM, and associating that backup file with the VM either by virsh-editing its xml or overwriting the original file with the backup file. This seems to be my experience in testing this, but there are very few references to this tool compared to the many lengthy discussions about snapshots and other approaches which is a bit puzzling. It would be great to have this understanding confirmed or refined! Many thanks for any pointers

2 4

libvirt-guests on Redhat
by Joe Muller 06 Jan '25

06 Jan '25

Hello, Per libvirt and Redhat (up to RHEL7) documentation, the libvirt-guests daemon should be enabled to ensure that guest VMs are properly shutdown/suspended if the KVM host is shutdown/rebooted. I understand that in addition to other changes, Redhat moved from the 'monolithic' libvirtd service model to the 'modular' multiple service model. Out of the box, the systemd libvirt-guests.service is disabled, and testing shows that the associated qemu-kvm processes for virtual machines are simply killed as the various virt* services are stopped as part of the systemd shutdown target. Did Redhat just decide to fork libvirt and do their own thing, or there some equivalent way to get the same clean shutdown behavior which libvirt-guests used to provide? Relevant documentation: --- https://libvirt.org/daemons.html https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/9/html/co… -- Joe Muller System Administrator Sonic.

2 1

Best practice to manage network
by linux＠hklb.ch 03 Jan '25

03 Jan '25

Hi, First, sorry if the topic has already been discussed recently (the only thread I found related to my problem was created in 2010..) I have a hypervisor with KVM and LXC installed on a Debian 12, and I'm using libvirt to create my VM. All my networks are defined in my /etc/network/interfaces.d/* (I'm using openvswitch with specific options, such as port mirroring/patch/...) , and I'm configuring the network on my VM XML definition like this : <interface type='bridge'> <mac address='52:54:00:ab:c3:d3'/> <source bridge='prod'/> <vlan> <tag id='55'/> </vlan> <virtualport type='openvswitch'> <parameters interfaceid='331d973c-0c5b-4d3c-b2ad-590f908e680d'/> </virtualport> <target dev='vnet180'/> <model type='virtio'/> <mtu size='9216'/> <alias name='net1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </interface> It is working perfectly, until I'm restarting my network (using ifreload, ifup, systemctl network restart, ...) - all my VM come unreachable... To make it work again, I also need to restart libvirtd Is it still expected to have this behavior ? What would be a better way to configure the network ? Thanks in advance Lucas

2 1

kernel panic when hotswapping nvidia gpu via gpu passthrough in a windows vm
by 5zj1zxgr＠anonaddy.me 30 Dec '24

30 Dec '24

This may be more issue with nvidia drivers, but just wanted to check if anyone tried to perform gpu passthrough with an nvidia gpu that is used as a secondary gpu on a nvidia "prime" laptop - laptop with both an integrated intel iGPU and a secondary nvidia dGPU. I was able SOMETIMES to perform GPU passthrough successfully with a VM, but SOMETIMES it led to kernel panics. It usually happens when virtual machine shuts down and nvidia dGPU is returned to the host. Dec 30 14:29:53 host kernel: ACPI Error: Needed [Integer/String/Buffer], found [Reference] 00000000f0ed5a9d (20240827/exresop-388) Dec 30 14:29:53 host kernel: ACPI Error: AE_AML_OPERAND_TYPE, While resolving operands for [And] (20240827/dswexec-433) Dec 30 14:29:53 host kernel: ACPI Error: Aborting method \_SB.PCI0.LPCB.EC0._Q20 due to previous error (AE_AML_OPERAND_TYPE) (20240827/psparse-529) Dec 30 14:29:56 host kernel: ACPI Error: Needed [Integer/String/Buffer], found [Reference] 0000000059be91f0 (20240827/exresop-388) Dec 30 14:29:56 host kernel: ACPI Error: AE_AML_OPERAND_TYPE, While resolving operands for [And] (20240827/dswexec-433) Dec 30 14:29:56 host kernel: ACPI Error: Aborting method \_SB.PCI0.LPCB.EC0._Q20 due to previous error (AE_AML_OPERAND_TYPE) (20240827/psparse-529) Any advice would be really appreciated.

1 0

Re: Features from Capabilities
by Simon Fairweather 23 Dec '24

23 Dec '24

For HyperV looks like I can build a list from the schema, but for CPU options would I use the xml in /usr/share/libvirt/cpu

1 0

Features from Capabilities
by Simon Fairweather 22 Dec '24

22 Dec '24

HI, Is there any documentation or somewhere I can look at features that could be selected? I have started to create an extract from capabilities and domcapabilities to provide a list I can use for selection. Example these are the following entries I have selected to allow running of Hyperv in Server 2019 <features> <acpi/> <apic/> <hyperv mode='custom'> <relaxed state='on'/> <vapic state='on'/> <spinlocks state='on' retries='8191'/> <vendor_id state='on' value='none'/> </hyperv> </features> <cpu mode='custom' match='strict' check='none'> <model fallback='forbid'>Skylake-Client-IBRS</model> <topology sockets='1' dies='1' clusters='1' cores='4' threads='1'/> </cpu> <clock offset='localtime'> <timer name='hpet' present='no'/> <timer name='hypervclock' present='no'/> </clock> But looking to find a list of options that would be valid to be specified. Regards Simon.

1 0

Overriding qemu.conf libvirt-qemu user per-domain
by Pascal Proulx 20 Dec '24

20 Dec '24

Hello, How can I override the libvirt-qemu user defined in /etc/libvirt/qemu.conf using a per-domain (virtual machine) override using the domain XML definitions? I can find qemu arg overrides but not this and I may have missed it. Thank you

2 2

List of valid arch's for virt-install
by Jeffrey Walton 13 Dec '24

13 Dec '24

Hi Everyone, I'm converting about 150 Virtual Box vm's to QEMU/KVM using `qemu-img convert` followed by a `virt-install`. The vm's are a mix of amd64, i686 and mx32, and they will run on a amd64 machine. I want to use the `virt-install --arch` option during the conversion. The man page for virt-install [1] does not list valid arch's. (I can't find a page for virt-install at libvirt.org) Where can I find the list of valid arch's for virt-install? A related question is, if I also specify `--cpu host` to ensure the host cpu's ISA's are available to the vm, then will that conflict with the `--arch arch` option? Thanks in advance. [1] https://manpages.org/virt-install

3 4

Running libvirt without dnsmasq
by procmem＠riseup.net 10 Dec '24

10 Dec '24

Hi, we are trying to document a way for our users to run libvirt without dnsmasq to reduce attack surface on the host. We are aware that the default network uses it but plan to disable that and use our own custom configured networks instead. Uninstalling dnsmasq causes libvirt to refuse to start even if the default network is no longer running. Is this possible or is this something that needs code changes upstream?

5 8