- Users - libvirt List Archives

Re: Encrypting boot partition Libvirt not showing the OS booting up
by john doe 13 Oct '20

13 Oct '20

On 10/12/2020 11:53 AM, john doe wrote: > On 10/12/2020 11:37 AM, Peter Krempa wrote: >> On Mon, Oct 12, 2020 at 11:27:20 +0200, john doe wrote: >>> Hi, thank you for your answer, I'm sending this privatly as you asked >>> for private information. >>> Can I ask you to keep those information private? >>> >>> On 10/12/2020 10:29 AM, Peter Krempa wrote: >>>> On Mon, Oct 12, 2020 at 10:03:15 +0200, john doe wrote: >>>>> Hi, >>>>> >>>>> I have installed Debian Buster with encrypted LVM so apon installation >>>>> my root partition is encrypted. >>>>> So far so good but as soon as I encrypt the boot partition, after >>>>> reboot >>>>> the OS won't start. >>>>> If I start the drive directly with qemu, it works but it looks like >>>>> Libvirt is somehow not able to deel with it. >>>> >>>> This is not enough information to diagnose the problem. >>>> >>>> We'll need the following: >>>> >>>> 1) Did you encrypt the partition using the debian installer >>>> >>> >>> No, I did it after installation following the instructions at (1). >>> >>>> 2) what vm XML you used: >>>> a) during installation >>> >>> The domain xml file was created by virt-install with the following >>> command: >>> $ virsh destroy try01; virsh undefine try01; time virt-install >>> --name=try01 --ram=1024 --noreboot --cpuset=auto --cpu host >>> --vcpus=1,maxvcpus=4 --disk=path=/mnt/usbkey01/machines/try/try01,size=6 >>> --graphic none --pxe --os-variant=debian10 --network >>> bridge=br0,mac=0e:35:32:84:c3:f3 --filesystem >>> type=mount,mode=mapped,source=/mnt/usbkey01/public,target=public_dir >>> >>>> b) when trying to finally boot the vm >>>> >>> >>> Attached as 'try01.xml' obtained by doing 'virsh dumpxml try01 > >>> try01.xml'. >>> >>>> 3) what qemu command line you've used at the point you claim it worked >>>> >>> >>> qemu-system-x86_64 -drive file=/mnt/usbkey01/machines/try/try01 -m 1024 >>> -boot c -accel kvm -machine q35 -nographic >>> >>>> 4) what is the error/final state when the VM fails to boot with libvirt >>>> >>> >>> After having encrypted the boot partition: >>> >>> $virsh console try01 >>> root@0e-35-32-84-c3-f3:# [ 208.513259] watchdog: watchdog0: watchdog >>> did not stop! >>> [ 208.855971] reboot: Restarting system >>> >>> >>> $ qemu-system-x86_64 -drive file=/mnt/usbkey01/machines/try/try01 -m >>> 1024 -boot c -accel kvm -machine q35 -nographic >>> SeaBIOS (version 1.12.0-1) >>> >>> >>> iPXE (http://ipxe.org) 00:02.0 C980 PCI2.10 PnP PMM+3FF8FE80+3FECFE80 >>> C980 >>> >>> >>> >>> Booting from Hard Disk... >>> Attempting to decrypt master key... >>> Enter passphrase for hd0,msdos1 (43a322dfc8ba4628b80afc66d49642a7): >>> >>> >>> As you can see above, if I invoked qemu directly, I get prompted for the >>> boot passthrase but I'm not getting it when using libvirt. >> >> Okay, so the root cause ... or "problem" here is that you don't see the >> console via 'virsh console' after the guest OS rebooted. >> >> My suspicion according to the VM XML is that the VM restart triggered a >> restart of the qemu process and thus our internal handler of the console >> passthrough disconnected. >> >> Please try a "virsh destroy $VM" if the VM is running/stuck waiting for >> the password without actually showing it and then start it using >> >> virsh start --console $VM >> >> This will start it and connect to the console immediately. >> >> Please report your findings, we might want to discuss what happens when >> a console is connected and the guest uses a setting of: >> >> <on_reboot>restart</on_reboot> >> >> when the VM is rebooted. >> > > Still no luck: > > $ virsh destroy try01; virsh start --console try01 > error: Failed to destroy domain try01 > error: Requested operation is not valid: domain is not running > > Domain try01 started > Connected to domain try01 > Escape character is ^] > > > > > > I did not modify anything during the time I sent my answer and seeing > yours. > I sent privately the requested xml file to 'Peter Krempa <pkrempa(a)redhat.com>'. Peter Krempa 's privately answered me back suggesting to add the following in the domain xml file: <bios useserial='yes'/> under <os> such as ... <os> <type arch='x86_64' machine='pc-q35-3.1'>hvm</type> <boot dev='hd'/> <bios useserial='yes'/> </os> This does not help at all and still give the output sent previously. -- John Doe

5 12

Re: Attached disk blockpull
by Peter Krempa 13 Oct '20

13 Oct '20

This is a libvirt question, so asking it on the qemu-block might not get you an answer that quick, ... or ever if I didn't notice your other question also addressed incorrectly. [adding libvirt-users(a)redhat.com to cc] On Tue, Sep 01, 2020 at 10:57:34 -0400, Yoonho Park wrote: > I am trying to perform a blockpull on an attached disk, but I cannot do > this using "virsh attach-disk" specifying a new, empty overlay file created > with "qemu-img create". The VM does not pick up the backing_file path from Could you please elaborate on the exact steps? specifically the arguments for qemu-img create. My feeling is that you didn't specify the backing image format (-F) when creating the overlay, but it's hard to tell from this very sparse question. > the qcow2 overlay file, and there seems to be no way to specify the backing > path through the attach-disk command itself. Instead the backing path has virsh attach-disk is a syntax-sugar wrapper which generates the XML document which is in fact used to attach disk. You can use virsh attach-disk --print-xml to see what would be used. > to be specified in the xml file used to create the VM, which precludes > attaching a new disk to a running VM. Is this a bug? Is it possible to using virsh attach-device allows you to pass in a whole <devices> subelement such as <disk> with any configuration you are able to specify when defining the XML. The virsh attach-disk is limited to just "basic" config options. > attach a disk to a running VM, and specify its backing path, using qemu > directly? This is with qemu 4.2.0 and libvirt 6.0.0.

1 0

Failed to get udev device for syspath '/sys/devices/virtual/dmi/id'
by Mathieu Tarral 12 Oct '20

12 Oct '20

Hi, I'm facing an issue with libvirt and the LIBXL driver, failing when searching for DMI data in /sys. info : libvirt version: 5.0.0, package: 4+deb10u1 (Guido Günther <agx(a)sigxcpu.org> Thu, 05 Dec 2019 00:22:14 +0100) error : udevGetDMIData:1719 : internal error: Failed to get udev device for syspath '/sys/devices/virtual/dmi/id' or '/sys/class/dmi/id' error : libxlDriverConfigNew:1803 : Unable to configure libxl's memory management parameters error : virStateInitialize:662 : Initialization of LIBXL state driver failed: internal error: Failed to get udev device for syspath '/sys/devices/virtual/dmi/id' or '/sys/class/dmi/id' error : daemonRunStateInit:799 : Driver state initialization failed The relevant function udevGetDMIData in libvirt: https://github.com/libvirt/libvirt/blob/cb09344a2cccc0cc9bcefa3cb53d7af45ba… I don't understand what's happening, as this initialization failure only happens when I boot on the Xen kernel. Also, the /sys/class/dmi* paths are not exposed on Xen. Note: Running on Xen 4.12.1 Any ideas ? Thanks !

4 5

Encrypting boot partition Libvirt not showing the OS booting up
by john doe 12 Oct '20

12 Oct '20

Hi, I have installed Debian Buster with encrypted LVM so apon installation my root partition is encrypted. So far so good but as soon as I encrypt the boot partition, after reboot the OS won't start. If I start the drive directly with qemu, it works but it looks like Libvirt is somehow not able to deel with it. What am I missing? -- John Doe

2 1

Building libvirt library without libvirtd or virsh
by Román González 09 Oct '20

09 Oct '20

Hello there, I'm trying to play with musl and libvirt to see if I'm able to build a libvirt client binary without dynamic lib dependencies. I have two questions: 1) to your knowledge, is this exercise futile? 2) Do you know if there is a way to *only* compile the library bits? I want to reduce the number of dependencies in the build, and only construct the libvirt libraries, not libvirtd nor virsh. Thanks in advance for any help you can provide. Roman.-

3 4

Is it possible that "virsh destroy" does not stop a domain ?
by Lentes, Bernd 08 Oct '20

08 Oct '20

Hi, Is it possible that "virsh destroy" does not stop a domain ? I'm asking because i have some domains running in a two-node HA-Cluster (pacemaker). And sometimes one node get fenced (killed) because it couldn't stop a domain. That's very ugly. This is also the reason why i asked before what "virsh destroy" really does ? IIRC a kill -9 can't terminate a process which is in "D" state (uninterruptible sleep). So if the process of the domain is in "D" state, it can't be finished. Right ? Pacemaker tries to shutdown or destroy a domain with a resource agent, which is a shell script, similar to an init script. Here is an excerp from the resource agent for virtual domains: force_stop() { local out ex translate local status=0 ocf_log info "Issuing forced shutdown (destroy) request for domain ${DOMAIN_NAME}." out=$(LANG=C virsh $VIRSH_OPTIONS destroy ${DOMAIN_NAME} 2>&1) # hier wird die domain destroyed ex=$? translate=$(echo $out|tr 'A-Z' 'a-z') echo >&2 "$translate" case $ex$translate in *"error:"*"domain is not running"*|*"error:"*"domain not found"*|\ *"error:"*"failed to get domain"*) : ;; # unexpected path to the intended outcome, all is well sucess [!0]*) ocf_exit_reason "forced stop failed" # <============ fail of destroy seems to be possible return $OCF_ERR_GENERIC ;; 0*) while [ $status != $OCF_NOT_RUNNING ]; do VirtualDomain_status status=$? done ;; esac return $OCF_SUCCESS } The function force_stop is responsible for stop/destroy the domain. And it cares about a non-working "virsh destroy". Is there a developer who can explain what "virsh destroy" really does ? Or is there another ML for the developers ? Bernd -- Bernd Lentes Systemadministration Institute for Metabolism and Cell Death (MCD) Building 25 - office 122 HelmholtzZentrum München bernd.lentes(a)helmholtz-muenchen.de phone: +49 89 3187 1241 phone: +49 89 3187 3827 fax: +49 89 3187 2294 http://www.helmholtz-muenchen.de/mcd stay healthy Helmholtz Zentrum MÃ¼nchen Helmholtz Zentrum MÃ¼nchen

3 3

PCI Passthrough and Surprise Hotplug
by Marc Smith 08 Oct '20

08 Oct '20

Hi, I'm using QEMU/KVM on RHEL (CentOS) 7.8.2003: # cat /etc/redhat-release CentOS Linux release 7.8.2003 I'm passing an NVMe drive into a Linux KVM virtual machine (<type arch='x86_64' machine='pc-i440fx-rhel7.0.0'>hvm</type>) which has the following 'hostdev' entry: <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x42' slot='0x00' function='0x0'/> </source> <alias name='hostdev5'/> <rom bar='off'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x0f' function='0x0'/> </hostdev> This all works fine during normal operation, but I noticed when we remove the NVMe drive (surprise hotplug event), the PCIe EP then seems "stuck"... here we see the link-down event on the host (when the drive is removed): [67720.177959] pciehp 0000:40:01.2:pcie004: Slot(238-1): Link Down [67720.178027] vfio-pci 0000:42:00.0: Relaying device request to user (#0) And naturally, inside of the Linux VM, we see the NVMe controller drop: [ 1203.491536] nvme nvme1: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0xffff [ 1203.522759] blk_update_request: I/O error, dev nvme1n2, sector 33554304 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 [ 1203.560505] nvme 0000:01:0f.0: Refused to change power state, currently in D3 [ 1203.561104] nvme nvme1: Removing after probe failure status: -19 [ 1203.583506] Buffer I/O error on dev nvme1n2, logical block 4194288, async page read [ 1203.583514] blk_update_request: I/O error, dev nvme1n1, sector 33554304 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 We see this EP is found at IOMMU group '76': # readlink /sys/bus/pci/devices/0000\:42\:00.0/iommu_group ../../../../kernel/iommu_groups/76 And it is no longer bound to the 'vfio-pci' driver (expected) on the host. I was expecting to see all of the FD's to the /dev/vfio/NN character devices closed, but it seems they are still open: # lsof | grep "vfio/76" qemu-kvm 242364 qemu 70u CHR 235,4 0t0 3925324 /dev/vfio/76 qemu-kvm 242364 242502 qemu 70u CHR 235,4 0t0 3925324 /dev/vfio/76 qemu-kvm 242364 242511 qemu 70u CHR 235,4 0t0 3925324 /dev/vfio/76 qemu-kvm 242364 242518 qemu 70u CHR 235,4 0t0 3925324 /dev/vfio/76 qemu-kvm 242364 242531 qemu 70u CHR 235,4 0t0 3925324 /dev/vfio/76 qemu-kvm 242364 242533 qemu 70u CHR 235,4 0t0 3925324 /dev/vfio/76 qemu-kvm 242364 242542 qemu 70u CHR 235,4 0t0 3925324 /dev/vfio/76 qemu-kvm 242364 242550 qemu 70u CHR 235,4 0t0 3925324 /dev/vfio/76 qemu-kvm 242364 242554 qemu 70u CHR 235,4 0t0 3925324 /dev/vfio/76 SPICE 242364 242559 qemu 70u CHR 235,4 0t0 3925324 /dev/vfio/76 After the NVMe drive was removed for 100 seconds, we see the following kernel messages on the host: [67820.179749] vfio-pci 0000:42:00.0: Relaying device request to user (#10) [67900.272468] vfio_bar_restore: 0000:42:00.0 reset recovery - restoring bars [67900.272652] vfio_bar_restore: 0000:42:00.0 reset recovery - restoring bars [67900.319284] vfio_bar_restore: 0000:42:00.0 reset recovery - restoring bars I also noticed these messages related to the EP that is down currently that seem to continue indefinitely on the host (every 100 seconds): [67920.181882] vfio-pci 0000:42:00.0: Relaying device request to user (#20) [68020.184945] vfio-pci 0000:42:00.0: Relaying device request to user (#30) [68120.188209] vfio-pci 0000:42:00.0: Relaying device request to user (#40) [68220.190397] vfio-pci 0000:42:00.0: Relaying device request to user (#50) [68320.192575] vfio-pci 0000:42:00.0: Relaying device request to user (#60) But perhaps that is expected behavior. In any case, the problem comes when I re-insert the NVMe drive into the system... on the host, we see the link-up event: [68418.595101] pciehp 0000:40:01.2:pcie004: Slot(238-1): Link Up But the device is not bound to the 'vfio-pci' driver: # ls -ltr /sys/bus/pci/devices/0000\:42\:00.0/driver ls: cannot access /sys/bus/pci/devices/0000:42:00.0/driver: No such file or directory And appears to fail when attempting to bind to it manually: # echo "0000:42:00.0" > /sys/bus/pci/drivers/vfio-pci/bind -bash: echo: write error: No such device Device is enabled: # cat /sys/bus/pci/devices/0000\:42\:00.0/enable 1 So, wondering if this is expected behavior? Stopping the VM and starting it (virsh destroy/start) allows the device to work in the VM again, but for my particular use case, this is not an option. Need the surprise hotplug functionality to work with the PCIe EP passed into the VM. And perhaps this is an issue elsewhere (eg, vfio-pci). Any tips/suggestions on where to dig more would be appreciated. Thanks for your time. --Marc

2 2

[libvirt-users] Live migrate + change interface name
by Marc Roos 07 Oct '20

07 Oct '20

Is it possible to do a live migrate of a guest, having on the from host a source_device=eth2 and to host a source_dev=eth1?

2 4

time in domain very unstable
by Lentes, Bernd 07 Oct '20

07 Oct '20

Hi, i have a domain (SLES 10 SP4) running with KVM. Time is very wrong when booting unless ntp synchronizes. What can i do ? Bernd -- Bernd Lentes Systemadministration Institute for Metabolism and Cell Death (MCD) Building 25 - office 122 HelmholtzZentrum München bernd.lentes(a)helmholtz-muenchen.de phone: +49 89 3187 1241 phone: +49 89 3187 3827 fax: +49 89 3187 2294 http://www.helmholtz-muenchen.de/mcd stay healthy Helmholtz Zentrum MÃ¼nchen Helmholtz Zentrum MÃ¼nchen

2 3

Unable to start or create any vm's on the qemu/kvm network
by muke101 06 Oct '20

06 Oct '20

Hi, for some reason my libvirt install has suddenly stopped working with any vm's on the qemu/kvm network, and I can't seem to figure out how to fix it or even anything that indicates what's causing the problem. I've posted my problem to discord servers, subreddits, irc's and the server fault stackexchange to no avail, so I'm hoping that if I share my stackexchnage post here someone might be able to take a look and figure out the problem? Here's the post with more details: https://serverfault.com/questions/1036370/error-when-starting-vm-on-the-qem… Any help would be appreciated, thanks.

2 1