[resent] virt-manager connection fails with 'qemu unexpectedly closed the monitor'

Hi! I recently ran into a problem when connecting to libvirtd 6.9.0 on Debian unstable and trying to import an existing image with Windows 7. Upon finishing the wizard and starting the instance, the import process fails with the following error message: Unable to complete install: 'internal error: qemu unexpectedly closed the monitor' Traceback (most recent call last): File "/usr/share/virt-manager/virtManager/asyncjob.py", line 65, in cb_wrapper callback(asyncjob, *args, **kwargs) File "/usr/share/virt-manager/virtManager/createvm.py", line 2081, in _do_async_install installer.start_install(guest, meter=meter) File "/usr/share/virt-manager/virtinst/install/installer.py", line 731, in start_install domain = self._create_guest( File "/usr/share/virt-manager/virtinst/install/installer.py", line 679, in _create_guest domain = self.conn.createXML(install_xml or final_xml, 0) File "/usr/lib64/python3.8/site-packages/libvirt.py", line 4366, in createXML raise libvirtError('virDomainCreateXML() failed') libvirt.libvirtError: internal error: qemu unexpectedly closed the monitor Since this error message is rather generic, I don't know where to start debugging. Does anyone know how to increase verbosity here to get an error message that might be more helpful? Thanks, Adrian -- .''`. John Paul Adrian Glaubitz : :' : Debian Developer - glaubitz@debian.org `. `' Freie Universitaet Berlin - glaubitz@physik.fu-berlin.de `- GPG: 62FF 8A75 84E0 2956 9546 0006 7426 3B37 F5B5 F913

On 12/20/20 6:53 AM, John Paul Adrian Glaubitz wrote:
Hi!
I recently ran into a problem when connecting to libvirtd 6.9.0 on Debian unstable and trying to import an existing image with Windows 7.
Upon finishing the wizard and starting the instance, the import process fails with the following error message:
Unable to complete install: 'internal error: qemu unexpectedly closed the monitor'
Traceback (most recent call last): File "/usr/share/virt-manager/virtManager/asyncjob.py", line 65, in cb_wrapper callback(asyncjob, *args, **kwargs) File "/usr/share/virt-manager/virtManager/createvm.py", line 2081, in _do_async_install installer.start_install(guest, meter=meter) File "/usr/share/virt-manager/virtinst/install/installer.py", line 731, in start_install domain = self._create_guest( File "/usr/share/virt-manager/virtinst/install/installer.py", line 679, in _create_guest domain = self.conn.createXML(install_xml or final_xml, 0) File "/usr/lib64/python3.8/site-packages/libvirt.py", line 4366, in createXML raise libvirtError('virDomainCreateXML() failed') libvirt.libvirtError: internal error: qemu unexpectedly closed the monitor
Since this error message is rather generic, I don't know where to start debugging.
Does anyone know how to increase verbosity here to get an error message that might be more helpful?
The first step would be to look in /var/log/libvirt/qemu/$guestname.log. If qemu is encountering some error, it should be logging a message there before it exits. (unless the error is a segfault; in that case I guess you'll need to look for a coredump. It will also list exactly the qemu commandline that was used. If the message there isn't illuminating enough and you come back here, it would be helpful to "bring along" that qemu commandline (and any accompanying error) as well as the output of "virsh dumpxml $guestname"

(CC'ing a friend who's run into the problem as well) Hi Laine! On 12/20/20 7:37 PM, Laine Stump wrote:
The first step would be to look in /var/log/libvirt/qemu/$guestname.log. If qemu is encountering some error, it should be logging a message there before it exits. (unless the error is a segfault; in that case I guess you'll need to look for a coredump. It will also list exactly the qemu commandline that was used.
Except for some unrelated warnings, qemu does not emit any error message: 2020-12-21 11:10:37.580+0000: starting up libvirt version: 6.9.0, package: 1+b2 (amd64 / i386 Build Daemon (x86-ubc-01) <buildd_amd64-x86-ubc-01@buildd.debian.org> Mon, 07 Dec 2020 09:45:52 +0000), qemu version: 5.2.0Debian 1:5.2+dfsg-2, kernel: 5.9.0-5-amd64, hostname: z6.physik.fu-berlin.de LC_ALL=C \ PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \ HOME=/var/lib/libvirt/qemu/domain-1-windows7 \ XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-1-windows7/.local/share \ XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-1-windows7/.cache \ XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-1-windows7/.config \ QEMU_AUDIO_DRV=spice \ /usr/bin/qemu-system-x86_64 \ -name guest=windows7,debug-threads=on \ -S \ -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-1-windows7/master-key.aes \ -machine pc-q35-5.2,accel=kvm,usb=off,vmport=off,dump-guest-core=off,memory-backend=pc.ram \ -cpu Broadwell-IBRS,vme=on,ss=on,vmx=on,pdcm=on,f16c=on,rdrand=on,hypervisor=on,arat=on,tsc-adjust=on,umip=on,md-clear=on,stibp=on,arch-capabilities=on,ssbd=on,xsaveopt=on,pdpe1gb=on,abm=on,ibpb=on,amd-stibp=on,amd-ssbd=on,skip-l1dfl-vmentry=on,pschange-mc-no=on,hv-time,hv-relaxed,hv-vapic,hv-spinlocks=0x1fff \ -m 4096 \ -object memory-backend-ram,id=pc.ram,size=4294967296 \ -overcommit mem-lock=off \ -smp 2,sockets=2,cores=1,threads=1 \ -uuid bb32a4b5-7b2c-4ac2-9a65-8fd2bb9cdb86 \ -no-user-config \ -nodefaults \ -chardev socket,id=charmonitor,fd=33,server,nowait \ -mon chardev=charmonitor,id=monitor,mode=control \ -rtc base=localtime,driftfix=slew \ -global kvm-pit.lost_tick_policy=delay \ -no-hpet \ -no-shutdown \ -global ICH9-LPC.disable_s3=1 \ -global ICH9-LPC.disable_s4=1 \ -boot strict=on \ -device pcie-root-port,port=0x10,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x2 \ -device pcie-root-port,port=0x11,chassis=2,id=pci.2,bus=pcie.0,addr=0x2.0x1 \ -device pcie-root-port,port=0x12,chassis=3,id=pci.3,bus=pcie.0,addr=0x2.0x2 \ -device pcie-root-port,port=0x13,chassis=4,id=pci.4,bus=pcie.0,addr=0x2.0x3 \ -device ich9-usb-ehci1,id=usb,bus=pcie.0,addr=0x1d.0x7 \ -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pcie.0,multifunction=on,addr=0x1d \ -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pcie.0,addr=0x1d.0x1 \ -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pcie.0,addr=0x1d.0x2 \ -device virtio-serial-pci,id=virtio-serial0,bus=pci.2,addr=0x0 \ -blockdev '{"driver":"file","filename":"/data/kvm/windows7.img","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}' \ -blockdev '{"node-name":"libvirt-1-format","read-only":false,"driver":"raw","file":"libvirt-1-storage"}' \ -device ide-hd,bus=ide.0,drive=libvirt-1-format,id=sata0-0-0,bootindex=1 \ -netdev tap,fd=35,id=hostnet0 \ -device e1000e,netdev=hostnet0,id=net0,mac=52:54:00:4b:c5:26,bus=pci.1,addr=0x0 \ -chardev pty,id=charserial0 \ -device isa-serial,chardev=charserial0,id=serial0 \ -chardev spicevmc,id=charchannel0,name=vdagent \ -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 \ -device usb-tablet,id=input0,bus=usb.0,port=1 \ -spice port=5900,addr=127.0.0.1,disable-ticketing,seamless-migration=on \ -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=16,max_outputs=1,bus=pcie.0,addr=0x1 \ -device ich9-intel-hda,id=sound0,bus=pcie.0,addr=0x1b \ -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 \ -chardev spicevmc,id=charredir0,name=usbredir \ -device usb-redir,chardev=charredir0,id=redir0,bus=usb.0,port=2 \ -chardev spicevmc,id=charredir1,name=usbredir \ -device usb-redir,chardev=charredir1,id=redir1,bus=usb.0,port=3 \ -device virtio-balloon-pci,id=balloon0,bus=pci.3,addr=0x0 \ -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \ -msg timestamp=on char device redirected to /dev/pts/0 (label charserial0) 2020-12-21T11:10:37.974205Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12] 2020-12-21T11:10:37.974284Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13] 2020-12-21T11:10:37.976069Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12] 2020-12-21T11:10:37.976089Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13] 2020-12-21 11:10:38.207+0000: shutting down, reason=failed
If the message there isn't illuminating enough and you come back here, it would be helpful to "bring along" that qemu commandline (and any accompanying error) as well as the output of "virsh dumpxml $guestname"
Since libvirt won't create the guest instance in the first place, dumping the XML of the config like that won't work. However, it's possible to view the XML from the preview window in the machine details: <domain type="kvm"> <name>windows7</name> <uuid>bb32a4b5-7b2c-4ac2-9a65-8fd2bb9cdb86</uuid> <metadata> <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0"> <libosinfo:os id="http://microsoft.com/win/7"/> </libosinfo:libosinfo> </metadata> <memory>4194304</memory> <currentMemory>4194304</currentMemory> <vcpu>2</vcpu> <os> <type arch="x86_64" machine="q35">hvm</type> <boot dev="hd"/> </os> <features> <acpi/> <apic/> <hyperv> <relaxed state="on"/> <vapic state="on"/> <spinlocks state="on" retries="8191"/> </hyperv> <vmport state="off"/> </features> <cpu mode="host-model"/> <clock offset="localtime"> <timer name="rtc" tickpolicy="catchup"/> <timer name="pit" tickpolicy="delay"/> <timer name="hpet" present="no"/> <timer name="hypervclock" present="yes"/> </clock> <pm> <suspend-to-mem enabled="no"/> <suspend-to-disk enabled="no"/> </pm> <devices> <emulator>/usr/bin/qemu-system-x86_64</emulator> <disk type="file" device="disk"> <driver name="qemu" type="raw"/> <source file="/data/kvm/windows7.img"/> <target dev="sda" bus="sata"/> </disk> <controller type="usb" model="ich9-ehci1"/> <controller type="usb" model="ich9-uhci1"> <master startport="0"/> </controller> <controller type="usb" model="ich9-uhci2"> <master startport="2"/> </controller> <controller type="usb" model="ich9-uhci3"> <master startport="4"/> </controller> <interface type="network"> <source network="default"/> <mac address="52:54:00:4b:c5:26"/> <model type="e1000e"/> </interface> <console type="pty"/> <channel type="spicevmc"> <target type="virtio" name="com.redhat.spice.0"/> </channel> <input type="tablet" bus="usb"/> <graphics type="spice" port="-1" tlsPort="-1" autoport="yes"/> <sound model="ich9"/> <video> <model type="qxl"/> </video> <redirdev bus="usb" type="spicevmc"/> <redirdev bus="usb" type="spicevmc"/> </devices> </domain> Adrian -- .''`. John Paul Adrian Glaubitz : :' : Debian Developer - glaubitz@debian.org `. `' Freie Universitaet Berlin - glaubitz@physik.fu-berlin.de `- GPG: 62FF 8A75 84E0 2956 9546 0006 7426 3B37 F5B5 F913

On 12/21/20 6:15 AM, John Paul Adrian Glaubitz wrote:
(CC'ing a friend who's run into the problem as well)
Hi Laine!
On 12/20/20 7:37 PM, Laine Stump wrote:
The first step would be to look in /var/log/libvirt/qemu/$guestname.log. If qemu is encountering some error, it should be logging a message there before it exits. (unless the error is a segfault; in that case I guess you'll need to look for a coredump. It will also list exactly the qemu commandline that was used.
Except for some unrelated warnings, qemu does not emit any error message: [...]
2020-12-21 11:10:37.580+0000: starting up libvirt version: 6.9.0, package: 1+b2 (amd64 / i386 Build Daemon (x86-ubc-01) <buildd_amd64-x86-ubc-01@buildd.debian.org> Mon, 07 Dec 2020 09:45:52 +0000), qemu version: 5.2.0Debian 1:5.2+dfsg-2, kernel: 5.9.0-5-amd64, hostname: z6.physik.fu-berlin.de
At least you're running pretty much up-to-date of both libvirt and qemu, so you won't have to suffer through someone saying "come back when you've updated your software!" :-) (although I suppose it's possible you've run into some recent regression that I haven't heard of, and all the people who have heard of it aren't responding because they gone off on their Christmas holidays already).
LC_ALL=C \ PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \ HOME=/var/lib/libvirt/qemu/domain-1-windows7 \ XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-1-windows7/.local/share \ XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-1-windows7/.cache \ XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-1-windows7/.config \ QEMU_AUDIO_DRV=spice \ /usr/bin/qemu-system-x86_64 \ -name guest=windows7,debug-threads=on \ -S \ -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-1-windows7/master-key.aes \ -machine pc-q35-5.2,accel=kvm,usb=off,vmport=off,dump-guest-core=off,memory-backend=pc.ram \ -cpu Broadwell-IBRS,vme=on,ss=on,vmx=on,pdcm=on,f16c=on,rdrand=on,hypervisor=on,arat=on,tsc-adjust=on,umip=on,md-clear=on,stibp=on,arch-capabilities=on,ssbd=on,xsaveopt=on,pdpe1gb=on,abm=on,ibpb=on,amd-stibp=on,amd-ssbd=on,skip-l1dfl-vmentry=on,pschange-mc-no=on,hv-time,hv-relaxed,hv-vapic,hv-spinlocks=0x1fff \ -m 4096 \ -object memory-backend-ram,id=pc.ram,size=4294967296 \ -overcommit mem-lock=off \ -smp 2,sockets=2,cores=1,threads=1 \ -uuid bb32a4b5-7b2c-4ac2-9a65-8fd2bb9cdb86 \ -no-user-config \ -nodefaults \ -chardev socket,id=charmonitor,fd=33,server,nowait \ -mon chardev=charmonitor,id=monitor,mode=control \ -rtc base=localtime,driftfix=slew \ -global kvm-pit.lost_tick_policy=delay \ -no-hpet \ -no-shutdown \ -global ICH9-LPC.disable_s3=1 \ -global ICH9-LPC.disable_s4=1 \ -boot strict=on \ -device pcie-root-port,port=0x10,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x2 \ -device pcie-root-port,port=0x11,chassis=2,id=pci.2,bus=pcie.0,addr=0x2.0x1 \ -device pcie-root-port,port=0x12,chassis=3,id=pci.3,bus=pcie.0,addr=0x2.0x2 \ -device pcie-root-port,port=0x13,chassis=4,id=pci.4,bus=pcie.0,addr=0x2.0x3 \ -device ich9-usb-ehci1,id=usb,bus=pcie.0,addr=0x1d.0x7 \ -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pcie.0,multifunction=on,addr=0x1d \ -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pcie.0,addr=0x1d.0x1 \ -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pcie.0,addr=0x1d.0x2 \ -device virtio-serial-pci,id=virtio-serial0,bus=pci.2,addr=0x0 \ -blockdev '{"driver":"file","filename":"/data/kvm/windows7.img","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}' \ -blockdev '{"node-name":"libvirt-1-format","read-only":false,"driver":"raw","file":"libvirt-1-storage"}' \ -device ide-hd,bus=ide.0,drive=libvirt-1-format,id=sata0-0-0,bootindex=1 \ -netdev tap,fd=35,id=hostnet0 \ -device e1000e,netdev=hostnet0,id=net0,mac=52:54:00:4b:c5:26,bus=pci.1,addr=0x0 \ -chardev pty,id=charserial0 \ -device isa-serial,chardev=charserial0,id=serial0 \ -chardev spicevmc,id=charchannel0,name=vdagent \ -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 \ -device usb-tablet,id=input0,bus=usb.0,port=1 \ -spice port=5900,addr=127.0.0.1,disable-ticketing,seamless-migration=on \ -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=16,max_outputs=1,bus=pcie.0,addr=0x1 \ -device ich9-intel-hda,id=sound0,bus=pcie.0,addr=0x1b \ -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 \ -chardev spicevmc,id=charredir0,name=usbredir \ -device usb-redir,chardev=charredir0,id=redir0,bus=usb.0,port=2 \ -chardev spicevmc,id=charredir1,name=usbredir \ -device usb-redir,chardev=charredir1,id=redir1,bus=usb.0,port=3 \ -device virtio-balloon-pci,id=balloon0,bus=pci.3,addr=0x0 \ -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \ -msg timestamp=on
Maybe try running this command directly from the commandline (after removing the "-netdev tap" and "-device e1000e" lines, since they require pre-creation/opening of a tap device) and see if it stays up or dies. If it dies, then retry it removing more and more of the commandline options until it runs successfully (note that many of the options come in pairs - e.g. the "-device ich9-intel-hda" and "-device hda-duplex" lines must be removed together, similarly for the sets of "-chardev spicevmc" and "-device usb-redir"). If you can keep track, use a bisecting strategy to save time. Once you've narrowed it down to a single option, maybe it will be easier to diagnose. A similar strategy you could use at the libvirt level would be to use "virsh define" (run as root) and give it the XML you captured as the definition. Then do the same thing - repeatedly try to "virsh start", and then when it fails, use "virsh edit" to remove some of the devices/options and try again. Good luck!
char device redirected to /dev/pts/0 (label charserial0) 2020-12-21T11:10:37.974205Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12] 2020-12-21T11:10:37.974284Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13] 2020-12-21T11:10:37.976069Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12] 2020-12-21T11:10:37.976089Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13]
One suggestion I saw for avoiding these warnings is to set "host-passthrough" for the CPU model rather than "host-model". No idea if it works though.
2020-12-21 11:10:38.207+0000: shutting down, reason=failed
If the message there isn't illuminating enough and you come back here, it would be helpful to "bring along" that qemu commandline (and any accompanying error) as well as the output of "virsh dumpxml $guestname"
Since libvirt won't create the guest instance in the first place, dumping the XML of the config like that won't work. However, it's possible to view the XML from the preview window in the machine details:
<domain type="kvm"> <name>windows7</name> <uuid>bb32a4b5-7b2c-4ac2-9a65-8fd2bb9cdb86</uuid> <metadata> <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0"> <libosinfo:os id="http://microsoft.com/win/7"/> </libosinfo:libosinfo> </metadata> <memory>4194304</memory> <currentMemory>4194304</currentMemory> <vcpu>2</vcpu> <os> <type arch="x86_64" machine="q35">hvm</type> <boot dev="hd"/> </os> <features> <acpi/> <apic/> <hyperv> <relaxed state="on"/> <vapic state="on"/> <spinlocks state="on" retries="8191"/> </hyperv> <vmport state="off"/> </features> <cpu mode="host-model"/> <clock offset="localtime"> <timer name="rtc" tickpolicy="catchup"/> <timer name="pit" tickpolicy="delay"/> <timer name="hpet" present="no"/> <timer name="hypervclock" present="yes"/> </clock> <pm> <suspend-to-mem enabled="no"/> <suspend-to-disk enabled="no"/> </pm> <devices> <emulator>/usr/bin/qemu-system-x86_64</emulator> <disk type="file" device="disk"> <driver name="qemu" type="raw"/> <source file="/data/kvm/windows7.img"/> <target dev="sda" bus="sata"/> </disk> <controller type="usb" model="ich9-ehci1"/> <controller type="usb" model="ich9-uhci1"> <master startport="0"/> </controller> <controller type="usb" model="ich9-uhci2"> <master startport="2"/> </controller> <controller type="usb" model="ich9-uhci3"> <master startport="4"/> </controller> <interface type="network"> <source network="default"/> <mac address="52:54:00:4b:c5:26"/> <model type="e1000e"/> </interface> <console type="pty"/> <channel type="spicevmc"> <target type="virtio" name="com.redhat.spice.0"/> </channel> <input type="tablet" bus="usb"/> <graphics type="spice" port="-1" tlsPort="-1" autoport="yes"/> <sound model="ich9"/> <video> <model type="qxl"/> </video> <redirdev bus="usb" type="spicevmc"/> <redirdev bus="usb" type="spicevmc"/> </devices> </domain>
Adrian

Hello! On 12/20/20 12:53 PM, John Paul Adrian Glaubitz wrote:
I recently ran into a problem when connecting to libvirtd 6.9.0 on Debian unstable and trying to import an existing image with Windows 7.
Upon finishing the wizard and starting the instance, the import process fails with the following error message:
Unable to complete install: 'internal error: qemu unexpectedly closed the monitor' (...) Does anyone know how to increase verbosity here to get an error message that might be more helpful?
Just as a heads-up: This problem got resolved after a dist-upgrade which didn't upgrade libvirt but some Python packages so I assume that one or more Python packages needed an update or rebuild with Python 3.9. Thanks, Adrian -- .''`. John Paul Adrian Glaubitz : :' : Debian Developer - glaubitz@debian.org `. `' Freie Universitaet Berlin - glaubitz@physik.fu-berlin.de `- GPG: 62FF 8A75 84E0 2956 9546 0006 7426 3B37 F5B5 F913
participants (2)
-
John Paul Adrian Glaubitz
-
Laine Stump