Greetings, a few weeks ago I've upgraded my system, this resulted with qemu and libvirt being upgraded to 6.0.0 and ~7.5.0 respectfully. I have two vms running on my system, router and streamer. the router vm works great, the streamer vm doesn't. after the streamer vm start, the monitor screen gets black and thats it. no relevant error are found in the log, see: http://dpaste.com/ERWDEJQPC the xml ca be found at https://dpaste.com/FQDN6NTN2 in contrast the following oneliner works: qemu-system-x86_64 \ -machine pc-q35-5.0,accel=kvm,usb=off,smm=on,dump-guest-core=off \ -cpu host,migratable=on \ -m 15360 \ -smp 4,sockets=1,dies=1,cores=2,threads=2 \ -drive file=/home/streamer/streamer.img.qcow2.new,if=virtio,format=qcow2 \ -device vfio-pci,host=0000:00:02.0,romfile=/home/streamer/gpu-8086:5912-uefi.rom,multifunction=on \ -device vfio-pci,host=0000:00:1f.3,multifunction=on \ -usb \ -device usb-host,vendorid=0x046d,productid=0xc52e \ -device usb-host,vendorid=0x2548,productid=0x1002 \ -display none \ -netdev tap,id=hostnet0,ifname=virtsw-streamer,script=no,downscript=no \ -device e1000e,netdev=hostnet0,id=net0,mac=52:54:00:5a:4c:8c \ -blockdev '{"driver":"file","filename":"/usr/share/edk2-ovmf/OVMF_CODE.secboot.fd","node-name":"libvirt-pflash0-storage","auto-read-only":true,"discard":"unmap"}' \ -blockdev '{"driver":"file","filename":"/var/lib/libvirt/qemu/nvram/streamer-vm-q35_VARS.fd","node-name":"libvirt-pflash1-storage","auto-read-only":true,"discard":"unmap"}' another issue encountered caused by this param: "-audiodev id=audio1,driver=none" which is auto added to the qemu line. if I add the line to the above oneliner, the guest doesn't detects the pt sound card. without it it does and it works.

help on solving both issues will be appreciated. Thanks, Dagg

Greetings Martin, > Sent: Monday, August 02, 2021 at 12:17 PM > From: "Martin Kletzander" <mkletzan(a)redhat.com&gt; > To: "daggs" <daggs(a)gmx.com&gt; > Cc: libvirt-users(a)redhat.com > Subject: Re: issues with vm after upgrade > > On Sat, Jul 31, 2021 at 12:19:59PM +0200, daggs wrote: > >Greetings, > > > >a few weeks ago I've upgraded my system, this resulted with qemu and libvirt being upgraded to 6.0.0 and ~7.5.0 respectfully. > >I have two vms running on my system, router and streamer. > >the router vm works great, the streamer vm doesn't. > >after the streamer vm start, the monitor screen gets black and thats it. no relevant error are found in the log, see: http://dpaste.com/ERWDEJQPC > >the xml ca be found at https://dpaste.com/FQDN6NTN2 > >in contrast the following oneliner works: qemu-system-x86_64 \ > >-machine pc-q35-5.0,accel=kvm,usb=off,smm=on,dump-guest-core=off \ > >-cpu host,migratable=on \ > >-m 15360 \ > >-smp 4,sockets=1,dies=1,cores=2,threads=2 \ > >-drive file=/home/streamer/streamer.img.qcow2.new,if=virtio,format=qcow2 \ > >-device vfio-pci,host=0000:00:02.0,romfile=/home/streamer/gpu-8086:5912-uefi.rom,multifunction=on \ > >-device vfio-pci,host=0000:00:1f.3,multifunction=on \ > >-usb \ > >-device usb-host,vendorid=0x046d,productid=0xc52e \ > >-device usb-host,vendorid=0x2548,productid=0x1002 \ > >-display none \ > >-netdev tap,id=hostnet0,ifname=virtsw-streamer,script=no,downscript=no \ > >-device e1000e,netdev=hostnet0,id=net0,mac=52:54:00:5a:4c:8c \ > >-blockdev '{"driver":"file","filename":"/usr/share/edk2-ovmf/OVMF_CODE.secboot.fd","node-name":"libvirt-pflash0-storage","auto-read-only":true,"discard":"unmap"}' \ > >-blockdev '{"driver":"file","filename":"/var/lib/libvirt/qemu/nvram/streamer-vm-q35_VARS.fd","node-name":"libvirt-pflash1-storage","auto-read-only":true,"discard":"unmap"}' > > > >another issue encountered caused by this param: "-audiodev id=audio1,driver=none" which is auto added to the qemu line. > >if I add the line to the above oneliner, the guest doesn't detects the pt sound card. without it it does and it works. > > > > The links have expired, so I cannot look at the XML. However do you > have any <audio/> in the XML at all? I believe that without it we are > disabling any audio backends since no audio HW was requested. > here are the links: 1. xml: https://dpaste.com/D6JANX3Z3 2. log: https://dpaste.com/FMUDZY9PD as for the audio entry, " <audio id='1' type='none'/>" is in the xml I didn't added it

Thanks, Dagg

Greetings Martin, > Sent: Tuesday, August 03, 2021 at 11:37 AM > From: "Martin Kletzander" <mkletzan(a)redhat.com&gt; > To: "daggs" <daggs(a)gmx.com&gt; > Cc: libvirt-users(a)redhat.com > Subject: Re: issues with vm after upgrade > > On Mon, Aug 02, 2021 at 04:33:20PM +0200, daggs wrote: > >Greetings Martin, > > > >> Sent: Monday, August 02, 2021 at 12:17 PM > >> From: "Martin Kletzander" <mkletzan(a)redhat.com&gt; > >> To: "daggs" <daggs(a)gmx.com&gt; > >> Cc: libvirt-users(a)redhat.com > >> Subject: Re: issues with vm after upgrade > >> > >> On Sat, Jul 31, 2021 at 12:19:59PM +0200, daggs wrote: > >> >Greetings, > >> > > >> >a few weeks ago I've upgraded my system, this resulted with qemu and libvirt being upgraded to 6.0.0 and ~7.5.0 respectfully. > >> >I have two vms running on my system, router and streamer. > >> >the router vm works great, the streamer vm doesn't. > >> >after the streamer vm start, the monitor screen gets black and thats it. no relevant error are found in the log, see: http://dpaste.com/ERWDEJQPC > >> >the xml ca be found at https://dpaste.com/FQDN6NTN2 > >> >in contrast the following oneliner works: qemu-system-x86_64 \ > >> >-machine pc-q35-5.0,accel=kvm,usb=off,smm=on,dump-guest-core=off \ > >> >-cpu host,migratable=on \ > >> >-m 15360 \ > >> >-smp 4,sockets=1,dies=1,cores=2,threads=2 \ > >> >-drive file=/home/streamer/streamer.img.qcow2.new,if=virtio,format=qcow2 \ > >> >-device vfio-pci,host=0000:00:02.0,romfile=/home/streamer/gpu-8086:5912-uefi.rom,multifunction=on \ > >> >-device vfio-pci,host=0000:00:1f.3,multifunction=on \ > >> >-usb \ > >> >-device usb-host,vendorid=0x046d,productid=0xc52e \ > >> >-device usb-host,vendorid=0x2548,productid=0x1002 \ > >> >-display none \ > >> >-netdev tap,id=hostnet0,ifname=virtsw-streamer,script=no,downscript=no \ > >> >-device e1000e,netdev=hostnet0,id=net0,mac=52:54:00:5a:4c:8c \ > >> >-blockdev '{"driver":"file","filename":"/usr/share/edk2-ovmf/OVMF_CODE.secboot.fd","node-name":"libvirt-pflash0-storage","auto-read-only":true,"discard":"unmap"}' \ > >> >-blockdev '{"driver":"file","filename":"/var/lib/libvirt/qemu/nvram/streamer-vm-q35_VARS.fd","node-name":"libvirt-pflash1-storage","auto-read-only":true,"discard":"unmap"}' > >> > > >> >another issue encountered caused by this param: "-audiodev id=audio1,driver=none" which is auto added to the qemu line. > >> >if I add the line to the above oneliner, the guest doesn't detects the pt sound card. without it it does and it works. > >> > > >> > >> The links have expired, so I cannot look at the XML. However do you > >> have any <audio/> in the XML at all? I believe that without it we are > >> disabling any audio backends since no audio HW was requested. > >> > > > >here are the links: > >1. xml: https://dpaste.com/D6JANX3Z3 > >2. log: https://dpaste.com/FMUDZY9PD > > > >as for the audio entry, " <audio id='1' type='none'/>" is in the xml > >I didn't added it > > > > So let get this straight. You want the VM to use the HW GPU that you > assigned to it and also use the audio on that GPU (e.g. HDMI or DP > output). When you add the "-audiodev driver=none" the VM cannot see the > audio device on that GPU card? Or is that unrelated? > > There were some strict definitions added for audio devices lately, I > presume to make sure that no data gets leaked in/out the VM and in order > to get audio anywhere else than from/to VNC/spice you'd need to > configure that. It should not affect device assignment as far as I > understand it, although there might be some misunderstanding. > I'll try to explain, I'm passing the computer's O/B gpu and O/B sound card to the vm so I can use them inside the vm which serves as streamer. all was working well until I've upgraded both qemu and libvirt (target versions are started in the original mail) since then, the vm doesn't boot at all. I was able to bisect the qemu cmd libvirt generates partially and go the vm up. this points to a config issue. as part of the testing I've found out that when the vm boots, it has no sound card. following your suggestion, I've found that the xml has this: <audio id='1' type='none'/> I have never added it to the xml so I assume that it was added as part of the upgrade. based on this I assume that there is another config which prevents my vm from booting. unfortunately, I don't have the xml prior to the upgrade to diff them.

Greetings Daniel, > Sent: Tuesday, August 03, 2021 at 4:12 PM > From: "Daniel P. Berrange" <dan(a)berrange.com&gt; > To: "daggs" <daggs(a)gmx.com&gt; > Cc: "Martin Kletzander" <mkletzan(a)redhat.com&gt;, libvirt-users(a)redhat.com > Subject: Re: issues with vm after upgrade > > The <audio> element just refers to the *host* backend used for audio > playback. It would not affect guest hardware. Further, this has always > existed - it just wasn't exposed in the XML previously. > > the upgrade changed something, here is the qemu cmd before the upgrade: https://dpaste.com/F2N5T8CT8 here is after https://dpaste.com/F2N5T8CT8

> Sent: Tuesday, August 03, 2021 at 6:29 PM > From: "Daniel P. Berrange" <dan(a)berrange.com&gt; > To: "daggs" <daggs(a)gmx.com&gt; > Cc: "Martin Kletzander" <mkletzan(a)redhat.com&gt;, libvirt-users(a)redhat.com > Subject: Re: issues with vm after upgrade > > On Tue, Aug 03, 2021 at 05:21:52PM +0200, daggs wrote: > > Greetings Daniel, > > > > > Sent: Tuesday, August 03, 2021 at 4:12 PM > > > From: "Daniel P. Berrange" <dan(a)berrange.com&gt; > > > To: "daggs" <daggs(a)gmx.com&gt; > > > Cc: "Martin Kletzander" <mkletzan(a)redhat.com&gt;, libvirt-users(a)redhat.com > > > Subject: Re: issues with vm after upgrade > > > > > > The <audio> element just refers to the *host* backend used for audio > > > playback. It would not affect guest hardware. Further, this has always > > > existed - it just wasn't exposed in the XML previously. > > > > > > > > > > the upgrade changed something, here is the qemu cmd before the upgrade: https://dpaste.com/F2N5T8CT8 > > here is after https://dpaste.com/F2N5T8CT8 > > Those links are both the same I'm afraid > Duh! my bad! good log: http://dpaste.com/F2N5T8CT8 bad log: http://dpaste.com/6ECUHD2J8

Sent: Tuesday, August 03, 2021 at 6:51 PM From: "daggs" <daggs(a)gmx.com&gt; To: dan(a)berrange.com Cc: "Martin Kletzander" <mkletzan(a)redhat.com&gt;, libvirt-users(a)redhat.com Subject: Re: issues with vm after upgrade Greetings Daniel, > Sent: Tuesday, August 03, 2021 at 6:39 PM > From: "Daniel P. Berrange" <dan(a)berrange.com&gt; > To: "daggs" <daggs(a)gmx.com&gt; > Cc: "Martin Kletzander" <mkletzan(a)redhat.com&gt;, libvirt-users(a)redhat.com > Subject: Re: issues with vm after upgrade > > On Tue, Aug 03, 2021 at 05:34:53PM +0200, daggs wrote: > > > Sent: Tuesday, August 03, 2021 at 6:29 PM > > > From: "Daniel P. Berrange" <dan(a)berrange.com&gt; > > > To: "daggs" <daggs(a)gmx.com&gt; > > > Cc: "Martin Kletzander" <mkletzan(a)redhat.com&gt;, libvirt-users(a)redhat.com > > > Subject: Re: issues with vm after upgrade > > > > > > On Tue, Aug 03, 2021 at 05:21:52PM +0200, daggs wrote: > > > > Greetings Daniel, > > > > > > > > > Sent: Tuesday, August 03, 2021 at 4:12 PM > > > > > From: "Daniel P. Berrange" <dan(a)berrange.com&gt; > > > > > To: "daggs" <daggs(a)gmx.com&gt; > > > > > Cc: "Martin Kletzander" <mkletzan(a)redhat.com&gt;, libvirt-users(a)redhat.com > > > > > Subject: Re: issues with vm after upgrade > > > > > > > > > > The <audio> element just refers to the *host* backend used for audio > > > > > playback. It would not affect guest hardware. Further, this has always > > > > > existed - it just wasn't exposed in the XML previously. > > > > > > > > > > > > > > > > > > the upgrade changed something, here is the qemu cmd before the upgrade: https://dpaste.com/F2N5T8CT8 > > > > here is after https://dpaste.com/F2N5T8CT8 > > > > > > Those links are both the same I'm afraid > > > > > > > Duh! my bad! > > good log: http://dpaste.com/F2N5T8CT8 > > bad log: http://dpaste.com/6ECUHD2J8 > > The new log has a CLI flag > > -audiodev id=audio1,driver=none > > but the old log has an env variable > > QEMU_AUDIO_DRV=none > > which should be functionally identical, as QEMU will parse them both > to the same internal config. > > The obvious difference in the logs which can cause your guest to fail > is the different QEMU version. The old log shows QEMU 5.2.0, while the > new log shows QEMU 6.0.0 > thanks for the help, I went to look why the efi fw and found out that the nvram entry in /etc/libvirt/eqmu.conf was deleted upon update. I'm sure fixing this will solve he boot issue, hopefully audio issue too. Thanks, Dagg.

Sent: Wednesday, August 04, 2021 at 11:11 AM From: "Martin Kletzander" <mkletzan(a)redhat.com&gt; To: "daggs" <daggs(a)gmx.com&gt; Cc: libvirt-users(a)redhat.com, dan(a)berrange.com Subject: Re: issues with vm after upgrade On Tue, Aug 03, 2021 at 08:47:20PM +0200, daggs wrote: >> Sent: Tuesday, August 03, 2021 at 6:51 PM >> From: "daggs" <daggs(a)gmx.com&gt; >> To: dan(a)berrange.com >> Cc: "Martin Kletzander" <mkletzan(a)redhat.com&gt;, libvirt-users(a)redhat.com >> Subject: Re: issues with vm after upgrade >> >> Greetings Daniel, >> >> > Sent: Tuesday, August 03, 2021 at 6:39 PM >> > From: "Daniel P. Berrange" <dan(a)berrange.com&gt; >> > To: "daggs" <daggs(a)gmx.com&gt; >> > Cc: "Martin Kletzander" <mkletzan(a)redhat.com&gt;, libvirt-users(a)redhat.com >> > Subject: Re: issues with vm after upgrade >> > >> > On Tue, Aug 03, 2021 at 05:34:53PM +0200, daggs wrote: >> > > > Sent: Tuesday, August 03, 2021 at 6:29 PM >> > > > From: "Daniel P. Berrange" <dan(a)berrange.com&gt; >> > > > To: "daggs" <daggs(a)gmx.com&gt; >> > > > Cc: "Martin Kletzander" <mkletzan(a)redhat.com&gt;, libvirt-users(a)redhat.com >> > > > Subject: Re: issues with vm after upgrade >> > > > >> > > > On Tue, Aug 03, 2021 at 05:21:52PM +0200, daggs wrote: >> > > > > Greetings Daniel, >> > > > > >> > > > > > Sent: Tuesday, August 03, 2021 at 4:12 PM >> > > > > > From: "Daniel P. Berrange" <dan(a)berrange.com&gt; >> > > > > > To: "daggs" <daggs(a)gmx.com&gt; >> > > > > > Cc: "Martin Kletzander" <mkletzan(a)redhat.com&gt;, libvirt-users(a)redhat.com >> > > > > > Subject: Re: issues with vm after upgrade >> > > > > > >> > > > > > The <audio> element just refers to the *host* backend used for audio >> > > > > > playback. It would not affect guest hardware. Further, this has always >> > > > > > existed - it just wasn't exposed in the XML previously. >> > > > > > >> > > > > > >> > > > > >> > > > > the upgrade changed something, here is the qemu cmd before the upgrade: https://dpaste.com/F2N5T8CT8 >> > > > > here is after https://dpaste.com/F2N5T8CT8 >> > > > >> > > > Those links are both the same I'm afraid >> > > > >> > > >> > > Duh! my bad! >> > > good log: http://dpaste.com/F2N5T8CT8 >> > > bad log: http://dpaste.com/6ECUHD2J8 >> > >> > The new log has a CLI flag >> > >> > -audiodev id=audio1,driver=none >> > >> > but the old log has an env variable >> > >> > QEMU_AUDIO_DRV=none >> > >> > which should be functionally identical, as QEMU will parse them both >> > to the same internal config. >> > >> > The obvious difference in the logs which can cause your guest to fail >> > is the different QEMU version. The old log shows QEMU 5.2.0, while the >> > new log shows QEMU 6.0.0 >> > >> >> thanks for the help, I went to look why the efi fw and found out that the nvram entry in /etc/libvirt/eqmu.conf was deleted upon update. >> I'm sure fixing this will solve he boot issue, hopefully audio issue too. >> >> Thanks, >> >> Dagg. >> > >unfortunately, that didn't helped, vm still wont come up, latest log at http://dpaste.com/2XZA4VQZA >any ideas? > Seems like the issue is: 2021-07-16T10:29:19.259409Z qemu-system-x86_64: vfio: Cannot reset device 0000:00:1f.3, no available reset mechanism. 2021-07-16T10:29:19.369391Z qemu-system-x86_64: vfio: Cannot reset device 0000:00:1f.3, no available reset mechanism. did you upgrade anything else?

Sent: Wednesday, August 04, 2021 at 11:52 AM From: "Martin Kletzander" <mkletzan(a)redhat.com&gt; To: "daggs" <daggs(a)gmx.com&gt; Cc: libvirt-users(a)redhat.com, dan(a)berrange.com Subject: Re: issues with vm after upgrade On Wed, Aug 04, 2021 at 10:30:29AM +0200, daggs wrote: >Greetings Martin , > >> Sent: Wednesday, August 04, 2021 at 11:11 AM >> From: "Martin Kletzander" <mkletzan(a)redhat.com&gt; >> To: "daggs" <daggs(a)gmx.com&gt; >> Cc: libvirt-users(a)redhat.com, dan(a)berrange.com >> Subject: Re: issues with vm after upgrade >> >> On Tue, Aug 03, 2021 at 08:47:20PM +0200, daggs wrote: >> >> Sent: Tuesday, August 03, 2021 at 6:51 PM >> >> From: "daggs" <daggs(a)gmx.com&gt; >> >> To: dan(a)berrange.com >> >> Cc: "Martin Kletzander" <mkletzan(a)redhat.com&gt;, libvirt-users(a)redhat.com >> >> Subject: Re: issues with vm after upgrade >> >> >> >> Greetings Daniel, >> >> >> >> > Sent: Tuesday, August 03, 2021 at 6:39 PM >> >> > From: "Daniel P. Berrange" <dan(a)berrange.com&gt; >> >> > To: "daggs" <daggs(a)gmx.com&gt; >> >> > Cc: "Martin Kletzander" <mkletzan(a)redhat.com&gt;, libvirt-users(a)redhat.com >> >> > Subject: Re: issues with vm after upgrade >> >> > >> >> > On Tue, Aug 03, 2021 at 05:34:53PM +0200, daggs wrote: >> >> > > > Sent: Tuesday, August 03, 2021 at 6:29 PM >> >> > > > From: "Daniel P. Berrange" <dan(a)berrange.com&gt; >> >> > > > To: "daggs" <daggs(a)gmx.com&gt; >> >> > > > Cc: "Martin Kletzander" <mkletzan(a)redhat.com&gt;, libvirt-users(a)redhat.com >> >> > > > Subject: Re: issues with vm after upgrade >> >> > > > >> >> > > > On Tue, Aug 03, 2021 at 05:21:52PM +0200, daggs wrote: >> >> > > > > Greetings Daniel, >> >> > > > > >> >> > > > > > Sent: Tuesday, August 03, 2021 at 4:12 PM >> >> > > > > > From: "Daniel P. Berrange" <dan(a)berrange.com&gt; >> >> > > > > > To: "daggs" <daggs(a)gmx.com&gt; >> >> > > > > > Cc: "Martin Kletzander" <mkletzan(a)redhat.com&gt;, libvirt-users(a)redhat.com >> >> > > > > > Subject: Re: issues with vm after upgrade >> >> > > > > > >> >> > > > > > The <audio> element just refers to the *host* backend used for audio >> >> > > > > > playback. It would not affect guest hardware. Further, this has always >> >> > > > > > existed - it just wasn't exposed in the XML previously. >> >> > > > > > >> >> > > > > > >> >> > > > > >> >> > > > > the upgrade changed something, here is the qemu cmd before the upgrade: https://dpaste.com/F2N5T8CT8 >> >> > > > > here is after https://dpaste.com/F2N5T8CT8 >> >> > > > >> >> > > > Those links are both the same I'm afraid >> >> > > > >> >> > > >> >> > > Duh! my bad! >> >> > > good log: http://dpaste.com/F2N5T8CT8 >> >> > > bad log: http://dpaste.com/6ECUHD2J8 >> >> > >> >> > The new log has a CLI flag >> >> > >> >> > -audiodev id=audio1,driver=none >> >> > >> >> > but the old log has an env variable >> >> > >> >> > QEMU_AUDIO_DRV=none >> >> > >> >> > which should be functionally identical, as QEMU will parse them both >> >> > to the same internal config. >> >> > >> >> > The obvious difference in the logs which can cause your guest to fail >> >> > is the different QEMU version. The old log shows QEMU 5.2.0, while the >> >> > new log shows QEMU 6.0.0 >> >> > >> >> >> >> thanks for the help, I went to look why the efi fw and found out that the nvram entry in /etc/libvirt/eqmu.conf was deleted upon update. >> >> I'm sure fixing this will solve he boot issue, hopefully audio issue too. >> >> >> >> Thanks, >> >> >> >> Dagg. >> >> >> > >> >unfortunately, that didn't helped, vm still wont come up, latest log at http://dpaste.com/2XZA4VQZA >> >any ideas? >> > >> >> Seems like the issue is: >> >> 2021-07-16T10:29:19.259409Z qemu-system-x86_64: vfio: Cannot reset device 0000:00:1f.3, no available reset mechanism. >> 2021-07-16T10:29:19.369391Z qemu-system-x86_64: vfio: Cannot reset device 0000:00:1f.3, no available reset mechanism. >> >> did you upgrade anything else? >> > >are you sure? you can see the same prints in the good log at http://dpaste.com/F2N5T8CT8 > Oh, sorry, my bad. Then I do not see why it would not start. You said you managed to make the VM start on your own, what was the change that made it boot? Maybe there's a bug somewhere in qemu...

Greetings Martin, Dan > Sent: Wednesday, August 04, 2021 at 1:54 PM > From: "daggs" <daggs(a)gmx.com&gt; > To: "Martin Kletzander" <mkletzan(a)redhat.com&gt; > Cc: libvirt-users(a)redhat.com, dan(a)berrange.com > Subject: Re: issues with vm after upgrade > > Greetings Martin, > > > Sent: Wednesday, August 04, 2021 at 11:52 AM > > From: "Martin Kletzander" <mkletzan(a)redhat.com&gt; > > To: "daggs" <daggs(a)gmx.com&gt; > > Cc: libvirt-users(a)redhat.com, dan(a)berrange.com > > Subject: Re: issues with vm after upgrade > > > > On Wed, Aug 04, 2021 at 10:30:29AM +0200, daggs wrote: > > >Greetings Martin , > > > > > >> Sent: Wednesday, August 04, 2021 at 11:11 AM > > >> From: "Martin Kletzander" <mkletzan(a)redhat.com&gt; > > >> To: "daggs" <daggs(a)gmx.com&gt; > > >> Cc: libvirt-users(a)redhat.com, dan(a)berrange.com > > >> Subject: Re: issues with vm after upgrade > > >> > > >> On Tue, Aug 03, 2021 at 08:47:20PM +0200, daggs wrote: > > >> >> Sent: Tuesday, August 03, 2021 at 6:51 PM > > >> >> From: "daggs" <daggs(a)gmx.com&gt; > > >> >> To: dan(a)berrange.com > > >> >> Cc: "Martin Kletzander" <mkletzan(a)redhat.com&gt;, libvirt-users(a)redhat.com > > >> >> Subject: Re: issues with vm after upgrade > > >> >> > > >> >> Greetings Daniel, > > >> >> > > >> >> > Sent: Tuesday, August 03, 2021 at 6:39 PM > > >> >> > From: "Daniel P. Berrange" <dan(a)berrange.com&gt; > > >> >> > To: "daggs" <daggs(a)gmx.com&gt; > > >> >> > Cc: "Martin Kletzander" <mkletzan(a)redhat.com&gt;, libvirt-users(a)redhat.com > > >> >> > Subject: Re: issues with vm after upgrade > > >> >> > > > >> >> > On Tue, Aug 03, 2021 at 05:34:53PM +0200, daggs wrote: > > >> >> > > > Sent: Tuesday, August 03, 2021 at 6:29 PM > > >> >> > > > From: "Daniel P. Berrange" <dan(a)berrange.com&gt; > > >> >> > > > To: "daggs" <daggs(a)gmx.com&gt; > > >> >> > > > Cc: "Martin Kletzander" <mkletzan(a)redhat.com&gt;, libvirt-users(a)redhat.com > > >> >> > > > Subject: Re: issues with vm after upgrade > > >> >> > > > > > >> >> > > > On Tue, Aug 03, 2021 at 05:21:52PM +0200, daggs wrote: > > >> >> > > > > Greetings Daniel, > > >> >> > > > > > > >> >> > > > > > Sent: Tuesday, August 03, 2021 at 4:12 PM > > >> >> > > > > > From: "Daniel P. Berrange" <dan(a)berrange.com&gt; > > >> >> > > > > > To: "daggs" <daggs(a)gmx.com&gt; > > >> >> > > > > > Cc: "Martin Kletzander" <mkletzan(a)redhat.com&gt;, libvirt-users(a)redhat.com > > >> >> > > > > > Subject: Re: issues with vm after upgrade > > >> >> > > > > > > > >> >> > > > > > The <audio> element just refers to the *host* backend used for audio > > >> >> > > > > > playback. It would not affect guest hardware. Further, this has always > > >> >> > > > > > existed - it just wasn't exposed in the XML previously. > > >> >> > > > > > > > >> >> > > > > > > > >> >> > > > > > > >> >> > > > > the upgrade changed something, here is the qemu cmd before the upgrade: https://dpaste.com/F2N5T8CT8 > > >> >> > > > > here is after https://dpaste.com/F2N5T8CT8 > > >> >> > > > > > >> >> > > > Those links are both the same I'm afraid > > >> >> > > > > > >> >> > > > > >> >> > > Duh! my bad! > > >> >> > > good log: http://dpaste.com/F2N5T8CT8 > > >> >> > > bad log: http://dpaste.com/6ECUHD2J8 > > >> >> > > > >> >> > The new log has a CLI flag > > >> >> > > > >> >> > -audiodev id=audio1,driver=none > > >> >> > > > >> >> > but the old log has an env variable > > >> >> > > > >> >> > QEMU_AUDIO_DRV=none > > >> >> > > > >> >> > which should be functionally identical, as QEMU will parse them both > > >> >> > to the same internal config. > > >> >> > > > >> >> > The obvious difference in the logs which can cause your guest to fail > > >> >> > is the different QEMU version. The old log shows QEMU 5.2.0, while the > > >> >> > new log shows QEMU 6.0.0 > > >> >> > > > >> >> > > >> >> thanks for the help, I went to look why the efi fw and found out that the nvram entry in /etc/libvirt/eqmu.conf was deleted upon update. > > >> >> I'm sure fixing this will solve he boot issue, hopefully audio issue too. > > >> >> > > >> >> Thanks, > > >> >> > > >> >> Dagg. > > >> >> > > >> > > > >> >unfortunately, that didn't helped, vm still wont come up, latest log at http://dpaste.com/2XZA4VQZA > > >> >any ideas? > > >> > > > >> > > >> Seems like the issue is: > > >> > > >> 2021-07-16T10:29:19.259409Z qemu-system-x86_64: vfio: Cannot reset device 0000:00:1f.3, no available reset mechanism. > > >> 2021-07-16T10:29:19.369391Z qemu-system-x86_64: vfio: Cannot reset device 0000:00:1f.3, no available reset mechanism. > > >> > > >> did you upgrade anything else? > > >> > > > > > >are you sure? you can see the same prints in the good log at http://dpaste.com/F2N5T8CT8 > > > > > > > Oh, sorry, my bad. Then I do not see why it would not start. You said > > you managed to make the VM start on your own, what was the change that > > made it boot? Maybe there's a bug somewhere in qemu... > > > > since the upgrade I wasn't able to boot it using libvirt. however the following qemucmd works: > qemu-system-x86_64 \ > -machine pc-q35-5.0,accel=kvm,usb=off,smm=on,dump-guest-core=off \ > -cpu host,migratable=on \ > -m 15360 \ > -smp 4,sockets=1,dies=1,cores=2,threads=2 \ > -drive file=/home/streamer/streamer.img.qcow2.new,if=virtio,format=qcow2 \ > -device vfio-pci,host=0000:00:02.0,romfile=/home/streamer/gpu-8086:5912-uefi.rom,multifunction=on \ > -device vfio-pci,host=0000:00:1f.3,multifunction=on \ > -usb \ > -device usb-host,vendorid=0x046d,productid=0xc52e \ > -device usb-host,vendorid=0x2548,productid=0x1002 \ > -display none \ > -netdev tap,id=hostnet0,ifname=virtsw-streamer,script=no,downscript=no \ > -device e1000e,netdev=hostnet0,id=net0,mac=52:54:00:5a:4c:8c \ > -blockdev '{"driver":"file","filename":"/usr/share/edk2-ovmf/OVMF_CODE.secboot.fd","node-name":"libvirt-pflash0-storage","auto-read-only":true,"discard":"unmap"}' \ > -blockdev '{"driver":"file","filename":"/var/lib/libvirt/qemu/nvram/streamer-vm-q35_VARS.fd","node-name":"libvirt-pflash1-storage","auto-readonly":true,"discard":"unmap"}' > > Dagg > I was able to get the vm running again by creating the xml again. the only issue I'm facing is the fact that only the one playback device is visible: **** List of PLAYBACK Hardware Devices **** card 0: PCH [HDA Intel PCH], device 0: ALC892 Analog [ALC892 Analog] Subdevices: 1/1 Subdevice #0: subdevice #0 prior to the upgrade I had hdmi entries too. I've check with two different oses and the issue persists. as mentioned before, if I drop this line: -audiodev id=audio1,driver=none from the cmd, I see all the playback channels. any idea?

Greetings Martin, > Sent: Wednesday, August 11, 2021 at 10:14 AM > From: "Martin Kletzander" <mkletzan(a)redhat.com&gt; > To: "daggs" <daggs(a)gmx.com&gt; > Cc: dan(a)berrange.com, libvirt-users(a)redhat.com > Subject: Re: issues with vm after upgrade > > On Tue, Aug 10, 2021 at 09:21:00PM +0200, daggs wrote: > >Greetings Martin, Dan > > > >> Sent: Wednesday, August 04, 2021 at 1:54 PM > >> From: "daggs" <daggs(a)gmx.com&gt; > >> To: "Martin Kletzander" <mkletzan(a)redhat.com&gt; > >> Cc: libvirt-users(a)redhat.com, dan(a)berrange.com > >> Subject: Re: issues with vm after upgrade > >> > >> Greetings Martin, > >> > >> > Sent: Wednesday, August 04, 2021 at 11:52 AM > >> > From: "Martin Kletzander" <mkletzan(a)redhat.com&gt; > >> > To: "daggs" <daggs(a)gmx.com&gt; > >> > Cc: libvirt-users(a)redhat.com, dan(a)berrange.com > >> > Subject: Re: issues with vm after upgrade > >> > > >> > On Wed, Aug 04, 2021 at 10:30:29AM +0200, daggs wrote: > >> > >Greetings Martin , > >> > > > >> > >> Sent: Wednesday, August 04, 2021 at 11:11 AM > >> > >> From: "Martin Kletzander" <mkletzan(a)redhat.com&gt; > >> > >> To: "daggs" <daggs(a)gmx.com&gt; > >> > >> Cc: libvirt-users(a)redhat.com, dan(a)berrange.com > >> > >> Subject: Re: issues with vm after upgrade > >> > >> > >> > >> On Tue, Aug 03, 2021 at 08:47:20PM +0200, daggs wrote: > >> > >> >> Sent: Tuesday, August 03, 2021 at 6:51 PM > >> > >> >> From: "daggs" <daggs(a)gmx.com&gt; > >> > >> >> To: dan(a)berrange.com > >> > >> >> Cc: "Martin Kletzander" <mkletzan(a)redhat.com&gt;, libvirt-users(a)redhat.com > >> > >> >> Subject: Re: issues with vm after upgrade > >> > >> >> > >> > >> >> Greetings Daniel, > >> > >> >> > >> > >> >> > Sent: Tuesday, August 03, 2021 at 6:39 PM > >> > >> >> > From: "Daniel P. Berrange" <dan(a)berrange.com&gt; > >> > >> >> > To: "daggs" <daggs(a)gmx.com&gt; > >> > >> >> > Cc: "Martin Kletzander" <mkletzan(a)redhat.com&gt;, libvirt-users(a)redhat.com > >> > >> >> > Subject: Re: issues with vm after upgrade > >> > >> >> > > >> > >> >> > On Tue, Aug 03, 2021 at 05:34:53PM +0200, daggs wrote: > >> > >> >> > > > Sent: Tuesday, August 03, 2021 at 6:29 PM > >> > >> >> > > > From: "Daniel P. Berrange" <dan(a)berrange.com&gt; > >> > >> >> > > > To: "daggs" <daggs(a)gmx.com&gt; > >> > >> >> > > > Cc: "Martin Kletzander" <mkletzan(a)redhat.com&gt;, libvirt-users(a)redhat.com > >> > >> >> > > > Subject: Re: issues with vm after upgrade > >> > >> >> > > > > >> > >> >> > > > On Tue, Aug 03, 2021 at 05:21:52PM +0200, daggs wrote: > >> > >> >> > > > > Greetings Daniel, > >> > >> >> > > > > > >> > >> >> > > > > > Sent: Tuesday, August 03, 2021 at 4:12 PM > >> > >> >> > > > > > From: "Daniel P. Berrange" <dan(a)berrange.com&gt; > >> > >> >> > > > > > To: "daggs" <daggs(a)gmx.com&gt; > >> > >> >> > > > > > Cc: "Martin Kletzander" <mkletzan(a)redhat.com&gt;, libvirt-users(a)redhat.com > >> > >> >> > > > > > Subject: Re: issues with vm after upgrade > >> > >> >> > > > > > > >> > >> >> > > > > > The <audio> element just refers to the *host* backend used for audio > >> > >> >> > > > > > playback. It would not affect guest hardware. Further, this has always > >> > >> >> > > > > > existed - it just wasn't exposed in the XML previously. > >> > >> >> > > > > > > >> > >> >> > > > > > > >> > >> >> > > > > > >> > >> >> > > > > the upgrade changed something, here is the qemu cmd before the upgrade: https://dpaste.com/F2N5T8CT8 > >> > >> >> > > > > here is after https://dpaste.com/F2N5T8CT8 > >> > >> >> > > > > >> > >> >> > > > Those links are both the same I'm afraid > >> > >> >> > > > > >> > >> >> > > > >> > >> >> > > Duh! my bad! > >> > >> >> > > good log: http://dpaste.com/F2N5T8CT8 > >> > >> >> > > bad log: http://dpaste.com/6ECUHD2J8 > >> > >> >> > > >> > >> >> > The new log has a CLI flag > >> > >> >> > > >> > >> >> > -audiodev id=audio1,driver=none > >> > >> >> > > >> > >> >> > but the old log has an env variable > >> > >> >> > > >> > >> >> > QEMU_AUDIO_DRV=none > >> > >> >> > > >> > >> >> > which should be functionally identical, as QEMU will parse them both > >> > >> >> > to the same internal config. > >> > >> >> > > >> > >> >> > The obvious difference in the logs which can cause your guest to fail > >> > >> >> > is the different QEMU version. The old log shows QEMU 5.2.0, while the > >> > >> >> > new log shows QEMU 6.0.0 > >> > >> >> > > >> > >> >> > >> > >> >> thanks for the help, I went to look why the efi fw and found out that the nvram entry in /etc/libvirt/eqmu.conf was deleted upon update. > >> > >> >> I'm sure fixing this will solve he boot issue, hopefully audio issue too. > >> > >> >> > >> > >> >> Thanks, > >> > >> >> > >> > >> >> Dagg. > >> > >> >> > >> > >> > > >> > >> >unfortunately, that didn't helped, vm still wont come up, latest log at http://dpaste.com/2XZA4VQZA > >> > >> >any ideas? > >> > >> > > >> > >> > >> > >> Seems like the issue is: > >> > >> > >> > >> 2021-07-16T10:29:19.259409Z qemu-system-x86_64: vfio: Cannot reset device 0000:00:1f.3, no available reset mechanism. > >> > >> 2021-07-16T10:29:19.369391Z qemu-system-x86_64: vfio: Cannot reset device 0000:00:1f.3, no available reset mechanism. > >> > >> > >> > >> did you upgrade anything else? > >> > >> > >> > > > >> > >are you sure? you can see the same prints in the good log at http://dpaste.com/F2N5T8CT8 > >> > > > >> > > >> > Oh, sorry, my bad. Then I do not see why it would not start. You said > >> > you managed to make the VM start on your own, what was the change that > >> > made it boot? Maybe there's a bug somewhere in qemu... > >> > > >> > >> since the upgrade I wasn't able to boot it using libvirt. however the following qemucmd works: > >> qemu-system-x86_64 \ > >> -machine pc-q35-5.0,accel=kvm,usb=off,smm=on,dump-guest-core=off \ > >> -cpu host,migratable=on \ > >> -m 15360 \ > >> -smp 4,sockets=1,dies=1,cores=2,threads=2 \ > >> -drive file=/home/streamer/streamer.img.qcow2.new,if=virtio,format=qcow2 \ > >> -device vfio-pci,host=0000:00:02.0,romfile=/home/streamer/gpu-8086:5912-uefi.rom,multifunction=on \ > >> -device vfio-pci,host=0000:00:1f.3,multifunction=on \ > >> -usb \ > >> -device usb-host,vendorid=0x046d,productid=0xc52e \ > >> -device usb-host,vendorid=0x2548,productid=0x1002 \ > >> -display none \ > >> -netdev tap,id=hostnet0,ifname=virtsw-streamer,script=no,downscript=no \ > >> -device e1000e,netdev=hostnet0,id=net0,mac=52:54:00:5a:4c:8c \ > >> -blockdev '{"driver":"file","filename":"/usr/share/edk2-ovmf/OVMF_CODE.secboot.fd","node-name":"libvirt-pflash0-storage","auto-read-only":true,"discard":"unmap"}' \ > >> -blockdev '{"driver":"file","filename":"/var/lib/libvirt/qemu/nvram/streamer-vm-q35_VARS.fd","node-name":"libvirt-pflash1-storage","auto-readonly":true,"discard":"unmap"}' > >> > >> Dagg > >> > > > >I was able to get the vm running again by creating the xml again. the only issue I'm facing is the fact that only the one playback device is visible: > >**** List of PLAYBACK Hardware Devices **** > >card 0: PCH [HDA Intel PCH], device 0: ALC892 Analog [ALC892 Analog] > > Subdevices: 1/1 > > Subdevice #0: subdevice #0 > >prior to the upgrade I had hdmi entries too. > >I've check with two different oses and the issue persists. > >as mentioned before, if I drop this line: -audiodev id=audio1,driver=none from the cmd, I see all the playback channels. > > > >any idea? > > > > So to clean up this thread let me reply to the two issues separately. > > 1) if your device-assigned audio is not visible with '-audiodev > id=audio1,driver=none', but *is* visible with QEMU_AUDIO_DRV=none, > then that is a bug in QEMU and needs to be dealt with there. I'll ping the eqmu devs on this. > > 2) To your issue with starting the domain it would be good to know what > is the error you get from virsh (or however you are starting the > domain) and the debug logs of libvirtd, ideally just for the part of > the domain starting. that is the issue, there wasn't any error. the vm just didn't booted.

I can diff the original xml with the new one to see the diffs and post them here if you wish

Dagg.

Sent: Wednesday, August 11, 2021 at 6:08 PM From: "daggs" <daggs(a)gmx.com&gt; To: "Martin Kletzander" <mkletzan(a)redhat.com&gt; Cc: dan(a)berrange.com, libvirt-users(a)redhat.com Subject: Re: issues with vm after upgrade Greetings Martin, > Sent: Wednesday, August 11, 2021 at 4:13 PM > From: "Martin Kletzander" <mkletzan(a)redhat.com&gt; > To: "daggs" <daggs(a)gmx.com&gt; > Cc: dan(a)berrange.com, libvirt-users(a)redhat.com > Subject: Re: issues with vm after upgrade > > On Wed, Aug 11, 2021 at 03:09:34PM +0200, daggs wrote: > >Greetings Martin, > > > >> Sent: Wednesday, August 11, 2021 at 10:14 AM > >> From: "Martin Kletzander" <mkletzan(a)redhat.com&gt; > >> To: "daggs" <daggs(a)gmx.com&gt; > >> Cc: dan(a)berrange.com, libvirt-users(a)redhat.com > >> Subject: Re: issues with vm after upgrade > >> > >> On Tue, Aug 10, 2021 at 09:21:00PM +0200, daggs wrote: > >> >Greetings Martin, Dan > >> > > >> >> Sent: Wednesday, August 04, 2021 at 1:54 PM > >> >> From: "daggs" <daggs(a)gmx.com&gt; > >> >> To: "Martin Kletzander" <mkletzan(a)redhat.com&gt; > >> >> Cc: libvirt-users(a)redhat.com, dan(a)berrange.com > >> >> Subject: Re: issues with vm after upgrade > >> >> > >> >> Greetings Martin, > >> >> > >> >> > Sent: Wednesday, August 04, 2021 at 11:52 AM > >> >> > From: "Martin Kletzander" <mkletzan(a)redhat.com&gt; > >> >> > To: "daggs" <daggs(a)gmx.com&gt; > >> >> > Cc: libvirt-users(a)redhat.com, dan(a)berrange.com > >> >> > Subject: Re: issues with vm after upgrade > >> >> > > >> >> > On Wed, Aug 04, 2021 at 10:30:29AM +0200, daggs wrote: > >> >> > >Greetings Martin , > >> >> > > > >> >> > >> Sent: Wednesday, August 04, 2021 at 11:11 AM > >> >> > >> From: "Martin Kletzander" <mkletzan(a)redhat.com&gt; > >> >> > >> To: "daggs" <daggs(a)gmx.com&gt; > >> >> > >> Cc: libvirt-users(a)redhat.com, dan(a)berrange.com > >> >> > >> Subject: Re: issues with vm after upgrade > >> >> > >> > >> >> > >> On Tue, Aug 03, 2021 at 08:47:20PM +0200, daggs wrote: > >> >> > >> >> Sent: Tuesday, August 03, 2021 at 6:51 PM > >> >> > >> >> From: "daggs" <daggs(a)gmx.com&gt; > >> >> > >> >> To: dan(a)berrange.com > >> >> > >> >> Cc: "Martin Kletzander" <mkletzan(a)redhat.com&gt;, libvirt-users(a)redhat.com > >> >> > >> >> Subject: Re: issues with vm after upgrade > >> >> > >> >> > >> >> > >> >> Greetings Daniel, > >> >> > >> >> > >> >> > >> >> > Sent: Tuesday, August 03, 2021 at 6:39 PM > >> >> > >> >> > From: "Daniel P. Berrange" <dan(a)berrange.com&gt; > >> >> > >> >> > To: "daggs" <daggs(a)gmx.com&gt; > >> >> > >> >> > Cc: "Martin Kletzander" <mkletzan(a)redhat.com&gt;, libvirt-users(a)redhat.com > >> >> > >> >> > Subject: Re: issues with vm after upgrade > >> >> > >> >> > > >> >> > >> >> > On Tue, Aug 03, 2021 at 05:34:53PM +0200, daggs wrote: > >> >> > >> >> > > > Sent: Tuesday, August 03, 2021 at 6:29 PM > >> >> > >> >> > > > From: "Daniel P. Berrange" <dan(a)berrange.com&gt; > >> >> > >> >> > > > To: "daggs" <daggs(a)gmx.com&gt; > >> >> > >> >> > > > Cc: "Martin Kletzander" <mkletzan(a)redhat.com&gt;, libvirt-users(a)redhat.com > >> >> > >> >> > > > Subject: Re: issues with vm after upgrade > >> >> > >> >> > > > > >> >> > >> >> > > > On Tue, Aug 03, 2021 at 05:21:52PM +0200, daggs wrote: > >> >> > >> >> > > > > Greetings Daniel, > >> >> > >> >> > > > > > >> >> > >> >> > > > > > Sent: Tuesday, August 03, 2021 at 4:12 PM > >> >> > >> >> > > > > > From: "Daniel P. Berrange" <dan(a)berrange.com&gt; > >> >> > >> >> > > > > > To: "daggs" <daggs(a)gmx.com&gt; > >> >> > >> >> > > > > > Cc: "Martin Kletzander" <mkletzan(a)redhat.com&gt;, libvirt-users(a)redhat.com > >> >> > >> >> > > > > > Subject: Re: issues with vm after upgrade > >> >> > >> >> > > > > > > >> >> > >> >> > > > > > The <audio> element just refers to the *host* backend used for audio > >> >> > >> >> > > > > > playback. It would not affect guest hardware. Further, this has always > >> >> > >> >> > > > > > existed - it just wasn't exposed in the XML previously. > >> >> > >> >> > > > > > > >> >> > >> >> > > > > > > >> >> > >> >> > > > > > >> >> > >> >> > > > > the upgrade changed something, here is the qemu cmd before the upgrade: https://dpaste.com/F2N5T8CT8 > >> >> > >> >> > > > > here is after https://dpaste.com/F2N5T8CT8 > >> >> > >> >> > > > > >> >> > >> >> > > > Those links are both the same I'm afraid > >> >> > >> >> > > > > >> >> > >> >> > > > >> >> > >> >> > > Duh! my bad! > >> >> > >> >> > > good log: http://dpaste.com/F2N5T8CT8 > >> >> > >> >> > > bad log: http://dpaste.com/6ECUHD2J8 > >> >> > >> >> > > >> >> > >> >> > The new log has a CLI flag > >> >> > >> >> > > >> >> > >> >> > -audiodev id=audio1,driver=none > >> >> > >> >> > > >> >> > >> >> > but the old log has an env variable > >> >> > >> >> > > >> >> > >> >> > QEMU_AUDIO_DRV=none > >> >> > >> >> > > >> >> > >> >> > which should be functionally identical, as QEMU will parse them both > >> >> > >> >> > to the same internal config. > >> >> > >> >> > > >> >> > >> >> > The obvious difference in the logs which can cause your guest to fail > >> >> > >> >> > is the different QEMU version. The old log shows QEMU 5.2.0, while the > >> >> > >> >> > new log shows QEMU 6.0.0 > >> >> > >> >> > > >> >> > >> >> > >> >> > >> >> thanks for the help, I went to look why the efi fw and found out that the nvram entry in /etc/libvirt/eqmu.conf was deleted upon update. > >> >> > >> >> I'm sure fixing this will solve he boot issue, hopefully audio issue too. > >> >> > >> >> > >> >> > >> >> Thanks, > >> >> > >> >> > >> >> > >> >> Dagg. > >> >> > >> >> > >> >> > >> > > >> >> > >> >unfortunately, that didn't helped, vm still wont come up, latest log at http://dpaste.com/2XZA4VQZA > >> >> > >> >any ideas? > >> >> > >> > > >> >> > >> > >> >> > >> Seems like the issue is: > >> >> > >> > >> >> > >> 2021-07-16T10:29:19.259409Z qemu-system-x86_64: vfio: Cannot reset device 0000:00:1f.3, no available reset mechanism. > >> >> > >> 2021-07-16T10:29:19.369391Z qemu-system-x86_64: vfio: Cannot reset device 0000:00:1f.3, no available reset mechanism. > >> >> > >> > >> >> > >> did you upgrade anything else? > >> >> > >> > >> >> > > > >> >> > >are you sure? you can see the same prints in the good log at http://dpaste.com/F2N5T8CT8 > >> >> > > > >> >> > > >> >> > Oh, sorry, my bad. Then I do not see why it would not start. You said > >> >> > you managed to make the VM start on your own, what was the change that > >> >> > made it boot? Maybe there's a bug somewhere in qemu... > >> >> > > >> >> > >> >> since the upgrade I wasn't able to boot it using libvirt. however the following qemucmd works: > >> >> qemu-system-x86_64 \ > >> >> -machine pc-q35-5.0,accel=kvm,usb=off,smm=on,dump-guest-core=off \ > >> >> -cpu host,migratable=on \ > >> >> -m 15360 \ > >> >> -smp 4,sockets=1,dies=1,cores=2,threads=2 \ > >> >> -drive file=/home/streamer/streamer.img.qcow2.new,if=virtio,format=qcow2 \ > >> >> -device vfio-pci,host=0000:00:02.0,romfile=/home/streamer/gpu-8086:5912-uefi.rom,multifunction=on \ > >> >> -device vfio-pci,host=0000:00:1f.3,multifunction=on \ > >> >> -usb \ > >> >> -device usb-host,vendorid=0x046d,productid=0xc52e \ > >> >> -device usb-host,vendorid=0x2548,productid=0x1002 \ > >> >> -display none \ > >> >> -netdev tap,id=hostnet0,ifname=virtsw-streamer,script=no,downscript=no \ > >> >> -device e1000e,netdev=hostnet0,id=net0,mac=52:54:00:5a:4c:8c \ > >> >> -blockdev '{"driver":"file","filename":"/usr/share/edk2-ovmf/OVMF_CODE.secboot.fd","node-name":"libvirt-pflash0-storage","auto-read-only":true,"discard":"unmap"}' \ > >> >> -blockdev '{"driver":"file","filename":"/var/lib/libvirt/qemu/nvram/streamer-vm-q35_VARS.fd","node-name":"libvirt-pflash1-storage","auto-readonly":true,"discard":"unmap"}' > >> >> > >> >> Dagg > >> >> > >> > > >> >I was able to get the vm running again by creating the xml again. the only issue I'm facing is the fact that only the one playback device is visible: > >> >**** List of PLAYBACK Hardware Devices **** > >> >card 0: PCH [HDA Intel PCH], device 0: ALC892 Analog [ALC892 Analog] > >> > Subdevices: 1/1 > >> > Subdevice #0: subdevice #0 > >> >prior to the upgrade I had hdmi entries too. > >> >I've check with two different oses and the issue persists. > >> >as mentioned before, if I drop this line: -audiodev id=audio1,driver=none from the cmd, I see all the playback channels. > >> > > >> >any idea? > >> > > >> > >> So to clean up this thread let me reply to the two issues separately. > >> > >> 1) if your device-assigned audio is not visible with '-audiodev > >> id=audio1,driver=none', but *is* visible with QEMU_AUDIO_DRV=none, > >> then that is a bug in QEMU and needs to be dealt with there. > >I'll ping the eqmu devs on this. > > > >> > >> 2) To your issue with starting the domain it would be good to know what > >> is the error you get from virsh (or however you are starting the > >> domain) and the debug logs of libvirtd, ideally just for the part of > >> the domain starting. > >that is the issue, there wasn't any error. the vm just didn't booted. > > Oh, so I misunderstood. What was the state of the VM in libvirt? > "paused" or "running"? Was there serial console working? it was marked as running and there was no serial > > >I can diff the original xml with the new one to see the diffs and post them here if you wish > > > > Would be nice to see if there are any differences. The newly created > one works then? > I'll sent it later today

Sent: Thursday, August 12, 2021 at 11:49 AM From: "Martin Kletzander" <mkletzan(a)redhat.com&gt; To: "daggs" <daggs(a)gmx.com&gt; Cc: dan(a)berrange.com, libvirt-users(a)redhat.com Subject: Re: issues with vm after upgrade On Wed, Aug 11, 2021 at 08:53:10PM +0200, daggs wrote: >Greetings Martin, > > >> Sent: Wednesday, August 11, 2021 at 6:08 PM >> From: "daggs" <daggs(a)gmx.com&gt; >> To: "Martin Kletzander" <mkletzan(a)redhat.com&gt; >> Cc: dan(a)berrange.com, libvirt-users(a)redhat.com >> Subject: Re: issues with vm after upgrade >> >> Greetings Martin, >> >> > Sent: Wednesday, August 11, 2021 at 4:13 PM >> > From: "Martin Kletzander" <mkletzan(a)redhat.com&gt; >> > To: "daggs" <daggs(a)gmx.com&gt; >> > Cc: dan(a)berrange.com, libvirt-users(a)redhat.com >> > Subject: Re: issues with vm after upgrade >> > >> > On Wed, Aug 11, 2021 at 03:09:34PM +0200, daggs wrote: >> > >Greetings Martin, >> > > >> > >> Sent: Wednesday, August 11, 2021 at 10:14 AM >> > >> From: "Martin Kletzander" <mkletzan(a)redhat.com&gt; >> > >> To: "daggs" <daggs(a)gmx.com&gt; >> > >> Cc: dan(a)berrange.com, libvirt-users(a)redhat.com >> > >> Subject: Re: issues with vm after upgrade >> > >> [...] >> > >> >> > >> 2) To your issue with starting the domain it would be good to know what >> > >> is the error you get from virsh (or however you are starting the >> > >> domain) and the debug logs of libvirtd, ideally just for the part of >> > >> the domain starting. >> > >that is the issue, there wasn't any error. the vm just didn't booted. >> > >> > Oh, so I misunderstood. What was the state of the VM in libvirt? >> > "paused" or "running"? Was there serial console working? >> it was marked as running and there was no serial >> That's a pity we could not examine what was actually happening. >> > >> > >I can diff the original xml with the new one to see the diffs and post them here if you wish >> > > >> > >> > Would be nice to see if there are any differences. The newly created >> > one works then? >> > >> >> I'll sent it later today >> > >here: https://dpaste.com/5VBUU8Z9W > Unfortunately there are many differences there. The machine type changes _something_ in qemu, there is different PCI(e) topology, and I do not think I will be able to figure this out without the non-working machine. So if your current setup works for you right now I'd leave figuring out the previous issue to others, if there is anyone wanting to figure out if there is some libvirt issue. Have a nice day

Greetings Martin, > Sent: Thursday, August 12, 2021 at 2:07 PM > From: "daggs" <daggs(a)gmx.com&gt; > To: "Martin Kletzander" <mkletzan(a)redhat.com&gt; > Cc: dan(a)berrange.com, libvirt-users(a)redhat.com > Subject: Re: issues with vm after upgrade > >> Sent: Thursday, August 12, 2021 at 11:49 AM >> From: "Martin Kletzander" <mkletzan(a)redhat.com&gt; >> To: "daggs" <daggs(a)gmx.com&gt; >> Cc: dan(a)berrange.com, libvirt-users(a)redhat.com >> Subject: Re: issues with vm after upgrade >> >> On Wed, Aug 11, 2021 at 08:53:10PM +0200, daggs wrote: >>> Greetings Martin, >>> >>> >>>> Sent: Wednesday, August 11, 2021 at 6:08 PM >>>> From: "daggs" <daggs(a)gmx.com&gt; >>>> To: "Martin Kletzander" <mkletzan(a)redhat.com&gt; >>>> Cc: dan(a)berrange.com, libvirt-users(a)redhat.com >>>> Subject: Re: issues with vm after upgrade >>>> >>>> Greetings Martin, >>>> >>>>> Sent: Wednesday, August 11, 2021 at 4:13 PM >>>>> From: "Martin Kletzander" <mkletzan(a)redhat.com&gt; >>>>> To: "daggs" <daggs(a)gmx.com&gt; >>>>> Cc: dan(a)berrange.com, libvirt-users(a)redhat.com >>>>> Subject: Re: issues with vm after upgrade >>>>> >>>>> On Wed, Aug 11, 2021 at 03:09:34PM +0200, daggs wrote: >>>>>> Greetings Martin, >>>>>> >>>>>>> Sent: Wednesday, August 11, 2021 at 10:14 AM >>>>>>> From: "Martin Kletzander" <mkletzan(a)redhat.com&gt; >>>>>>> To: "daggs" <daggs(a)gmx.com&gt; >>>>>>> Cc: dan(a)berrange.com, libvirt-users(a)redhat.com >>>>>>> Subject: Re: issues with vm after upgrade >>>>>>> >> >> [...] >> >>>>>>> >>>>>>> 2) To your issue with starting the domain it would be good to know what >>>>>>> is the error you get from virsh (or however you are starting the >>>>>>> domain) and the debug logs of libvirtd, ideally just for the part of >>>>>>> the domain starting. >>>>>> that is the issue, there wasn't any error. the vm just didn't booted. >>>>> >>>>> Oh, so I misunderstood. What was the state of the VM in libvirt? >>>>> "paused" or "running"? Was there serial console working? >>>> it was marked as running and there was no serial >>>> >> >> That's a pity we could not examine what was actually happening. >> >>>>> >>>>>> I can diff the original xml with the new one to see the diffs and post them here if you wish >>>>>> >>>>> >>>>> Would be nice to see if there are any differences. The newly created >>>>> one works then? >>>>> >>>> >>>> I'll sent it later today >>>> >>> >>> here: https://dpaste.com/5VBUU8Z9W >>> >> >> Unfortunately there are many differences there. The machine type >> changes _something_ in qemu, there is different PCI(e) topology, and I >> do not think I will be able to figure this out without the non-working >> machine. >> >> So if your current setup works for you right now I'd leave figuring out >> the previous issue to others, if there is anyone wanting to figure out >> if there is some libvirt issue. >> >> Have a nice day >> > > my current setup works beside the hdmi audio, this I still need to investigate. > > thanks for your help. > > Dagg > just to update, I've solved the sound issue, frankly, I don't understand how the guest showed a soundcard in the first place. from what I gather, libvirt sets the -nodefaults flag to prepare the vm's properties from scratch. in this situation, the sound card is a function in the host machine's pci tree. when libvirt created the pci tree for the guest, it placed the card as a function of a device as well, in my case 02:00.2 however it didn't created a device at 02:00.0.

> From: "daggs" <daggs(a)gmx.com&gt; >> From: "Martin Kletzander" <mkletzan(a)redhat.com&gt; >> On Wed, Aug 11, 2021 at 03:09:34PM +0200, daggs wrote: >>> I can diff the original xml with the new one to see the diffs and

>>> >> >> Would be nice to see if there are any differences. The newly created >> one works then? > > I'll sent it later today > here: https://dpaste.com/5VBUU8Z9W

my fix was to move the device to 00:1f.4 in the guest.

I won't be surprised this was the issue why the vm didn't booted after the upgrade with the old xml.

Greetings Laine, > Sent: Monday, August 16, 2021 at 12:57 AM > From: "Laine Stump" <laine(a)redhat.com&gt; > To: "daggs" <daggs(a)gmx.com&gt; > Cc: "Martin Kletzander" <mkletzan(a)redhat.com&gt;, libvirt-users(a)redhat.com > Subject: Re: issues with vm after upgrade > > > > On 8/14/21 6:05 AM, daggs wrote: >> Greetings Martin, >> >>> Sent: Thursday, August 12, 2021 at 2:07 PM >>> From: "daggs" <daggs(a)gmx.com&gt; >>> To: "Martin Kletzander" <mkletzan(a)redhat.com&gt; >>> Cc: dan(a)berrange.com, libvirt-users(a)redhat.com >>> Subject: Re: issues with vm after upgrade >>> >>>> Sent: Thursday, August 12, 2021 at 11:49 AM >>>> From: "Martin Kletzander" <mkletzan(a)redhat.com&gt; >>>> To: "daggs" <daggs(a)gmx.com&gt; >>>> Cc: dan(a)berrange.com, libvirt-users(a)redhat.com >>>> Subject: Re: issues with vm after upgrade >>>> >>>> On Wed, Aug 11, 2021 at 08:53:10PM +0200, daggs wrote: >>>>> Greetings Martin, >>>>> >>>>> >>>>>> Sent: Wednesday, August 11, 2021 at 6:08 PM >>>>>> From: "daggs" <daggs(a)gmx.com&gt; >>>>>> To: "Martin Kletzander" <mkletzan(a)redhat.com&gt; >>>>>> Cc: dan(a)berrange.com, libvirt-users(a)redhat.com >>>>>> Subject: Re: issues with vm after upgrade >>>>>> >>>>>> Greetings Martin, >>>>>> >>>>>>> Sent: Wednesday, August 11, 2021 at 4:13 PM >>>>>>> From: "Martin Kletzander" <mkletzan(a)redhat.com&gt; >>>>>>> To: "daggs" <daggs(a)gmx.com&gt; >>>>>>> Cc: dan(a)berrange.com, libvirt-users(a)redhat.com >>>>>>> Subject: Re: issues with vm after upgrade >>>>>>> >>>>>>> On Wed, Aug 11, 2021 at 03:09:34PM +0200, daggs wrote: >>>>>>>> Greetings Martin, >>>>>>>> >>>>>>>>> Sent: Wednesday, August 11, 2021 at 10:14 AM >>>>>>>>> From: "Martin Kletzander" <mkletzan(a)redhat.com&gt; >>>>>>>>> To: "daggs" <daggs(a)gmx.com&gt; >>>>>>>>> Cc: dan(a)berrange.com, libvirt-users(a)redhat.com >>>>>>>>> Subject: Re: issues with vm after upgrade >>>>>>>>> >>>> >>>> [...] >>>> >>>>>>>>> >>>>>>>>> 2) To your issue with starting the domain it would be good to know what >>>>>>>>> is the error you get from virsh (or however you are starting the >>>>>>>>> domain) and the debug logs of libvirtd, ideally just for the part of >>>>>>>>> the domain starting. >>>>>>>> that is the issue, there wasn't any error. the vm just didn't booted. >>>>>>> >>>>>>> Oh, so I misunderstood. What was the state of the VM in libvirt? >>>>>>> "paused" or "running"? Was there serial console working? >>>>>> it was marked as running and there was no serial >>>>>> >>>> >>>> That's a pity we could not examine what was actually happening. >>>> >>>>>>> >>>>>>>> I can diff the original xml with the new one to see the diffs and post them here if you wish >>>>>>>> >>>>>>> >>>>>>> Would be nice to see if there are any differences. The newly created >>>>>>> one works then? >>>>>>> >>>>>> >>>>>> I'll sent it later today >>>>>> >>>>> >>>>> here: https://dpaste.com/5VBUU8Z9W >>>>> >>>> >>>> Unfortunately there are many differences there. The machine type >>>> changes _something_ in qemu, there is different PCI(e) topology, and I >>>> do not think I will be able to figure this out without the non-working >>>> machine. >>>> >>>> So if your current setup works for you right now I'd leave figuring out >>>> the previous issue to others, if there is anyone wanting to figure out >>>> if there is some libvirt issue. >>>> >>>> Have a nice day >>>> >>> >>> my current setup works beside the hdmi audio, this I still need to investigate. >>> >>> thanks for your help. >>> >>> Dagg >>> >> >> just to update, I've solved the sound issue, frankly, I don't understand how the guest showed a soundcard in the first place. >> from what I gather, libvirt sets the -nodefaults flag to prepare the vm's properties from scratch. >> in this situation, the sound card is a function in the host machine's pci tree. >> when libvirt created the pci tree for the guest, it placed the card as a function of a device as well, in my case 02:00.2 >> however it didn't created a device at 02:00.0. > > Are you basing this claim on the libvirt XML? Or on what you see with > lspci in the guest? > > When libvirt is assigning PCI addresses to devices in a guest, it will > never auto-assign a non-0 function. This will only happen if the user > explicitly requests it (and even then, iirc, libvirt should generate an > error if function 0 of the same slot has no device - something to the > effect of "no device on function 0 of a multifunction device"). > > Anyway, when I looked back at the XML diff you posted earlier (see > below), I didn't see any hostdev device assigned to 02:00.2. What I > *did* see was that in both the old and the new version of the diff, the > hostdev devices were assigned to function 0 of different *slots* on a > dmi-to-pci-bridge controller, which should cause no problems (unless > there is a bug in QEMU's dmi-to-pci-bridge). (The important thing, > though, is that there is no hostdev device on a non-0 function, and when > it is on a non-0 slot, that's because it's on a dmi-to-pci-bridge (which > has 32 slots).

I saw it in guest,

I'd assume that if libvirt defines a device on a specific bdf, the guest will not change it.

infact, over the last 10 years I've booted thousand of systems both bare metal and visualized and never encountered such scenario. that said, it might be a bug in qemu. > what I did saw is that on the old vm in guest, after the upgrade the sound card was defined as a function of the scsi virtblk controller and the new vm placed it as a function of non existent device.

> > > On the topic of having a dmi-to-pci-bridge show up in your XML: I don't > remember what versions the changes were in (it was at least a year or > two ago), but only a fairly old version of libvirt woud do that - 1) > recent libvirt will assume that any hostdev PCI device is a PCIe device, > so it will add a pcie-root-port and assign the hostdev device to slot 0 > of that root-port, and even before that 2) we switched from using > dmi-to-pci-bridge to using pcie-to-pci-bridge quite some time ago as well. as stated in the original mail, the issue started after a major version upgrade of both libvirt and qemu, I'm currently using latest stable afaik.

> > So if you're generating new XML based on config that doesn't have pci > controllers already in it, and you're seeing hostdevs (or any other PCI > devices) assigned to an automatically-added dmi-to-pci-bridge, then your > libvirt version is severely out of date. here are the version I'm using: # emerge --search app-emulation/libvirt app-emulation/qemu [ Results for search key : app-emulation/libvirt ] Searching... * app-emulation/libvirt Latest version available: 7.5.0 Latest version installed: 7.5.0 Size of files: 9749 KiB Homepage: https://www.libvirt.org/ https://gitlab.com/libvirt/libvirt/ Description: C toolkit to manipulate virtual machines License: LGPL-2.1 [ Applications found : 1 ] [ Results for search key : app-emulation/qemu ] Searching... * app-emulation/qemu Latest version available: 6.0.0-r52 Latest version installed: 6.0.0-r52 Size of files: 22724 KiB Homepage: http://www.qemu.org http://www.linux-kvm.org Description: QEMU + Kernel-based Virtual Machine userland tools License: GPL-2 LGPL-2 BSD-2 [ Applications found : 1 ] > > > On 8/11/21 2:53 PM, daggs wrote: > >> From: "daggs" <daggs(a)gmx.com&gt; > >>> From: "Martin Kletzander" <mkletzan(a)redhat.com&gt; > >>> On Wed, Aug 11, 2021 at 03:09:34PM +0200, daggs wrote: > >>>> I can diff the original xml with the new one to see the diffs and > post them here if you wish > >>>> > >>> > >>> Would be nice to see if there are any differences. The newly created > >>> one works then? > >> > >> I'll sent it later today > >> > > > > here: https://dpaste.com/5VBUU8Z9W > > >> my fix was to move the device to 00:1f.4 in the guest. > > That's an interesting choice :-). You could have just put it on function > 0 of some other unused slot (or a non-0 function of the slot the GPU is > assigned to). 00:1f is used for integrated devices on the Q35 chipset - > it's nice that QEMU's emulation code was written to allowing adding more > devices on that slot, but I wouldn't have been surprised if it had > caused problems... 10 years of working in a virtualization company has taught me that somethings, keeping the pci structure close as much as possible to the original is the best way to go. that is why I chose it a s func, it is a func on the host mahcine.

> > >> I won't be surprised this was the issue why the vm didn't booted after the upgrade with the old xml. > > Well, if your XML had a device assigned to a non-0 function of a slot > and no device in function 0 of that slot, it would have failed to work > previously as well (my recollection is that in this case it's more a > problem of the guest OS not probing non-0 functions when there is > nothing on function 0, and not with anything done by QEMU). > > here is the xml of the machine after I've recreated it, it worked but no sound: https://dpaste.com/BB9EDY6BK I used virt-manager. note that the sound card pt is placed as a func in bus 0x8 which doesn't exists.

Greetings Laine, > Sent: Wednesday, August 25, 2021 at 7:53 PM > From: "Laine Stump" <laine(a)redhat.com&gt; > To: "daggs" <daggs(a)gmx.com&gt; > Cc: "Martin Kletzander" <mkletzan(a)redhat.com&gt;, libvirt-users(a)redhat.com > Subject: Re: issues with vm after upgrade > > On 8/20/21 12:07 PM, daggs wrote: >> Greetings Laine, >> >>> Sent: Monday, August 16, 2021 at 12:57 AM >>> From: "Laine Stump" <laine(a)redhat.com&gt; >>> To: "daggs" <daggs(a)gmx.com&gt; >>> Cc: "Martin Kletzander" <mkletzan(a)redhat.com&gt;, libvirt-users(a)redhat.com >>> Subject: Re: issues with vm after upgrade >>> >>> >>> >>> On 8/14/21 6:05 AM, daggs wrote: >>>> Greetings Martin, >>>> >>>>> Sent: Thursday, August 12, 2021 at 2:07 PM >>>>> From: "daggs" <daggs(a)gmx.com&gt; >>>>> To: "Martin Kletzander" <mkletzan(a)redhat.com&gt; >>>>> Cc: dan(a)berrange.com, libvirt-users(a)redhat.com >>>>> Subject: Re: issues with vm after upgrade >>>>> >>>>>> Sent: Thursday, August 12, 2021 at 11:49 AM >>>>>> From: "Martin Kletzander" <mkletzan(a)redhat.com&gt; >>>>>> To: "daggs" <daggs(a)gmx.com&gt; >>>>>> Cc: dan(a)berrange.com, libvirt-users(a)redhat.com >>>>>> Subject: Re: issues with vm after upgrade >>>>>> >>>>>> On Wed, Aug 11, 2021 at 08:53:10PM +0200, daggs wrote: >>>>>>> Greetings Martin, >>>>>>> >>>>>>> >>>>>>>> Sent: Wednesday, August 11, 2021 at 6:08 PM >>>>>>>> From: "daggs" <daggs(a)gmx.com&gt; >>>>>>>> To: "Martin Kletzander" <mkletzan(a)redhat.com&gt; >>>>>>>> Cc: dan(a)berrange.com, libvirt-users(a)redhat.com >>>>>>>> Subject: Re: issues with vm after upgrade >>>>>>>> >>>>>>>> Greetings Martin, >>>>>>>> >>>>>>>>> Sent: Wednesday, August 11, 2021 at 4:13 PM >>>>>>>>> From: "Martin Kletzander" <mkletzan(a)redhat.com&gt; >>>>>>>>> To: "daggs" <daggs(a)gmx.com&gt; >>>>>>>>> Cc: dan(a)berrange.com, libvirt-users(a)redhat.com >>>>>>>>> Subject: Re: issues with vm after upgrade >>>>>>>>> >>>>>>>>> On Wed, Aug 11, 2021 at 03:09:34PM +0200, daggs wrote: >>>>>>>>>> Greetings Martin, >>>>>>>>>> >>>>>>>>>>> Sent: Wednesday, August 11, 2021 at 10:14 AM >>>>>>>>>>> From: "Martin Kletzander" <mkletzan(a)redhat.com&gt; >>>>>>>>>>> To: "daggs" <daggs(a)gmx.com&gt; >>>>>>>>>>> Cc: dan(a)berrange.com, libvirt-users(a)redhat.com >>>>>>>>>>> Subject: Re: issues with vm after upgrade >>>>>>>>>>> >>>>>> >>>>>> [...] >>>>>> >>>>>>>>>>> >>>>>>>>>>> 2) To your issue with starting the domain it would be good to know what >>>>>>>>>>> is the error you get from virsh (or however you are starting the >>>>>>>>>>> domain) and the debug logs of libvirtd, ideally just for the part of >>>>>>>>>>> the domain starting. >>>>>>>>>> that is the issue, there wasn't any error. the vm just didn't booted. >>>>>>>>> >>>>>>>>> Oh, so I misunderstood. What was the state of the VM in libvirt? >>>>>>>>> "paused" or "running"? Was there serial console working? >>>>>>>> it was marked as running and there was no serial >>>>>>>> >>>>>> >>>>>> That's a pity we could not examine what was actually happening. >>>>>> >>>>>>>>> >>>>>>>>>> I can diff the original xml with the new one to see the diffs and post them here if you wish >>>>>>>>>> >>>>>>>>> >>>>>>>>> Would be nice to see if there are any differences. The newly created >>>>>>>>> one works then? >>>>>>>>> >>>>>>>> >>>>>>>> I'll sent it later today >>>>>>>> >>>>>>> >>>>>>> here: https://dpaste.com/5VBUU8Z9W >>>>>>> >>>>>> >>>>>> Unfortunately there are many differences there. The machine type >>>>>> changes _something_ in qemu, there is different PCI(e) topology, and I >>>>>> do not think I will be able to figure this out without the non-working >>>>>> machine. >>>>>> >>>>>> So if your current setup works for you right now I'd leave figuring out >>>>>> the previous issue to others, if there is anyone wanting to figure out >>>>>> if there is some libvirt issue. >>>>>> >>>>>> Have a nice day >>>>>> >>>>> >>>>> my current setup works beside the hdmi audio, this I still need to investigate. >>>>> >>>>> thanks for your help. >>>>> >>>>> Dagg >>>>> >>>> >>>> just to update, I've solved the sound issue, frankly, I don't understand how the guest showed a soundcard in the first place. >>>> from what I gather, libvirt sets the -nodefaults flag to prepare the vm's properties from scratch. >>>> in this situation, the sound card is a function in the host machine's pci tree. >>>> when libvirt created the pci tree for the guest, it placed the card as a function of a device as well, in my case 02:00.2 >>>> however it didn't created a device at 02:00.0. >>> >>> Are you basing this claim on the libvirt XML? Or on what you see with >>> lspci in the guest? >>> >>> When libvirt is assigning PCI addresses to devices in a guest, it will >>> never auto-assign a non-0 function. This will only happen if the user >>> explicitly requests it (and even then, iirc, libvirt should generate an >>> error if function 0 of the same slot has no device - something to the >>> effect of "no device on function 0 of a multifunction device"). >>> >>> Anyway, when I looked back at the XML diff you posted earlier (see >>> below), I didn't see any hostdev device assigned to 02:00.2. What I >>> *did* see was that in both the old and the new version of the diff, the >>> hostdev devices were assigned to function 0 of different *slots* on a >>> dmi-to-pci-bridge controller, which should cause no problems (unless >>> there is a bug in QEMU's dmi-to-pci-bridge). (The important thing, >>> though, is that there is no hostdev device on a non-0 function, and when >>> it is on a non-0 slot, that's because it's on a dmi-to-pci-bridge (which >>> has 32 slots). > >> I saw it in guest, > > But I didn't see it in the XML diffs that you had posted. as mentioned below, here is the xml of the new vm but with the sound problem: https://dpaste.com/BB9EDY6BK the relevant entry is at https://dpaste.com/BB9EDY6BK#line-130 you can see that the bdf is 08:01.0, note that there is no device defined at 08:00.0. I may be wrong and there is no need for such device but if I run lspci on both my linux system, I don't see device with such scenario.

note that the new vm was created using virt-manager, so the address wasn't allocated by me > >> I'd assume that if libvirt defines a device on a specific bdf, the guest will not change it. > > That's not exactly true - the bus "number" in libvirt isn't given to > qemu as an actual number, but as an alphanumeric device id (called > "alias name" in libvirt XML). QEMU doesn't have any concept of "bus > number", because (afaiu) there is no way to convey such info to the > guest firmware/OS; instead, QEMU creates a topology of interconnected > controllers, the firmware and/or OS traverses this topology and assigns > numbers to the encountered controllers as it sees fit. so bottom line, libvirt defines the "order" of he device and qemu creates however he wants but maintains libvirt's "order"?

> > So you may have PCI controllers with indexes 1, 2, and 3 in your libvirt > config, but those will be described on the QEMU commandline as > controllers "pcie.1", "pcie.2", and "pcie.3": > > -device > pcie-root-port,port=0x8,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x1 > \ > -device pcie-root-port,port=0x9,chassis=2,id=pci.2,bus=pcie.0,addr=0x1.0x1 \ > -device pcie-root-port,port=0xa,chassis=3,id=pci.3,bus=pcie.0,addr=0x1.0x2 \ > > and when a PCI device is attached to one of these controllers, the QEMU > commandline uses the id name of the controller, not a bus number: > > -device vfio-pci,host=0000:05:00.0,bus=pci.1,multifunction=on,addr=0x0 \ > -device vfio-pci,host=0000:05:00.1,id=hostdev1,bus=pci.1,addr=0x0.0x1 \ > > It is a nice coincidence that the OSes I've seen happen to traverse the > PCI topology in a manner that results in the guest OS numbering the > buses the same as they are numbered in libvirt XML that has had PCI > addresses auto-assigned by libvirt, but it is trivial to make this *not* > happen. For example, if you changed the config so that the bus with > index='2' (pcie.2) was attached to pcie.0, addr=0x1.0x1 (i.e. change its > PCI address to <address type='pci' bus='0' slot='1' function='1'/>" , > and the bus with index='3' was attached to pcie.0, addr=0x1.0x2, then > the guest would number "pcie.2" as bus 1, and "pcie.1" as bus 2. > > And of course the guest OS is free to traverse the controller topology > in any manner it wants, so the bus numbering in the guest could be > different even if the libvirt-generated QEMU commandline was the same. > > *HOWEVER*, slot (device) and function number are specified on the QEMU > commandline numerically, and they will appear in the guest exactly as > they are in the libvirt XML. > > > >> infact, over the last 10 years I've booted thousand of systems both bare metal and visualized and never encountered such scenario. >> that said, it might be a bug in qemu. > >> what I did saw is that on the old vm in guest, after the upgrade the sound card was defined as a function of the scsi virtblk controller and the new vm placed >> it as a function of non existent device. > > I would be very interested in seeing the libvirt XML, QEMU commandline, > and guest-side output of "lspci" for this. I can't think of any way this > could happen without a serious bug *somewhere* (or manual intervention > in the PCI addresses in the guest). of the system with the sound issue? if so, I'll need to digg in my logs, the most problematic part is the guest lspci

> >> >>> >>> >>> On the topic of having a dmi-to-pci-bridge show up in your XML: I don't >>> remember what versions the changes were in (it was at least a year or >>> two ago), but only a fairly old version of libvirt woud do that - 1) >>> recent libvirt will assume that any hostdev PCI device is a PCIe device, >>> so it will add a pcie-root-port and assign the hostdev device to slot 0 >>> of that root-port, and even before that 2) we switched from using >>> dmi-to-pci-bridge to using pcie-to-pci-bridge quite some time ago as well. >> as stated in the original mail, the issue started after a major version upgrade of both libvirt and qemu, >> I'm currently using latest stable afaik. > > Right. If your guest was defined the first time using a much older > libvirt, then devices would have been assigned to an auto-created > dmi-to-pci-bridge at that time, and if you don't change (or remove) the > PCI addresses of the devices or the bridge, then that will all be > maintained whenever you restart the guest, ragardless of libvirt > upgrades. But this again points out that the guest-side PCI addresses > (which are determined by the PCI addresses in the libvirt config) should > not change when upgrading libvirt (NOTE: 1) libvirt will only > auto-assign a new PCI address to a device if it doesn't already have a > PCI address assigned to it, and 2) libvirt *never* auto-assigns a non-0 > function except when adding a pcie-root-port (and in that case it will > always first assign something to function 0)) then I wonder how the upgrade broke the system, in contrast, the other vm I'm running (router with 5 nics in pt) worked out of the box

> > >> >>> >>> So if you're generating new XML based on config that doesn't have pci >>> controllers already in it, and you're seeing hostdevs (or any other PCI >>> devices) assigned to an automatically-added dmi-to-pci-bridge, then your >>> libvirt version is severely out of date. >> here are the version I'm using: >> # emerge --search app-emulation/libvirt app-emulation/qemu >> >> [ Results for search key : app-emulation/libvirt ] >> Searching... >> >> * app-emulation/libvirt >> Latest version available: 7.5.0 >> Latest version installed: 7.5.0 >> Size of files: 9749 KiB >> Homepage: https://www.libvirt.org/ https://gitlab.com/libvirt/libvirt/ >> Description: C toolkit to manipulate virtual machines >> License: LGPL-2.1 >> >> [ Applications found : 1 ] >> >> >> [ Results for search key : app-emulation/qemu ] >> Searching... >> >> * app-emulation/qemu >> Latest version available: 6.0.0-r52 >> Latest version installed: 6.0.0-r52 >> Size of files: 22724 KiB >> Homepage: http://www.qemu.org http://www.linux-kvm.org >> Description: QEMU + Kernel-based Virtual Machine userland tools >> License: GPL-2 LGPL-2 BSD-2 >> >> [ Applications found : 1 ] >> >>> >>> >>> On 8/11/21 2:53 PM, daggs wrote: >>> >> From: "daggs" <daggs(a)gmx.com&gt; >>> >>> From: "Martin Kletzander" <mkletzan(a)redhat.com&gt; >>> >>> On Wed, Aug 11, 2021 at 03:09:34PM +0200, daggs wrote: >>> >>>> I can diff the original xml with the new one to see the diffs and >>> post them here if you wish >>> >>>> >>> >>> >>> >>> Would be nice to see if there are any differences. The newly created >>> >>> one works then? >>> >> >>> >> I'll sent it later today >>> >> >>> > >>> > here: https://dpaste.com/5VBUU8Z9W >>> >>> >>>> my fix was to move the device to 00:1f.4 in the guest. >>> >>> That's an interesting choice :-). You could have just put it on function >>> 0 of some other unused slot (or a non-0 function of the slot the GPU is >>> assigned to). 00:1f is used for integrated devices on the Q35 chipset - >>> it's nice that QEMU's emulation code was written to allowing adding more >>> devices on that slot, but I wouldn't have been surprised if it had >>> caused problems... >> 10 years of working in a virtualization company has taught me that somethings, keeping the pci structure close as much as possible >> to the original is the best way to go. >> that is why I chose it a s func, it is a func on the host mahcine. > > It wasn't a function of a slot that also contains integrated chipset > devices though.... Oh, wait. According to the XML you reference down > below, it looks like the audio device you're assigning to the guest *is > itself* integrated on the chipset of the host, is that right? > > (It's interesting that this function of slot 1F is apparently in a > different IOMMU group than the other functions of slot 1F. I would have > guessed they would all be in the same IOMMU group, resulting in an > inability to assign this one function to the guest without at least > disabling the other devices on slot 1F (by binding them to the vfio-pci > driver) I'm using the pcie acs override patch to split the soundcard and the nic as they belong in different vms.

> >> >>> >>> >>>> I won't be surprised this was the issue why the vm didn't booted after the upgrade with the old xml. >>> >>> Well, if your XML had a device assigned to a non-0 function of a slot >>> and no device in function 0 of that slot, it would have failed to work >>> previously as well (my recollection is that in this case it's more a >>> problem of the guest OS not probing non-0 functions when there is >>> nothing on function 0, and not with anything done by QEMU). >>> >>> >> here is the xml of the machine after I've recreated it, it worked but no sound: https://dpaste.com/BB9EDY6BK >> I used virt-manager. note that the sound card pt is placed as a func in bus 0x8 which doesn't exists. > > This doesn't show any devices assigned to non-0 functions in the guest > (which is the part of what you said in previous messages that sounded > wrong to me). (except for the SATA controller, which is listed in the > libvirt config only for informational purposes, as it is hardcoded into > the basic q35 virtual machine and can't be removed). > > What is does show is that there is a device a 00:1F.3 *on the host* that > is being assigned to 08:01.00 (slot 1, function 0 of the > pcie-pci-bridge) in the guest. I'm guessing this is the audio device? > Also in this version of the XML, there is no longer a dmi-to-pci-bridge, > but there is instead a pcie-to-pci-bridge, implying that you've > redefined the guest config, resulting in PCI address auto-assignment > being re-run (at least relative to the config you referenced last week > that had a dmi-to-pci-bridge). > > It's possible that the audio device's driver just doesn't like the > device being on a standard PCI (i.e. non-PCIe) slot in the guest > somehow, since it's a chipset-integrated PCIe device on the host. I > haven't heard of that being the case in the past, but it's possible. > > > Anyway, at this point I've lost track of all the changes that have > happened (your update entailed much more than just updating the libvirt > package - your guest config was also changed/redefined) so I don't know > how much more effort should be expended with post-mortem, especially > since you now have it working. One thing that I would note is that we > should probably be auto-assigning integrated chipset devices to > pcie-root-ports rather than to a pci-bridge (I thought we already did > that, but I can see how we might not). > > I'll try to sum it up, prior to upgrade, both vms worked. after the upgrade, only the router vm worked, the streamer one started and never ran, I've started experimenting with qemu cmdline invocation, I've got to a situation where the vm was up and running with every thing defined beside the nic link active. this lead me to believe that the issue is with my config. I've tried to downgrade to previous versions however the vm still didn't booted. I've then reached the assumption that following the upgrade, the vm's xml was changed. because I didn't had the previous xml, I cannot verify. my next step was to recreate the vm's xml using virt-manager under the assumption that new config defined by virt-manager will work. that was partially correct, after the recreation was completed, the vm booted however the hdmi sound wasn't found. this sent me back to the qemu cmdline as I had a working cmd line invocation. I've started adding and removing entries from the generated qemu line to the working cmd and found out that the issue was caused because of the -nodefaults switch in qemu. this lead me to inspect the pci tree created and found out that in my working scenario, the sound card was placed in 00:02.0 and in the malfunctioning scenario,

the sound card was placed at 02:01.0 (got screen shots of it) this lead me to the conclusion that the pci config might cause it, moving it to 1f.x worked