[libvirt-users] Duplicate reboot events
Hi, Running on: $ cat /etc/redhat-release CentOS Linux release 7.3.1611 (Core) And: $ rpm -qa |grep libvirt libvirt-daemon-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-qemu-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-secret-2.0.0-10.el7_3.5.x86_64 libvirt-client-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-network-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-storage-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-config-nwfilter-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-config-network-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-nodedev-2.0.0-10.el7_3.5.x86_64 libvirt-2.0.0-10.el7_3.5.x86_64 libvirt-python-2.0.0-2.el7.x86_64 libvirt-daemon-driver-nwfilter-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-lxc-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-interface-2.0.0-10.el7_3.5.x86_64 I’m seeing duplicated reboot events when using the python event api. Use is simplified down to the simple test case attached. Running with that results in: $ sudo ./libvirt_events_single.py Reboot: Domain i-06945b37(21) 1494958504.72 Reboot: Domain i-06945b37(21) 1494958504.74 This is reproducible 100% of the time by doing /sbin/reboot from within a VM started with qemu-kvm. Any ideas on whats going on? Let me know if any other information would be useful. Thanks Martins
On Tue, May 16, 2017 at 06:41:05PM +0000, Innus, Martins wrote:
Hi,
Running on:
$ cat /etc/redhat-release CentOS Linux release 7.3.1611 (Core)
And:
$ rpm -qa |grep libvirt libvirt-daemon-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-qemu-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-secret-2.0.0-10.el7_3.5.x86_64 libvirt-client-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-network-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-storage-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-config-nwfilter-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-config-network-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-nodedev-2.0.0-10.el7_3.5.x86_64 libvirt-2.0.0-10.el7_3.5.x86_64 libvirt-python-2.0.0-2.el7.x86_64 libvirt-daemon-driver-nwfilter-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-lxc-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-interface-2.0.0-10.el7_3.5.x86_64
I’m seeing duplicated reboot events when using the python event api. Use is simplified down to the simple test case attached. Running with that results in:
$ sudo ./libvirt_events_single.py Reboot: Domain i-06945b37(21) 1494958504.72 Reboot: Domain i-06945b37(21) 1494958504.74
Can you try with newer libvirt? Does this happen with 'virsh event --loop --event lifecycle'? Is there anything fishy going on when looking at the console (VN/spice) during that reboot?
This is reproducible 100% of the time by doing /sbin/reboot from within a VM started with qemu-kvm. Any ideas on whats going on?
Let me know if any other information would be useful.
Thanks
Martins
#!/usr/bin/env python
import libvirt import time
def RBcallback(conn, dom, opaque): print "Reboot: Domain %s(%s)" % (dom.name(), dom.ID()) print time.time()
if __name__ == '__main__':
libvirt.virEventRegisterDefaultImpl()
conn = libvirt.openReadOnly('qemu:///system')
conn.domainEventRegisterAny(None, libvirt.VIR_DOMAIN_EVENT_ID_REBOOT, RBcallback, None)
while True: libvirt.virEventRunDefaultImpl()
_______________________________________________ libvirt-users mailing list libvirt-users@redhat.com https://www.redhat.com/mailman/listinfo/libvirt-users
On May 16, 2017, at 3:49 PM, Martin Kletzander <mkletzan@redhat.com> wrote:
On Tue, May 16, 2017 at 06:41:05PM +0000, Innus, Martins wrote:
Hi,
Running on:
$ cat /etc/redhat-release CentOS Linux release 7.3.1611 (Core)
And:
$ rpm -qa |grep libvirt libvirt-daemon-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-qemu-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-secret-2.0.0-10.el7_3.5.x86_64 libvirt-client-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-network-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-storage-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-config-nwfilter-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-config-network-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-nodedev-2.0.0-10.el7_3.5.x86_64 libvirt-2.0.0-10.el7_3.5.x86_64 libvirt-python-2.0.0-2.el7.x86_64 libvirt-daemon-driver-nwfilter-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-lxc-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-interface-2.0.0-10.el7_3.5.x86_64
I’m seeing duplicated reboot events when using the python event api. Use is simplified down to the simple test case attached. Running with that results in:
$ sudo ./libvirt_events_single.py Reboot: Domain i-06945b37(21) 1494958504.72 Reboot: Domain i-06945b37(21) 1494958504.74
Can you try with newer libvirt?
I’m not sure. I will have to see if I can build it, unless there is a repo somewhere with centos7 rpms. I assume by this you mean that you don’t see this with current libvirt?
Does this happen with 'virsh event --loop --event lifecycle’?
“lifecycle” shows no output, but with “reboot” or “all”: $ sudo virsh event --loop --all event 'reboot' for domain i-06945b37 event 'reboot' for domain i-06945b37
Is there anything fishy going on when looking at the console (VN/spice) during that reboot?
Don’t see anything unusual. I can send you the full console output if you want, but I don’t see anything strange. Thanks Martins
This is reproducible 100% of the time by doing /sbin/reboot from within a VM started with qemu-kvm. Any ideas on whats going on?
Let me know if any other information would be useful.
Thanks
Martins
#!/usr/bin/env python
import libvirt import time
def RBcallback(conn, dom, opaque): print "Reboot: Domain %s(%s)" % (dom.name(), dom.ID()) print time.time()
if __name__ == '__main__':
libvirt.virEventRegisterDefaultImpl()
conn = libvirt.openReadOnly('qemu:///system')
conn.domainEventRegisterAny(None, libvirt.VIR_DOMAIN_EVENT_ID_REBOOT, RBcallback, None)
while True: libvirt.virEventRunDefaultImpl()
_______________________________________________ libvirt-users mailing list libvirt-users@redhat.com https://www.redhat.com/mailman/listinfo/libvirt-users
On Tue, May 16, 2017 at 08:11:54PM +0000, Innus, Martins wrote:
On May 16, 2017, at 3:49 PM, Martin Kletzander <mkletzan@redhat.com> wrote:
On Tue, May 16, 2017 at 06:41:05PM +0000, Innus, Martins wrote:
Hi,
Running on:
$ cat /etc/redhat-release CentOS Linux release 7.3.1611 (Core)
And:
$ rpm -qa |grep libvirt libvirt-daemon-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-qemu-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-secret-2.0.0-10.el7_3.5.x86_64 libvirt-client-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-network-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-storage-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-config-nwfilter-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-config-network-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-nodedev-2.0.0-10.el7_3.5.x86_64 libvirt-2.0.0-10.el7_3.5.x86_64 libvirt-python-2.0.0-2.el7.x86_64 libvirt-daemon-driver-nwfilter-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-lxc-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-interface-2.0.0-10.el7_3.5.x86_64
I’m seeing duplicated reboot events when using the python event api. Use is simplified down to the simple test case attached. Running with that results in:
$ sudo ./libvirt_events_single.py Reboot: Domain i-06945b37(21) 1494958504.72 Reboot: Domain i-06945b37(21) 1494958504.74
Can you try with newer libvirt?
I’m not sure. I will have to see if I can build it, unless there is a repo somewhere with centos7 rpms. I assume by this you mean that you don’t see this with current libvirt?
Does this happen with 'virsh event --loop --event lifecycle’?
“lifecycle” shows no output, but with “reboot” or “all”:
$ sudo virsh event --loop --all event 'reboot' for domain i-06945b37 event 'reboot' for domain i-06945b37
Is there anything fishy going on when looking at the console (VN/spice) during that reboot?
Don’t see anything unusual. I can send you the full console output if you want, but I don’t see anything strange.
I meant the graphical terminal, but that won't help much. I tried gathering as much info. So we just need to figure out whether QEMU sends us the event multiple times. Could you either set up debug logs [1] and then look for that in them or use the qemu-monitor.stp systemtap script to show you what's happening on the monitor? If it comes from qemu two times, than there's not much we can do about it. Martin [1] http://wiki.libvirt.org/page/DebugLogs
On May 16, 2017, at 4:28 PM, Martin Kletzander <mkletzan@redhat.com> wrote:
On Tue, May 16, 2017 at 08:11:54PM +0000, Innus, Martins wrote:
On May 16, 2017, at 3:49 PM, Martin Kletzander <mkletzan@redhat.com> wrote:
On Tue, May 16, 2017 at 06:41:05PM +0000, Innus, Martins wrote:
Hi,
Running on:
$ cat /etc/redhat-release CentOS Linux release 7.3.1611 (Core)
And:
$ rpm -qa |grep libvirt libvirt-daemon-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-qemu-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-secret-2.0.0-10.el7_3.5.x86_64 libvirt-client-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-network-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-storage-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-config-nwfilter-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-config-network-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-nodedev-2.0.0-10.el7_3.5.x86_64 libvirt-2.0.0-10.el7_3.5.x86_64 libvirt-python-2.0.0-2.el7.x86_64 libvirt-daemon-driver-nwfilter-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-lxc-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-interface-2.0.0-10.el7_3.5.x86_64
I’m seeing duplicated reboot events when using the python event api. Use is simplified down to the simple test case attached. Running with that results in:
$ sudo ./libvirt_events_single.py Reboot: Domain i-06945b37(21) 1494958504.72 Reboot: Domain i-06945b37(21) 1494958504.74
Can you try with newer libvirt?
I’m not sure. I will have to see if I can build it, unless there is a repo somewhere with centos7 rpms. I assume by this you mean that you don’t see this with current libvirt?
Does this happen with 'virsh event --loop --event lifecycle’?
“lifecycle” shows no output, but with “reboot” or “all”:
$ sudo virsh event --loop --all event 'reboot' for domain i-06945b37 event 'reboot' for domain i-06945b37
Is there anything fishy going on when looking at the console (VN/spice) during that reboot?
Don’t see anything unusual. I can send you the full console output if you want, but I don’t see anything strange.
I meant the graphical terminal, but that won't help much. I tried gathering as much info. So we just need to figure out whether QEMU sends us the event multiple times. Could you either set up debug logs [1] and then look for that in them or use the qemu-monitor.stp systemtap script to show you what's happening on the monitor? If it comes from qemu two times, than there's not much we can do about it.
So I guess with: $ sudo virsh qemu-monitor-event i-06945b37 --pretty --loop event RESET at 1495030343.874769 for domain i-06945b37: <null> event RESET at 1495030343.896221 for domain i-06945b37: <null> And the following in the debug logs: 2017-05-17 14:21:03.369+0000: 31471: info : qemuMonitorJSONIOProcessLine:206 : QEMU_MONITOR_RECV_EVENT: mon=0x7f5a24000f50 event={"timestamp": {"seconds": 1495030863, "microseconds": 369485}, "event": "RESET"} 2017-05-17 14:21:03.384+0000: 31471: info : qemuMonitorJSONIOProcessLine:206 : QEMU_MONITOR_RECV_EVENT: mon=0x7f5a24000f50 event={"timestamp": {"seconds": 1495030863, "microseconds": 383626}, "event": "RESET”} That means its coming from QEMU right? Any suggestions besides bugging the QEMU folks? Thanks Martins
On May 16, 2017, at 4:28 PM, Martin Kletzander <mkletzan@redhat.com<mailto:mkletzan@redhat.com>> wrote: On Tue, May 16, 2017 at 08:11:54PM +0000, Innus, Martins wrote: On May 16, 2017, at 3:49 PM, Martin Kletzander <mkletzan@redhat.com<mailto:mkletzan@redhat.com>> wrote: On Tue, May 16, 2017 at 06:41:05PM +0000, Innus, Martins wrote: Hi, Running on: $ cat /etc/redhat-release CentOS Linux release 7.3.1611 (Core) And: $ rpm -qa |grep libvirt libvirt-daemon-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-qemu-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-secret-2.0.0-10.el7_3.5.x86_64 libvirt-client-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-network-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-storage-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-config-nwfilter-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-config-network-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-nodedev-2.0.0-10.el7_3.5.x86_64 libvirt-2.0.0-10.el7_3.5.x86_64 libvirt-python-2.0.0-2.el7.x86_64 libvirt-daemon-driver-nwfilter-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-lxc-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-interface-2.0.0-10.el7_3.5.x86_64 I’m seeing duplicated reboot events when using the python event api. Use is simplified down to the simple test case attached. Running with that results in: $ sudo ./libvirt_events_single.py Reboot: Domain i-06945b37(21) 1494958504.72 Reboot: Domain i-06945b37(21) 1494958504.74 Can you try with newer libvirt? I’m not sure. I will have to see if I can build it, unless there is a repo somewhere with centos7 rpms. I assume by this you mean that you don’t see this with current libvirt? Does this happen with 'virsh event --loop --event lifecycle’? “lifecycle” shows no output, but with “reboot” or “all”: $ sudo virsh event --loop --all event 'reboot' for domain i-06945b37 event 'reboot' for domain i-06945b37 Is there anything fishy going on when looking at the console (VN/spice) during that reboot? Don’t see anything unusual. I can send you the full console output if you want, but I don’t see anything strange. I meant the graphical terminal, but that won't help much. I tried gathering as much info. So we just need to figure out whether QEMU sends us the event multiple times. Could you either set up debug logs [1] and then look for that in them or use the qemu-monitor.stp systemtap script to show you what's happening on the monitor? If it comes from qemu two times, than there's not much we can do about it. Nevermind, as far as I can tell, this is a bug in QEMU and is closed as a WONTFIX: https://bugzilla.redhat.com/show_bug.cgi?id=951525 Thanks for the debugging help. Martins
On Wed, May 17, 2017 at 03:14:18PM +0000, Innus, Martins wrote:
Nevermind, as far as I can tell, this is a bug in QEMU and is closed as a WONTFIX: https://bugzilla.redhat.com/show_bug.cgi?id=951525
Thanks for the debugging help.
As you found out in the other mail, yes it is. I didn't know about the bug, though. Anyway, at least we are sure it's not in libvirt as well. Thanks for the patience. Martin
participants (2)
-
Innus, Martins -
Martin Kletzander