8:44 p.m.
Hi,
i have several domains running on a 2-node HA-cluster.
Each night i create snapshots of the domains, after copying the consistent raw file to a
CIFS server i blockcommit the changes into the raw files.
That's running quite well.
But recent the blockcommit didn't work for one domain:
I create a logfile from the whole procedure:
===============================================================
...
Sat Jun 1 03:05:24 CEST 2019
Target Source
------------------------------------------------
vdb /mnt/snap/severin.sn
hdc -
/usr/bin/virsh blockcommit severin /mnt/snap/severin.sn --verbose --active --pivot
Block commit: [ 0 %]Block commit: [ 15 %]Block commit: [ 28 %]Block commit: [ 35 %]Block
commit: [ 43 %]Block commit: [ 53 %]Block commit: [ 63 %]Block commit: [ 73 %]Block
commit: [ 82 %]Block commit: [ 89 %]Block commit: [ 98 %]Block commit: [100 %]Target
Source
------------------------------------------------
vdb /mnt/snap/severin.sn
...
==============================================================
The libvirtd-log says (it's UTC IIRC):
=============================================================
...
2019-05-31 20:31:34.481+0000: 4170: error : qemuMonitorIO:719 : internal error: End of
file from qemu monitor
2019-06-01 01:05:32.233+0000: 4170: error : qemuMonitorIO:719 : internal error: End of
file from qemu monitor
2019-06-01 01:05:43.804+0000: 22605: warning : qemuGetProcessInfo:1461 : cannot parse
process status data
2019-06-01 01:05:43.848+0000: 22596: warning : qemuGetProcessInfo:1461 : cannot parse
process status data
2019-06-01 01:06:11.438+0000: 26112: warning : qemuDomainObjBeginJobInternal:4865 : Cannot
start job (destroy, none) for doma
in severin; current job is (modify, none) owned by (5372
remoteDispatchDomainBlockJobAbort, 0 <null>) for (39s, 0s)
2019-06-01 01:06:11.438+0000: 26112: error : qemuDomainObjBeginJobInternal:4877 : Timed
out during operation: cannot acquire
state change lock (held by remoteDispatchDomainBlockJobAbort)
2019-06-01 01:06:13.976+0000: 5369: warning : qemuGetProcessInfo:1461 : cannot parse
process status data
2019-06-01 01:06:14.028+0000: 22596: warning : qemuGetProcessInfo:1461 : cannot parse
process status data
2019-06-01 01:06:44.165+0000: 5371: warning : qemuGetProcessInfo:1461 : cannot parse
process status data
2019-06-01 01:06:44.218+0000: 22605: warning : qemuGetProcessInfo:1461 : cannot parse
process status data
2019-06-01 01:07:14.343+0000: 5369: warning : qemuGetProcessInfo:1461 : cannot parse
process status data
2019-06-01 01:07:14.387+0000: 22598: warning : qemuGetProcessInfo:1461 : cannot parse
process status data
2019-06-01 01:07:44.495+0000: 22605: warning : qemuGetProcessInfo:1461 : cannot parse
process status data
...
===========================================================
and "cannot parse process status data" continuously until the end of the
logfile.
The syslog from the domain itself didn't reveal anything, it just continues to run.
The libvirt log from the domains just says:
qemu-system-x86_64: block/mirror.c:864: mirror_run: Assertion
`((&bs->tracked_requests)->lh_first == ((void *)0))' failed.
Hosts are SLES 12 SP4 with libvirt-daemon-4.0.0-8.9.1.x86_64.
Bernd
--
Bernd Lentes
Systemadministration
Institut für Entwicklungsgenetik
Gebäude 35.34 - Raum 208
HelmholtzZentrum münchen
bernd.lentes(a)helmholtz-muenchen.de
phone: +49 89 3187 1241
phone: +49 89 3187 3827
fax: +49 89 3187 2294
http://www.helmholtz-muenchen.de/idg
wer Fehler macht kann etwas lernen
wer nichts macht kann auch nichts lernen
Helmholtz Zentrum Muenchen
Deutsches Forschungszentrum fuer Gesundheit und Umwelt (GmbH)
Ingolstaedter Landstr. 1
85764 Neuherberg
www.helmholtz-muenchen.de
Stellv. Aufsichtsratsvorsitzender: MinDirig. Dr. Manfred Wolter
Geschaeftsfuehrung: Prof. Dr. med. Dr. h.c. Matthias Tschoep, Heinrich Bassler, Kerstin
Guenther
Registergericht: Amtsgericht Muenchen HRB 6466
USt-IdNr: DE 129521671
3:27 p.m.
On Tue, Jun 04, 2019 at 14:44:29 +0200, Lentes, Bernd wrote:
Hi,
Hi,
i have several domains running on a 2-node HA-cluster.
Each night i create snapshots of the domains, after copying the consistent raw file to a
CIFS server i blockcommit the changes into the raw files.
That's running quite well.
But recent the blockcommit didn't work for one domain:
I create a logfile from the whole procedure:
===============================================================
...
Sat Jun 1 03:05:24 CEST 2019
Target Source
------------------------------------------------
vdb /mnt/snap/severin.sn
hdc -
/usr/bin/virsh blockcommit severin /mnt/snap/severin.sn --verbose --active --pivot
Block commit: [ 0 %]Block commit: [ 15 %]Block commit: [ 28 %]Block commit: [ 35 %]Block
commit: [ 43 %]Block commit: [ 53 %]Block commit: [ 63 %]Block commit: [ 73 %]Block
commit: [ 82 %]Block commit: [ 89 %]Block commit: [ 98 %]Block commit: [100 %]Target
Source
------------------------------------------------
vdb /mnt/snap/severin.sn
...
==============================================================
The libvirtd-log says (it's UTC IIRC):
=============================================================
...
2019-05-31 20:31:34.481+0000: 4170: error : qemuMonitorIO:719 : internal error: End of
file from qemu monitor
2019-06-01 01:05:32.233+0000: 4170: error : qemuMonitorIO:719 : internal error: End of
file from qemu monitor
This message is printed if qemu crashes for some reason and then closes
the monitor socket unexpectedly.
2019-06-01 01:05:43.804+0000: 22605: warning :
qemuGetProcessInfo:1461 : cannot parse process status data
2019-06-01 01:05:43.848+0000: 22596: warning : qemuGetProcessInfo:1461 : cannot parse
process status data
2019-06-01 01:06:11.438+0000: 26112: warning : qemuDomainObjBeginJobInternal:4865 :
Cannot start job (destroy, none) for doma
in severin; current job is (modify, none) owned by (5372
remoteDispatchDomainBlockJobAbort, 0 <null>) for (39s, 0s)
2019-06-01 01:06:11.438+0000: 26112: error : qemuDomainObjBeginJobInternal:4877 : Timed
out during operation: cannot acquire
state change lock (held by remoteDispatchDomainBlockJobAbort)
So this means that the virDomainBlockJobAbort API which is also used for
--pivot got stuck for some time.
This is kind of strange if the VM crashed, there might also be a bug in
the synchronous block job handling, but it's hard to tell from this log.
2019-06-01 01:06:13.976+0000: 5369: warning : qemuGetProcessInfo:1461
: cannot parse process status data
2019-06-01 01:06:14.028+0000: 22596: warning : qemuGetProcessInfo:1461 : cannot parse
process status data
2019-06-01 01:06:44.165+0000: 5371: warning : qemuGetProcessInfo:1461 : cannot parse
process status data
2019-06-01 01:06:44.218+0000: 22605: warning : qemuGetProcessInfo:1461 : cannot parse
process status data
2019-06-01 01:07:14.343+0000: 5369: warning : qemuGetProcessInfo:1461 : cannot parse
process status data
2019-06-01 01:07:14.387+0000: 22598: warning : qemuGetProcessInfo:1461 : cannot parse
process status data
2019-06-01 01:07:44.495+0000: 22605: warning : qemuGetProcessInfo:1461 : cannot parse
process status data
...
===========================================================
and "cannot parse process status data" continuously until the end of the
logfile.
The syslog from the domain itself didn't reveal anything, it just continues to run.
The libvirt log from the domains just says:
qemu-system-x86_64: block/mirror.c:864: mirror_run: Assertion
`((&bs->tracked_requests)->lh_first == ((void *)0))' failed.
So that's interresting. Usually assertion failure in qemu leads to
calling abort() and thus the vm would have crashed. Didn't you HA
solution restart it?
At any rate it would be really beneficial if you could collect debug
logs for libvirtd which also contain the monitor interactions with qemu:
https://wiki.libvirt.org/page/DebugLogs
The qemu assertion failure above should ideally be reported to qemu, but
if you are able to reproduce the problem with libvirtd debug logs
enabled I can extract more useful info from there which the qemu project
would ask you anyways.
7:33 p.m.
Hi Peter,
thanks for your help.
----- On Jun 5, 2019, at 9:27 AM, Peter Krempa pkrempa(a)redhat.com wrote:
> =============================================================
> ...
> 2019-05-31 20:31:34.481+0000: 4170: error : qemuMonitorIO:719 : internal error:
> End of file from qemu monitor
> 2019-06-01 01:05:32.233+0000: 4170: error : qemuMonitorIO:719 : internal error:
> End of file from qemu monitor
This message is printed if qemu crashes for some reason and then closes
the monitor socket unexpectedly.
> 2019-06-01 01:05:43.804+0000: 22605: warning : qemuGetProcessInfo:1461 : cannot
> parse process status data
> 2019-06-01 01:05:43.848+0000: 22596: warning : qemuGetProcessInfo:1461 : cannot
> parse process status data
> 2019-06-01 01:06:11.438+0000: 26112: warning :
> qemuDomainObjBeginJobInternal:4865 : Cannot start job (destroy, none) for doma
> in severin; current job is (modify, none) owned by (5372
> remoteDispatchDomainBlockJobAbort, 0 <null>) for (39s, 0s)
> 2019-06-01 01:06:11.438+0000: 26112: error : qemuDomainObjBeginJobInternal:4877
> : Timed out during operation: cannot acquire
> state change lock (held by remoteDispatchDomainBlockJobAbort)
So this means that the virDomainBlockJobAbort API which is also used for
--pivot got stuck for some time.
This is kind of strange if the VM crashed, there might also be a bug in
the synchronous block job handling, but it's hard to tell from this log.
The VM didn't crash. It kept running.
See "last":
root pts/49 ha-idg-2.scidom. Tue Jun 4 14:02 - 13:18 (23:16)
root pts/47 pc60337.scidom.d Mon Jun 3 15:13 still logged in
reboot system boot 2.6.4-52-smp Wed May 15 20:19 (20+17:02)
reboot system boot 2.6.4-52-smp Fri Mar 15 17:38 (81+18:44)
reboot system boot 2.6.4-52-smp Wed Feb 27 20:29 (15+21:04)
> The syslog from the domain itself didn't reveal anything, it
just continues to
> run.
> The libvirt log from the domains just says:
> qemu-system-x86_64: block/mirror.c:864: mirror_run: Assertion
> `((&bs->tracked_requests)->lh_first == ((void *)0))' failed.
So that's interresting. Usually assertion failure in qemu leads to
calling abort() and thus the vm would have crashed. Didn't you HA
solution restart it?
No. As said the VM didn't crash. It kept running.
At any rate it would be really beneficial if you could collect debug
logs for libvirtd which also contain the monitor interactions with qemu:
https://wiki.libvirt.org/page/DebugLogs
The qemu assertion failure above should ideally be reported to qemu, but
if you are able to reproduce the problem with libvirtd debug logs
enabled I can extract more useful info from there which the qemu project
would ask you anyways.
I can't reproduce it. It seems to happen accidentally. But i can collect the logs. Do
they get very large ?
I can contact you the next time it happen. Is that ok for you ?
Bernd
Helmholtz Zentrum Muenchen
Deutsches Forschungszentrum fuer Gesundheit und Umwelt (GmbH)
Ingolstaedter Landstr. 1
85764 Neuherberg
www.helmholtz-muenchen.de
Stellv. Aufsichtsratsvorsitzender: MinDirig. Dr. Manfred Wolter
Geschaeftsfuehrung: Prof. Dr. med. Dr. h.c. Matthias Tschoep, Heinrich Bassler, Kerstin
Guenther
Registergericht: Amtsgericht Muenchen HRB 6466
USt-IdNr: DE 129521671
10:49 p.m.
On Wed, Jun 05, 2019 at 13:33:49 +0200, Lentes, Bernd wrote:
Hi Peter,
thanks for your help.
----- On Jun 5, 2019, at 9:27 AM, Peter Krempa pkrempa(a)redhat.com wrote:
[...]
>
> So that's interresting. Usually assertion failure in qemu leads to
> calling abort() and thus the vm would have crashed. Didn't you HA
> solution restart it?
No. As said the VM didn't crash. It kept running.
That's interresting. I hope you manage to reproduce it then.
> At any rate it would be really beneficial if you could collect debug
> logs for libvirtd which also contain the monitor interactions with qemu:
>
> https://wiki.libvirt.org/page/DebugLogs
>
> The qemu assertion failure above should ideally be reported to qemu, but
> if you are able to reproduce the problem with libvirtd debug logs
> enabled I can extract more useful info from there which the qemu project
> would ask you anyways.
I can't reproduce it. It seems to happen accidentally. But i can collect the logs. Do
they get very large ?
I can contact you the next time it happen. Is that ok for you ?
Unfortunately they do get very large if there's some monitoring
gathering stats through libvirt, but it's okay to nuke them prior
to attempting the block commit, or daily or so.
Please do contact me if you gather anything interresting.
12:02 a.m.
----- On Jun 5, 2019, at 4:49 PM, Peter Krempa pkrempa(a)redhat.com wrote:
On Wed, Jun 05, 2019 at 13:33:49 +0200, Lentes, Bernd wrote:
> Hi Peter,
>
> thanks for your help.
>
> ----- On Jun 5, 2019, at 9:27 AM, Peter Krempa pkrempa(a)redhat.com wrote:
[...]
>
> >
> > So that's interresting. Usually assertion failure in qemu leads to
> > calling abort() and thus the vm would have crashed. Didn't you HA
> > solution restart it?
>
> No. As said the VM didn't crash. It kept running.
That's interresting. I hope you manage to reproduce it then.
> I can't reproduce it. It seems to happen accidentally. But i
can collect the
> logs. Do they get very large ?
> I can contact you the next time it happen. Is that ok for you ?
Unfortunately they do get very large if there's some monitoring
gathering stats through libvirt, but it's okay to nuke them prior
to attempting the block commit, or daily or so.
Please do contact me if you gather anything interresting.
Hi,
i followed https://wiki.libvirt.org/page/DebugLogs.
Where do i have to set LIBVIRT_LOG_OUTPUTS="1:file:/tmp/libvirt_client.log" ?
Also in /etc/libvirt/libvirtd.conf ?
Bernd
Helmholtz Zentrum Muenchen
Deutsches Forschungszentrum fuer Gesundheit und Umwelt (GmbH)
Ingolstaedter Landstr. 1
85764 Neuherberg
www.helmholtz-muenchen.de
Stellv. Aufsichtsratsvorsitzender: MinDirig. Dr. Manfred Wolter
Geschaeftsfuehrung: Prof. Dr. med. Dr. h.c. Matthias Tschoep, Heinrich Bassler, Kerstin
Guenther
Registergericht: Amtsgericht Muenchen HRB 6466
USt-IdNr: DE 129521671
3:17 p.m.
On Wed, Jun 05, 2019 at 18:02:06 +0200, Lentes, Bernd wrote:
[...]
Hi,
i followed https://wiki.libvirt.org/page/DebugLogs.
Where do i have to set LIBVIRT_LOG_OUTPUTS="1:file:/tmp/libvirt_client.log" ?
Also in /etc/libvirt/libvirtd.conf ?
No, that's an environment variable if you want to debug the client, but
that's not necessary in this case. Only the daemon logs are
interresting.
1:19 a.m.
----- On Jun 5, 2019, at 4:49 PM, Peter Krempa pkrempa(a)redhat.com wrote:
On Wed, Jun 05, 2019 at 13:33:49 +0200, Lentes, Bernd wrote:
> Hi Peter,
>
> thanks for your help.
>
> ----- On Jun 5, 2019, at 9:27 AM, Peter Krempa pkrempa(a)redhat.com wrote:
[...]
>
> >
> > So that's interresting. Usually assertion failure in qemu leads to
> > calling abort() and thus the vm would have crashed. Didn't you HA
> > solution restart it?
>
> No. As said the VM didn't crash. It kept running.
That's interresting. I hope you manage to reproduce it then.
>
> > At any rate it would be really beneficial if you could collect debug
> > logs for libvirtd which also contain the monitor interactions with qemu:
> >
> > https://wiki.libvirt.org/page/DebugLogs
> >
> > The qemu assertion failure above should ideally be reported to qemu, but
> > if you are able to reproduce the problem with libvirtd debug logs
> > enabled I can extract more useful info from there which the qemu project
> > would ask you anyways.
>
> I can't reproduce it. It seems to happen accidentally. But i can collect the
> logs. Do they get very large ?
> I can contact you the next time it happen. Is that ok for you ?
Unfortunately they do get very large if there's some monitoring
gathering stats through libvirt, but it's okay to nuke them prior
to attempting the block commit, or daily or so.
Please do contact me if you gather anything interresting.
Hi,
it happened again.
Following the log of my script it started on 8th of june at 5:59:09 (UTC+2) to blockcommit
the domain.
These are the related lines in libvirtd.log:
===================================================
2019-06-07 20:30:57.170+0000: 30299: error : qemuMonitorIO:719 : internal error: End of
file from qemu monitor
2019-06-08 03:59:17.690+0000: 30299: error : qemuMonitorIO:719 : internal error: End of
file from qemu monitor
2019-06-08 03:59:26.145+0000: 30300: warning : qemuGetProcessInfo:1461 : cannot parse
process status data
2019-06-08 03:59:26.191+0000: 30303: warning : qemuGetProcessInfo:1461 : cannot parse
process status data
2019-06-08 03:59:56.095+0000: 27956: warning : qemuDomainObjBeginJobInternal:4865 : Cannot
start job (destroy, none) for domain severin; current job is (modify, none) owned by
(13061 remoteDispatchDomainBlockJobAbort, 0 <null>) for (38s,
0s)
2019-06-08 03:59:56.095+0000: 27956: error : qemuDomainObjBeginJobInternal:4877 : Timed
out during operation: cannot acquire state change lock (held by
remoteDispatchDomainBlockJobAbort)
2019-06-08 03:59:56.325+0000: 13060: warning : qemuGetProcessInfo:1461 : cannot parse
process status data
2019-06-08 03:59:56.372+0000: 30304: warning : qemuGetProcessInfo:1461 : cannot parse
process status data
2019-06-08 04:00:26.503+0000: 13060: warning : qemuGetProcessInfo:1461 : cannot parse
process status data
====================================================
Since then the script is stuck.
Thanks for your help.
Bernd
Helmholtz Zentrum Muenchen
Deutsches Forschungszentrum fuer Gesundheit und Umwelt (GmbH)
Ingolstaedter Landstr. 1
85764 Neuherberg
www.helmholtz-muenchen.de
Aufsichtsratsvorsitzende: MinDir'in Prof. Dr. Veronika von Messling
Geschaeftsfuehrung: Prof. Dr. med. Dr. h.c. Matthias Tschoep, Heinrich Bassler, Kerstin
Guenther
Registergericht: Amtsgericht Muenchen HRB 6466
USt-IdNr: DE 129521671
3:56 p.m.
On Tue, Jun 11, 2019 at 19:19:05 +0200, Lentes, Bernd wrote:
[...]
Hi,
Hi,
it happened again.
Following the log of my script it started on 8th of june at 5:59:09 (UTC+2) to
blockcommit the domain.
These are the related lines in libvirtd.log:
===================================================
Thanks for comming back to me with the information.
Unfortunately this is not a full debug log but I can try to tell you
what I see here:
2019-06-07 20:30:57.170+0000: 30299: error : qemuMonitorIO:719 :
internal error: End of file from qemu monitor
2019-06-08 03:59:17.690+0000: 30299: error : qemuMonitorIO:719 : internal error: End of
file from qemu monitor
So this looks like qemu crashed. Or at least it's the usual symptom we
get. Is there anything in /var/log/libvirt/qemu/$VMNAME.log?
2019-06-08 03:59:26.145+0000: 30300: warning :
qemuGetProcessInfo:1461 : cannot parse process status data
2019-06-08 03:59:26.191+0000: 30303: warning : qemuGetProcessInfo:1461 : cannot parse
process status data
2019-06-08 03:59:56.095+0000: 27956: warning : qemuDomainObjBeginJobInternal:4865 :
Cannot start job (destroy, none) for domain severin; current job is (modify, none) owned
by (13061 remoteDispatchDomainBlockJobAbort, 0 <null>) for (38s,
0s)
And this looks to me as if the Abort job can't be interrupted properly
while waiting synchronously for the job to finish. This seems to be the
problem. If the VM indeed crashed there's a problem in job waiting
apparently.
I'd still really like to have debug logs in this case to really see what
happened.
7:08 p.m.
----- On Jun 13, 2019, at 9:56 AM, Peter Krempa pkrempa(a)redhat.com wrote:
Thanks for comming back to me with the information.
Unfortunately this is not a full debug log but I can try to tell you
what I see here:
I configured libvirtd that way:
ha-idg-1:~ # grep -Ev '^$|#' /etc/libvirt/libvirtd.conf
log_level = 1
log_filters="1:qemu 3:remote 4:event 3:util.json 3:rpc"
log_outputs="1:file:/var/log/libvirt/libvirtd.log"
keepalive_interval = -1
That's what i found on https://wiki.libvirt.org/page/DebugLogs .
Isn't that correct ? That should create informative logfiles.
The other host has excat the same configuration but produce much bigger logfiles !?!
I have libvirt-daemon-4.0.0-8.12.1.x86_64.
> 2019-06-07 20:30:57.170+0000: 30299: error : qemuMonitorIO:719 : internal error:
> End of file from qemu monitor
> 2019-06-08 03:59:17.690+0000: 30299: error : qemuMonitorIO:719 : internal error:
> End of file from qemu monitor
So this looks like qemu crashed. Or at least it's the usual symptom we
get. Is there anything in /var/log/libvirt/qemu/$VMNAME.log?
That's all:
qemu-system-x86_64: block/mirror.c:864: mirror_run: Assertion
`((&bs->tracked_requests)->lh_first == ((void *)0))' failed.
> 2019-06-08 03:59:26.145+0000: 30300: warning : qemuGetProcessInfo:1461 : cannot
> parse process status data
> 2019-06-08 03:59:26.191+0000: 30303: warning : qemuGetProcessInfo:1461 : cannot
> parse process status data
> 2019-06-08 03:59:56.095+0000: 27956: warning :
> qemuDomainObjBeginJobInternal:4865 : Cannot start job (destroy, none) for
> domain severin; current job is (modify, none) owned by (13061
> remoteDispatchDomainBlockJobAbort, 0 <null>) for (38s,
> 0s)
And this looks to me as if the Abort job can't be interrupted properly
while waiting synchronously for the job to finish. This seems to be the
problem. If the VM indeed crashed there's a problem in job waiting
apparently.
I'd still really like to have debug logs in this case to really see what
happened.
I configured logging as i found on https://wiki.libvirt.org/page/DebugLogs.
What else can i do ?
Bernd
Bernd
Helmholtz Zentrum Muenchen
Deutsches Forschungszentrum fuer Gesundheit und Umwelt (GmbH)
Ingolstaedter Landstr. 1
85764 Neuherberg
www.helmholtz-muenchen.de
Aufsichtsratsvorsitzende: MinDir'in Prof. Dr. Veronika von Messling
Geschaeftsfuehrung: Prof. Dr. med. Dr. h.c. Matthias Tschoep, Heinrich Bassler, Kerstin
Guenther
Registergericht: Amtsgericht Muenchen HRB 6466
USt-IdNr: DE 129521671
10:01 p.m.
----- On Jun 13, 2019, at 1:08 PM, Bernd Lentes bernd.lentes(a)helmholtz-muenchen.de wrote:
I found further information in /var/log/messages for both occurrences:
2019-06-01T03:05:31.620725+02:00 ha-idg-2 systemd-coredump[14253]: Core Dumping has been
disabled for process 30590 (qemu-system-x86).
2019-06-01T03:05:31.712673+02:00 ha-idg-2 systemd-coredump[14253]: Process 30590
(qemu-system-x86) of user 488 dumped core.
2019-06-01T03:05:32.173272+02:00 ha-idg-2 kernel: [294682.387828] br0: port 4(vnet2)
entered disabled state
2019-06-01T03:05:32.177111+02:00 ha-idg-2 kernel: [294682.388384] device vnet2 left
promiscuous mode
2019-06-01T03:05:32.177122+02:00 ha-idg-2 kernel: [294682.388391] br0: port 4(vnet2)
entered disabled state
2019-06-01T03:05:32.208916+02:00 ha-idg-2 wickedd[2954]: error retrieving tap attribute
from sysfs
2019-06-01T03:05:41.395685+02:00 ha-idg-2 systemd-machined[2824]: Machine qemu-31-severin
terminated.
2019-06-08T05:59:17.502899+02:00 ha-idg-1 systemd-coredump[31089]: Core Dumping has been
disabled for process 19489 (qemu-system-x86).
2019-06-08T05:59:17.523050+02:00 ha-idg-1 systemd-coredump[31089]: Process 19489
(qemu-system-x86) of user 489 dumped core.
2019-06-08T05:59:17.650334+02:00 ha-idg-1 kernel: [999258.577132] br0: port 9(vnet7)
entered disabled state
2019-06-08T05:59:17.650354+02:00 ha-idg-1 kernel: [999258.578103] device vnet7 left
promiscuous mode
2019-06-08T05:59:17.650355+02:00 ha-idg-1 kernel: [999258.578108] br0: port 9(vnet7)
entered disabled state
2019-06-08T05:59:25.983702+02:00 ha-idg-1 systemd-machined[1383]: Machine qemu-205-severin
terminated.
Core Dumping is disabled, but nevertheless a core dump has been created ?
Where could i find it ?
Would it be useful to provide it ?
Bernd
Helmholtz Zentrum Muenchen
Deutsches Forschungszentrum fuer Gesundheit und Umwelt (GmbH)
Ingolstaedter Landstr. 1
85764 Neuherberg
www.helmholtz-muenchen.de
Aufsichtsratsvorsitzende: MinDir'in Prof. Dr. Veronika von Messling
Geschaeftsfuehrung: Prof. Dr. med. Dr. h.c. Matthias Tschoep, Heinrich Bassler, Kerstin
Guenther
Registergericht: Amtsgericht Muenchen HRB 6466
USt-IdNr: DE 129521671
3:14 p.m.
On Thu, Jun 13, 2019 at 16:01:18 +0200, Lentes, Bernd wrote:
----- On Jun 13, 2019, at 1:08 PM, Bernd Lentes bernd.lentes(a)helmholtz-muenchen.de
wrote:
I found further information in /var/log/messages for both occurrences:
2019-06-01T03:05:31.620725+02:00 ha-idg-2 systemd-coredump[14253]: Core Dumping has been
disabled for process 30590 (qemu-system-x86).
2019-06-01T03:05:31.712673+02:00 ha-idg-2 systemd-coredump[14253]: Process 30590
(qemu-system-x86) of user 488 dumped core.
2019-06-01T03:05:32.173272+02:00 ha-idg-2 kernel: [294682.387828] br0: port 4(vnet2)
entered disabled state
2019-06-01T03:05:32.177111+02:00 ha-idg-2 kernel: [294682.388384] device vnet2 left
promiscuous mode
2019-06-01T03:05:32.177122+02:00 ha-idg-2 kernel: [294682.388391] br0: port 4(vnet2)
entered disabled state
2019-06-01T03:05:32.208916+02:00 ha-idg-2 wickedd[2954]: error retrieving tap attribute
from sysfs
2019-06-01T03:05:41.395685+02:00 ha-idg-2 systemd-machined[2824]: Machine qemu-31-severin
terminated.
2019-06-08T05:59:17.502899+02:00 ha-idg-1 systemd-coredump[31089]: Core Dumping has been
disabled for process 19489 (qemu-system-x86).
2019-06-08T05:59:17.523050+02:00 ha-idg-1 systemd-coredump[31089]: Process 19489
(qemu-system-x86) of user 489 dumped core.
2019-06-08T05:59:17.650334+02:00 ha-idg-1 kernel: [999258.577132] br0: port 9(vnet7)
entered disabled state
2019-06-08T05:59:17.650354+02:00 ha-idg-1 kernel: [999258.578103] device vnet7 left
promiscuous mode
2019-06-08T05:59:17.650355+02:00 ha-idg-1 kernel: [999258.578108] br0: port 9(vnet7)
entered disabled state
2019-06-08T05:59:25.983702+02:00 ha-idg-1 systemd-machined[1383]: Machine
qemu-205-severin terminated.
Core Dumping is disabled, but nevertheless a core dump has been created ?
Where could i find it ?
Would it be useful to provide it ?
So this really hints to qemu crashing. It certainly will be beneficial
to collect the backtrace, but you really should report this (including
the error message from the vm log file) to the qemu team.
They might have even fixed it by now, so a plain update might help.
9:33 p.m.
----- On Jun 14, 2019, at 9:14 AM, Peter Krempa pkrempa(a)redhat.com wrote:
On Thu, Jun 13, 2019 at 16:01:18 +0200, Lentes, Bernd wrote:
>
> ----- On Jun 13, 2019, at 1:08 PM, Bernd Lentes
> bernd.lentes(a)helmholtz-muenchen.de wrote:
>
> I found further information in /var/log/messages for both occurrences:
>
> 2019-06-01T03:05:31.620725+02:00 ha-idg-2 systemd-coredump[14253]: Core Dumping
> has been disabled for process 30590 (qemu-system-x86).
> 2019-06-01T03:05:31.712673+02:00 ha-idg-2 systemd-coredump[14253]: Process 30590
> (qemu-system-x86) of user 488 dumped core.
> 2019-06-01T03:05:32.173272+02:00 ha-idg-2 kernel: [294682.387828] br0: port
> 4(vnet2) entered disabled state
> 2019-06-01T03:05:32.177111+02:00 ha-idg-2 kernel: [294682.388384] device vnet2
> left promiscuous mode
> 2019-06-01T03:05:32.177122+02:00 ha-idg-2 kernel: [294682.388391] br0: port
> 4(vnet2) entered disabled state
> 2019-06-01T03:05:32.208916+02:00 ha-idg-2 wickedd[2954]: error retrieving tap
> attribute from sysfs
> 2019-06-01T03:05:41.395685+02:00 ha-idg-2 systemd-machined[2824]: Machine
> qemu-31-severin terminated.
>
>
> 2019-06-08T05:59:17.502899+02:00 ha-idg-1 systemd-coredump[31089]: Core Dumping
> has been disabled for process 19489 (qemu-system-x86).
> 2019-06-08T05:59:17.523050+02:00 ha-idg-1 systemd-coredump[31089]: Process 19489
> (qemu-system-x86) of user 489 dumped core.
> 2019-06-08T05:59:17.650334+02:00 ha-idg-1 kernel: [999258.577132] br0: port
> 9(vnet7) entered disabled state
> 2019-06-08T05:59:17.650354+02:00 ha-idg-1 kernel: [999258.578103] device vnet7
> left promiscuous mode
> 2019-06-08T05:59:17.650355+02:00 ha-idg-1 kernel: [999258.578108] br0: port
> 9(vnet7) entered disabled state
> 2019-06-08T05:59:25.983702+02:00 ha-idg-1 systemd-machined[1383]: Machine
> qemu-205-severin terminated.
>
> Core Dumping is disabled, but nevertheless a core dump has been created ?
> Where could i find it ?
> Would it be useful to provide it ?
So this really hints to qemu crashing. It certainly will be beneficial
to collect the backtrace, but you really should report this (including
the error message from the vm log file) to the qemu team.
They might have even fixed it by now, so a plain update might help.
Hi Peter,
thanks for your help. I'll continue on the Qemu ML:
https://lists.nongnu.org/archive/html/qemu-discuss/2019-06/msg00014.html
Bernd
Helmholtz Zentrum Muenchen
Deutsches Forschungszentrum fuer Gesundheit und Umwelt (GmbH)
Ingolstaedter Landstr. 1
85764 Neuherberg
www.helmholtz-muenchen.de
Aufsichtsratsvorsitzende: MinDir'in Prof. Dr. Veronika von Messling
Geschaeftsfuehrung: Prof. Dr. med. Dr. h.c. Matthias Tschoep, Heinrich Bassler, Kerstin
Guenther
Registergericht: Amtsgericht Muenchen HRB 6466
USt-IdNr: DE 129521671
1969
days inactive
1979
days old
11 comments
2 participants
participants (2)
-
Lentes, Bernd -
Peter Krempa