On Tue, Apr 04, 2023 at 16:28:18 +0200, André Malm wrote:

Hello,

For some vms the virsh backup-begin sometimes shuts off the vm and returns "error: operation failed: domain is not running" although it was clearly in state running (or paused).

Is the idea that you should guest-fsfreeze-freeze / virsh suspend before virsh backup-begin? I have tried with both with the same results.

Freezing the guest filesystems is a good idea to increase the data consistency of the backup, but is not necessary. Nor it should have any influence on the lifecycle of the VM.

What could be causing the machine to shut off?

The VM most likely crashed, or was turned off in a different way. Try running virsh domstate --reason $VMNAME to see what the reason for the current state is.

(preferrably don't top-post on technical lists) On Wed, Apr 05, 2023 at 07:44:21 +0200, André Malm wrote:

The reason given is shut off (crashed).

So something virsh backup-begin does is causing he guest to crash?

The backup operation is quite complex so it is possible. Please have a look into /var/log/libvirt/qemu/$VMNAME.log to see whether qemu logged something like an assertion failure before crashing. Additionally you can have a look into 'coredumpctl' whether there are any recorded crashes of 'qemu-system-x86_64' and also possibly collect the backtrace. Also make sure to try updating the qemu package and see whether the bug reproduces. If yes, please collect the stack/back-trace, versions of qemu and libvirt, the contents of the VM log file and also ideally configure libvirt for debug logging and collect the debug log as well: https://www.libvirt.org/kbase/debuglogs.html

Den 2023-04-04 kl. 16:58, skrev Peter Krempa:

...

On Tue, Apr 04, 2023 at 16:28:18 +0200, André Malm wrote:

...

Hello,

For some vms the virsh backup-begin sometimes shuts off the vm and returns "error: operation failed: domain is not running" although it was clearly in state running (or paused).

Is the idea that you should guest-fsfreeze-freeze / virsh suspend before virsh backup-begin? I have tried with both with the same results. Freezing the guest filesystems is a good idea to increase the data consistency of the backup, but is not necessary. Nor it should have any influence on the lifecycle of the VM.

...

What could be causing the machine to shut off? The VM most likely crashed, or was turned off in a different way.

Try running

virsh domstate --reason $VMNAME

to see what the reason for the current state is.

On Wed, Apr 05, 2023 at 17:37:31 +0200, André Malm wrote:

Den 2023-04-05 kl. 09:47, skrev Peter Krempa:

...

The backup operation is quite complex so it is possible. Please have a look into /var/log/libvirt/qemu/$VMNAME.log to see whether qemu logged something like an assertion failure before crashing.

Additionally you can have a look into 'coredumpctl' whether there are any recorded crashes of 'qemu-system-x86_64' and also possibly collect the backtrace.

Also make sure to try updating the qemu package and see whether the bug reproduces.

If yes, please collect the stack/back-trace, versions of qemu and libvirt, the contents of the VM log file and also ideally configure libvirt for debug logging and collect the debug log as well:

https://www.libvirt.org/kbase/debuglogs.html

In the $VMNAME.log: qemu-system-x86_64: ../../block/qcow2.c:5175: qcow2_get_specific_info: Assertion `false' failed.

I'm running libvirt 8.0.0, which is the latest version for my dist (ubuntu 22.04). If required how would I properly upgrade?

Looking at https://github.com/qemu/qemu/blob/0c8022876f2183f93e23a7314862140c94ee62e7/b... which would be the version of qcow2.c for 8.0.0 there seem to be some issue with qcow2 compat.

Huh, that is weird. Both images seem to be qcow2v3 so it's weird that the code reaches the assertion. I think at this point you should report an issue with qemu: https://gitlab.com/qemu-project/qemu/-/issues or report it on the qemu-block@nongnu.org mailing list You'll be asked for what operations lead to the failure so please make sure to collect the libvirt debug log as I've requested. I can help the qemu team to analyze it so make sure to mention me (or my gitlab handle 'pipo.sk') on the issue.