Hanging migration leaving domain in paused state
by Yangchen Ye
Hi,
We are running libvirt 8.0.0, and sometimes live migration could not finish
(because the guest is dirtying the memory too fast). We implemented a
monitor that increases max downtime when it observed that "Data Remaining"
bumps up. But we found a strange sequence of events from the monitor, which
leads to a paused domain on the destination hypervisor:
The monitor sees Data Remaining bumping up and increases max downtime up to
20 seconds, but weird thing is that after a period of time, it started
reporting "Data Remaining" and "Data Total" is both 0, but the migration
job is still unfinished:
"Migration in progress - DataTotal: 85
904728064, DataRemaining: 22201458688, TimeElapsed: 20005, MaxDowntime:
500, DirtyRate: 0"
"Migration in progress - DataTotal:
85904728064, DataRemaining: 43801825280, TimeElapsed: 10005, MaxDowntime:
500, DirtyRate: 0"
"Migration in progress - DataTotal:
85904728064, DataRemaining: 52382912512, TimeElapsed: 10004, MaxDowntime:
500, DirtyRate: 0" (DataRemaining bumps up, we start increasing max
downtime)
"Migration in progress - DataTotal:
85904728064, DataRemaining: 4219596800, TimeElapsed: 40004, MaxDowntime:
1500, DirtyRate: 0" (Last poll where we see the job info)
After which monitor logs
"Migration in progress - DataTotal:
0, DataRemaining: 0, TimeElapsed: 40004, MaxDowntime: 13500, DirtyRate: 0"
The domain is always running on the source hypervisor but there is a paused
domain on the destination hypervisor which is paused at start up.
Trying to understand what might have happened:
- Is this a known issue for live migrating high memory activity guests and
the way we interact with libvirt?
- What is the recommended way to ensure that a started live migration
always run to completion if we don't care about downtime?
Appreciate any help here
Yangchen Ye
1 day, 2 hours
How to recover from "Display output is not active"
by Jeffrey Walton
Hi Everyone,
I'm working on Ubuntu 22.04.5, amd64, fully patched. It provides QEMU
1:6.2+dfsg-2ubuntu6.26 and libvirt 8.0.0-1ubuntu7.11.
The problem I am experiencing is, when the host machine blanks the
monitor, I lose the display on the guest when I sit back down at the
computer and wake the monitor. The error message (?) in the guest
window is, "Display output is not active." The condition happens with
both VIRTIO and VGA models. Also see <https://imgur.com/a/tEp73k2>.
I have not found a workaround to stop it from happening or recover
after it happens. I have to Force Off the VM and then restart the VM
to the display back.
My host power configuration is: disable sleep, suspend, hibernate and
hybrid-sleep via systemd's systemctl. However, I allow the monitor to
be turned off after 15 minutes of inactivity. My guest power
configuration is whatever the distro provides.
Based on my searching, others have experienced the problem. And one
fellow says switching to the VGA model fixed it for him. (But no joy
for me).
Does anyone have any suggestions for a work around?
Thanks in advance. (And my apologies if this is the wrong list).
Jeff
1 week
How to disable sysctl calls performed by libvirt?
by Chimmie Firefly
Hello,
I have an issue whenever using `/proc/sys/ipv6/conf` as read-only mountpoint.
While the libvirt by itself is running fine, I'm unable to create NAT network interfaces because of the following error:
```
2025-07-05 19:33:32.821+0000: 100: error : networkSetIPv6Sysctl:1847 : cannot write to '/proc/sys/net/ipv6/conf/virbr0/disable_ipv6' on bridge 'virbr0': Read-only file system
2025-07-05 19:33:32.841+0000: 100: error : virNetDevSetIFFlag:601 : Cannot get interface flags on 'virbr0': No such device
2025-07-05 19:33:32.947+0000: 100: error : virNetlinkDelLink:688 : error destroying network device virbr0: No such device
```
While using iproute2 and bridge utils, I'm able to create and configure new bridge device, along with attaching the VM to the interface.
I've used the following commands:
```
sysctl -w net.ipv4.ip_forward=1
brctl addbr br0
ip link set br0 up
ip addr add 192.168.100.1/24 dev br0
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
iptables -A FORWARD -i br0 -o eth0 -j ACCEPT
iptables -A FORWARD -i eth0 -o br0 -m state --state RELATED,ESTABLISHED -j ACCEPT
```
Is libvirt allowing to disable sysctl calls in the configuration files? If so, what should I set in the config?
1 week, 4 days
List checkpoints
by francisco.garcia＠baculasystems.com
Hello,
I'm using checkpoints to back up Libvirt domains. I'm using the "virsh backup-begin" tool and specifying the dombackup and checkpoint XML files. Everything works correctly (I perform full and incremental backups) without any issues.
When I list the checkpoints using the "virsh checkpoint-list DOMAIN --parent --topological" tool, it correctly shows me the entire chain of full and incremental backups.
However, when I perform a differential backup, setting the parent of that checkpoint to the last full checkpoint, I have problems when listing them. It tells me that the parent of the differential checkpoint is the last incremental checkpoint, instead of the full checkpoint. And I don't understand why, because when I perform the backup-begin, I specify the full checkpoint in the "incremental" label, and the bitmap generated from the disk is all the data from the full checkpoint on.
I put the xmls that I used:
FULL BACKUP:
dombackup.xml:
```
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<domainbackup mode="pull">
<server name="localhost" port="10809"/>
<disks>
<disk name="vda" type="file">
<scratch file="/tmp/libvirt-full-30-vda-debian12.scratch"/>
</disk>
<disk name="vdb" type="file">
<scratch file="/tmp/libvirt-full-30-vdb-debian12.scratch"/>
</disk>
</disks>
</domainbackup>
```
checkpoint.xml
```
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<domaincheckpoint>
<name>Full-30-Debian12</name>
<disks>
<disk name="vda" bitmap="Full-30-Debian12" checkpoint="bitmap"/>
<disk name="vdb" bitmap="Full-30-Debian12" checkpoint="bitmap"/>
<disk name="sda" checkpoint="no"/>
</disks>
</domaincheckpoint>
```
--------------------------------------------------------
INCREMENTAL BACKUP:
dombackup.xml
```
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<domainbackup mode="pull">
<server name="localhost" port="10809"/>
<incremental>Full-30-Debian12</incremental>
<disks>
<disk name="vda" type="file">
<scratch file="/tmp/libvirt-incremental-35-vda-debian12.scratch"/>
</disk>
<disk name="vdb" type="file">
<scratch file="/tmp/libvirt-incremental-35-vdb-debian12.scratch"/>
</disk>
</disks>
</domainbackup>
```
checkpoint.xml
```
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<domaincheckpoint>
<name>Incremental-31-Debian12</name>
<parent>
<name>Full-30-Debian12</name>
</parent>
<disks>
<disk name="vda" bitmap="Incremental-31-Debian12" checkpoint="bitmap"/>
<disk name="vdb" bitmap="Incremental-31-Debian12" checkpoint="bitmap"/>
<disk name="sda" checkpoint="no"/>
</disks>
</domaincheckpoint>
```
-----------------------------------------------------------------
DIFFERENTIAL BACKUP:
dombackup.xml
```
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<domainbackup mode="pull">
<server name="localhost" port="10809"/>
<incremental>Full-30-Debian12</incremental>
<disks>
<disk name="vda" type="file">
<scratch file="/tmp/libvirt-differential-36-vda-debian12.scratch"/>
</disk>
<disk name="vdb" type="file">
<scratch file="/tmp/libvirt-differential-36-vdb-debian12.scratch"/>
</disk>
</disks>
</domainbackup>
```
checkpoint.xml
```
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<domaincheckpoint>
<name>Differential-32-Debian12</name>
<parent>
<name>Full-30-Debian12</name>
</parent>
<disks>
<disk name="vda" bitmap="Differential-32-Debian12" checkpoint="bitmap"/>
<disk name="vdb" bitmap="Differential-32-Debian12" checkpoint="bitmap"/>
<disk name="sda" checkpoint="no"/>
</disks>
</domaincheckpoint>
```
-----------------------------------------------------------------------
After apply several "virsh backup-begin" with these files:
```
virsh checkpoint-list debian12 --topological --parent
Name Creation Time Parent
--------------------------------------------------------------------------------------------------------------------------------
Full-30-Debian12 2025-07-09 17:08:31 +0200
Incremental-31-Debian12 2025-07-09 17:34:56 +0200 Full-30-Debian12
Differential-32-Debian12 2025-07-09 17:37:51 +0200 Incremental-31-Debian12
```
I do not know why the parent of "Differential-32-Debian12" is "Incremental-31-Debian12" instead of "Full-30-Debian12". Any ideas?
1 week, 6 days
fsck unable to fix fs issue
by ola .
I am able to boot VMs by using rbd as the root disk. When I restart and stop the VM everything works fine however, anytime the host goes down say due to a power outage, when next I try to boot the VM, I run into a situation where the root disk gets corrupted and get stuck at "initramfs". I have tried to fix this but to no avail. Here are the errors I get when I try yo use fix the fs issue with fsck manually.
done.
Begin: Running /scripts/init-premount ... done.
Begin: Mounting root file system ... Begin: Running /scripts/local-top ... done.
Begin: Running /scripts/local-premount ... [ 7.760625] Btrfs loaded, crc32c=crc32c-intel, zoned=yes, fsverity=yes
Scanning for Btrfs filesystems
done.
Begin: Will now check root file system ... fsck from util-linux 2.37.2
[/usr/sbin/fsck.ext4 (1) -- /dev/vda1] fsck.ext4 -a -C0 /dev/vda1
[ 7.866954] blk_update_request: I/O error, dev vda, sector 0 op 0x1:(WRITE) flags 0x800 phys_seg 0 prio class 0
cloudimg-rootfs: recovering journal
[ 8.164279] blk_update_request: I/O error, dev vda, sector 227328 op 0x1:(WRITE) flags 0x800 phys_seg 24 prio class 0
[ 8.168272] Buffer I/O error on dev vda1, logical block 0, lost async page write
[ 8.170413] Buffer I/O error on dev vda1, logical block 1, lost async page write
[ 8.172545] Buffer I/O error on dev vda1, logical block 2, lost async page write
[ 8.174601] Buffer I/O error on dev vda1, logical block 3, lost async page write
[ 8.176651] Buffer I/O error on dev vda1, logical block 4, lost async page write
[ 8.178694] Buffer I/O error on dev vda1, logical block 5, lost async page write
[ 8.180601] Buffer I/O error on dev vda1, logical block 6, lost async page write
[ 8.182641] Buffer I/O error on dev vda1, logical block 7, lost async page write
[ 8.184710] Buffer I/O error on dev vda1, logical block 8, lost async page write
[ 8.186744] Buffer I/O error on dev vda1, logical block 9, lost async page write
[ 8.188748] blk_update_request: I/O error, dev vda, sector 229392 op 0x1:(WRITE) flags 0x800 phys_seg 8 prio class 0
[ 8.191433] blk_update_request: I/O error, dev vda, sector 229440 op 0x1:(WRITE) flags 0x800 phys_seg 32 prio class 0
[ 8.194204] blk_update_request: I/O error, dev vda, sector 229480 op 0x1:(WRITE) flags 0x800 phys_seg 16 prio class 0
[ 8.196976] blk_update_request: I/O error, dev vda, sector 229512 op 0x1:(WRITE) flags 0x800 phys_seg 8 prio class 0
[ 8.243612] blk_update_request: I/O error, dev vda, sector 229544 op 0x1:(WRITE) flags 0x800 phys_seg 8 prio class 0
[ 8.246068] blk_update_request: I/O error, dev vda, sector 229640 op 0x1:(WRITE) flags 0x800 phys_seg 32 prio class 0
[ 8.248668] blk_update_request: I/O error, dev vda, sector 229688 op 0x1:(WRITE) flags 0x800 phys_seg 8 prio class 0
[ 8.251174] blk_update_request: I/O error, dev vda, sector 229704 op 0x1:(WRITE) flags 0x800 phys_seg 8 prio class 0
fsck.ext4: Input/output error while recovering journal of cloudimg-rootfs
fsck.ext4: unable to set superblock flags on cloudimg-rootfs
cloudimg-rootfs: ********** WARNING: Filesystem still has errors **********
fsck exited with status code 12
done.
Failure: File system check of the root filesystem failed
The root filesystem on /dev/vda1 requires a manual fsck
BusyBox v1.30.1 (Ubuntu 1:1.30.1-7ubuntu3.1) built-in shell (ash)
Enter 'help' for a list of built-in commands.
(initramfs) fsck.ext4 -f -y /dev/vda1
e2fsck 1.46.5 (30-Dec-2021)
[ 24.286341] print_req_error: 174 callbacks suppressed
[ 24.286358] blk_update_request: I/O error, dev vda, sector 0 op 0x1:(WRITE) flags 0x800 phys_seg 0 prio class 0
cloudimg-rootfs: recovering journal
[ 24.552343] blk_update_request: I/O error, dev vda, sector 227328 op 0x1:(WRITE) flags 0x800 phys_seg 24 prio class 0
[ 24.556674] buffer_io_error: 5222 callbacks suppressed
[ 24.558925] Buffer I/O error on dev vda1, logical block 0, lost async page write
[ 24.562116] Buffer I/O error on dev vda1, logical block 1, lost async page write
[ 24.565161] Buffer I/O error on dev vda1, logical block 2, lost async page write
[ 24.567872] Buffer I/O error on dev vda1, logical block 3, lost async page write
[ 24.570586] Buffer I/O error on dev vda1, logical block 4, lost async page write
[ 24.573418] Buffer I/O error on dev vda1, logical block 5, lost async page write
[ 24.575940] Buffer I/O error on dev vda1, logical block 6, lost async page write
[ 24.578622] Buffer I/O error on dev vda1, logical block 7, lost async page write
[ 24.581386] Buffer I/O error on dev vda1, logical block 8, lost async page write
[ 24.583873] Buffer I/O error on dev vda1, logical block 9, lost async page write
[ 24.586410] blk_update_request: I/O error, dev vda, sector 229392 op 0x1:(WRITE) flags 0x800 phys_seg 8 prio class 0
[ 24.589821] blk_update_request: I/O error, dev vda, sector 229440 op 0x1:(WRITE) flags 0x800 phys_seg 32 prio class 0
[ 24.593380] blk_update_request: I/O error, dev vda, sector 229480 op 0x1:(WRITE) flags 0x800 phys_seg 16 prio class 0
[ 24.596615] blk_update_request: I/O error, dev vda, sector 229512 op 0x1:(WRITE) flags 0x800 phys_seg 8 prio class 0
[ 24.643829] blk_update_request: I/O error, dev vda, sector 229544 op 0x1:(WRITE) flags 0x800 phys_seg 8 prio class 0
[ 24.646924] blk_update_request: I/O error, dev vda, sector 229640 op 0x1:(WRITE) flags 0x800 phys_seg 32 prio class 0
[ 24.650051] blk_update_request: I/O error, dev vda, sector 229688 op 0x1:(WRITE) flags 0x800 phys_seg 8 prio class 0
[ 24.653128] blk_update_request: I/O error, dev vda, sector 229704 op 0x1:(WRITE) flags 0x800 phys_seg 8 prio class 0
fsck.ext4: Input/output error while recovering journal of cloudimg-rootfs
fsck.ext4: unable to set superblock flags on cloudimg-rootfs
cloudimg-rootfs: ********** WARNING: Filesystem still has errors **********
This is what my rbd disk looks like
<disk type='network' device='disk'>
<driver name='qemu' type='raw' cache='none' io='native'/>
<auth username='dove'>
<secret type='ceph' uuid='b608caae-5eb4-45cc-bfd4-0b4ac11c7613'/>
</auth>
<source protocol='rbd' name='vms/wing-64700f1d-8c469a54-3f50-4d1e-9db2-2b6ea5f3d14a'>
<host name='x.168.1.x' port='6789'/>
</source>
<target dev='vda' bus='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</disk>
So, my questions are;
- How do I prevent this from happening as i have tried different options like changing the "cache" value for the disk template?
- How can this be fixed?
Thanks
2 weeks