Re: virsh migrate fails when --copy-storage-all option is given!
CC: libvirt devel list Hi Kevin/Peter, Thank you so much for addressing this issue. I tried out few more things and here is my analysis: Even when I removed the readonly option from guest xml I was still seeing the issue in migration,In the qemu-commandline I could still see auto-read-only option being set as true by default. Then I tried giving the auto-read-only as false in the guest xml with qemu-commandline param, it was actually getting set to false and the migration worked! Steps I tried: 1) Started the guest with adding the snippet in the guest xml with parameters as: <qemu:commandline> <qemu:arg value='-blockdev'/> <qemu:arg value='driver=file,filename=/disk_nfs/nfs/migrate_root.qcow2,node-name=drivefile,auto-read-only=false'/> <qemu:arg value='-blockdev'/> <qemu:arg value='driver=qcow2,file=drivefile,node-name=drive0'/> <qemu:arg value='-device'/> <qemu:arg value='virtio-blk-pci,drive=drive0,id=virtio-disk0,bus=pci.0,addr=0x5'/> </qemu:commandline> 2) Started the migration and it worked. Could anyone please clarify from libvirt side what is the change required? Thanks, Anushree-Mathur On 04/06/25 6:57 PM, Peter Krempa wrote:
On Wed, Jun 04, 2025 at 14:41:54 +0200, Kevin Wolf wrote:
Am 28.05.2025 um 17:34 hat Peter Xu geschrieben:
Copy Kevin.
On Wed, May 28, 2025 at 07:21:12PM +0530, Anushree Mathur wrote:
Hi all,
When I am trying to migrate the guest from host1 to host2 with the command line as follows:
date;virsh migrate --live --domain guest1 qemu+ssh://dest/system --verbose --undefinesource --persistent --auto-converge --postcopy --copy-storage-all;date
and it fails with the following error message-
error: internal error: unable to execute QEMU command 'block-export-add': Block node is read-only
HOST ENV:
qemu : QEMU emulator version 9.2.2 libvirt : libvirtd (libvirt) 11.1.0 Seen with upstream qemu also
Steps to reproduce: 1) Start the guest1 2) Migrate it with the command as
date;virsh migrate --live --domain guest1 qemu+ssh://dest/system --verbose --undefinesource --persistent --auto-converge --postcopy --copy-storage-all;date
3) It fails as follows: error: internal error: unable to execute QEMU command 'block-export-add': Block node is read-only I assume this is about an inactive block node. Probably on the destination, but that's not clear to me from the error message. Yes this would be on the destination. Libvirt exports the nodes on destination, source connects and does the blockjob.
The destination side is configured the same way as the source side so if the source disk is configured as read-write the destination should be as well
Things I analyzed- 1) This issue is not happening if I give --unsafe option in the virsh migrate command This is weird; this shouldn't have any impact.
What does this translate to on the QEMU command line?
2) O/P of qemu-monitor command also shows ro as false
virsh qemu-monitor-command guest1 --pretty --cmd '{ "execute": "query-block" it'd be impossible to execute this on the guest due to timing; you'll need to collect libvirt debug logs to do that:
https://www.libvirt.org/kbase/debuglogs.html#tl-dr-enable-debug-logs-for-mos...
I also thing this should be eventually filed in a
}' { "return": [ { "io-status": "ok", "device": "", "locked": false, "removable": false, "inserted": { "iops_rd": 0, "detect_zeroes": "off", "image": { "virtual-size": 21474836480, "filename": "/home/Anu/guest_anu.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 5226561536, "format-specific": { "type": "qcow2", "data": { "compat": "1.1", "compression-type": "zlib", "lazy-refcounts": false, "refcount-bits": 16, "corrupt": false, "extended-l2": false } }, "dirty-flag": false }, "iops_wr": 0, "ro": false, "node-name": "libvirt-1-format", "backing_file_depth": 0, "drv": "qcow2", "iops": 0, "bps_wr": 0, "write_threshold": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "cache": { "no-flush": false, "direct": false, "writeback": true }, "file": "/home/Anu/guest_anu.qcow2" }, "qdev": "/machine/peripheral/virtio-disk0/virtio-backend", "type": "unknown" } ], "id": "libvirt-26" } I assume this is still from the source where the image is still active. Yes; on the destination the process wouldn't be around long enough to call 'virsh qemu-monitor-command'
Also it doesn't contain the "active" field yet that was recently introduced, which could show something about this. I believe you would still get "read-only": false for an inactive image if it's supposed to be read-write after the migration completes.
3) Guest doesn't have any readonly
virsh dumpxml guest1 | grep readonly
4) Tried giving the proper permissions also
-rwxrwxrwx. 1 qemu qemu 4.9G Apr 28 15:06 guest_anu.qcow Is this on the destination? did you pre-create it yourself? otherwise libvirt is pre-creating that image for-non-shared-storage migration (--copy-storage-all) which should have proper permissions when it's created
5) Checked for the permission of the pool also that is also proper!
6) Found 1 older bug similar to this, pasting the link for reference:
https://patchwork.kernel.org/project/qemu-devel/patch/20170811164854.GG4162@... What's happening in detail is more of a virsh/libvirt question. CCing Peter Krempa, he might have an idea. Please collect the debug log; at least from the destination side of migration. That should show how the VM is prepared and qemu invoked.
On Mon, Jun 23, 2025 at 23:25:28 +0530, Anushree Mathur wrote:
CC: libvirt devel list
Hi Kevin/Peter,
Thank you so much for addressing this issue. I tried out few more things and here is my analysis:
Even when I removed the readonly option from guest xml I was still seeing the issue in migration,In the qemu-commandline I could still see auto-read-only option being set as true by default.
'auto-read-only' is very likely not the problem here. Due to the name it's often suspected but it's really a red herring mostly. What it does is that it instructs qemu just to automatically switch between read-only and read-write as it did in the pre-blockdev era.
Then I tried giving the auto-read-only as false in the guest xml with qemu-commandline param, it was actually getting set to false and the migration worked!
Okay, based on what you've wrote I'm now even more confused what you wanted to do. Based on previous report [1] you wanted to migrate with non-shared storage (presence of --copy-storage-all which copies all read-write disks to destination). But now ...
Steps I tried:
1) Started the guest with adding the snippet in the guest xml with parameters as:
<qemu:commandline> <qemu:arg value='-blockdev'/> <qemu:arg value='driver=file,filename=/disk_nfs/nfs/migrate_root.qcow2,node-name=drivefile,auto-read-only=false'/>
... this looks like shared storage ...
<qemu:arg value='-blockdev'/> <qemu:arg value='driver=qcow2,file=drivefile,node-name=drive0'/> <qemu:arg value='-device'/> <qemu:arg value='virtio-blk-pci,drive=drive0,id=virtio-disk0,bus=pci.0,addr=0x5'/> </qemu:commandline>
2) Started the migration and it worked.
... and especially since you added it via the qemu:arg backdoor, which libvirt doesn't in any way interpret, it'd mean that --copy-storage-all fully ignores what was declared here. Thus it's weird that this would actually help anything especially when you originally used option meant for non-shared storage. So, how did you migrate this? What did you want to achieve?
Could anyone please clarify from libvirt side what is the change required?
You need to first clarify what you are actually doing. This seems to be different from what you tried last time. For any further investigation I'll need: - the full XML of the VM - debug logs from the source [2] - debug logs from the destination - description what you are trying to achieve
Thanks, Anushree-Mathur
[2] https://www.libvirt.org/kbase/debuglogs.html
On 04/06/25 6:57 PM, Peter Krempa wrote:
On Wed, Jun 04, 2025 at 14:41:54 +0200, Kevin Wolf wrote:
Am 28.05.2025 um 17:34 hat Peter Xu geschrieben:
Copy Kevin.
On Wed, May 28, 2025 at 07:21:12PM +0530, Anushree Mathur wrote:
Hi all,
When I am trying to migrate the guest from host1 to host2 with the command line as follows:
date;virsh migrate --live --domain guest1 qemu+ssh://dest/system --verbose --undefinesource --persistent --auto-converge --postcopy --copy-storage-all;date
and it fails with the following error message-
error: internal error: unable to execute QEMU command 'block-export-add': Block node is read-only
HOST ENV:
qemu : QEMU emulator version 9.2.2 libvirt : libvirtd (libvirt) 11.1.0 Seen with upstream qemu also
Steps to reproduce: 1) Start the guest1 2) Migrate it with the command as
date;virsh migrate --live --domain guest1 qemu+ssh://dest/system --verbose --undefinesource --persistent --auto-converge --postcopy --copy-storage-all;date
[1]
3) It fails as follows: error: internal error: unable to execute QEMU command 'block-export-add': Block node is read-only
I assume this is about an inactive block node. Probably on the destination, but that's not clear to me from the error message. Yes this would be on the destination. Libvirt exports the nodes on destination, source connects and does the blockjob.
The destination side is configured the same way as the source side so if the source disk is configured as read-write the destination should be as well
Things I analyzed- 1) This issue is not happening if I give --unsafe option in the virsh migrate command This is weird; this shouldn't have any impact.
What does this translate to on the QEMU command line?
2) O/P of qemu-monitor command also shows ro as false
virsh qemu-monitor-command guest1 --pretty --cmd '{ "execute": "query-block" it'd be impossible to execute this on the guest due to timing; you'll need to collect libvirt debug logs to do that:
https://www.libvirt.org/kbase/debuglogs.html#tl-dr-enable-debug-logs-for-mos...
I also thing this should be eventually filed in a
}' { "return": [ { "io-status": "ok", "device": "", "locked": false, "removable": false, "inserted": { "iops_rd": 0, "detect_zeroes": "off", "image": { "virtual-size": 21474836480, "filename": "/home/Anu/guest_anu.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 5226561536, "format-specific": { "type": "qcow2", "data": { "compat": "1.1", "compression-type": "zlib", "lazy-refcounts": false, "refcount-bits": 16, "corrupt": false, "extended-l2": false } }, "dirty-flag": false }, "iops_wr": 0, "ro": false, "node-name": "libvirt-1-format", "backing_file_depth": 0, "drv": "qcow2", "iops": 0, "bps_wr": 0, "write_threshold": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "cache": { "no-flush": false, "direct": false, "writeback": true }, "file": "/home/Anu/guest_anu.qcow2" }, "qdev": "/machine/peripheral/virtio-disk0/virtio-backend", "type": "unknown" } ], "id": "libvirt-26" } I assume this is still from the source where the image is still active. Yes; on the destination the process wouldn't be around long enough to call 'virsh qemu-monitor-command'
Also it doesn't contain the "active" field yet that was recently introduced, which could show something about this. I believe you would still get "read-only": false for an inactive image if it's supposed to be read-write after the migration completes.
3) Guest doesn't have any readonly
virsh dumpxml guest1 | grep readonly
4) Tried giving the proper permissions also
-rwxrwxrwx. 1 qemu qemu 4.9G Apr 28 15:06 guest_anu.qcow Is this on the destination? did you pre-create it yourself? otherwise libvirt is pre-creating that image for-non-shared-storage migration (--copy-storage-all) which should have proper permissions when it's created
5) Checked for the permission of the pool also that is also proper!
6) Found 1 older bug similar to this, pasting the link for reference:
https://patchwork.kernel.org/project/qemu-devel/patch/20170811164854.GG4162@... What's happening in detail is more of a virsh/libvirt question. CCing Peter Krempa, he might have an idea. Please collect the debug log; at least from the destination side of migration. That should show how the VM is prepared and qemu invoked.
participants (2)
-
Anushree Mathur -
Peter Krempa