8:52 a.m.
Hi, all
I've been having an annoying issue with live VM + volume migration. When
I try to use |virDomainMigrate2| to migrate a VM and some of its volumes
(but not all) I get the following error:
|unable to execute QEMU command 'block-export-add': Block node is read-only |
Some additional context:
* I originally found this issue using the java bindings, but I've been
able to reproduce it using virsh;
* I'm using Libvirt 8.0.0 and Qemu 6.2.0;
* This only happens when the VM has volumes on NFS storage;
* The issue does not happen when migrating all the volumes that are on
NFS storage. Note that, if the VM has volumes that are on some other
type of storage, such as ISCSI, those do not need to be migrated.
However, migrating any volume will raise the error if the volumes on
NFS are not migrated as well;
* I've tried a number of valid combinations along the
|VIR_MIGRATE_LIVE| flag, using the flags: |VIR_MIGRATE_PAUSED|,
|VIR_MIGRATE_NON_SHARED_DISK|, |VIR_MIGRATE_NON_SHARED_INC|,
|VIR_MIGRATE_UNSAFE|, |VIR_MIGRATE_COMPRESSED|.
The easiest way I've found to reproduce this error was using the
following steps:
1. Create a VM using virt-manager, it must have at least two volumes,
at least one of them must be on NFS storage. Here's the XML of one
of the VMs I've tested with:
|<domain type='kvm' id='4'> <name>ubuntu22.04</name>
<uuid>9965e409-c89d-408c-a9f7-488184566d56</uuid> <metadata>
<libosinfo:libosinfo
xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0"...
<libosinfo:os id="http://ubuntu.com/ubuntu/22.04"/>
</libosinfo:libosinfo> </metadata> <memory
unit='KiB'>2097152</memory>
<currentMemory unit='KiB'>2097152</currentMemory> <vcpu
placement='static'>1</vcpu> <resource>
<partition>/machine</partition>
</resource> <os> <type arch='x86_64'
machine='pc-q35-6.2'>hvm</type>
<boot dev='hd'/> </os> <features> <acpi/> <apic/>
<vmport state='off'/>
</features> <cpu mode='host-passthrough' check='none'
migratable='on'/>
<clock offset='utc'> <timer name='rtc'
tickpolicy='catchup'/> <timer
name='pit' tickpolicy='delay'/> <timer name='hpet'
present='no'/>
</clock> <on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot> <on_crash>destroy</on_crash>
<pm>
<suspend-to-mem enabled='no'/> <suspend-to-disk enabled='no'/>
</pm>
<devices> <emulator>/usr/bin/qemu-system-x86_64</emulator> <disk
type='file' device='disk'> <driver name='qemu'
type='qcow2'/> <source
file='/mnt/nfs-storage/template.qcow2' index='1'/>
<backingStore/>
<target dev='vda' bus='virtio'/> <alias
name='virtio-disk0'/> <address
type='pci' domain='0x0000' bus='0x04' slot='0x00'
function='0x0'/>
</disk> <disk type='file' device='disk'> <driver
name='qemu'
type='qcow2'/> <source file='/mnt/nfs-storage/vol.qcow2'
index='2'/>
<backingStore/> <target dev='vdb' bus='virtio'/> <alias
name='virtio-disk1'/> <address type='pci' domain='0x0000'
bus='0x08'
slot='0x00' function='0x0'/> </disk> <controller
type='usb' index='0'
model='qemu-xhci' ports='15'> <alias name='usb'/>
<address type='pci'
domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
</controller>
<controller type='pci' index='0' model='pcie-root'>
<alias
name='pcie.0'/> </controller> <controller type='pci'
index='1'
model='pcie-root-port'> <model name='pcie-root-port'/> <target
chassis='1' port='0x10'/> <alias name='pci.1'/>
<address type='pci'
domain='0x0000' bus='0x00' slot='0x02' function='0x0'
multifunction='on'/> </controller> <controller type='pci'
index='2'
model='pcie-root-port'> <model name='pcie-root-port'/> <target
chassis='2' port='0x11'/> <alias name='pci.2'/>
<address type='pci'
domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
</controller>
<controller type='pci' index='3' model='pcie-root-port'>
<model
name='pcie-root-port'/> <target chassis='3' port='0x12'/>
<alias
name='pci.3'/> <address type='pci' domain='0x0000'
bus='0x00'
slot='0x02' function='0x2'/> </controller> <controller
type='pci'
index='4' model='pcie-root-port'> <model
name='pcie-root-port'/> <target
chassis='4' port='0x13'/> <alias name='pci.4'/>
<address type='pci'
domain='0x0000' bus='0x00' slot='0x02' function='0x3'/>
</controller>
<controller type='pci' index='5' model='pcie-root-port'>
<model
name='pcie-root-port'/> <target chassis='5' port='0x14'/>
<alias
name='pci.5'/> <address type='pci' domain='0x0000'
bus='0x00'
slot='0x02' function='0x4'/> </controller> <controller
type='pci'
index='6' model='pcie-root-port'> <model
name='pcie-root-port'/> <target
chassis='6' port='0x15'/> <alias name='pci.6'/>
<address type='pci'
domain='0x0000' bus='0x00' slot='0x02' function='0x5'/>
</controller>
<controller type='pci' index='7' model='pcie-root-port'>
<model
name='pcie-root-port'/> <target chassis='7' port='0x16'/>
<alias
name='pci.7'/> <address type='pci' domain='0x0000'
bus='0x00'
slot='0x02' function='0x6'/> </controller> <controller
type='pci'
index='8' model='pcie-root-port'> <model
name='pcie-root-port'/> <target
chassis='8' port='0x17'/> <alias name='pci.8'/>
<address type='pci'
domain='0x0000' bus='0x00' slot='0x02' function='0x7'/>
</controller>
<controller type='pci' index='9' model='pcie-root-port'>
<model
name='pcie-root-port'/> <target chassis='9' port='0x18'/>
<alias
name='pci.9'/> <address type='pci' domain='0x0000'
bus='0x00'
slot='0x03' function='0x0' multifunction='on'/>
</controller>
<controller type='pci' index='10' model='pcie-root-port'>
<model
name='pcie-root-port'/> <target chassis='10'
port='0x19'/> <alias
name='pci.10'/> <address type='pci' domain='0x0000'
bus='0x00'
slot='0x03' function='0x1'/> </controller> <controller
type='pci'
index='11' model='pcie-root-port'> <model
name='pcie-root-port'/>
<target chassis='11' port='0x1a'/> <alias
name='pci.11'/> <address
type='pci' domain='0x0000' bus='0x00' slot='0x03'
function='0x2'/>
</controller> <controller type='pci' index='12'
model='pcie-root-port'>
<model name='pcie-root-port'/> <target chassis='12'
port='0x1b'/> <alias
name='pci.12'/> <address type='pci' domain='0x0000'
bus='0x00'
slot='0x03' function='0x3'/> </controller> <controller
type='pci'
index='13' model='pcie-root-port'> <model
name='pcie-root-port'/>
<target chassis='13' port='0x1c'/> <alias
name='pci.13'/> <address
type='pci' domain='0x0000' bus='0x00' slot='0x03'
function='0x4'/>
</controller> <controller type='pci' index='14'
model='pcie-root-port'>
<model name='pcie-root-port'/> <target chassis='14'
port='0x1d'/> <alias
name='pci.14'/> <address type='pci' domain='0x0000'
bus='0x00'
slot='0x03' function='0x5'/> </controller> <controller
type='sata'
index='0'> <alias name='ide'/> <address type='pci'
domain='0x0000'
bus='0x00' slot='0x1f' function='0x2'/> </controller>
<controller
type='virtio-serial' index='0'> <alias
name='virtio-serial0'/> <address
type='pci' domain='0x0000' bus='0x03' slot='0x00'
function='0x0'/>
</controller> <controller type='scsi' index='0'
model='virtio-scsi'>
<alias name='scsi0'/> <address type='pci' domain='0x0000'
bus='0x07'
slot='0x00' function='0x0'/> </controller> <interface
type='bridge'>
<mac address='52:54:00:c4:a7:6d'/> <source bridge='br-test'/>
<target
dev='vnet6'/> <model type='virtio'/> <alias
name='net0'/> <address
type='pci' domain='0x0000' bus='0x01' slot='0x00'
function='0x0'/>
</interface> <serial type='pty'> <source
path='/dev/pts/3'/> <target
type='isa-serial' port='0'> <model name='isa-serial'/>
</target> <alias
name='serial0'/> </serial> <console type='pty'
tty='/dev/pts/3'> <source
path='/dev/pts/3'/> <target type='serial' port='0'/>
<alias
name='serial0'/> </console> <channel type='unix'>
<source mode='bind'
path='/var/lib/libvirt/qemu/channel/target/domain-4-ubuntu22.04/org.qemu.guest_agent.0'/>
<target type='virtio' name='org.qemu.guest_agent.0'
state='disconnected'/> <alias name='channel0'/> <address
type='virtio-serial' controller='0' bus='0' port='1'/>
</channel>
<channel type='spicevmc'> <target type='virtio'
name='com.redhat.spice.0' state='connected'/> <alias
name='channel1'/>
<address type='virtio-serial' controller='0' bus='0'
port='2'/>
</channel> <input type='tablet' bus='usb'> <alias
name='input0'/>
<address type='usb' bus='0' port='1'/> </input>
<input type='mouse'
bus='ps2'> <alias name='input1'/> </input> <input
type='keyboard'
bus='ps2'> <alias name='input2'/> </input> <graphics
type='spice'
port='5901' autoport='yes' listen='127.0.0.1'> <listen
type='address'
address='127.0.0.1'/> </graphics> <sound model='ich9'>
<alias
name='sound0'/> <address type='pci' domain='0x0000'
bus='0x00'
slot='0x1b' function='0x0'/> </sound> <audio id='1'
type='spice'/>
<video> <model type='qxl' ram='65536' vram='65536'
vgamem='16384'
heads='1' primary='yes'/> <alias name='video0'/>
<address type='pci'
domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
</video>
<redirdev bus='usb' type='spicevmc'> <alias
name='redir0'/> <address
type='usb' bus='0' port='2'/> </redirdev> <redirdev
bus='usb'
type='spicevmc'> <alias name='redir1'/> <address
type='usb' bus='0'
port='3'/> </redirdev> <memballoon model='virtio'>
<alias
name='balloon0'/> <address type='pci' domain='0x0000'
bus='0x05'
slot='0x00' function='0x0'/> </memballoon> <rng
model='virtio'> <backend
model='random'>/dev/urandom</backend> <alias name='rng0'/>
<address
type='pci' domain='0x0000' bus='0x06' slot='0x00'
function='0x0'/>
</rng> </devices> <seclabel type='dynamic' model='dac'
relabel='yes'>
<label>+0:+0</label> <imagelabel>+0:+0</imagelabel>
</seclabel> </domain> |
2. Use |virsh migrate --domain <name> --copy-storage-all --live
--desturi <other_host_uri> --xml ./newxml| to migrate the VM and one
of its volumes, while leaving a volume that is on NFS without
migration. I've tested migrating the volume to another NFS storage,
as well as migrating to a ISCSI storage that is mounted on both
hosts, the result is the same. Here's the content of the |newxml|
for this example:
|<domain type='kvm' id='4'> <name>ubuntu22.04</name>
<uuid>9965e409-c89d-408c-a9f7-488184566d56</uuid> <metadata>
<libosinfo:libosinfo
xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0"...
<libosinfo:os id="http://ubuntu.com/ubuntu/22.04"/>
</libosinfo:libosinfo> </metadata> <memory
unit='KiB'>2097152</memory>
<currentMemory unit='KiB'>2097152</currentMemory> <vcpu
placement='static'>1</vcpu> <resource>
<partition>/machine</partition>
</resource> <os> <type arch='x86_64'
machine='pc-q35-6.2'>hvm</type>
<boot dev='hd'/> </os> <features> <acpi/> <apic/>
<vmport state='off'/>
</features> <cpu mode='host-passthrough' check='none'
migratable='on'/>
<clock offset='utc'> <timer name='rtc'
tickpolicy='catchup'/> <timer
name='pit' tickpolicy='delay'/> <timer name='hpet'
present='no'/>
</clock> <on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot> <on_crash>destroy</on_crash>
<pm>
<suspend-to-mem enabled='no'/> <suspend-to-disk enabled='no'/>
</pm>
<devices> <emulator>/usr/bin/qemu-system-x86_64</emulator> <disk
type='file' device='disk'> <driver name='qemu'
type='qcow2'
cache='none'/> <source file='/mnt/nfs-storage/template1.qcow2'
index='1'/> <backingStore/> <target dev='vda'
bus='virtio'/> <alias
name='virtio-disk0'/> <address type='pci' domain='0x0000'
bus='0x04'
slot='0x00' function='0x0'/> </disk> <disk type='file'
device='disk'>
<driver name='qemu' type='qcow2' cache='none'/> <source
file='/mnt/other-storage/vol.qcow2' index='2'/> <backingStore/>
<target
dev='vdb' bus='virtio'/> <alias name='virtio-disk1'/>
<address
type='pci' domain='0x0000' bus='0x08' slot='0x00'
function='0x0'/>
</disk> <controller type='usb' index='0'
model='qemu-xhci' ports='15'>
<alias name='usb'/> <address type='pci' domain='0x0000'
bus='0x02'
slot='0x00' function='0x0'/> </controller> <controller
type='pci'
index='0' model='pcie-root'> <alias name='pcie.0'/>
</controller>
<controller type='pci' index='1' model='pcie-root-port'>
<model
name='pcie-root-port'/> <target chassis='1' port='0x10'/>
<alias
name='pci.1'/> <address type='pci' domain='0x0000'
bus='0x00'
slot='0x02' function='0x0' multifunction='on'/>
</controller>
<controller type='pci' index='2' model='pcie-root-port'>
<model
name='pcie-root-port'/> <target chassis='2' port='0x11'/>
<alias
name='pci.2'/> <address type='pci' domain='0x0000'
bus='0x00'
slot='0x02' function='0x1'/> </controller> <controller
type='pci'
index='3' model='pcie-root-port'> <model
name='pcie-root-port'/> <target
chassis='3' port='0x12'/> <alias name='pci.3'/>
<address type='pci'
domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
</controller>
<controller type='pci' index='4' model='pcie-root-port'>
<model
name='pcie-root-port'/> <target chassis='4' port='0x13'/>
<alias
name='pci.4'/> <address type='pci' domain='0x0000'
bus='0x00'
slot='0x02' function='0x3'/> </controller> <controller
type='pci'
index='5' model='pcie-root-port'> <model
name='pcie-root-port'/> <target
chassis='5' port='0x14'/> <alias name='pci.5'/>
<address type='pci'
domain='0x0000' bus='0x00' slot='0x02' function='0x4'/>
</controller>
<controller type='pci' index='6' model='pcie-root-port'>
<model
name='pcie-root-port'/> <target chassis='6' port='0x15'/>
<alias
name='pci.6'/> <address type='pci' domain='0x0000'
bus='0x00'
slot='0x02' function='0x5'/> </controller> <controller
type='pci'
index='7' model='pcie-root-port'> <model
name='pcie-root-port'/> <target
chassis='7' port='0x16'/> <alias name='pci.7'/>
<address type='pci'
domain='0x0000' bus='0x00' slot='0x02' function='0x6'/>
</controller>
<controller type='pci' index='8' model='pcie-root-port'>
<model
name='pcie-root-port'/> <target chassis='8' port='0x17'/>
<alias
name='pci.8'/> <address type='pci' domain='0x0000'
bus='0x00'
slot='0x02' function='0x7'/> </controller> <controller
type='pci'
index='9' model='pcie-root-port'> <model
name='pcie-root-port'/> <target
chassis='9' port='0x18'/> <alias name='pci.9'/>
<address type='pci'
domain='0x0000' bus='0x00' slot='0x03' function='0x0'
multifunction='on'/> </controller> <controller type='pci'
index='10'
model='pcie-root-port'> <model name='pcie-root-port'/> <target
chassis='10' port='0x19'/> <alias name='pci.10'/>
<address type='pci'
domain='0x0000' bus='0x00' slot='0x03' function='0x1'/>
</controller>
<controller type='pci' index='11' model='pcie-root-port'>
<model
name='pcie-root-port'/> <target chassis='11'
port='0x1a'/> <alias
name='pci.11'/> <address type='pci' domain='0x0000'
bus='0x00'
slot='0x03' function='0x2'/> </controller> <controller
type='pci'
index='12' model='pcie-root-port'> <model
name='pcie-root-port'/>
<target chassis='12' port='0x1b'/> <alias
name='pci.12'/> <address
type='pci' domain='0x0000' bus='0x00' slot='0x03'
function='0x3'/>
</controller> <controller type='pci' index='13'
model='pcie-root-port'>
<model name='pcie-root-port'/> <target chassis='13'
port='0x1c'/> <alias
name='pci.13'/> <address type='pci' domain='0x0000'
bus='0x00'
slot='0x03' function='0x4'/> </controller> <controller
type='pci'
index='14' model='pcie-root-port'> <model
name='pcie-root-port'/>
<target chassis='14' port='0x1d'/> <alias
name='pci.14'/> <address
type='pci' domain='0x0000' bus='0x00' slot='0x03'
function='0x5'/>
</controller> <controller type='sata' index='0'> <alias
name='ide'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x1f'
function='0x2'/> </controller> <controller
type='virtio-serial'
index='0'> <alias name='virtio-serial0'/> <address
type='pci'
domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
</controller>
<controller type='scsi' index='0' model='virtio-scsi'>
<alias
name='scsi0'/> <address type='pci' domain='0x0000'
bus='0x07'
slot='0x00' function='0x0'/> </controller> <interface
type='bridge'>
<mac address='52:54:00:c4:a7:6d'/> <source bridge='br-test'/>
<target
dev='vnet6'/> <model type='virtio'/> <alias
name='net0'/> <address
type='pci' domain='0x0000' bus='0x01' slot='0x00'
function='0x0'/>
</interface> <serial type='pty'> <source
path='/dev/pts/3'/> <target
type='isa-serial' port='0'> <model name='isa-serial'/>
</target> <alias
name='serial0'/> </serial> <console type='pty'
tty='/dev/pts/3'> <source
path='/dev/pts/3'/> <target type='serial' port='0'/>
<alias
name='serial0'/> </console> <channel type='unix'>
<source mode='bind'
path='/var/lib/libvirt/qemu/channel/target/domain-4-ubuntu22.04/org.qemu.guest_agent.0'/>
<target type='virtio' name='org.qemu.guest_agent.0'
state='disconnected'/> <alias name='channel0'/> <address
type='virtio-serial' controller='0' bus='0' port='1'/>
</channel>
<channel type='spicevmc'> <target type='virtio'
name='com.redhat.spice.0' state='connected'/> <alias
name='channel1'/>
<address type='virtio-serial' controller='0' bus='0'
port='2'/>
</channel> <input type='tablet' bus='usb'> <alias
name='input0'/>
<address type='usb' bus='0' port='1'/> </input>
<input type='mouse'
bus='ps2'> <alias name='input1'/> </input> <input
type='keyboard'
bus='ps2'> <alias name='input2'/> </input> <graphics
type='spice'
port='5901' autoport='yes' listen='127.0.0.1'> <listen
type='address'
address='127.0.0.1'/> </graphics> <sound model='ich9'>
<alias
name='sound0'/> <address type='pci' domain='0x0000'
bus='0x00'
slot='0x1b' function='0x0'/> </sound> <audio id='1'
type='spice'/>
<video> <model type='qxl' ram='65536' vram='65536'
vgamem='16384'
heads='1' primary='yes'/> <alias name='video0'/>
<address type='pci'
domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
</video>
<redirdev bus='usb' type='spicevmc'> <alias
name='redir0'/> <address
type='usb' bus='0' port='2'/> </redirdev> <redirdev
bus='usb'
type='spicevmc'> <alias name='redir1'/> <address
type='usb' bus='0'
port='3'/> </redirdev> <memballoon model='virtio'>
<alias
name='balloon0'/> <address type='pci' domain='0x0000'
bus='0x05'
slot='0x00' function='0x0'/> </memballoon> <rng
model='virtio'> <backend
model='random'>/dev/urandom</backend> <alias name='rng0'/>
<address
type='pci' domain='0x0000' bus='0x06' slot='0x00'
function='0x0'/>
</rng> </devices> <seclabel type='dynamic' model='dac'
relabel='yes'>
<label>+0:+0</label> <imagelabel>+0:+0</imagelabel>
</seclabel> </domain> |
Right after the command is sent, the |unable to execute QEMU command
'block-export-add': Block node is read-only| happens. If I do the same
steps, but migrate all volumes that are on NFS (even if they stay on the
same storage, only changing names), the error does not happen.
Here are some other examples just to clarify when the error happens,
examples with green headers do not present the error, while examples
with red headers do:
Volume Origin Destination
ROOT iSCSI1 iSCSI2
DATA1 iSCSI1 ---
DATA2 iSCSI1 ---
Volume Origin Destination
ROOT iSCSI1 ---
DATA1 iSCSI1 iSCSI2
DATA2 iSCSI2 iSCSI1
Volume Origin Destination
ROOT NFS1 NFS2
DATA1 NFS2 NFS1
DATA2 iSCSI1 ---
Volume Origin Destination
ROOT iSCSI1 NFS2
DATA1 NFS2 ---
DATA2 NFS2 ---
Volume Origin Destination
ROOT iSCSI1 iSCSI1
DATA1 NFS1 ---
DATA2 NFS1 ---
Volume Origin Destination
ROOT iSCSI1 ---
DATA1 NFS2 NFS2
DATA2 NFS2 ---
Has anyone experienced this before? If so, is there another workaround?
As migrating all the volumes that are on NFS storage might be costly in
terms of time and storage/network resources.
Best regards,
João Jandre.
7:41 p.m.
Hello João
When trying to migrating a volume located on NFS storage, it will try to
read and write itself which is impossible, and the following error is
expected:
unable to execute QEMU command 'block-export-add': Block node is read-only
You can specify which volumes to migrate by virsh option. For example, if
my vm has three volumes, vda and vdb are on shared storage and can be
accessed on both src and dest hosts, vdc and vdd are on local storage, I
can use such option: --migrate-disks vdc,vdd
BR,
Fangge Jin
On Wed, Jan 24, 2024 at 11:02 PM João Jandre <joao(a)apache.org> wrote:
Hi, all
I've been having an annoying issue with live VM + volume migration. When I
try to use virDomainMigrate2 to migrate a VM and some of its volumes (but
not all) I get the following error:
unable to execute QEMU command 'block-export-add': Block node is read-only
Some additional context:
- I originally found this issue using the java bindings, but I've been
able to reproduce it using virsh;
- I'm using Libvirt 8.0.0 and Qemu 6.2.0;
- This only happens when the VM has volumes on NFS storage;
- The issue does not happen when migrating all the volumes that are on
NFS storage. Note that, if the VM has volumes that are on some other type
of storage, such as ISCSI, those do not need to be migrated. However,
migrating any volume will raise the error if the volumes on NFS are not
migrated as well;
- I've tried a number of valid combinations along the VIR_MIGRATE_LIVE
flag, using the flags: VIR_MIGRATE_PAUSED, VIR_MIGRATE_NON_SHARED_DISK,
VIR_MIGRATE_NON_SHARED_INC, VIR_MIGRATE_UNSAFE, VIR_MIGRATE_COMPRESSED.
The easiest way I've found to reproduce this error was using the following
steps:
1. Create a VM using virt-manager, it must have at least two volumes,
at least one of them must be on NFS storage. Here's the XML of one of the
VMs I've tested with:
<domain type='kvm' id='4'>
<name>ubuntu22.04</name>
<uuid>9965e409-c89d-408c-a9f7-488184566d56</uuid>
<metadata>
<libosinfo:libosinfo
xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0"
<http://libosinfo.org/xmlns/libvirt/domain/1.0>>
<libosinfo:os id="http://ubuntu.com/ubuntu/22.04"
<http://ubuntu.com/ubuntu/22.04>/>
</libosinfo:libosinfo>
</metadata>
<memory unit='KiB'>2097152</memory>
<currentMemory unit='KiB'>2097152</currentMemory>
<vcpu placement='static'>1</vcpu>
<resource>
<partition>/machine</partition>
</resource>
<os>
<type arch='x86_64' machine='pc-q35-6.2'>hvm</type>
<boot dev='hd'/>
</os>
<features>
<acpi/>
<apic/>
<vmport state='off'/>
</features>
<cpu mode='host-passthrough' check='none'
migratable='on'/>
<clock offset='utc'>
<timer name='rtc' tickpolicy='catchup'/>
<timer name='pit' tickpolicy='delay'/>
<timer name='hpet' present='no'/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<pm>
<suspend-to-mem enabled='no'/>
<suspend-to-disk enabled='no'/>
</pm>
<devices>
<emulator>/usr/bin/qemu-system-x86_64</emulator>
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2'/>
<source file='/mnt/nfs-storage/template.qcow2' index='1'/>
<backingStore/>
<target dev='vda' bus='virtio'/>
<alias name='virtio-disk0'/>
<address type='pci' domain='0x0000' bus='0x04'
slot='0x00' function='0x0'/>
</disk>
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2'/>
<source file='/mnt/nfs-storage/vol.qcow2' index='2'/>
<backingStore/>
<target dev='vdb' bus='virtio'/>
<alias name='virtio-disk1'/>
<address type='pci' domain='0x0000' bus='0x08'
slot='0x00' function='0x0'/>
</disk>
<controller type='usb' index='0' model='qemu-xhci'
ports='15'>
<alias name='usb'/>
<address type='pci' domain='0x0000' bus='0x02'
slot='0x00' function='0x0'/>
</controller>
<controller type='pci' index='0' model='pcie-root'>
<alias name='pcie.0'/>
</controller>
<controller type='pci' index='1'
model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='1' port='0x10'/>
<alias name='pci.1'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x02' function='0x0' multifunction='on'/>
</controller>
<controller type='pci' index='2'
model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='2' port='0x11'/>
<alias name='pci.2'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x02' function='0x1'/>
</controller>
<controller type='pci' index='3'
model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='3' port='0x12'/>
<alias name='pci.3'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x02' function='0x2'/>
</controller>
<controller type='pci' index='4'
model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='4' port='0x13'/>
<alias name='pci.4'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x02' function='0x3'/>
</controller>
<controller type='pci' index='5'
model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='5' port='0x14'/>
<alias name='pci.5'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x02' function='0x4'/>
</controller>
<controller type='pci' index='6'
model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='6' port='0x15'/>
<alias name='pci.6'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x02' function='0x5'/>
</controller>
<controller type='pci' index='7'
model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='7' port='0x16'/>
<alias name='pci.7'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x02' function='0x6'/>
</controller>
<controller type='pci' index='8'
model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='8' port='0x17'/>
<alias name='pci.8'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x02' function='0x7'/>
</controller>
<controller type='pci' index='9'
model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='9' port='0x18'/>
<alias name='pci.9'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x03' function='0x0' multifunction='on'/>
</controller>
<controller type='pci' index='10'
model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='10' port='0x19'/>
<alias name='pci.10'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x03' function='0x1'/>
</controller>
<controller type='pci' index='11'
model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='11' port='0x1a'/>
<alias name='pci.11'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x03' function='0x2'/>
</controller>
<controller type='pci' index='12'
model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='12' port='0x1b'/>
<alias name='pci.12'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x03' function='0x3'/>
</controller>
<controller type='pci' index='13'
model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='13' port='0x1c'/>
<alias name='pci.13'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x03' function='0x4'/>
</controller>
<controller type='pci' index='14'
model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='14' port='0x1d'/>
<alias name='pci.14'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x03' function='0x5'/>
</controller>
<controller type='sata' index='0'>
<alias name='ide'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x1f' function='0x2'/>
</controller>
<controller type='virtio-serial' index='0'>
<alias name='virtio-serial0'/>
<address type='pci' domain='0x0000' bus='0x03'
slot='0x00' function='0x0'/>
</controller>
<controller type='scsi' index='0' model='virtio-scsi'>
<alias name='scsi0'/>
<address type='pci' domain='0x0000' bus='0x07'
slot='0x00' function='0x0'/>
</controller>
<interface type='bridge'>
<mac address='52:54:00:c4:a7:6d'/>
<source bridge='br-test'/>
<target dev='vnet6'/>
<model type='virtio'/>
<alias name='net0'/>
<address type='pci' domain='0x0000' bus='0x01'
slot='0x00' function='0x0'/>
</interface>
<serial type='pty'>
<source path='/dev/pts/3'/>
<target type='isa-serial' port='0'>
<model name='isa-serial'/>
</target>
<alias name='serial0'/>
</serial>
<console type='pty' tty='/dev/pts/3'>
<source path='/dev/pts/3'/>
<target type='serial' port='0'/>
<alias name='serial0'/>
</console>
<channel type='unix'>
<source mode='bind'
path='/var/lib/libvirt/qemu/channel/target/domain-4-ubuntu22.04/org.qemu.guest_agent.0'/>
<target type='virtio' name='org.qemu.guest_agent.0'
state='disconnected'/>
<alias name='channel0'/>
<address type='virtio-serial' controller='0' bus='0'
port='1'/>
</channel>
<channel type='spicevmc'>
<target type='virtio' name='com.redhat.spice.0'
state='connected'/>
<alias name='channel1'/>
<address type='virtio-serial' controller='0' bus='0'
port='2'/>
</channel>
<input type='tablet' bus='usb'>
<alias name='input0'/>
<address type='usb' bus='0' port='1'/>
</input>
<input type='mouse' bus='ps2'>
<alias name='input1'/>
</input>
<input type='keyboard' bus='ps2'>
<alias name='input2'/>
</input>
<graphics type='spice' port='5901' autoport='yes'
listen='127.0.0.1'>
<listen type='address' address='127.0.0.1'/>
</graphics>
<sound model='ich9'>
<alias name='sound0'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x1b' function='0x0'/>
</sound>
<audio id='1' type='spice'/>
<video>
<model type='qxl' ram='65536' vram='65536'
vgamem='16384' heads='1' primary='yes'/>
<alias name='video0'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x01' function='0x0'/>
</video>
<redirdev bus='usb' type='spicevmc'>
<alias name='redir0'/>
<address type='usb' bus='0' port='2'/>
</redirdev>
<redirdev bus='usb' type='spicevmc'>
<alias name='redir1'/>
<address type='usb' bus='0' port='3'/>
</redirdev>
<memballoon model='virtio'>
<alias name='balloon0'/>
<address type='pci' domain='0x0000' bus='0x05'
slot='0x00' function='0x0'/>
</memballoon>
<rng model='virtio'>
<backend model='random'>/dev/urandom</backend>
<alias name='rng0'/>
<address type='pci' domain='0x0000' bus='0x06'
slot='0x00' function='0x0'/>
</rng>
</devices>
<seclabel type='dynamic' model='dac' relabel='yes'>
<label>+0:+0</label>
<imagelabel>+0:+0</imagelabel>
</seclabel>
</domain>
1. Use virsh migrate --domain <name> --copy-storage-all --live
--desturi <other_host_uri> --xml ./newxml to migrate the VM and one of
its volumes, while leaving a volume that is on NFS without migration. I've
tested migrating the volume to another NFS storage, as well as migrating to
a ISCSI storage that is mounted on both hosts, the result is the same.
Here's the content of the newxml for this example:
<domain type='kvm' id='4'>
<name>ubuntu22.04</name>
<uuid>9965e409-c89d-408c-a9f7-488184566d56</uuid>
<metadata>
<libosinfo:libosinfo
xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0"
<http://libosinfo.org/xmlns/libvirt/domain/1.0>>
<libosinfo:os id="http://ubuntu.com/ubuntu/22.04"
<http://ubuntu.com/ubuntu/22.04>/>
</libosinfo:libosinfo>
</metadata>
<memory unit='KiB'>2097152</memory>
<currentMemory unit='KiB'>2097152</currentMemory>
<vcpu placement='static'>1</vcpu>
<resource>
<partition>/machine</partition>
</resource>
<os>
<type arch='x86_64' machine='pc-q35-6.2'>hvm</type>
<boot dev='hd'/>
</os>
<features>
<acpi/>
<apic/>
<vmport state='off'/>
</features>
<cpu mode='host-passthrough' check='none'
migratable='on'/>
<clock offset='utc'>
<timer name='rtc' tickpolicy='catchup'/>
<timer name='pit' tickpolicy='delay'/>
<timer name='hpet' present='no'/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<pm>
<suspend-to-mem enabled='no'/>
<suspend-to-disk enabled='no'/>
</pm>
<devices>
<emulator>/usr/bin/qemu-system-x86_64</emulator>
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2' cache='none'/>
<source file='/mnt/nfs-storage/template1.qcow2' index='1'/>
<backingStore/>
<target dev='vda' bus='virtio'/>
<alias name='virtio-disk0'/>
<address type='pci' domain='0x0000' bus='0x04'
slot='0x00' function='0x0'/>
</disk>
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2' cache='none'/>
<source file='/mnt/other-storage/vol.qcow2' index='2'/>
<backingStore/>
<target dev='vdb' bus='virtio'/>
<alias name='virtio-disk1'/>
<address type='pci' domain='0x0000' bus='0x08'
slot='0x00' function='0x0'/>
</disk>
<controller type='usb' index='0' model='qemu-xhci'
ports='15'>
<alias name='usb'/>
<address type='pci' domain='0x0000' bus='0x02'
slot='0x00' function='0x0'/>
</controller>
<controller type='pci' index='0' model='pcie-root'>
<alias name='pcie.0'/>
</controller>
<controller type='pci' index='1'
model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='1' port='0x10'/>
<alias name='pci.1'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x02' function='0x0' multifunction='on'/>
</controller>
<controller type='pci' index='2'
model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='2' port='0x11'/>
<alias name='pci.2'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x02' function='0x1'/>
</controller>
<controller type='pci' index='3'
model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='3' port='0x12'/>
<alias name='pci.3'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x02' function='0x2'/>
</controller>
<controller type='pci' index='4'
model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='4' port='0x13'/>
<alias name='pci.4'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x02' function='0x3'/>
</controller>
<controller type='pci' index='5'
model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='5' port='0x14'/>
<alias name='pci.5'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x02' function='0x4'/>
</controller>
<controller type='pci' index='6'
model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='6' port='0x15'/>
<alias name='pci.6'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x02' function='0x5'/>
</controller>
<controller type='pci' index='7'
model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='7' port='0x16'/>
<alias name='pci.7'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x02' function='0x6'/>
</controller>
<controller type='pci' index='8'
model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='8' port='0x17'/>
<alias name='pci.8'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x02' function='0x7'/>
</controller>
<controller type='pci' index='9'
model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='9' port='0x18'/>
<alias name='pci.9'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x03' function='0x0' multifunction='on'/>
</controller>
<controller type='pci' index='10'
model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='10' port='0x19'/>
<alias name='pci.10'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x03' function='0x1'/>
</controller>
<controller type='pci' index='11'
model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='11' port='0x1a'/>
<alias name='pci.11'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x03' function='0x2'/>
</controller>
<controller type='pci' index='12'
model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='12' port='0x1b'/>
<alias name='pci.12'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x03' function='0x3'/>
</controller>
<controller type='pci' index='13'
model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='13' port='0x1c'/>
<alias name='pci.13'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x03' function='0x4'/>
</controller>
<controller type='pci' index='14'
model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='14' port='0x1d'/>
<alias name='pci.14'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x03' function='0x5'/>
</controller>
<controller type='sata' index='0'>
<alias name='ide'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x1f' function='0x2'/>
</controller>
<controller type='virtio-serial' index='0'>
<alias name='virtio-serial0'/>
<address type='pci' domain='0x0000' bus='0x03'
slot='0x00' function='0x0'/>
</controller>
<controller type='scsi' index='0' model='virtio-scsi'>
<alias name='scsi0'/>
<address type='pci' domain='0x0000' bus='0x07'
slot='0x00' function='0x0'/>
</controller>
<interface type='bridge'>
<mac address='52:54:00:c4:a7:6d'/>
<source bridge='br-test'/>
<target dev='vnet6'/>
<model type='virtio'/>
<alias name='net0'/>
<address type='pci' domain='0x0000' bus='0x01'
slot='0x00' function='0x0'/>
</interface>
<serial type='pty'>
<source path='/dev/pts/3'/>
<target type='isa-serial' port='0'>
<model name='isa-serial'/>
</target>
<alias name='serial0'/>
</serial>
<console type='pty' tty='/dev/pts/3'>
<source path='/dev/pts/3'/>
<target type='serial' port='0'/>
<alias name='serial0'/>
</console>
<channel type='unix'>
<source mode='bind'
path='/var/lib/libvirt/qemu/channel/target/domain-4-ubuntu22.04/org.qemu.guest_agent.0'/>
<target type='virtio' name='org.qemu.guest_agent.0'
state='disconnected'/>
<alias name='channel0'/>
<address type='virtio-serial' controller='0' bus='0'
port='1'/>
</channel>
<channel type='spicevmc'>
<target type='virtio' name='com.redhat.spice.0'
state='connected'/>
<alias name='channel1'/>
<address type='virtio-serial' controller='0' bus='0'
port='2'/>
</channel>
<input type='tablet' bus='usb'>
<alias name='input0'/>
<address type='usb' bus='0' port='1'/>
</input>
<input type='mouse' bus='ps2'>
<alias name='input1'/>
</input>
<input type='keyboard' bus='ps2'>
<alias name='input2'/>
</input>
<graphics type='spice' port='5901' autoport='yes'
listen='127.0.0.1'>
<listen type='address' address='127.0.0.1'/>
</graphics>
<sound model='ich9'>
<alias name='sound0'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x1b' function='0x0'/>
</sound>
<audio id='1' type='spice'/>
<video>
<model type='qxl' ram='65536' vram='65536'
vgamem='16384' heads='1' primary='yes'/>
<alias name='video0'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x01' function='0x0'/>
</video>
<redirdev bus='usb' type='spicevmc'>
<alias name='redir0'/>
<address type='usb' bus='0' port='2'/>
</redirdev>
<redirdev bus='usb' type='spicevmc'>
<alias name='redir1'/>
<address type='usb' bus='0' port='3'/>
</redirdev>
<memballoon model='virtio'>
<alias name='balloon0'/>
<address type='pci' domain='0x0000' bus='0x05'
slot='0x00' function='0x0'/>
</memballoon>
<rng model='virtio'>
<backend model='random'>/dev/urandom</backend>
<alias name='rng0'/>
<address type='pci' domain='0x0000' bus='0x06'
slot='0x00' function='0x0'/>
</rng>
</devices>
<seclabel type='dynamic' model='dac' relabel='yes'>
<label>+0:+0</label>
<imagelabel>+0:+0</imagelabel>
</seclabel>
</domain>
Right after the command is sent, the unable to execute QEMU command
'block-export-add': Block node is read-only happens. If I do the same
steps, but migrate all volumes that are on NFS (even if they stay on the
same storage, only changing names), the error does not happen.
Here are some other examples just to clarify when the error happens,
examples with green headers do not present the error, while examples with
red headers do:
Volume Origin Destination
ROOT iSCSI1 iSCSI2
DATA1 iSCSI1 ---
DATA2 iSCSI1 ---
Volume Origin Destination
ROOT iSCSI1 ---
DATA1 iSCSI1 iSCSI2
DATA2 iSCSI2 iSCSI1
Volume Origin Destination
ROOT NFS1 NFS2
DATA1 NFS2 NFS1
DATA2 iSCSI1 ---
Volume Origin Destination
ROOT iSCSI1 NFS2
DATA1 NFS2 ---
DATA2 NFS2 ---
Volume Origin Destination
ROOT iSCSI1 iSCSI1
DATA1 NFS1 ---
DATA2 NFS1 ---
Volume Origin Destination
ROOT iSCSI1 ---
DATA1 NFS2 NFS2
DATA2 NFS2 ---
Has anyone experienced this before? If so, is there another workaround? As
migrating all the volumes that are on NFS storage might be costly in terms
of time and storage/network resources.
Best regards,
João Jandre.
_______________________________________________
Users mailing list -- users(a)lists.libvirt.org
To unsubscribe send an email to users-leave(a)lists.libvirt.org
302
days inactive
303
days old
1 comments
2 participants
participants (2)
-
Fangge Jin -
João Jandre