[libvirt] Live migration fails with "Migration without shared storage is unsafe" when using netapp storage on CentOS 7 and Libvirt 4.5.0
Hello everyone, We've been using libvirt version 0.10.2 for a long time (on CentOS 6 OS) and now we have started creating new KVM cluster all based on CentOS 7 OS and libvirt version 4.5.0 (that came with CentOS repo). TLDR Version: - When we try to migrate using "virsh migrate test --live qemu+ssh://kvm_aquila-02/system" we get an error "error: Unsafe migration: Migration without shared storage is unsafe". - When we try to migrate using same method but adding "--unsafe" migration works. - When we try to migrate a VPS that is using QCOW image on a shared GFS2 image, mapped on both servers, migration is successful. We have two servers in a cluster so far, they are connected to NetApp storage and all luns that VPS server should be using are mapped to both servers. Long version with background: We have plenty CentOS 6 Servers which work perfectly fine with libvirt version up to 0.10.2. All connected also to a NetApp storage and all luns are mapped to all servers. LUN's (disks) are visible under /dev/mapper directory. When we create VPS, we point to raw disk to the same location, /dev/mapper. For example, VPS "test" would use "/dev/mapper/test" for disk with "cache=none" in xml file. If we issue "virsh migrate test --live qemu+ssh://kvm_server-0X/system", it will successfully migrate VPS to another hypervisor without errors, because disk already exists there. We also have a LUN mapped to each hypervisor and GFS2 established in a cluster between hypervisors where we keep qcow images. We can also migrate those machines without issues. On new CentOS 7 hypervisors, we have pretty much the exact same setup. NetApp storage connected to servers. All VPS LUN's mapped to all hypervisors. Clustered GFS2 volume. When we define VPS with approximately same XML, it will NOT migrate it and will issue out an error stated above. "--unsafe" parameter seems to work fine. BUT. We want to setup a proper libvirt/kvm cluster with pcs. I've defined a pcs resource for each VPS and live migration also doesn't work if i try "pcs resource move" command. It will not migrate it "live" as it should, and as it does with QCOW images, probably because of the same reason - because it's unable to perform live migrate via libvirt without "unsafe". If there would be any trick to add "--unsafe" parameter to pcs i would do it, but I'm not aware of it. Still, seems like a bug that libvirt would not realize that netapp storage is present in this configuration. Is this maybe corrected in later versions of libvirt? Unfortunately, 4.5.0 is latest in CentOS repo but i would try to compile it if needed. Any help would be greatly appreciated and if more info is needed i will be more than happy to provide. Thanks in advance, Marko Todoric
Hello everyone once again. Regarding an email bellow, i found why is this happening. Namely I've tried updating libvirt and all it's components to version 4.10.0, but still the same issue happened. I've reviewed xml files between CentOS 6 hypervisors and new one on CentOS 7. And basically this is the difference when i comes to disk setup: CentOS 6 (Libvirt 0.10.2) <disk type='block' device='disk'> <driver name='qemu' type='raw' cache='none' io='native'/> <source dev='/dev/mapper/tables1'/> <target dev='vda' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </disk> CentOS 7 (Libvirt 4.10.0) <disk type='file' device='disk'> <driver name='qemu' type='raw' cache='none' io='native'/> <source file='/dev/mapper/proba'/> <target dev='vda' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </disk> So basically, even if i put disk as a path to block device, libvirt will recognize it as a "file" instead of "block" device. When i changed the xml file and tried to migrate with options bellow - all worked well! Now, is this a bug in libvirt causing it not to recognize block device or am i doing something wrong? We have been creating VM's in CentOS 7 the same way we did in CentOS 6 hypervisors, and this is the way: virt-install --connect=qemu+ssh://kvm_aquila-01/system \ --name=proba \ --hvm \ --graphics vnc \ --disk path=/dev/mapper/proba,bus=virtio,cache=none \ --vcpus 2 \ --ram=2048 \ --location=http://192.168.54.182/mirror \ --extra-args "ksdevice=eth0 inst.ks=http://192.168.53.99/ks/proba.cfg" \ --network bridge=br7,model=virtio \ --boot hd,cdrom,menu=on \ --os-type=linux \ --os-variant=rhel7.0 \ --metadata description="LinuxTim - Proba" \ --noreboot How do we make libvirt automatically default to block device instead of file when it comes to disk type? Srdačan pozdrav, Marko Todorić Inženjer za razvoj i održavanje servisa [cid:part1.25AC1762.814452EE@oriontelekom.rs] Orion telekom d.o.o. Beograd Naselje Zemun Polje, Mala pruga 8 11080 Zemun, Srbija m: +381 64 8379 337 e: marko.todoric@oriontelekom.rs<mailto:dusan.radisavljevic@oriontelekom.rs> w: www.oriontelekom.rs<http://www.oriontelekom.rs> On 6/25/19 5:16 PM, Marko Todoric wrote: Hello everyone, We've been using libvirt version 0.10.2 for a long time (on CentOS 6 OS) and now we have started creating new KVM cluster all based on CentOS 7 OS and libvirt version 4.5.0 (that came with CentOS repo). TLDR Version: - When we try to migrate using "virsh migrate test --live qemu+ssh://kvm_aquila-02/system" we get an error "error: Unsafe migration: Migration without shared storage is unsafe". - When we try to migrate using same method but adding "--unsafe" migration works. - When we try to migrate a VPS that is using QCOW image on a shared GFS2 image, mapped on both servers, migration is successful. We have two servers in a cluster so far, they are connected to NetApp storage and all luns that VPS server should be using are mapped to both servers. Long version with background: We have plenty CentOS 6 Servers which work perfectly fine with libvirt version up to 0.10.2. All connected also to a NetApp storage and all luns are mapped to all servers. LUN's (disks) are visible under /dev/mapper directory. When we create VPS, we point to raw disk to the same location, /dev/mapper. For example, VPS "test" would use "/dev/mapper/test" for disk with "cache=none" in xml file. If we issue "virsh migrate test --live qemu+ssh://kvm_server-0X/system", it will successfully migrate VPS to another hypervisor without errors, because disk already exists there. We also have a LUN mapped to each hypervisor and GFS2 established in a cluster between hypervisors where we keep qcow images. We can also migrate those machines without issues. On new CentOS 7 hypervisors, we have pretty much the exact same setup. NetApp storage connected to servers. All VPS LUN's mapped to all hypervisors. Clustered GFS2 volume. When we define VPS with approximately same XML, it will NOT migrate it and will issue out an error stated above. "--unsafe" parameter seems to work fine. BUT. We want to setup a proper libvirt/kvm cluster with pcs. I've defined a pcs resource for each VPS and live migration also doesn't work if i try "pcs resource move" command. It will not migrate it "live" as it should, and as it does with QCOW images, probably because of the same reason - because it's unable to perform live migrate via libvirt without "unsafe". If there would be any trick to add "--unsafe" parameter to pcs i would do it, but I'm not aware of it. Still, seems like a bug that libvirt would not realize that netapp storage is present in this configuration. Is this maybe corrected in later versions of libvirt? Unfortunately, 4.5.0 is latest in CentOS repo but i would try to compile it if needed. Any help would be greatly appreciated and if more info is needed i will be more than happy to provide. Thanks in advance, Marko Todoric
participants (1)
-
Marko Todorić