Hello list, In short: I am experiencing a problem where trying to start a VM which is allready running on a different libvirt host. Starting it fails, thanks to sanlock, but the disk dies on the original hypervisor and the vms filesystem is rendered read-only. Long version: i have the following setup: Two machines (H1 and H2) using CentOS 6.2 with the packages: qemu-kvm-0.12.1.2-2.209.el6_2.4.x86_64 libvirt-client-0.9.4-23.el6_2.7.x86_64 libvirt-0.9.4-23.el6_2.7.x86_64 libvirt-lock-sanlock-0.9.4-23.el6_2.7.x86_64 They share two NFS folders, one for filesystems ( net set up as a libvirt storage pool ) and one for sanlock. sanlock is started without watchdog. /etc/libvirt/qemu-sanlock.conf: auto_disk_leases = 1 disk_lease_dir = "/opt/mln/sanlock" host_id = 51 ( host_id is 52 on H2 ) cat /etc/libvirt/qemu.conf | grep sanlock lock_manager = "sanlock" A virtual machine is created ( with virsh create ) on H1 using the following xml: <domain type='kvm' > <name>rhel1.locktest</name> <memory>524288</memory> <vcpu></vcpu> <os> <type arch='x86_64' >hvm</type> <boot dev='hd'/> </os> <features> <acpi/> <apic/> <pae/> </features> <clock offset='utc'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/bin/qemu-kvm</emulator> <disk type='file' device='disk'> <driver name='qemu' type='raw'/> <source file='/opt/mln/projects/root/locktest/images/rhel1'/> <target dev='hda' bus='ide'/> <alias name='ide0-0-0'/> <address type='drive' controller='0' bus='0' unit='0'/> </disk> <disk type='block' device='cdrom'> <driver name='qemu' type='raw'/> <target dev='hdc' bus='ide'/> <readonly/> <alias name='ide0-1-0'/> <address type='drive' controller='0' bus='1' unit='0'/> </disk> <controller type='ide' index='0'> <alias name='ide0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <interface type='bridge'> <source bridge='br0'/> </interface> <serial type='pty'> <source path='/dev/pts/8'/> <target port='0'/> <alias name='serial0'/> </serial> <console type='pty' tty='/dev/pts/8'> <source path='/dev/pts/8'/> <target type='serial' port='0'/> <alias name='serial0'/> </console> <input type='mouse' bus='ps2'/> <sound model='ich6'> <alias name='sound0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </sound> <graphics type='spice' /> <video> <model type='qxl' heads='1'/> <alias name='video0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </video> <memballoon model='virtio'> <alias name='balloon0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </memballoon> </devices> <seclabel type='dynamic' model='selinux' relabel='yes'> <label>system_u:system_r:svirt_t:s0:c141,c961</label> <imagelabel>system_u:object_r:svirt_image_t:s0:c141,c961</imagelabel> </seclabel> </domain> Here is the qemu log ( with debugging enabled ) from H1: 2012-04-08 22:03:13.346: starting up LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=spice /usr/bin/qemu-kvm -S -M rhel6.2.0 -enable-kvm -m 512 -smp 1,sockets=1,cores=1,threads=1 -name rhel1.locktest -uuid e3bc07fe-8b01-5f90-7c91-ad154630379e -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/rhel1.locktest.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=readline -rtc base=utc -drive file=/opt/mln/projects/root/locktest/images/rhel1,if=none,id=drive-ide0-0-0,format=raw -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 -drive if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -device rtl8139,vlan=0,id=net0,mac=52:54:00:b6:8d:14,bus=pci.0,addr=0x3 -net tap,fd=25,vlan=0,name=hostnet0 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -usb -spice port=5900,addr=127.0.0.1,disable-ticketing -vga qxl -global qxl-vga.vram_size=67108864 -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 2012-04-08 20:03:13.351+0000: 3485: info : libvirt version: 0.9.4, package: 23.el6_2.7 (CentOS BuildSystem <http://bugs.centos.org>;, 2012-03-26-14:12:59, c6b5.bsys.dev.centos.org) 2012-04-08 20:03:13.351+0000: 3485: debug : virCommandHook:1920 : Run hook 0x484a70 0x7faa81356610 2012-04-08 20:03:13.351+0000: 3485: debug : qemuProcessHook:2147 : Obtaining domain lock 2012-04-08 20:03:13.351+0000: 3485: debug : virDomainLockManagerNew:123 : plugin=0x7faa74018450 dom=0x7faa680008c0 withResources=1 2012-04-08 20:03:13.351+0000: 3485: debug : virLockManagerNew:291 : plugin=0x7faa74018450 type=0 nparams=4 params=0x7faa81355a60 flags=0 2012-04-08 20:03:13.351+0000: 3485: debug : virLockManagerLogParams:98 : key=uuid type=uuid value=e3bc07fe-8b01-5f90-7c91-ad154630379e 2012-04-08 20:03:13.351+0000: 3485: debug : virLockManagerLogParams:94 : key=name type=string value=rhel1.locktest 2012-04-08 20:03:13.351+0000: 3485: debug : virLockManagerLogParams:82 : key=id type=uint value=2 2012-04-08 20:03:13.351+0000: 3485: debug : virLockManagerLogParams:82 : key=pid type=uint value=3485 2012-04-08 20:03:13.351+0000: 3485: debug : virDomainLockManagerNew:135 : Adding leases 2012-04-08 20:03:13.351+0000: 3485: debug : virDomainLockManagerNew:140 : Adding disks 2012-04-08 20:03:13.351+0000: 3485: debug : virDomainLockManagerAddDisk:86 : Add disk /opt/mln/projects/root/locktest/images/rhel1 2012-04-08 20:03:13.351+0000: 3485: debug : virLockManagerAddResource:320 : lock=0x7faa680021d0 type=0 name=/opt/mln/projects/root/locktest/images/rhel1 nparams=0 params=(nil) flags=0 2012-04-08 20:03:13.352+0000: 3485: debug : virLockManagerAcquire:337 : lock=0x7faa680021d0 state='(null)' flags=3 fd=0x7faa81355bcc 2012-04-08 20:03:13.352+0000: 3485: debug : virLockManagerSanlockAcquire:725 : Register sanlock 3 2012-04-08 20:03:13.353+0000: 3485: debug : virLockManagerSanlockAcquire:783 : Acquire completed fd=3 2012-04-08 20:03:13.353+0000: 3485: debug : virLockManagerFree:374 : lock=0x7faa680021d0 2012-04-08 20:03:13.353+0000: 3485: debug : qemuProcessHook:2172 : Moving procss to cgroup 2012-04-08 20:03:13.353+0000: 3485: debug : virCgroupNew:602 : New group /libvirt/qemu/rhel1.locktest 2012-04-08 20:03:13.353+0000: 3485: debug : virCgroupDetect:261 : Detected mount/mapping 0:cpu at /cgroup/cpu in 2012-04-08 20:03:13.353+0000: 3485: debug : virCgroupDetect:261 : Detected mount/mapping 1:cpuacct at /cgroup/cpuacct in 2012-04-08 20:03:13.353+0000: 3485: debug : virCgroupDetect:261 : Detected mount/mapping 2:cpuset at /cgroup/cpuset in 2012-04-08 20:03:13.353+0000: 3485: debug : virCgroupDetect:261 : Detected mount/mapping 3:memory at /cgroup/memory in 2012-04-08 20:03:13.353+0000: 3485: debug : virCgroupDetect:261 : Detected mount/mapping 4:devices at /cgroup/devices in 2012-04-08 20:03:13.353+0000: 3485: debug : virCgroupDetect:261 : Detected mount/mapping 5:freezer at /cgroup/freezer in 2012-04-08 20:03:13.353+0000: 3485: debug : virCgroupDetect:261 : Detected mount/mapping 6:blkio at /cgroup/blkio in 2012-04-08 20:03:13.353+0000: 3485: debug : virCgroupMakeGroup:523 : Make group /libvirt/qemu/rhel1.locktest 2012-04-08 20:03:13.353+0000: 3485: debug : virCgroupMakeGroup:545 : Make controller /cgroup/cpu/libvirt/qemu/rhel1.locktest/ 2012-04-08 20:03:13.353+0000: 3485: debug : virCgroupMakeGroup:545 : Make controller /cgroup/cpuacct/libvirt/qemu/rhel1.locktest/ 2012-04-08 20:03:13.353+0000: 3485: debug : virCgroupMakeGroup:545 : Make controller /cgroup/cpuset/libvirt/qemu/rhel1.locktest/ 2012-04-08 20:03:13.353+0000: 3485: debug : virCgroupMakeGroup:545 : Make controller /cgroup/memory/libvirt/qemu/rhel1.locktest/ 2012-04-08 20:03:13.353+0000: 3485: debug : virCgroupMakeGroup:545 : Make controller /cgroup/devices/libvirt/qemu/rhel1.locktest/ 2012-04-08 20:03:13.353+0000: 3485: debug : virCgroupMakeGroup:545 : Make controller /cgroup/freezer/libvirt/qemu/rhel1.locktest/ 2012-04-08 20:03:13.353+0000: 3485: debug : virCgroupMakeGroup:545 : Make controller /cgroup/blkio/libvirt/qemu/rhel1.locktest/ 2012-04-08 20:03:13.353+0000: 3485: debug : virCgroupSetValueStr:319 : Set value '/cgroup/cpu/libvirt/qemu/rhel1.locktest/tasks' to '3485' 2012-04-08 20:03:13.359+0000: 3485: debug : virCgroupSetValueStr:319 : Set value '/cgroup/cpuacct/libvirt/qemu/rhel1.locktest/tasks' to '3485' 2012-04-08 20:03:13.367+0000: 3485: debug : virCgroupSetValueStr:319 : Set value '/cgroup/cpuset/libvirt/qemu/rhel1.locktest/tasks' to '3485' 2012-04-08 20:03:13.375+0000: 3485: debug : virCgroupSetValueStr:319 : Set value '/cgroup/memory/libvirt/qemu/rhel1.locktest/tasks' to '3485' 2012-04-08 20:03:13.383+0000: 3485: debug : virCgroupSetValueStr:319 : Set value '/cgroup/devices/libvirt/qemu/rhel1.locktest/tasks' to '3485' 2012-04-08 20:03:13.391+0000: 3485: debug : virCgroupSetValueStr:319 : Set value '/cgroup/freezer/libvirt/qemu/rhel1.locktest/tasks' to '3485' 2012-04-08 20:03:13.399+0000: 3485: debug : virCgroupSetValueStr:319 : Set value '/cgroup/blkio/libvirt/qemu/rhel1.locktest/tasks' to '3485' 2012-04-08 20:03:13.407+0000: 3485: debug : qemuProcessHook:2178 : Setup CPU affinity 2012-04-08 20:03:13.407+0000: 3485: debug : qemuProcessInitCpuAffinity:1464 : Setting CPU affinity 2012-04-08 20:03:13.407+0000: 3485: debug : qemuProcessHook:2185 : Setting up security labelling 2012-04-08 20:03:13.407+0000: 3485: debug : virSecurityDACSetProcessLabel:630 : Dropping privileges of VM to 107:107 2012-04-08 20:03:13.408+0000: 3485: debug : qemuProcessHook:2192 : Hook complete ret=0 2012-04-08 20:03:13.408+0000: 3485: debug : virCommandHook:1922 : Done hook 0 2012-04-08 20:03:13.408+0000: 3485: debug : virCommandHook:1935 : Notifying parent for handshake start on 27 2012-04-08 20:03:13.408+0000: 3485: debug : virCommandHook:1956 : Waiting on parent for handshake complete on 28 2012-04-08 20:03:13.409+0000: 3485: debug : virCommandHook:1972 : Hook is done 0 char device redirected to /dev/pts/1 do_spice_init: starting 0.8.3 spice_server_add_interface: SPICE_INTERFACE_MIGRATION spice_server_add_interface: SPICE_INTERFACE_KEYBOARD spice_server_add_interface: SPICE_INTERFACE_MOUSE spice_server_add_interface: SPICE_INTERFACE_QXL red_worker_main: begin spice_server_add_interface: SPICE_INTERFACE_PLAYBACK spice_server_add_interface: SPICE_INTERFACE_RECORD handle_dev_input: start I can log in to the VM and everything is fine. Next, i try to start the VM on H2, i get the following error: error: Failed to create domain from libvirt/rhel1.locktest.xml error: internal error Failed to acquire lock: error -243 All seems well, but now the VMs filesystem is in read-only state.... This is the qemu log from H2: 2012-04-08 22:05:27.833: starting up LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=spice /usr/bin/qemu-kvm -S -M rhel6.2.0 -enable-kvm -m 512 -smp 1,sockets=1,cores=1,threads=1 -name rhel1.locktest -uuid b1aa8f3f-4391-1014-3759-79d47a4dffdd -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/rhel1.locktest.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=readline -rtc base=utc -drive file=/opt/mln/projects/root/locktest/images/rhel1,if=none,id=drive-ide0-0-0,format=raw -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 -drive if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -device rtl8139,vlan=0,id=net0,mac=52:54:00:d7:a4:4d,bus=pci.0,addr=0x3 -net tap,fd=25,vlan=0,name=hostnet0 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -usb -spice port=5900,addr=127.0.0.1,disable-ticketing -vga qxl -global qxl-vga.vram_size=67108864 -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 2012-04-08 20:05:27.847+0000: 5141: info : libvirt version: 0.9.4, package: 23.el6_2.7 (CentOS BuildSystem <http://bugs.centos.org>;, 2012-03-26-14:12:59, c6b5.bsys.dev.centos.org) 2012-04-08 20:05:27.847+0000: 5141: debug : virCommandHook:1920 : Run hook 0x484a70 0x7f9b3c673610 2012-04-08 20:05:27.848+0000: 5141: debug : qemuProcessHook:2147 : Obtaining domain lock 2012-04-08 20:05:27.848+0000: 5141: debug : virDomainLockManagerNew:123 : plugin=0x7f9b30016aa0 dom=0x7f9b2000b6a0 withResources=1 2012-04-08 20:05:27.848+0000: 5141: debug : virLockManagerNew:291 : plugin=0x7f9b30016aa0 type=0 nparams=4 params=0x7f9b3c672a60 flags=0 2012-04-08 20:05:27.848+0000: 5141: debug : virLockManagerLogParams:98 : key=uuid type=uuid value=b1aa8f3f-4391-1014-3759-79d47a4dffdd 2012-04-08 20:05:27.848+0000: 5141: debug : virLockManagerLogParams:94 : key=name type=string value=rhel1.locktest 2012-04-08 20:05:27.848+0000: 5141: debug : virLockManagerLogParams:82 : key=id type=uint value=9 2012-04-08 20:05:27.848+0000: 5141: debug : virLockManagerLogParams:82 : key=pid type=uint value=5141 2012-04-08 20:05:27.848+0000: 5141: debug : virDomainLockManagerNew:135 : Adding leases 2012-04-08 20:05:27.848+0000: 5141: debug : virDomainLockManagerNew:140 : Adding disks 2012-04-08 20:05:27.848+0000: 5141: debug : virDomainLockManagerAddDisk:86 : Add disk /opt/mln/projects/root/locktest/images/rhel1 2012-04-08 20:05:27.848+0000: 5141: debug : virLockManagerAddResource:320 : lock=0x7f9b2000dd90 type=0 name=/opt/mln/projects/root/locktest/images/rhel1 nparams=0 params=(nil) flags=0 2012-04-08 20:05:27.849+0000: 5141: debug : virLockManagerAcquire:337 : lock=0x7f9b2000dd90 state='(null)' flags=3 fd=0x7f9b3c672bcc 2012-04-08 20:05:27.850+0000: 5141: debug : virLockManagerSanlockAcquire:725 : Register sanlock 3 2012-04-08 20:05:27.850+0000: 5141: debug : virLockManagerSanlockAcquire:783 : Acquire completed fd=3 2012-04-08 20:05:27.850+0000: 5141: debug : virLockManagerFree:374 : lock=0x7f9b2000dd90 2012-04-08 20:05:27.850+0000: 5141: debug : qemuProcessHook:2172 : Moving procss to cgroup 2012-04-08 20:05:27.850+0000: 5141: debug : virCgroupNew:602 : New group /libvirt/qemu/rhel1.locktest 2012-04-08 20:05:27.851+0000: 5141: debug : virCgroupDetect:261 : Detected mount/mapping 0:cpu at /cgroup/cpu in 2012-04-08 20:05:27.851+0000: 5141: debug : virCgroupDetect:261 : Detected mount/mapping 1:cpuacct at /cgroup/cpuacct in 2012-04-08 20:05:27.851+0000: 5141: debug : virCgroupDetect:261 : Detected mount/mapping 2:cpuset at /cgroup/cpuset in 2012-04-08 20:05:27.851+0000: 5141: debug : virCgroupDetect:261 : Detected mount/mapping 3:memory at /cgroup/memory in 2012-04-08 20:05:27.851+0000: 5141: debug : virCgroupDetect:261 : Detected mount/mapping 4:devices at /cgroup/devices in 2012-04-08 20:05:27.851+0000: 5141: debug : virCgroupDetect:261 : Detected mount/mapping 5:freezer at /cgroup/freezer in 2012-04-08 20:05:27.851+0000: 5141: debug : virCgroupDetect:261 : Detected mount/mapping 6:blkio at /cgroup/blkio in 2012-04-08 20:05:27.851+0000: 5141: debug : virCgroupMakeGroup:523 : Make group /libvirt/qemu/rhel1.locktest 2012-04-08 20:05:27.851+0000: 5141: debug : virCgroupMakeGroup:545 : Make controller /cgroup/cpu/libvirt/qemu/rhel1.locktest/ 2012-04-08 20:05:27.851+0000: 5141: debug : virCgroupMakeGroup:545 : Make controller /cgroup/cpuacct/libvirt/qemu/rhel1.locktest/ 2012-04-08 20:05:27.851+0000: 5141: debug : virCgroupMakeGroup:545 : Make controller /cgroup/cpuset/libvirt/qemu/rhel1.locktest/ 2012-04-08 20:05:27.851+0000: 5141: debug : virCgroupMakeGroup:545 : Make controller /cgroup/memory/libvirt/qemu/rhel1.locktest/ 2012-04-08 20:05:27.851+0000: 5141: debug : virCgroupMakeGroup:545 : Make controller /cgroup/devices/libvirt/qemu/rhel1.locktest/ 2012-04-08 20:05:27.851+0000: 5141: debug : virCgroupMakeGroup:545 : Make controller /cgroup/freezer/libvirt/qemu/rhel1.locktest/ 2012-04-08 20:05:27.851+0000: 5141: debug : virCgroupMakeGroup:545 : Make controller /cgroup/blkio/libvirt/qemu/rhel1.locktest/ 2012-04-08 20:05:27.851+0000: 5141: debug : virCgroupSetValueStr:319 : Set value '/cgroup/cpu/libvirt/qemu/rhel1.locktest/tasks' to '5141' 2012-04-08 20:05:27.865+0000: 5141: debug : virCgroupSetValueStr:319 : Set value '/cgroup/cpuacct/libvirt/qemu/rhel1.locktest/tasks' to '5141' 2012-04-08 20:05:27.873+0000: 5141: debug : virCgroupSetValueStr:319 : Set value '/cgroup/cpuset/libvirt/qemu/rhel1.locktest/tasks' to '5141' 2012-04-08 20:05:27.881+0000: 5141: debug : virCgroupSetValueStr:319 : Set value '/cgroup/memory/libvirt/qemu/rhel1.locktest/tasks' to '5141' 2012-04-08 20:05:27.893+0000: 5141: debug : virCgroupSetValueStr:319 : Set value '/cgroup/devices/libvirt/qemu/rhel1.locktest/tasks' to '5141' 2012-04-08 20:05:27.901+0000: 5141: debug : virCgroupSetValueStr:319 : Set value '/cgroup/freezer/libvirt/qemu/rhel1.locktest/tasks' to '5141' 2012-04-08 20:05:27.909+0000: 5141: debug : virCgroupSetValueStr:319 : Set value '/cgroup/blkio/libvirt/qemu/rhel1.locktest/tasks' to '5141' 2012-04-08 20:05:27.917+0000: 5141: debug : qemuProcessHook:2178 : Setup CPU affinity 2012-04-08 20:05:27.917+0000: 5141: debug : qemuProcessInitCpuAffinity:1464 : Setting CPU affinity 2012-04-08 20:05:27.918+0000: 5141: debug : qemuProcessHook:2185 : Setting up security labelling 2012-04-08 20:05:27.918+0000: 5141: debug : virSecurityDACSetProcessLabel:630 : Dropping privileges of VM to 107:107 2012-04-08 20:05:27.918+0000: 5141: debug : qemuProcessHook:2192 : Hook complete ret=0 2012-04-08 20:05:27.918+0000: 5141: debug : virCommandHook:1922 : Done hook 0 2012-04-08 20:05:27.918+0000: 5141: debug : virCommandHook:1935 : Notifying parent for handshake start on 27 2012-04-08 20:05:27.919+0000: 5141: debug : virCommandHook:1956 : Waiting on parent for handshake complete on 28 2012-04-08 20:05:27.920+0000: 5141: debug : virCommandHook:1972 : Hook is done 0 char device redirected to /dev/pts/1 do_spice_init: starting 0.8.3 spice_server_add_interface: SPICE_INTERFACE_MIGRATION spice_server_add_interface: SPICE_INTERFACE_KEYBOARD spice_server_add_interface: SPICE_INTERFACE_MOUSE spice_server_add_interface: SPICE_INTERFACE_QXL red_worker_main: begin spice_server_add_interface: SPICE_INTERFACE_PLAYBACK spice_server_add_interface: SPICE_INTERFACE_RECORD 2012-04-08 22:05:28.141: shutting down qemu: terminating on signal 15 from pid 3138 Back on H1 there are multiple lines in the qemu log for the VM: block I/O error in device 'drive-ide0-0-0': Input/output error (5) block I/O error in device 'drive-ide0-0-0': Input/output error (5) block I/O error in device 'drive-ide0-0-0': Input/output error (5) block I/O error in device 'drive-ide0-0-0': Input/output error (5) block I/O error in device 'drive-ide0-0-0': Input/output error (5) Can someone please help me understand why the filesystem dies on the original host? Is this an NFS problem? What should I do differently? Many thanks in advance, K