Re: [libvirt-users] [virt-tools-list] live-migrating a VM whose current host's storage is fubar'ed
On 04/15/2014 05:16 PM, Digimer wrote:
Hi all,
For those on the linux-cluster list, this is a similar question to my post[1] there.
I recently had a client destroy their node's RAID array (they yanked hard drives until the array was destroyed). The VMs running on the node continued to operate for another hour, impressively. Their storage was based on DRBD, so the VM's storage just started going to/from the other node's storage.
The problem was when I tried to migrate them off of the damaged node to a healthy node. Without being able to write to the logs (I assume), the migration failed. In the end, I had to force-off the damaged host and reboot the VMs on the backup node. This caused an outage which, I think, could have been avoided.
So my question is; Can VMs be migrated if/when the host's FS is read-only/destroyed? If so, how?
Thanks!
1. https://www.redhat.com/archives/linux-cluster/2014-April/msg00010.html
Probably a better question for libvirt lists, CCd, but I don't think anyone tests a scenario like that so there's bound to be problems. - Cole
On 16/04/14 02:17 PM, Cole Robinson wrote:
On 04/15/2014 05:16 PM, Digimer wrote:
Hi all,
For those on the linux-cluster list, this is a similar question to my post[1] there.
I recently had a client destroy their node's RAID array (they yanked hard drives until the array was destroyed). The VMs running on the node continued to operate for another hour, impressively. Their storage was based on DRBD, so the VM's storage just started going to/from the other node's storage.
The problem was when I tried to migrate them off of the damaged node to a healthy node. Without being able to write to the logs (I assume), the migration failed. In the end, I had to force-off the damaged host and reboot the VMs on the backup node. This caused an outage which, I think, could have been avoided.
So my question is; Can VMs be migrated if/when the host's FS is read-only/destroyed? If so, how?
Thanks!
1. https://www.redhat.com/archives/linux-cluster/2014-April/msg00010.html
Probably a better question for libvirt lists, CCd, but I don't think anyone tests a scenario like that so there's bound to be problems.
- Cole
Thanks for cc'ing. Ya, I'm not surprised that I had problems. I guess now that I've hit it though, and given that it's not impossible to have this happen again, it would be worth looking in to a way to handle it. I suspect something like "--no-logging" to disable attempts to write to local disk during a migration would be enough. -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education?
participants (2)
-
Cole Robinson -
Digimer