[libvirt-users] VM running with storage image file deleted
Hello, I have a strange issue I'd like to have some insights on. We have 2 running VM, the storage pool is over NFS. Unfortunately, the backend storage folder has been deleted (so the image file in it as well, obviously) ... Yet, the 2 VM are still up and running, with their root filesystem being mounted "rw". - how does that work? I was expecting that with the file image being deleted, the VM would not work at all. - since the VM is still running, is there a way to create a snapshot of it in order to re-create the image file on another storage pool ? I'm currently exploring "virsh snapshot-create-as ...", but if access to the image file is needed, it will fail. Thanks. -- Nicolas
On 10.01.19 11:19, Roosen, Nicolas wrote:
Hello, I have a strange issue I'd like to have some insights on.
We have 2 running VM, the storage pool is over NFS. Unfortunately, the backend storage folder has been deleted (so the image file in it as well, obviously) ...
Yet, the 2 VM are still up and running, with their root filesystem being mounted "rw".
- how does that work? I was expecting that with the file image being deleted, the VM would not work at all.
The file was "unlinked" but still exists. https://superuser.com/questions/283102/how-to-recover-deleted-file-if-it-is-... I would consider doing a backup from inside the VM, too.
- since the VM is still running, is there a way to create a snapshot of it in order to re-create the image file on another storage pool ? I'm currently exploring "virsh snapshot-create-as ...", but if access to the image file is needed, it will fail.
Note: the file will be lost, when it is released by the QEMU process.
On 1/10/19 11:19 AM, Roosen, Nicolas wrote:
Hello, I have a strange issue I'd like to have some insights on.
We have 2 running VM, the storage pool is over NFS. Unfortunately, the backend storage folder has been deleted (so the image file in it as well, obviously) ...
Yet, the 2 VM are still up and running, with their root filesystem being mounted "rw".
- how does that work? I was expecting that with the file image being deleted, the VM would not work at all.
- since the VM is still running, is there a way to create a snapshot of it in order to re-create the image file on another storage pool ? I'm currently exploring "virsh snapshot-create-as ...", but if access to the image file is needed, it will fail.
On Linux a file is not deleted right away if there's a process that has the file opened, which is your case. The deletion is deferred until the last process closed the file. In order to rescue the file you could get the PID of qemu that still has the disk open, and then find the FD corresponding to the file and with some magic you should be able to recover the file: # pgrep qemu 221472 # ls -l /proc/221472/fd/ | grep deleted lr-x------ 1 root root 64 Jan 10 11:31 29 -> /var/lib/libvirt/images/fd.img (deleted) (Here, 29 is the FD we're looking for) # dd if=/proc/221472/fd/29 of=blah.img snapshot-create-* won't work because that will instruct qemu to open the file again which will fail because the filename is gone. Michal
Thanks Michal, this is good to know. Unfortunately, "dd" complains that the file is a "Stale file handle". dd if=/proc/470/fs/27 of=/data/vmguest.img dd: failed to open '/proc/470/fs/27': Stale file handle The file was on a NFS server, but the process "qemu" is running on the VM host. Duno if it makes a difference, but anyway, it seems I cannot "dd" the FD file. I'll try a rsync of the root FS of the running guest, to "backup" as much as I can. Nicolas ________________________________ From: Michal Privoznik <mprivozn@redhat.com> Sent: Thursday, January 10, 2019 11:35 To: Roosen, Nicolas; libvirt-users@redhat.com Subject: Re: [libvirt-users] VM running with storage image file deleted On 1/10/19 11:19 AM, Roosen, Nicolas wrote:
Hello, I have a strange issue I'd like to have some insights on.
We have 2 running VM, the storage pool is over NFS. Unfortunately, the backend storage folder has been deleted (so the image file in it as well, obviously) ...
Yet, the 2 VM are still up and running, with their root filesystem being mounted "rw".
- how does that work? I was expecting that with the file image being deleted, the VM would not work at all.
- since the VM is still running, is there a way to create a snapshot of it in order to re-create the image file on another storage pool ? I'm currently exploring "virsh snapshot-create-as ...", but if access to the image file is needed, it will fail.
On Linux a file is not deleted right away if there's a process that has the file opened, which is your case. The deletion is deferred until the last process closed the file. In order to rescue the file you could get the PID of qemu that still has the disk open, and then find the FD corresponding to the file and with some magic you should be able to recover the file: # pgrep qemu 221472 # ls -l /proc/221472/fd/ | grep deleted lr-x------ 1 root root 64 Jan 10 11:31 29 -> /var/lib/libvirt/images/fd.img (deleted) (Here, 29 is the FD we're looking for) # dd if=/proc/221472/fd/29 of=blah.img snapshot-create-* won't work because that will instruct qemu to open the file again which will fail because the filename is gone. Michal
On Thu, Jan 10, 2019 at 11:35:13 +0100, Michal Privoznik wrote:
On 1/10/19 11:19 AM, Roosen, Nicolas wrote:
Hello, I have a strange issue I'd like to have some insights on.
We have 2 running VM, the storage pool is over NFS. Unfortunately, the backend storage folder has been deleted (so the image file in it as well, obviously) ...
Yet, the 2 VM are still up and running, with their root filesystem being mounted "rw".
- how does that work? I was expecting that with the file image being deleted, the VM would not work at all.
- since the VM is still running, is there a way to create a snapshot of it in order to re-create the image file on another storage pool ? I'm currently exploring "virsh snapshot-create-as ...", but if access to the image file is needed, it will fail.
On Linux a file is not deleted right away if there's a process that has the file opened, which is your case. The deletion is deferred until the last process closed the file.
In order to rescue the file you could get the PID of qemu that still has the disk open, and then find the FD corresponding to the file and with some magic you should be able to recover the file:
# pgrep qemu 221472
# ls -l /proc/221472/fd/ | grep deleted lr-x------ 1 root root 64 Jan 10 11:31 29 -> /var/lib/libvirt/images/fd.img (deleted)
(Here, 29 is the FD we're looking for)
# dd if=/proc/221472/fd/29 of=blah.img
snapshot-create-* won't work because that will instruct qemu to open the file again which will fail because the filename is gone.
You can use 'virsh blockcopy $domname $diskname /path/to/other/location --pivot --transient-job' to rescue this situation. The above command will copy the full block device from qemu's point of view (thus without any change) into the new location and then switch to it when done. Your VM will even continue running as it did but will use the new file.
participants (4)
-
Michael Hierweck -
Michal Privoznik -
Peter Krempa -
Roosen, Nicolas