[libvirt-users] re-create disk image file
Hi Everybody: I am curious about how to re-create a disk image file from a running guest when the original disk image file is accidentally deleted. This came up recently during a test scenario but I couldn't figure out how to do it. Thanks, Joe
On Mon, Nov 05, 2012 at 11:03:11AM -0800, Joe Linoff wrote:
I am curious about how to re-create a disk image file from a running guest when the original disk image file is accidentally deleted. This came up recently during a test scenario but I couldn't figure out how to do it.
If you have unlinked the image, but qemu-kvm (or whatever hypervisor you are running) still has it open, you should be able to to recover it by going to /proc/PID/fd and looking for a file descriptor number that is a symlink to your deleted image. Just copy it to a safe place. I would first run sync inside the guest and maybe suspend it. It would be nice if you could switch it to single user mode with all filesystems mounted read-only, but whatever you do, avoid destroying the domain. Or why not just attach a new disk to the guest and copy stuff inside the guest?
Hi Henrik: Thank you. Both suggestions are excellent. Have you noticed spam messages from this email list? I have gotten several in the past couple of days. Regards, Joe -----Original Message----- From: Henrik Ahlgren [mailto:pablo@seestieto.com] Sent: Monday, November 05, 2012 11:53 AM To: Joe Linoff Cc: libvirt-users@redhat.com Subject: Re: [libvirt-users] re-create disk image file On Mon, Nov 05, 2012 at 11:03:11AM -0800, Joe Linoff wrote:
I am curious about how to re-create a disk image file from a running guest when the original disk image file is accidentally deleted. This came up recently during a test scenario but I couldn't figure out how to do it.
If you have unlinked the image, but qemu-kvm (or whatever hypervisor you are running) still has it open, you should be able to to recover it by going to /proc/PID/fd and looking for a file descriptor number that is a symlink to your deleted image. Just copy it to a safe place. I would first run sync inside the guest and maybe suspend it. It would be nice if you could switch it to single user mode with all filesystems mounted read-only, but whatever you do, avoid destroying the domain. Or why not just attach a new disk to the guest and copy stuff inside the guest?
Or why not just attach a new disk to the guest and copy stuff inside
Hi Henrik: When I tried to copy the file descriptor file a "Stale NFS file handle" error was generated: $ cp -f /proc/13196/fd/9 /tmp/guest.img cp: cannot stat `/proc/13196/fd/9': Stale NFS file handle I suppose that makes sense because the file was accidently deleted but from the NFS server perspective it really isn't there. Do you know of any way to work around it? the guest? I understand how that would work to recover individual files/directories but I don't understand how I would be able to recover the entire image. Can you explain? Thanks, Joe -----Original Message----- From: libvirt-users-bounces@redhat.com [mailto:libvirt-users-bounces@redhat.com] On Behalf Of Joe Linoff Sent: Monday, November 05, 2012 11:55 AM To: Henrik Ahlgren Cc: libvirt-users@redhat.com Subject: Re: [libvirt-users] re-create disk image file Hi Henrik: Thank you. Both suggestions are excellent. Have you noticed spam messages from this email list? I have gotten several in the past couple of days. Regards, Joe -----Original Message----- From: Henrik Ahlgren [mailto:pablo@seestieto.com] Sent: Monday, November 05, 2012 11:53 AM To: Joe Linoff Cc: libvirt-users@redhat.com Subject: Re: [libvirt-users] re-create disk image file On Mon, Nov 05, 2012 at 11:03:11AM -0800, Joe Linoff wrote:
I am curious about how to re-create a disk image file from a running guest when the original disk image file is accidentally deleted. This came up recently during a test scenario but I couldn't figure out how to do it.
If you have unlinked the image, but qemu-kvm (or whatever hypervisor you are running) still has it open, you should be able to to recover it by going to /proc/PID/fd and looking for a file descriptor number that is a symlink to your deleted image. Just copy it to a safe place. I would first run sync inside the guest and maybe suspend it. It would be nice if you could switch it to single user mode with all filesystems mounted read-only, but whatever you do, avoid destroying the domain. Or why not just attach a new disk to the guest and copy stuff inside the guest? _______________________________________________ libvirt-users mailing list libvirt-users@redhat.com https://www.redhat.com/mailman/listinfo/libvirt-users
On 11/05/2012 01:25 PM, Joe Linoff wrote:
Hi Henrik:
When I tried to copy the file descriptor file a "Stale NFS file handle" error was generated:
$ cp -f /proc/13196/fd/9 /tmp/guest.img cp: cannot stat `/proc/13196/fd/9': Stale NFS file handle
I suppose that makes sense because the file was accidently deleted but from the NFS server perspective it really isn't there. Do you know of any way to work around it?
Or why not just attach a new disk to the guest and copy stuff inside the guest?
I understand how that would work to recover individual files/directories but I don't understand how I would be able to recover the entire image. Can you explain?
Wouldn't something like: dd /dev/vda /dev/vdb from within the guest do the trick of copying the entire image? But that's risky if the guest is actively modifying /dev/vda at the same time. What version of libvirt and qemu are you using? If at least 0.9.6 and 1.2 respectively, I think it should be possible to use 'virsh blockpull' to resolve your conundrum, as follows: # Create a new qcow2 file that wraps the deleted but still-open file virsh snapshot-create-as $dom --no-metadata --disk-only \ --diskspec /tmp/guest.img,file=/tmp/guest.img1 # Pull the contents into the new file virsh blockpull $dom /tmp/guest.img1 --wait --verbose at which point, the new file /tmp/guest.img1 should contain everything in use by the old deleted file, while no longer needing to keep the old file around, so it will finally be safe to stop your guest. -- Eric Blake eblake@redhat.com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Hi Eric: Thank you for your suggestions. Here is what happened when I tried the snapshot. I will try the dd a bit later. VERSIONS virsh: 0.9.10 qemu: 2.0.12 DOM: webdev $ ls -l /proc/13196/fd/9 lrwx------. 1 qemu qemu 64 Nov 5 11:57 /proc/13196/fd/9 -> /tools/vm/images/webdev.img (deleted) $ cp /proc/13196/fd/9 /tools/vm/images/webdev.img cp: cannot stat `/proc/13196/fd/9': Stale NFS file handle $ virsh snapshot-create-as webdev \ --no-metadata \ --disk-only \ --diskspec /tools/vm/images/webdev.img,file=/tools/vm/images/webdev.img1 error: unsupported configuration: no disk named '/tools/vm/images/webdev.img' I think that I must have some sort of typo but I am not familiar enough with snapshots to see it. -----Original Message----- From: Eric Blake [mailto:eblake@redhat.com] Sent: Monday, November 05, 2012 12:37 PM To: Joe Linoff Cc: Henrik Ahlgren; libvirt-users@redhat.com Subject: Re: [libvirt-users] re-create disk image file On 11/05/2012 01:25 PM, Joe Linoff wrote:
Hi Henrik:
When I tried to copy the file descriptor file a "Stale NFS file handle" error was generated:
$ cp -f /proc/13196/fd/9 /tmp/guest.img cp: cannot stat `/proc/13196/fd/9': Stale NFS file handle
I suppose that makes sense because the file was accidently deleted but
from the NFS server perspective it really isn't there. Do you know of any way to work around it?
Or why not just attach a new disk to the guest and copy stuff inside the guest?
I understand how that would work to recover individual files/directories but I don't understand how I would be able to recover the entire image. Can you explain?
Wouldn't something like: dd /dev/vda /dev/vdb from within the guest do the trick of copying the entire image? But that's risky if the guest is actively modifying /dev/vda at the same time. What version of libvirt and qemu are you using? If at least 0.9.6 and 1.2 respectively, I think it should be possible to use 'virsh blockpull' to resolve your conundrum, as follows: # Create a new qcow2 file that wraps the deleted but still-open file virsh snapshot-create-as $dom --no-metadata --disk-only \ --diskspec /tmp/guest.img,file=/tmp/guest.img1 # Pull the contents into the new file virsh blockpull $dom /tmp/guest.img1 --wait --verbose at which point, the new file /tmp/guest.img1 should contain everything in use by the old deleted file, while no longer needing to keep the old file around, so it will finally be safe to stop your guest. -- Eric Blake eblake@redhat.com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
On 11/05/2012 01:52 PM, Joe Linoff wrote:
Hi Eric:
[please don't top-post on technical lists]
Thank you for your suggestions. Here is what happened when I tried the snapshot. I will try the dd a bit later.
VERSIONS virsh: 0.9.10 qemu: 2.0.12
Huh? The current upstream version of qemu is 1.2; I don't recognize the 2.0.12 numbering.
DOM: webdev
$ ls -l /proc/13196/fd/9 lrwx------. 1 qemu qemu 64 Nov 5 11:57 /proc/13196/fd/9 -> /tools/vm/images/webdev.img (deleted)
$ cp /proc/13196/fd/9 /tools/vm/images/webdev.img cp: cannot stat `/proc/13196/fd/9': Stale NFS file handle
$ virsh snapshot-create-as webdev \ --no-metadata \ --disk-only \ --diskspec /tools/vm/images/webdev.img,file=/tools/vm/images/webdev.img1 error: unsupported configuration: no disk named '/tools/vm/images/webdev.img'
What does this say? virsh domblklist webdev That is the name of the disk where you are creating the snapshot. The --diskspec line needs two pieces of information: a designation for the current file being snapshotted (either the absolute path you used in your xml, or the shorthand name like 'vda' - valid names are found by the domblklist command); and the name of the new file to create the snapshot as. The fact that your error message complains about "no disk named '/tools/vm/images/webdev.img'" says that you probably didn't spell the file the same as it was named in the XML; seeing the domblklist command will make it clear what name you should be using instead.
# Pull the contents into the new file virsh blockpull $dom /tmp/guest.img1 --wait --verbose
For the blockpull command, the same usage applies - you name the disk by its path (but note that you just changed the path as a result of the snapshot-create-as command; as another domblklist would prove) or by its device name, such as vda. -- Eric Blake eblake@redhat.com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
[please don't top-post on technical lists]
Thank you for your suggestions. Here is what happened when I tried
Okay, sorry about that. the
snapshot. I will try the dd a bit later.
VERSIONS virsh: 0.9.10 qemu: 2.0.12
Huh? The current upstream version of qemu is 1.2; I don't recognize the 2.0.12 numbering.
Sorry about that. It was a typo. The version seems to be 0.12.1.2. from running $ /usr/libexec/qemu-kvm --version QEMU PC emulator version 0.12.1 (qemu-kvm-0.12.1.2), Copyright (c) 2003-2008 Fabrice Bellard
DOM: webdev
$ ls -l /proc/13196/fd/9 lrwx------. 1 qemu qemu 64 Nov 5 11:57 /proc/13196/fd/9 -> /tools/vm/images/webdev.img (deleted)
$ cp /proc/13196/fd/9 /tools/vm/images/webdev.img cp: cannot stat `/proc/13196/fd/9': Stale NFS file handle
$ virsh snapshot-create-as webdev \ --no-metadata \ --disk-only \ --diskspec /tools/vm/images/webdev.img,file=/tools/vm/images/webdev.img1 error: unsupported configuration: no disk named '/tools/vm/images/webdev.img'
What does this say?
virsh domblklist webdev
Target Source ------------------------------------------------ vda /var/lib/libvirt/images/webdev.img hdc -
That is the name of the disk where you are creating the snapshot. The --diskspec line needs two pieces of information: a designation for the current file being snapshotted (either the absolute path you used in your xml, or the shorthand name like 'vda' - valid names are found by the domblklist command); and the name of the new file to create the snapshot as.
The fact that your error message complains about "no disk named '/tools/vm/images/webdev.img'" says that you probably didn't spell the file the same as it was named in the XML; seeing the domblklist command will make it clear what name you should be using instead.
I see. Thank you for the clear explanation. It appears that the file name in the XML and the file name referenced by the fd are different: /var/lib/libvirt/images/webdev.img (XML, link, domblklist) /tools/vm/images/webdev.img (filer, fd reference) I modified the command to use /var/lib/libvirt/images/webdev.img but it had a different problem: # virsh snapshot-create-as webdev \ --no-metadata \ --disk-only \ --diskspec /var/lib/libvirt/images/webdev.img,file=/tools/vm/images/webdev.img1 error: operation failed: Failed to take snapshot: unknown command: 'snapshot_blkdev' I don't understand what the message is trying to tell me. Why would snapshot_blkdev not be available? Perhaps the version I have is too old?
# Pull the contents into the new file virsh blockpull $dom /tmp/guest.img1 --wait --verbose
For the blockpull command, the same usage applies - you name the disk by its path (but note that you just changed the path as a result of the snapshot-create-as command; as another domblklist would prove) or by its > device name, such as vda.
I see. BTW, I tried the dd option with another guest and it seemed to work. Thank you, Joe -- Eric Blake eblake@redhat.com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
On 11/05/2012 02:19 PM, Joe Linoff wrote:
Huh? The current upstream version of qemu is 1.2; I don't recognize the 2.0.12 numbering.
Sorry about that. It was a typo. The version seems to be 0.12.1.2. from running
$ /usr/libexec/qemu-kvm --version QEMU PC emulator version 0.12.1 (qemu-kvm-0.12.1.2), Copyright (c) 2003-2008 Fabrice Bellard
That sounds like RHEL 6 (or CentOS).
I see. Thank you for the clear explanation. It appears that the file name in the XML and the file name referenced by the fd are different:
/var/lib/libvirt/images/webdev.img (XML, link, domblklist) /tools/vm/images/webdev.img (filer, fd reference)
Sounds like one resolved symlinks while the other did not. Maybe it's worth enhancing libvirt to allow all names that resolve to the same file, rather than just the exact spelling used? At any rate, we solved this.
I modified the command to use /var/lib/libvirt/images/webdev.img but it had a different problem:
# virsh snapshot-create-as webdev \ --no-metadata \ --disk-only \ --diskspec /var/lib/libvirt/images/webdev.img,file=/tools/vm/images/webdev.img1 error: operation failed: Failed to take snapshot: unknown command: 'snapshot_blkdev'
I don't understand what the message is trying to tell me. Why would snapshot_blkdev not be available? Perhaps the version I have is too old?
Alas, your qemu is too old. Red Had did not ship live disk snapshots in RHEL 6.3; to get that functionality, you need RHEV 3.0. Wish I had better news, but at this point, an in-guest dd to a hot-plugged alternative disk is the best I can think of, since there's nothing you can do from the host side without qemu support.
BTW, I tried the dd option with another guest and it seemed to work.
Hope it works for your guest in question, then. Good luck. -- Eric Blake eblake@redhat.com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
That sounds like RHEL 6 (or CentOS).
Sounds like one resolved symlinks while the other did not. Maybe it's worth enhancing libvirt to allow all names that resolve to the same file, rather than just the exact spelling used? At any rate, we solved
Yup, CentOS 6.3. I should have mentioned that. this. Maybe but I don't think it is a high priority. Libvirt is pretty awesome the way it is. Yup, it is solved, thank you again.
Alas, your qemu is too old. Red Had did not ship live disk snapshots in RHEL 6.3; to get that functionality, you need RHEV 3.0. Wish I had better news, but at this point, an in-guest dd to a hot-plugged alternative disk is the best I can think of, since there's nothing you can do from the host side without qemu support.
That's unfortunate but at least I understand why.
Hope it works for your guest in question, then. Good luck.
Thank you. This mailing list has been great. Regards, Joe
participants (3)
-
Eric Blake -
Henrik Ahlgren -
Joe Linoff