Re: [libvirt-users] Using virsh blockcopy -- what's it supposed to accomplish?
On 12/24/14 4:42 AM, Kashyap Chamarthy wrote:
On Tue, Dec 23, 2014 at 12:38:57PM -0600, Gary R Hook wrote:
[. . .]
In my case, the block device is a QCOW2 disk image file. If I boot without using the disk image file which has the operating system, the domain will fail to boot, no?
I see you're playing with NBD disks. I'll admit, I haven't played much with QEMU NBD, will have to experiment post holidays.
Back from the holidays, and back on this issue. I've learned a lot. I've learned how to use the blockcopy command to create a local copy in a simple disk file: virsh dumpxml my_domain > my_domain.xml virsh undefine my_domain virsh blockcopy --domain my_domain vda $PWD/dsk.copy.qcow2 --wait --verbose --finish virsh define my_domain.xml and the resulting copy in dsk.copy.qcow2 is, indeed, bootable. It appears to be a perfect copy, as I expect it to be. But while I see (per Kashyap's article, etc) that it can be useful in certain scenarios, it's not interesting to me. I would like to my copy to be off-system, and was hoping to use the NBD interface to accomplish that. So I tried this (a variant of the above), working on the same system because it's easier: qemu-img create -f qcow2 /tmp/dsk.test.qcow2 qemu-nbd -f qcow2 -p11112 /tmp/dsk.test.qcow2 nbd-client localhost 11112 /dev/nbd2 virsh dumpxml my_domain > my_domain.xml virsh undefine my_domain virsh blockcopy --domain my_domain --wait --verbose --finish virsh define my_domain.xml nbd-client -d /dev/nbd2 and the qemu-nbd process exits, as I wish. I presume at this point that the new file has integrity. I can take the qcow2 file that belongs to the domain and serve it up via NBD: qemu-nbd --partition=1 -p11112 /path/to/my/qcow2/file.qcow2 nbd-client localhost 11112 /dev/nbd2 mount /dev/nbd2 -oloop /mnt/foo and lo! in /mnt/foo I found my root filesytem. Seems perfectly reasonable. If, however, I try to use my generated-via-NBD file, I get this: # qemu-nbd --partition=1 -p11112 $PWD/dsk.test.qcow2 & [1] 7672 # qemu-nbd: Could not find partition 1: Invalid argument [1]+ Exit 1 qemu-nbd --partition=1 -p11112 $PWD/dsk.test.qcow2 # qemu-nbd --partition=0 -p11112 $PWD/dsk.test.qcow2 & [1] 7686 # qemu-nbd: Invalid partition 0 ^C [1]+ Exit 1 qemu-nbd --partition=0 -p11112 $PWD/dsk.test.qcow2 # qemu-nbd --partition=2 -p11112 $PWD/dsk.test.qcow2 & [1] 7699 # qemu-nbd: Could not find partition 2: Invalid argument ^C [1]+ Exit 1 qemu-nbd --partition=2 -p11112 $PWD/dsk.test.qcow2 # qemu-nbd --partition=3 -p11112 $PWD/dsk.test.qcow2 & [1] 7830 # qemu-nbd: Could not find partition 3: Invalid argument [1]+ Exit 1 qemu-nbd --partition=3 -p11112 $PWD/dsk.test.qcow2 I don't know what has been created, but it's not a copy of the original guest's disk. There's no partition there, it seems. So yes, blockcopy works fine under certain conditions. But the NBD layer seems to really muck things up. Or, more likely, I'm doing things wrong. I'm hoping someone can point out something obvious. There's a recent thread about "Block Replication for Continuous Checkpointing" that is heading towards using NBD. I fail to understand how this is ever going to work, based on my explorations. -- Gary R Hook Senior Kernel Engineer NIMBOXX, Inc
On Thu, Jan 08, 2015 at 01:44:58PM -0600, Gary R Hook wrote:
On 12/24/14 4:42 AM, Kashyap Chamarthy wrote:
On Tue, Dec 23, 2014 at 12:38:57PM -0600, Gary R Hook wrote:
[. . .]
In my case, the block device is a QCOW2 disk image file. If I boot without using the disk image file which has the operating system, the domain will fail to boot, no?
I see you're playing with NBD disks. I'll admit, I haven't played much with QEMU NBD, will have to experiment post holidays.
Back from the holidays, and back on this issue. I've learned a lot.
I've learned how to use the blockcopy command to create a local copy in a simple disk file:
virsh dumpxml my_domain > my_domain.xml virsh undefine my_domain virsh blockcopy --domain my_domain vda $PWD/dsk.copy.qcow2 --wait --verbose --finish virsh define my_domain.xml
Yes, that's correct.
and the resulting copy in dsk.copy.qcow2 is, indeed, bootable. It appears to be a perfect copy, as I expect it to be.
Good, this time you find it bootable, compared to your previous test.
But while I see (per Kashyap's article, etc) that it can be useful in certain scenarios, it's not interesting to me. I would like to my copy to be off-system, and was hoping to use the NBD interface to accomplish that. So I tried this (a variant of the above), working on the same system because it's easier:
qemu-img create -f qcow2 /tmp/dsk.test.qcow2
A typo? You also need to provide a size here: $ qemu-img create -f qcow2 /tmp/dsk.test.qcow2 1G For the rest, I'm afraid I still didn't manage time to test the NBD scenario to give a meaningful response here. I'll let the others who deal with NBD more often respond to it.
qemu-nbd -f qcow2 -p11112 /tmp/dsk.test.qcow2 nbd-client localhost 11112 /dev/nbd2 virsh dumpxml my_domain > my_domain.xml virsh undefine my_domain virsh blockcopy --domain my_domain --wait --verbose --finish virsh define my_domain.xml nbd-client -d /dev/nbd2
and the qemu-nbd process exits, as I wish. I presume at this point that the new file has integrity.
I can take the qcow2 file that belongs to the domain and serve it up via NBD:
qemu-nbd --partition=1 -p11112 /path/to/my/qcow2/file.qcow2 nbd-client localhost 11112 /dev/nbd2 mount /dev/nbd2 -oloop /mnt/foo
and lo! in /mnt/foo I found my root filesytem. Seems perfectly reasonable.
If, however, I try to use my generated-via-NBD file, I get this:
# qemu-nbd --partition=1 -p11112 $PWD/dsk.test.qcow2 & [1] 7672 # qemu-nbd: Could not find partition 1: Invalid argument
[1]+ Exit 1 qemu-nbd --partition=1 -p11112 $PWD/dsk.test.qcow2 # qemu-nbd --partition=0 -p11112 $PWD/dsk.test.qcow2 & [1] 7686 # qemu-nbd: Invalid partition 0 ^C [1]+ Exit 1 qemu-nbd --partition=0 -p11112 $PWD/dsk.test.qcow2 # qemu-nbd --partition=2 -p11112 $PWD/dsk.test.qcow2 & [1] 7699 # qemu-nbd: Could not find partition 2: Invalid argument ^C [1]+ Exit 1 qemu-nbd --partition=2 -p11112 $PWD/dsk.test.qcow2 # qemu-nbd --partition=3 -p11112 $PWD/dsk.test.qcow2 & [1] 7830 # qemu-nbd: Could not find partition 3: Invalid argument
[1]+ Exit 1 qemu-nbd --partition=3 -p11112 $PWD/dsk.test.qcow2
I don't know what has been created, but it's not a copy of the original guest's disk. There's no partition there, it seems.
So yes, blockcopy works fine under certain conditions. But the NBD layer seems to really muck things up.
Or, more likely, I'm doing things wrong. I'm hoping someone can point out something obvious.
There's a recent thread about "Block Replication for Continuous Checkpointing" that is heading towards using NBD. I fail to understand how this is ever going to work, based on my explorations.
-- /kashyap
On 1/8/15 2:21 PM, Kashyap Chamarthy wrote:
qemu-img create -f qcow2 /tmp/dsk.test.qcow2
A typo? You also need to provide a size here:
$ qemu-img create -f qcow2 /tmp/dsk.test.qcow2 1G
Yes, my mistake. The size is set to the potential size of the source disk, which in this case is 20G.
For the rest, I'm afraid I still didn't manage time to test the NBD scenario to give a meaningful response here. I'll let the others who deal with NBD more often respond to it.
Well, that's the trick, right? No one that may have any experience has managed to pitch in to this conversation. I can add this now: qemu-nbd seems to function like nbd-server, and concern itself with serving a filesystem. What's need, however, is a simple _file_. The challenge is getting that NBD-served thing to be viewed the same as a local disk file. It would appear that the virsh option --raw is required. As far as I can tell, both virsh and qemu-nbd munge data with the intent of making it suitable for a qcow2 destination; the problem is that we don't need double-munging. So telling the blockcopy operation to "knock it off" seems to make it treat the NBD device the same as it does a simple disk file. Yea! So I find that the following:
qemu-nbd -f qcow2 -p11112 /tmp/dsk.test.qcow2 nbd-client localhost 11112 /dev/nbd2 virsh dumpxml my_domain > my_domain.xml virsh undefine my_domain virsh blockcopy --domain my_domain --wait --verbose --finish
only requires the addition of "--raw" to the above command. Or, rather, what I really need, which is virsh blockcopy --domain my_domain --raw which I can then control with subsequent commands. I'm kinda surprised no one else has tried to do this and lived to write about it. -- Gary R Hook Senior Kernel Engineer NIMBOXX, Inc
On Thu, Jan 08, 2015 at 06:04:20PM -0600, Gary R Hook wrote:
On 1/8/15 2:21 PM, Kashyap Chamarthy wrote:
qemu-img create -f qcow2 /tmp/dsk.test.qcow2
A typo? You also need to provide a size here:
$ qemu-img create -f qcow2 /tmp/dsk.test.qcow2 1G
Yes, my mistake. The size is set to the potential size of the source disk, which in this case is 20G.
For the rest, I'm afraid I still didn't manage time to test the NBD scenario to give a meaningful response here. I'll let the others who deal with NBD more often respond to it.
Well, that's the trick, right? No one that may have any experience has managed to pitch in to this conversation.
I can add this now: qemu-nbd seems to function like nbd-server, and concern itself with serving a filesystem. What's need, however, is a simple _file_. The challenge is getting that NBD-served thing to be viewed the same as a local disk file.
It would appear that the virsh option --raw is required. As far as I can tell, both virsh and qemu-nbd munge data with the intent of making it suitable for a qcow2 destination; the problem is that we don't need double-munging. So telling the blockcopy operation to "knock it off" seems to make it treat the NBD device the same as it does a simple disk file.
Yea!
So I find that the following:
qemu-nbd -f qcow2 -p11112 /tmp/dsk.test.qcow2 nbd-client localhost 11112 /dev/nbd2 virsh dumpxml my_domain > my_domain.xml virsh undefine my_domain virsh blockcopy --domain my_domain --wait --verbose --finish
only requires the addition of "--raw" to the above command. Or, rather, what I really need, which is
virsh blockcopy --domain my_domain --raw
which I can then control with subsequent commands.
Nice, I see that Eric and Paolo also concur with your observations.
I'm kinda surprised no one else has tried to do this and lived to write about it.
Well, the above is slightly an exaggerated statement, as you can guess from other responses in this thread, not everyone will invest the time to write everything they test explaining it to the world. Anyhow, the issue seems to be resolved, and as you suggested in your other response to write about this, that'd be useful. :-) -- /kashyap
On 01/08/2015 12:44 PM, Gary R Hook wrote:
I've learned how to use the blockcopy command to create a local copy in a simple disk file:
virsh dumpxml my_domain > my_domain.xml virsh undefine my_domain virsh blockcopy --domain my_domain vda $PWD/dsk.copy.qcow2 --wait --verbose --finish virsh define my_domain.xml
and the resulting copy in dsk.copy.qcow2 is, indeed, bootable. It appears to be a perfect copy, as I expect it to be.
But while I see (per Kashyap's article, etc) that it can be useful in certain scenarios, it's not interesting to me. I would like to my copy to be off-system,
Once you have the local dsk.copy.qcow2, you can then move it off-system and serve it up however you'd like, including by NBD.
and was hoping to use the NBD interface to accomplish that. So I tried this (a variant of the above), working on the same system because it's easier:
qemu-img create -f qcow2 /tmp/dsk.test.qcow2 qemu-nbd -f qcow2 -p11112 /tmp/dsk.test.qcow2 nbd-client localhost 11112 /dev/nbd2 virsh dumpxml my_domain > my_domain.xml virsh undefine my_domain virsh blockcopy --domain my_domain --wait --verbose --finish
Missing a destination? I'm guessing you meant /dev/nbd2 as the pre-existing destination?
virsh define my_domain.xml nbd-client -d /dev/nbd2
and the qemu-nbd process exits, as I wish. I presume at this point that the new file has integrity.
Where are you specifying the format? I have not personally played with NBD much. But here's my guess: Even though /tmp/dsk.test.qcow2 is a qcow2 file, the NBD server is serving up a RAW image through /dev/nbd2. Thus, if you are trying to treat /dev/nbd2 as the destination of your copy, you MUST tell qemu that the file format of the copy is to be raw (regardless of the file format of the original that is being copied from). If you omit the --raw (also spelled --format=raw in newer libvirt) parameter to the virsh blockcopy command, then libvirt has to guess at the destination format; if the source was qcow2, then libvirt will guess that the destination should be qcow2 as well. But writing qcow2 data to a raw NBD disk means you have created a nested file in /tmp/dsk.test.qcow2 - it is a qcow2 file whose contents are a qcow2 file whose contents are the raw data (not typical usage, and a bit weird to wrap your head around).
If, however, I try to use my generated-via-NBD file, I get this:
# qemu-nbd --partition=1 -p11112 $PWD/dsk.test.qcow2 & [1] 7672 # qemu-nbd: Could not find partition 1: Invalid argument
It goes back to your format nesting mistake above - you are serving up a nested qcow2 image rather than raw data, while the guest is trying to treat the served up image as data. Since the nested qcow2 data in sector 0 of the served-up data is not a partition table, it is no wonder that you can't find a partition.
I don't know what has been created, but it's not a copy of the original guest's disk. There's no partition there, it seems.
Or there is, but only if you peel off yet another layer of qcow2 :) Try doing 'virsh blockcopy --raw /dev/nbd2 ...', to see if I was right and that makes the difference. Remember, blockcopy CAN change things so that the destination (as seen by qemu) is a different format than the source, and in the case of doing a blockcopy from a qcow2 source to a raw-image NBD server, this is a case where you need to be explicit about the difference in format. -- Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 08/01/2015 22:04, Eric Blake wrote:
Where are you specifying the format? I have not personally played with NBD much. But here's my guess: Even though /tmp/dsk.test.qcow2 is a qcow2 file, the NBD server is serving up a RAW image through /dev/nbd2. Thus, if you are trying to treat /dev/nbd2 as the destination of your copy, you MUST tell qemu that the file format of the copy is to be raw (regardless of the file format of the original that is being copied from). If you omit the --raw (also spelled --format=raw in newer libvirt) parameter to the virsh blockcopy command, then libvirt has to guess at the destination format; if the source was qcow2, then libvirt will guess that the destination should be qcow2 as well. But writing qcow2 data to a raw NBD disk means you have created a nested file in /tmp/dsk.test.qcow2 - it is a qcow2 file whose contents are a qcow2 file whose contents are the raw data (not typical usage, and a bit weird to wrap your head around).
I think you're right. Paolo -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQEcBAEBAgAGBQJUrvL9AAoJEL/70l94x66DkusIAJX9/MtI65eOZeW57upOBxuX NOEZA34X8f/JY3WUI6Fhyfeoi/EZet4lEtPDnqG5isrYYhm7TMmJFCLt0fu4pr1Z KSOgMlGN/Ze5m+ty/XdoYn21Ujw/nmSplaMVAc0iqfSXUXnd15LWzICemjXNSnHS zOp0y7uLEM64EygW4vHrPAZZo5JCFj2JbQiRUiWjDyT14JeJ7EWRJ3vYSrZEAclB RVDlm/Q6p9wirFSNbjfZgo1Fn5cHvVvsGvmNvWwLOnijBGeQwOD5LaqSsuAD3ZLN QAQw77eLnVxwES+lR96vQcVWOKfR2pECD9pDEAwTVTqn/xbw0dqWuEwMSIZbfXU= =Asfg -----END PGP SIGNATURE-----
On 1/8/15 3:04 PM, Eric Blake wrote:
Where are you specifying the format? I have not personally played with NBD much.
This appears to be the pervasive situation. There's not much out there in google-land about this.
But here's my guess: Even though /tmp/dsk.test.qcow2 is a qcow2 file, the NBD server is serving up a RAW image through /dev/nbd2. Thus, if you are trying to treat /dev/nbd2 as the destination of your copy, you MUST tell qemu that the file format of the copy is to be raw (regardless of the file format of the original that is being copied from). If you omit the --raw (also spelled --format=raw in newer libvirt) parameter to the virsh blockcopy command, then libvirt has to guess at the destination format; if the source was qcow2, then libvirt will guess that the destination should be qcow2 as well. But writing qcow2 data to a raw NBD disk means you have created a nested file in /tmp/dsk.test.qcow2 - it is a qcow2 file whose contents are a qcow2 file whose contents are the raw data (not typical usage, and a bit weird to wrap your head around).
Yes, I _finally_ figured all of that out this afternoon after observing that --raw seemed to get everything working. Thank you for confirming the conclusions I arrived at independently. I should turn this experience into a guest blog post, I suppose. -- Gary R Hook Senior Kernel Engineer NIMBOXX, Inc
participants (4)
-
Eric Blake -
Gary R Hook -
Kashyap Chamarthy -
Paolo Bonzini