1:44 p.m.
On 12/24/14 4:42 AM, Kashyap Chamarthy wrote:
On Tue, Dec 23, 2014 at 12:38:57PM -0600, Gary R Hook wrote:
[. . .]
In my case, the block device is a QCOW2 disk image file. If I boot
without using the disk image file which has the operating system, the
domain will fail to boot, no?
I see you're playing with NBD disks. I'll admit, I haven't played much
with QEMU NBD, will have to experiment post holidays.
Back from the holidays, and back on this issue. I've learned a lot.
I've learned how to use the blockcopy command to create a local copy in
a simple disk file:
virsh dumpxml my_domain > my_domain.xml
virsh undefine my_domain
virsh blockcopy --domain my_domain vda $PWD/dsk.copy.qcow2 --wait
--verbose --finish
virsh define my_domain.xml
and the resulting copy in dsk.copy.qcow2 is, indeed, bootable. It
appears to be a perfect copy, as I expect it to be.
But while I see (per Kashyap's article, etc) that it can be useful in
certain scenarios, it's not interesting to me. I would like to my copy
to be off-system, and was hoping to use the NBD interface to accomplish
that. So I tried this (a variant of the above), working on the same
system because it's easier:
qemu-img create -f qcow2 /tmp/dsk.test.qcow2
qemu-nbd -f qcow2 -p11112 /tmp/dsk.test.qcow2
nbd-client localhost 11112 /dev/nbd2
virsh dumpxml my_domain > my_domain.xml
virsh undefine my_domain
virsh blockcopy --domain my_domain --wait --verbose --finish
virsh define my_domain.xml
nbd-client -d /dev/nbd2
and the qemu-nbd process exits, as I wish. I presume at this point that
the new file has integrity.
I can take the qcow2 file that belongs to the domain and serve it up via
NBD:
qemu-nbd --partition=1 -p11112 /path/to/my/qcow2/file.qcow2
nbd-client localhost 11112 /dev/nbd2
mount /dev/nbd2 -oloop /mnt/foo
and lo! in /mnt/foo I found my root filesytem. Seems perfectly reasonable.
If, however, I try to use my generated-via-NBD file, I get this:
# qemu-nbd --partition=1 -p11112 $PWD/dsk.test.qcow2 &
[1] 7672
# qemu-nbd: Could not find partition 1: Invalid argument
[1]+ Exit 1 qemu-nbd --partition=1 -p11112
$PWD/dsk.test.qcow2
# qemu-nbd --partition=0 -p11112 $PWD/dsk.test.qcow2 &
[1] 7686
# qemu-nbd: Invalid partition 0
^C
[1]+ Exit 1 qemu-nbd --partition=0 -p11112
$PWD/dsk.test.qcow2
# qemu-nbd --partition=2 -p11112 $PWD/dsk.test.qcow2 &
[1] 7699
# qemu-nbd: Could not find partition 2: Invalid argument
^C
[1]+ Exit 1 qemu-nbd --partition=2 -p11112
$PWD/dsk.test.qcow2
# qemu-nbd --partition=3 -p11112 $PWD/dsk.test.qcow2 &
[1] 7830
# qemu-nbd: Could not find partition 3: Invalid argument
[1]+ Exit 1 qemu-nbd --partition=3 -p11112
$PWD/dsk.test.qcow2
I don't know what has been created, but it's not a copy of the original
guest's disk. There's no partition there, it seems.
So yes, blockcopy works fine under certain conditions. But the NBD layer
seems to really muck things up.
Or, more likely, I'm doing things wrong. I'm hoping someone can point
out something obvious.
There's a recent thread about "Block Replication for Continuous
Checkpointing" that is heading towards using NBD. I fail to understand
how this is ever going to work, based on my explorations.
--
Gary R Hook
Senior Kernel Engineer
NIMBOXX, Inc
2:21 p.m.
New subject: [libvirt-users] Using virsh blockcopy -- what's it supposed to accomplish?
On Thu, Jan 08, 2015 at 01:44:58PM -0600, Gary R Hook wrote:
On 12/24/14 4:42 AM, Kashyap Chamarthy wrote:
>On Tue, Dec 23, 2014 at 12:38:57PM -0600, Gary R Hook wrote:
>
>[. . .]
>
>In my case, the block device is a QCOW2 disk image file. If I boot
>without using the disk image file which has the operating system, the
>domain will fail to boot, no?
>
>I see you're playing with NBD disks. I'll admit, I haven't played much
>with QEMU NBD, will have to experiment post holidays.
Back from the holidays, and back on this issue. I've learned a lot.
I've learned how to use the blockcopy command to create a local copy in a
simple disk file:
virsh dumpxml my_domain > my_domain.xml
virsh undefine my_domain
virsh blockcopy --domain my_domain vda $PWD/dsk.copy.qcow2 --wait --verbose
--finish
virsh define my_domain.xml
Yes, that's correct.
and the resulting copy in dsk.copy.qcow2 is, indeed, bootable. It
appears to
be a perfect copy, as I expect it to be.
Good, this time you find it bootable, compared to your previous test.
But while I see (per Kashyap's article, etc) that it can be
useful in
certain scenarios, it's not interesting to me. I would like to my copy to be
off-system, and was hoping to use the NBD interface to accomplish that. So I
tried this (a variant of the above), working on the same system because it's
easier:
qemu-img create -f qcow2 /tmp/dsk.test.qcow2
A typo? You also need to provide a size here:
$ qemu-img create -f qcow2 /tmp/dsk.test.qcow2 1G
For the rest, I'm afraid I still didn't manage time to test
the NBD scenario to give a meaningful response here. I'll let the others
who deal with NBD more often respond to it.
qemu-nbd -f qcow2 -p11112 /tmp/dsk.test.qcow2
nbd-client localhost 11112 /dev/nbd2
virsh dumpxml my_domain > my_domain.xml
virsh undefine my_domain
virsh blockcopy --domain my_domain --wait --verbose --finish
virsh define my_domain.xml
nbd-client -d /dev/nbd2
and the qemu-nbd process exits, as I wish. I presume at this point that the
new file has integrity.
I can take the qcow2 file that belongs to the domain and serve it up via
NBD:
qemu-nbd --partition=1 -p11112 /path/to/my/qcow2/file.qcow2
nbd-client localhost 11112 /dev/nbd2
mount /dev/nbd2 -oloop /mnt/foo
and lo! in /mnt/foo I found my root filesytem. Seems perfectly reasonable.
If, however, I try to use my generated-via-NBD file, I get this:
# qemu-nbd --partition=1 -p11112 $PWD/dsk.test.qcow2 &
[1] 7672
# qemu-nbd: Could not find partition 1: Invalid argument
[1]+ Exit 1 qemu-nbd --partition=1 -p11112
$PWD/dsk.test.qcow2
# qemu-nbd --partition=0 -p11112 $PWD/dsk.test.qcow2 &
[1] 7686
# qemu-nbd: Invalid partition 0
^C
[1]+ Exit 1 qemu-nbd --partition=0 -p11112
$PWD/dsk.test.qcow2
# qemu-nbd --partition=2 -p11112 $PWD/dsk.test.qcow2 &
[1] 7699
# qemu-nbd: Could not find partition 2: Invalid argument
^C
[1]+ Exit 1 qemu-nbd --partition=2 -p11112
$PWD/dsk.test.qcow2
# qemu-nbd --partition=3 -p11112 $PWD/dsk.test.qcow2 &
[1] 7830
# qemu-nbd: Could not find partition 3: Invalid argument
[1]+ Exit 1 qemu-nbd --partition=3 -p11112
$PWD/dsk.test.qcow2
I don't know what has been created, but it's not a copy of the original
guest's disk. There's no partition there, it seems.
So yes, blockcopy works fine under certain conditions. But the NBD layer
seems to really muck things up.
Or, more likely, I'm doing things wrong. I'm hoping someone can point out
something obvious.
There's a recent thread about "Block Replication for Continuous
Checkpointing" that is heading towards using NBD. I fail to understand how
this is ever going to work, based on my explorations.
--
/kashyap
6:04 p.m.
New subject: [libvirt-users] Using virsh blockcopy -- what's it supposed to accomplish?
On 1/8/15 2:21 PM, Kashyap Chamarthy wrote:
> qemu-img create -f qcow2 /tmp/dsk.test.qcow2
A typo? You also need to provide a size here:
$ qemu-img create -f qcow2 /tmp/dsk.test.qcow2 1G
Yes, my mistake. The size is set to the potential size of the source
disk, which in this case is 20G.
For the rest, I'm afraid I still didn't manage time to test
the NBD scenario to give a meaningful response here. I'll let the others
who deal with NBD more often respond to it.
Well, that's the trick, right? No one that may have any experience has
managed to pitch in to this conversation.
I can add this now: qemu-nbd seems to function like nbd-server, and
concern itself with serving a filesystem. What's need, however, is a
simple _file_. The challenge is getting that NBD-served thing to be
viewed the same as a local disk file.
It would appear that the virsh option --raw is required. As far as I can
tell, both virsh and qemu-nbd munge data with the intent of making it
suitable for a qcow2 destination; the problem is that we don't need
double-munging. So telling the blockcopy operation to "knock it off"
seems to make it treat the NBD device the same as it does a simple disk
file.
Yea!
So I find that the following:
> qemu-nbd -f qcow2 -p11112 /tmp/dsk.test.qcow2
> nbd-client localhost 11112 /dev/nbd2
> virsh dumpxml my_domain > my_domain.xml
> virsh undefine my_domain
> virsh blockcopy --domain my_domain --wait --verbose --finish
only requires the addition of "--raw" to the above command. Or, rather,
what I really need, which is
virsh blockcopy --domain my_domain --raw
which I can then control with subsequent commands.
I'm kinda surprised no one else has tried to do this and lived to write
about it.
--
Gary R Hook
Senior Kernel Engineer
NIMBOXX, Inc
2:30 a.m.
New subject: [libvirt-users] [Qemu-devel] Using virsh blockcopy -- what's it supposed to accomplish?
On Thu, Jan 08, 2015 at 06:04:20PM -0600, Gary R Hook wrote:
On 1/8/15 2:21 PM, Kashyap Chamarthy wrote:
>>qemu-img create -f qcow2 /tmp/dsk.test.qcow2
>
>A typo? You also need to provide a size here:
>
> $ qemu-img create -f qcow2 /tmp/dsk.test.qcow2 1G
Yes, my mistake. The size is set to the potential size of the source disk,
which in this case is 20G.
>For the rest, I'm afraid I still didn't manage time to test
>the NBD scenario to give a meaningful response here. I'll let the others
>who deal with NBD more often respond to it.
Well, that's the trick, right? No one that may have any experience has
managed to pitch in to this conversation.
I can add this now: qemu-nbd seems to function like nbd-server, and concern
itself with serving a filesystem. What's need, however, is a simple _file_.
The challenge is getting that NBD-served thing to be viewed the same as a
local disk file.
It would appear that the virsh option --raw is required. As far as I can
tell, both virsh and qemu-nbd munge data with the intent of making it
suitable for a qcow2 destination; the problem is that we don't need
double-munging. So telling the blockcopy operation to "knock it off" seems
to make it treat the NBD device the same as it does a simple disk file.
Yea!
So I find that the following:
>>qemu-nbd -f qcow2 -p11112 /tmp/dsk.test.qcow2
>>nbd-client localhost 11112 /dev/nbd2
>>virsh dumpxml my_domain > my_domain.xml
>>virsh undefine my_domain
>>virsh blockcopy --domain my_domain --wait --verbose --finish
only requires the addition of "--raw" to the above command. Or, rather, what
I really need, which is
virsh blockcopy --domain my_domain --raw
which I can then control with subsequent commands.
Nice, I see that Eric and Paolo also concur with your observations.
I'm kinda surprised no one else has tried to do this and lived to write
about it.
Well, the above is slightly an exaggerated statement, as you can guess
from other responses in this thread, not everyone will invest the time
to write everything they test explaining it to the world. Anyhow, the
issue seems to be resolved, and as you suggested in your other response
to write about this, that'd be useful. :-)
--
/kashyap
3:04 p.m.
New subject: [libvirt-users] Using virsh blockcopy -- what's it supposed to accomplish?
On 01/08/2015 12:44 PM, Gary R Hook wrote:
I've learned how to use the blockcopy command to create a local
copy in
a simple disk file:
virsh dumpxml my_domain > my_domain.xml
virsh undefine my_domain
virsh blockcopy --domain my_domain vda $PWD/dsk.copy.qcow2 --wait
--verbose --finish
virsh define my_domain.xml
and the resulting copy in dsk.copy.qcow2 is, indeed, bootable. It
appears to be a perfect copy, as I expect it to be.
But while I see (per Kashyap's article, etc) that it can be useful in
certain scenarios, it's not interesting to me. I would like to my copy
to be off-system,
Once you have the local dsk.copy.qcow2, you can then move it off-system
and serve it up however you'd like, including by NBD.
and was hoping to use the NBD interface to accomplish
that. So I tried this (a variant of the above), working on the same
system because it's easier:
qemu-img create -f qcow2 /tmp/dsk.test.qcow2
qemu-nbd -f qcow2 -p11112 /tmp/dsk.test.qcow2
nbd-client localhost 11112 /dev/nbd2
virsh dumpxml my_domain > my_domain.xml
virsh undefine my_domain
virsh blockcopy --domain my_domain --wait --verbose --finish
Missing a destination? I'm guessing you meant /dev/nbd2 as the
pre-existing destination?
virsh define my_domain.xml
nbd-client -d /dev/nbd2
and the qemu-nbd process exits, as I wish. I presume at this point that
the new file has integrity.
Where are you specifying the format? I have not personally played with
NBD much. But here's my guess: Even though /tmp/dsk.test.qcow2 is a
qcow2 file, the NBD server is serving up a RAW image through /dev/nbd2.
Thus, if you are trying to treat /dev/nbd2 as the destination of your
copy, you MUST tell qemu that the file format of the copy is to be raw
(regardless of the file format of the original that is being copied
from). If you omit the --raw (also spelled --format=raw in newer
libvirt) parameter to the virsh blockcopy command, then libvirt has to
guess at the destination format; if the source was qcow2, then libvirt
will guess that the destination should be qcow2 as well. But writing
qcow2 data to a raw NBD disk means you have created a nested file in
/tmp/dsk.test.qcow2 - it is a qcow2 file whose contents are a qcow2 file
whose contents are the raw data (not typical usage, and a bit weird to
wrap your head around).
If, however, I try to use my generated-via-NBD file, I get this:
# qemu-nbd --partition=1 -p11112 $PWD/dsk.test.qcow2 &
[1] 7672
# qemu-nbd: Could not find partition 1: Invalid argument
It goes back to your format nesting mistake above - you are serving up a
nested qcow2 image rather than raw data, while the guest is trying to
treat the served up image as data. Since the nested qcow2 data in
sector 0 of the served-up data is not a partition table, it is no wonder
that you can't find a partition.
I don't know what has been created, but it's not a copy of the original
guest's disk. There's no partition there, it seems.
Or there is, but only if you peel off yet another layer of qcow2 :)
Try doing 'virsh blockcopy --raw /dev/nbd2 ...', to see if I was right
and that makes the difference. Remember, blockcopy CAN change things so
that the destination (as seen by qemu) is a different format than the
source, and in the case of doing a blockcopy from a qcow2 source to a
raw-image NBD server, this is a case where you need to be explicit about
the difference in format.
--
Eric Blake eblake redhat com +1-919-301-3266
Libvirt virtualization library http://libvirt.org
3:13 p.m.
New subject: [libvirt-users] Using virsh blockcopy -- what's it supposed to accomplish?
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 08/01/2015 22:04, Eric Blake wrote:
Where are you specifying the format? I have not personally played
with NBD much. But here's my guess: Even though
/tmp/dsk.test.qcow2 is a qcow2 file, the NBD server is serving up a
RAW image through /dev/nbd2. Thus, if you are trying to treat
/dev/nbd2 as the destination of your copy, you MUST tell qemu that
the file format of the copy is to be raw (regardless of the file
format of the original that is being copied from). If you omit the
--raw (also spelled --format=raw in newer libvirt) parameter to the
virsh blockcopy command, then libvirt has to guess at the
destination format; if the source was qcow2, then libvirt will
guess that the destination should be qcow2 as well. But writing
qcow2 data to a raw NBD disk means you have created a nested file
in /tmp/dsk.test.qcow2 - it is a qcow2 file whose contents are a
qcow2 file whose contents are the raw data (not typical usage, and
a bit weird to wrap your head around).
I think you're right.
Paolo
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
iQEcBAEBAgAGBQJUrvL9AAoJEL/70l94x66DkusIAJX9/MtI65eOZeW57upOBxuX
NOEZA34X8f/JY3WUI6Fhyfeoi/EZet4lEtPDnqG5isrYYhm7TMmJFCLt0fu4pr1Z
KSOgMlGN/Ze5m+ty/XdoYn21Ujw/nmSplaMVAc0iqfSXUXnd15LWzICemjXNSnHS
zOp0y7uLEM64EygW4vHrPAZZo5JCFj2JbQiRUiWjDyT14JeJ7EWRJ3vYSrZEAclB
RVDlm/Q6p9wirFSNbjfZgo1Fn5cHvVvsGvmNvWwLOnijBGeQwOD5LaqSsuAD3ZLN
QAQw77eLnVxwES+lR96vQcVWOKfR2pECD9pDEAwTVTqn/xbw0dqWuEwMSIZbfXU=
=Asfg
-----END PGP SIGNATURE-----
6:18 p.m.
New subject: [libvirt-users] Using virsh blockcopy -- what's it supposed to accomplish?
On 1/8/15 3:04 PM, Eric Blake wrote:
Where are you specifying the format? I have not personally played with
NBD much.
This appears to be the pervasive situation. There's not much out there
in google-land about this.
But here's my guess: Even though /tmp/dsk.test.qcow2 is a
qcow2 file, the NBD server is serving up a RAW image through /dev/nbd2.
Thus, if you are trying to treat /dev/nbd2 as the destination of your
copy, you MUST tell qemu that the file format of the copy is to be raw
(regardless of the file format of the original that is being copied
from). If you omit the --raw (also spelled --format=raw in newer
libvirt) parameter to the virsh blockcopy command, then libvirt has to
guess at the destination format; if the source was qcow2, then libvirt
will guess that the destination should be qcow2 as well. But writing
qcow2 data to a raw NBD disk means you have created a nested file in
/tmp/dsk.test.qcow2 - it is a qcow2 file whose contents are a qcow2 file
whose contents are the raw data (not typical usage, and a bit weird to
wrap your head around).
Yes, I _finally_ figured all of that out this afternoon after observing
that --raw seemed to get everything working. Thank you for confirming
the conclusions I arrived at independently.
I should turn this experience into a guest blog post, I suppose.
--
Gary R Hook
Senior Kernel Engineer
NIMBOXX, Inc
3640
days inactive
3641
days old
6 comments
4 participants
participants (4)
-
Eric Blake -
Gary R Hook -
Kashyap Chamarthy -
Paolo Bonzini