3:06 p.m.
Hi! I have two pools pool_a and pool_b and need to copy volume from
pool_a to pool_b. Now i can use volume download/upload to transfer
data, but this is not optimal because all traffic goes via client.
Does it possible to signal libvirt copy volume from pool_a to pool_b
not transfer all data via client? Like server side copy?
If this is not possible - may be someone from redhat team can do that?
(Or provide some tips how can i do that?)
--
Vasiliy Tolstov,
e-mail: v.tolstov(a)selfip.ru
12:38 a.m.
On 04/14/2017 10:06 PM, Vasiliy Tolstov wrote:
Hi! I have two pools pool_a and pool_b and need to copy volume from
pool_a to pool_b. Now i can use volume download/upload to transfer
data, but this is not optimal because all traffic goes via client.
Does it possible to signal libvirt copy volume from pool_a to pool_b
not transfer all data via client? Like server side copy?
How about: virsh vol-create-from.
I've just realized that it does not preserve file sparseness O:-)
If this is not possible - may be someone from redhat team can do that?
(Or provide some tips how can i do that?)
Anyone can submit patches to libvirt. Not just Red Hat guys ;-) I'd love
to have more people with different backgrounds contributing. Simply
because together we know more.
Michal
3:47 a.m.
2017-04-17 8:38 GMT+03:00 Michal Privoznik <mprivozn(a)redhat.com>:
Anyone can submit patches to libvirt. Not just Red Hat guys ;-)
I'd love
to have more people with different backgrounds contributing. Simply
because together we know more.
I can take this (may be) what is the best way to implement this - use
optimal copy function tight with storage backend? Or create generic
variant that utilize vol download/upload and transfer data via local
libvirt daemon?
--
Vasiliy Tolstov,
e-mail: v.tolstov(a)selfip.ru
6:08 a.m.
On 04/17/2017 10:47 AM, Vasiliy Tolstov wrote:
2017-04-17 8:38 GMT+03:00 Michal Privoznik
<mprivozn(a)redhat.com>:
> Anyone can submit patches to libvirt. Not just Red Hat guys ;-) I'd love
> to have more people with different backgrounds contributing. Simply
> because together we know more.
I can take this (may be) what is the best way to implement this - use
optimal copy function tight with storage backend? Or create generic
variant that utilize vol download/upload and transfer data via local
libvirt daemon?
Before digging into implementation of a new functionality we should
check if the functionality is not already there. I mean - that 'virsh
vol-create-from' or libvirt.virStoragePool.createXMLFrom() or
virStorageVolCreateXMLFrom() - they should serve exactly what you need.
Don't they?
If not, then yes - we might focus on adding new API for that.
Michal
6:32 a.m.
2017-04-17 14:08 GMT+03:00 Michal Privoznik <mprivozn(a)redhat.com>:
Before digging into implementation of a new functionality we should
check if the functionality is not already there. I mean - that 'virsh
vol-create-from' or libvirt.virStoragePool.createXMLFrom() or
virStorageVolCreateXMLFrom() - they should serve exactly what you need.
Don't they?
If not, then yes - we might focus on adding new API for that.
This function now work across pools. So two volumes need to be in the
same pool. But i need to copy across pools
--
Vasiliy Tolstov,
e-mail: v.tolstov(a)selfip.ru
12:39 a.m.
On 04/17/2017 01:32 PM, Vasiliy Tolstov wrote:
2017-04-17 14:08 GMT+03:00 Michal Privoznik
<mprivozn(a)redhat.com>:
> Before digging into implementation of a new functionality we should
> check if the functionality is not already there. I mean - that 'virsh
> vol-create-from' or libvirt.virStoragePool.createXMLFrom() or
> virStorageVolCreateXMLFrom() - they should serve exactly what you need.
> Don't they?
>
> If not, then yes - we might focus on adding new API for that.
This function now work across pools. So two volumes need to be in the
same pool. But i need to copy across pools
Is that right? Here I'm successfully cloning a volume across two storage pools:
$ ipython
Python 3.4.5 (default, Mar 23 2017, 19:22:46)
Type "copyright", "credits" or "license" for more
information.
IPython 3.2.1 -- An enhanced Interactive Python.
? -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help -> Python's own help system.
object? -> Details about 'object', use 'object??' for extra details.
In [1]: import libvirt
In [2]: conn=libvirt.open("qemu:///system")
In [3]: poolD=conn.storagePoolLookupByName("default")
In [4]: poolT=conn.storagePoolLookupByName("tmp")
In [5]: volF=poolD.storageVolLookupByName("fedora.qcow2")
In [6]: poolT.createXMLFrom?
Signature: poolT.createXMLFrom(xmlDesc, clonevol, flags=0)
Docstring:
Create a storage volume in the parent pool, using the
'clonevol' volume as input. Information for the new
volume (name, perms) are passed via a typical volume
XML description.
Since 1.0.1 VIR_STORAGE_VOL_CREATE_PREALLOC_METADATA
in flags can be used to get higher performance with
qcow2 image files which don't support full preallocation,
by creating a sparse image file with metadata.
virStorageVolFree should be used to free the resources after the
storage volume object is no longer needed.
File: /usr/lib64/python3.4/site-packages/libvirt.py
Type: method
In [7]: volF.XMLDesc()
Out[7]: "<volume type='file'>\n
<name>fedora.qcow2</name>\n
<key>/var/lib/libvirt/images/fedora.qcow2</key>\n <source>\n
</source>\n <capacity unit='bytes'>21474836480</capacity>\n
<allocation unit='bytes'>15330648064</allocation>\n <physical
unit='bytes'>15330770944</physical>\n <target>\n
<path>/var/lib/libvirt/images/fedora.qcow2</path>\n <format
type='qcow2'/>\n <permissions>\n <mode>0644</mode>\n
<owner>0</owner>\n <group>0</group>\n
</permissions>\n <timestamps>\n
<atime>1492104221.232529688</atime>\n
<mtime>1491985789.153734999</mtime>\n
<ctime>1492104221.232529688</ctime>\n </timestamps>\n
<compat>1.1</compat>\n <features/>\n
</target>\n</volume>\n"
In [8]: poolT.createXMLFrom(volF.XMLDesc(), volF, 0)
Out[8]: <libvirt.virStorageVol at 0x7fa9fa279ac8>
Michal
6:03 a.m.
2017-04-18 8:39 GMT+03:00 Michal Privoznik <mprivozn(a)redhat.com>:
s that right? Here I'm successfully cloning a volume across two
storage pools:
Yes, it works but....
Source image:
image: /var/lib/libvirt/images/sda/143177
file format: qcow2
virtual size: 5.0G (5368709120 bytes)
disk size: 902M
cluster_size: 65536
Format specific information:
compat: 0.10
refcount bits: 16
Dest image:
image: /var/lib/libvirt/images/sda/test
file format: qcow2
virtual size: 5.0G (5368709120 bytes)
disk size: 902M
cluster_size: 65536
Format specific information:
compat: 0.10
refcount bits: 16
But
-rw------- 1 root 908M Apr 18 13:59 143177
-rw------- 1 root 902M Apr 18 13:59 test
Why size is changed on destination image? In case of cp
--spwarse=always size does not changed.
6:22 a.m.
2017-04-18 14:03 GMT+03:00 Vasiliy Tolstov <v.tolstov(a)selfip.ru>:
Why size is changed on destination image? In case of cp
--spwarse=always size does not changed.
This is qemu-img feature... Does libvirt garantie that successful
return from copy volume function not breaks dst volume? (md5 can't be
checked because file is changed)
--
Vasiliy Tolstov,
e-mail: v.tolstov(a)selfip.ru
6:31 a.m.
On 04/18/2017 01:03 PM, Vasiliy Tolstov wrote:
2017-04-18 8:39 GMT+03:00 Michal Privoznik
<mprivozn(a)redhat.com>:
> s that right? Here I'm successfully cloning a volume across two storage pools:
Yes, it works but....
Source image:
image: /var/lib/libvirt/images/sda/143177
file format: qcow2
virtual size: 5.0G (5368709120 bytes)
disk size: 902M
cluster_size: 65536
Format specific information:
compat: 0.10
refcount bits: 16
Dest image:
image: /var/lib/libvirt/images/sda/test
file format: qcow2
virtual size: 5.0G (5368709120 bytes)
disk size: 902M
cluster_size: 65536
Format specific information:
compat: 0.10
refcount bits: 16
But
-rw------- 1 root 908M Apr 18 13:59 143177
-rw------- 1 root 902M Apr 18 13:59 test
Why size is changed on destination image? In case of cp
--spwarse=always size does not changed.
That's because libvirt does not use `cp` to copy the disk images. We use
qemu-img for copying. And we do not guarantee that the disk structure
does not change. We guarantee that the data stored on the disk doesn't
change. Otherwise we wouldn't be able to convert disk images.
NB, with sparse files - it's not as straightforward either. Some
filesystems may decide that the hole is not big enough so they do
allocate it actually. XFS is a very good example of such behaviour. So
the data/hole structure of a file may not line up with the structure of
a copy. Although, the data as read from the files are the same.
Michal
7 a.m.
On Tue, Apr 18, 2017 at 01:31:11PM +0200, Michal Privoznik wrote:
On 04/18/2017 01:03 PM, Vasiliy Tolstov wrote:
> 2017-04-18 8:39 GMT+03:00 Michal Privoznik <mprivozn(a)redhat.com>:
> > s that right? Here I'm successfully cloning a volume across two storage
pools:
>
>
> Yes, it works but....
> Source image:
> image: /var/lib/libvirt/images/sda/143177
> file format: qcow2
> virtual size: 5.0G (5368709120 bytes)
> disk size: 902M
> cluster_size: 65536
> Format specific information:
> compat: 0.10
> refcount bits: 16
>
> Dest image:
> image: /var/lib/libvirt/images/sda/test
> file format: qcow2
> virtual size: 5.0G (5368709120 bytes)
> disk size: 902M
> cluster_size: 65536
> Format specific information:
> compat: 0.10
> refcount bits: 16
>
>
> But
> -rw------- 1 root 908M Apr 18 13:59 143177
> -rw------- 1 root 902M Apr 18 13:59 test
>
> Why size is changed on destination image? In case of cp
> --spwarse=always size does not changed.
>
That's because libvirt does not use `cp` to copy the disk images. We use
qemu-img for copying. And we do not guarantee that the disk structure does
not change. We guarantee that the data stored on the disk doesn't change.
Otherwise we wouldn't be able to convert disk images.
NB, with sparse files - it's not as straightforward either. Some filesystems
may decide that the hole is not big enough so they do allocate it actually.
XFS is a very good example of such behaviour. So the data/hole structure of
a file may not line up with the structure of a copy. Although, the data as
read from the files are the same.
XFS will also do speculative allocation of filesystem blocks which have
not yet had data written to them, on the basis that a future write might
need them soon. So you can't really compare allocation disk size at all.
Regards,
Daniel
--
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org -o- http://virt-manager.org :|
|: http://entangle-photo.org -o- http://search.cpan.org/~danberr/ :|
7:08 a.m.
2017-04-18 15:00 GMT+03:00 Daniel P. Berrange <berrange(a)redhat.com>:
XFS will also do speculative allocation of filesystem blocks which
have
not yet had data written to them, on the basis that a future write might
need them soon. So you can't really compare allocation disk size at all.
Ok thanks for info and for saving my time =)!
--
Vasiliy Tolstov,
e-mail: v.tolstov(a)selfip.ru
9 a.m.
On Tuesday, 18 April 2017 13:03:11 CEST Vasiliy Tolstov wrote:
2017-04-18 8:39 GMT+03:00 Michal Privoznik
<mprivozn(a)redhat.com>:
> s that right? Here I'm successfully cloning a volume across two storage pools:
Yes, it works but....
Source image:
image: /var/lib/libvirt/images/sda/143177
file format: qcow2
virtual size: 5.0G (5368709120 bytes)
disk size: 902M
cluster_size: 65536
Format specific information:
compat: 0.10
refcount bits: 16
Dest image:
image: /var/lib/libvirt/images/sda/test
file format: qcow2
virtual size: 5.0G (5368709120 bytes)
disk size: 902M
cluster_size: 65536
Format specific information:
compat: 0.10
refcount bits: 16
But
-rw------- 1 root 908M Apr 18 13:59 143177
-rw------- 1 root 902M Apr 18 13:59 test
Why size is changed on destination image? In case of cp
--spwarse=always size does not changed.
As Michal and Daniel said, this should not be a problem. Also, you can
check the content is the same using virt-diff (part of libguestfs):
# virt-diff \
-a /var/lib/libvirt/images/sda/143177 \
-A /var/lib/libvirt/images/sda/test > log
Most probably log should be empty, since the actual content of the
image did not change.
--
Pino Toscano
3:41 a.m.
2017-04-18 17:00 GMT+03:00 Pino Toscano <ptoscano(a)redhat.com>:
As Michal and Daniel said, this should not be a problem. Also, you
can
check the content is the same using virt-diff (part of libguestfs):
# virt-diff \
-a /var/lib/libvirt/images/sda/143177 \
-A /var/lib/libvirt/images/sda/test > log
Most probably log should be empty, since the actual content of the
image did not change.
Ok, thanks!
--
Vasiliy Tolstov,
e-mail: v.tolstov(a)selfip.ru
2774
days inactive
2779
days old
12 comments
4 participants
participants (4)
-
Daniel P. Berrange -
Michal Privoznik -
Pino Toscano -
Vasiliy Tolstov