11:20 p.m.
Hello,
I am curious why when using virt-sparsify the apparent-size of the file
gets reduced to the actual file size? Is there a way to get the
apparent-size left as the full size?
In the man page for virt-sparsify it has a section on
"IMPORTANT NOTE ABOUT SPARSE OUTPUT IMAGES" and its output seems to
indicate what I would expect where "ls -lh" shows the full apparent size
and "du -sh" shows the actual size. But my actual results from testing
are different.
In the following example input.qcow2 is before sparsify and output.qcow2
is after.
Doing ls reports full size for input but a small size for
the output:
root@testingbox:/tmp/test # ls -lh
total 3.1G
-rw-r--r-- 1 root root 512M Jun 14 22:30 file.img
-rw-r--r-- 1 root root 51G Jun 14 22:30 input.qcow2
-rw-r--r-- 1 root root 790M Jun 14 22:36 output.qcow2
Same when asking du for apparent-size:
root@testingbox:/tmp/test # du -h --apparent-size *
512M file.img
51G input.qcow2
790M output.qcow2
A normal du will show the actual size of the files:
root@testingbox:/tmp/test # du -h *
0 file.img
2.4G input.qcow2
790M output.qcow2
The only way I have found to check the full virtual size of the file
after virt-sparsify is with qemu-img:
root@testingbox:/tmp/test # qemu-img info input.qcow2
image: input.qcow2
file format: qcow2
virtual size: 50G (53687091200 bytes)
disk size: 2.3G
cluster_size: 65536
Format specific information:
compat: 1.1
lazy refcounts: true
root@testingbox:/tmp/test # qemu-img info output.qcow2
image: output.qcow2
file format: qcow2
virtual size: 50G (53687091200 bytes)
disk size: 789M
cluster_size: 65536
Format specific information:
compat: 1.1
lazy refcounts: false
Thanks!
4:11 p.m.
On Wed, Jun 15, 2016 at 10:20:54AM -0500, libvirt_users(a)skagitattic.com wrote:
Hello,
I am curious why when using virt-sparsify the apparent-size of the file
gets reduced to the actual file size? Is there a way to get the
apparent-size left as the full size?
In the man page for virt-sparsify it has a section on
"IMPORTANT NOTE ABOUT SPARSE OUTPUT IMAGES" and its output seems to
indicate what I would expect where "ls -lh" shows the full apparent size
and "du -sh" shows the actual size. But my actual results from testing
are different.
In the following example input.qcow2 is before sparsify and output.qcow2
is after.
Doing ls reports full size for input but a small size for
the output:
root@testingbox:/tmp/test # ls -lh
total 3.1G
-rw-r--r-- 1 root root 512M Jun 14 22:30 file.img
-rw-r--r-- 1 root root 51G Jun 14 22:30 input.qcow2
-rw-r--r-- 1 root root 790M Jun 14 22:36 output.qcow2
What is file.img?
Same when asking du for apparent-size:
root@testingbox:/tmp/test # du -h --apparent-size *
512M file.img
51G input.qcow2
790M output.qcow2
'du --apparent-size' and 'ls -l' are the same thing, so it's not
really surprising that you see the same numbers here.
A normal du will show the actual size of the files:
root@testingbox:/tmp/test # du -h *
0 file.img
2.4G input.qcow2
790M output.qcow2
The only way I have found to check the full virtual size of the file
after virt-sparsify is with qemu-img:
root@testingbox:/tmp/test # qemu-img info input.qcow2
image: input.qcow2
file format: qcow2
virtual size: 50G (53687091200 bytes)
disk size: 2.3G
cluster_size: 65536
Format specific information:
compat: 1.1
lazy refcounts: true
root@testingbox:/tmp/test # qemu-img info output.qcow2
image: output.qcow2
file format: qcow2
virtual size: 50G (53687091200 bytes)
disk size: 789M
cluster_size: 65536
Format specific information:
compat: 1.1
lazy refcounts: false
If I understand your question correctly, then everything is working as
it should.
For qcow2 files, the virtual size is not related to the "apparent"
size, as you can easily prove:
$ qemu-img create -f qcow2 huge.qcow2 1T
Formatting 'huge.qcow2', fmt=qcow2 size=1099511627776 encryption=off
cluster_size=65536 lazy_refcounts=off refcount_bits=16
$ ls -lh huge.qcow2
-rw-r--r--. 1 rjones rjones 208K Jun 17 09:09 huge.qcow2
If you use raw files, then the file apparent size and virtual size are
the same.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines. Supports shell scripting,
bindings from many languages. http://libguestfs.org
1:40 p.m.
Hello Rich,
Thanks for the reply and information.
What is file.img?
Its simply a file I made while testing sparse files "truncate -s 512M
file.img".
'du --apparent-size' and 'ls -l' are the same thing,
so it's not
really surprising that you see the same numbers here.
Indeed, I expected those to be the same, just figured I would show them
both for completeness.
For qcow2 files, the virtual size is not related to the
"apparent"
size, as you can easily prove:
So you are saying when I do the virt-sparsify its converting the image
from raw to cow2?
I studied the man page for virt-sparsify and tried again with
the flag "--format raw" (output2.cow2). This output files looks as I
first expected.
# ls -lrh
total 4.7G
-rw-r--r-- 1 root root 790M Jun 14 22:36 output.qcow2
-rw-r--r-- 1 root root 51G Jun 21 18:34 output2.qcow2
-rw-r--r-- 1 root root 51G Jun 14 22:30 input.qcow2
-rw-r--r-- 1 root root 512M Jun 14 22:30 file.img
# du -sh *
0 file.img
2.4G input.qcow2
1.6G output2.qcow2
790M output.qcow2
So I guess it was detecting the source wrong and doing a conversion?
(As it says "If this is not specified, then the input format is used."
in the man page)
Also seems odd that it is 1.6G rather then the 790M of the output in
qcow2 format. Does qcow2 do some compression or something?
Is there some way I can see if a file is raw or cow2? When I did
"qemu-img info" it reported both of them as cow2.
I ran the initial install like the following and it ended up
with the sparse file that reports full size with ls: "virt-install
--name test-install --memory 512 -l
'http://ftp.nl.debian.org/debian/dists/jessie/main/installer-amd64/'
--disk size=50 --nographics -x "console=ttyS0""
I tried again and got the same result as the initial input file. So
why is it naming it qcow2 if its actually raw? Or is there two
way's of doing formatting inside the qcow2 file, one raw and the other
called something else?
Is there any reason to use raw vs non-raw? From some reading online it
appears performance is better with raw files. Is there a downside to
using raw?
Thanks!
Keelan
On Fri, 17 Jun 2016 09:11:04 +0100
"Richard W.M. Jones" <rjones(a)redhat.com> wrote:
> On Wed, Jun 15, 2016 at 10:20:54AM -0500,
> libvirt_users(a)skagitattic.com wrote:
> > Hello,
> >
> > I am curious why when using virt-sparsify the apparent-size of the
> > file gets reduced to the actual file size? Is there a way to get
> > the apparent-size left as the full size?
> >
> > In the man page for virt-sparsify it has a section on
> > "IMPORTANT NOTE ABOUT SPARSE OUTPUT IMAGES" and its output seems to
> > indicate what I would expect where "ls -lh" shows the full apparent
> > size and "du -sh" shows the actual size. But my actual results
> > from testing are different.
> >
> > In the following example input.qcow2 is before sparsify and
> > output.qcow2 is after.
> >
> > Doing ls reports full size for input but a small size for
> > the output:
> > root@testingbox:/tmp/test # ls -lh
> > total 3.1G
> > -rw-r--r-- 1 root root 512M Jun 14 22:30 file.img
> > -rw-r--r-- 1 root root 51G Jun 14 22:30 input.qcow2
> > -rw-r--r-- 1 root root 790M Jun 14 22:36 output.qcow2
>
What is file.img?
>
> > Same when asking du for apparent-size:
> > root@testingbox:/tmp/test # du -h --apparent-size *
> > 512M file.img
> > 51G input.qcow2
> > 790M output.qcow2
>
'du --apparent-size' and 'ls -l' are the same thing,
so it's not
really surprising that you see the same numbers here.
>
> > A normal du will show the actual size of the files:
> > root@testingbox:/tmp/test # du -h *
> > 0 file.img
> > 2.4G input.qcow2
> > 790M output.qcow2
> >
> > The only way I have found to check the full virtual size of the file
> > after virt-sparsify is with qemu-img:
> > root@testingbox:/tmp/test # qemu-img info input.qcow2
> > image: input.qcow2
> > file format: qcow2
> > virtual size: 50G (53687091200 bytes)
> > disk size: 2.3G
> > cluster_size: 65536
> > Format specific information:
> > compat: 1.1
> > lazy refcounts: true
> > root@testingbox:/tmp/test # qemu-img info output.qcow2
> > image: output.qcow2
> > file format: qcow2
> > virtual size: 50G (53687091200 bytes)
> > disk size: 789M
> > cluster_size: 65536
> > Format specific information:
> > compat: 1.1
> > lazy refcounts: false
>
> If I understand your question correctly, then everything is working as
> it should.
>
For qcow2 files, the virtual size is not related to the
"apparent"
size, as you can easily prove:
>
> $ qemu-img create -f qcow2 huge.qcow2 1T
> Formatting 'huge.qcow2', fmt=qcow2 size=1099511627776 encryption=off
> cluster_size=65536 lazy_refcounts=off refcount_bits=16 $ ls -lh
> huge.qcow2 -rw-r--r--. 1 rjones rjones 208K Jun 17 09:09 huge.qcow2
>
> If you use raw files, then the file apparent size and virtual size are
> the same.
>
> Rich.
>
4:58 p.m.
On Wed, Jun 22, 2016 at 12:40:40AM -0500, libvirt_users(a)skagitattic.com wrote:
So you are saying when I do the virt-sparsify its converting the
image
from raw to [qcow2]?
No. virt-sparsify will use the same input and output formats, unless
you use the --convert option.
I studied the man page for virt-sparsify and tried again with
the flag "--format raw" (output2.cow2). This output files looks as I
first expected.
# ls -lrh
total 4.7G
-rw-r--r-- 1 root root 790M Jun 14 22:36 output.qcow2
-rw-r--r-- 1 root root 51G Jun 21 18:34 output2.qcow2
-rw-r--r-- 1 root root 51G Jun 14 22:30 input.qcow2
-rw-r--r-- 1 root root 512M Jun 14 22:30 file.img
# du -sh *
0 file.img
2.4G input.qcow2
1.6G output2.qcow2
790M output.qcow2
So I guess it was detecting the source wrong and doing a conversion?
(As it says "If this is not specified, then the input format is used."
in the man page)
Also seems odd that it is 1.6G rather then the 790M of the output in
qcow2 format. Does qcow2 do some compression or something?
I'm really confused about what you actually are doing.
Delete what you've done and start from the beginning. Describe
exactly how you created the guest. Use 'qemu-img info' to show the
format of the input file. Show precisely the virt-sparsify command
you are running. And use 'qemu-img info' on the output file too.
Is there any reason to use raw vs non-raw? From some reading online
it
appears performance is better with raw files. Is there a downside to
using raw?
Raw has fewer features, especially no simple snapshotting or support
for backing files.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html
6:56 a.m.
Delete what you've done and start from the beginning. Describe
exactly how you created the guest. Use 'qemu-img info' to show the
format of the input file. Show precisely the virt-sparsify command
you are running. And use 'qemu-img info' on the output file too.
Okay, here goes. Start off with an install
root@testingbox:~ # virt-install --name testimage --memory 512 -l
'http://ftp.nl.debian.org/debian/dists/jessie/main/installer-amd64/'
--disk size=50 --nographics -x "console=ttyS0"
Starting install...
Retrieving file
MANIFEST...
| 3.3 kB 00:00 ... Retrieving file
linux...
| 6.0 MB 00:03 ... Retrieving file
initrd.gz...
| 29 MB 00:09 ... Allocating
'testimage.qcow2'
| 50 GB 00:00 Creating
domain...
| 0 B 00:00 Connected to domain testimage Escape character is
[ 0.000000] Initializing cgroup subsys cpuset [ 0.000000]
Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup
subsys cpuacct [ 0.000000] Linux version 3.16.0-4-amd64
(debian-kernel(a)lists.debian.org) (gcc version 4.8.4 (Debian 4.8.4-1)
) #1 SMP Debian 3.16.7-ckt25-2 (2016-04-08)
.... Many more lines here and it turns interactive. Suffice it to
say I did a basic minimal install following the prompts. Nothing
special here.
This created an image that is 2.5G with a 50G apparent size and
identifies as qcow2 as shown below.
# ls -slh testimage.qcow2
2.5G -rw-r--r-- 1 root root 51G Jun 22 20:22 testimage.qcow2
# qemu-img info testimage.qcow2
image: testimage.qcow2
file format: qcow2
virtual size: 50G (53687091200 bytes)
disk size: 2.4G
cluster_size: 65536
Format specific information:
compat: 1.1
lazy refcounts: true
Now we sparsify (/bigtmp is on a larger partition then / to give
virt-sparsify enough room to work with so it does not complain).
root@testingbox: 08:59 PM # virt-sparsify testimage.qcow2
testimage1.qcow2 --tmp /bigtmp Input disk virtual size = 53687091200
bytes (50.0G) Create overlay file in /bigtmp to protect source
disk ... Examine source disk ...
100%
⟦▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒⟧
--:-- Fill free space in /dev/sda1 with zero ... 100%
⟦▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒⟧
00:00 Clearing Linux swap on /dev/sda5 ... 100%
⟦▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒⟧
--:-- Copy to destination and make sparse ...
Sparsify operation completed with no errors. Before deleting the
old disk, carefully check that the target disk boots and works
correctly.
root@testingbox: 09:04 PM #
This image reports still reports as qcow2 however the apparent and real
size is the same.
# ls -slh testimage1.qcow2
1.1G -rw-r--r-- 1 root root 1.1G Jun 22 21:04 testimage1.qcow2
# qemu-img info testimage1.qcow2
image: testimage1.qcow2
file format: qcow2
virtual size: 50G (53687091200 bytes)
disk size: 1.1G
cluster_size: 65536
Format specific information:
compat: 1.1
lazy refcounts: false
If we try it again but specify raw its MUCH faster
root@testingbox: 09:26 PM # virt-sparsify testimage.qcow2
testimage2.qcow2 --tmp /bigtmp --format raw Input disk virtual size
= 53687091200 bytes (50.0G) Create overlay file in /bigtmp to
protect source disk ... Examine source disk ...
Copy to destination and make sparse ...
Sparsify operation completed with no errors. Before deleting the
old disk, carefully check that the target disk boots and works
correctly.
root@testingbox: 09:27 PM #
This time it takes up more space and reports real and apparent size
differently. It still reports as qcow2 with qemu-img.
# ls -slh testimage2.qcow2
1.7G -rw-r--r-- 1 root root 51G Jun 22 21:27 testimage2.qcow2
# qemu-img info testimage2.qcow2
image: testimage2.qcow2
file format: qcow2
virtual size: 50G (53687091200 bytes)
disk size: 1.7G
cluster_size: 65536
Format specific information:
compat: 1.1
lazy refcounts: true
The only things I see different from the finished files is that the raw
one uses a bit more space, the apparent size is not set to the
virtual size on the default sparsify one and the lazy refcounts is
different.
Is there something more detailed then "qemu-img info" that I could tell
the difference with?
Raw has fewer features, especially no simple snapshotting or support
for backing files.
Is simple snapshotting using qemu-img snapshot or something else? I
tried making a snapshot with both test images and they both appeared to
work.
# qemu-img snapshot -c test testimage1.qcow2
# qemu-img snapshot -c test testimage2.qcow2
# qemu-img info testimage2.qcow2
image: testimage2.qcow2
file format: qcow2
virtual size: 50G (53687091200 bytes)
disk size: 1.7G
cluster_size: 65536
Snapshot list:
ID TAG VM SIZE DATE VM
CLOCK 1 test 0 2016-06-22 22:41:52
00:00:00.000 Format specific information:
compat: 1.1
lazy refcounts: true
# qemu-img info testimage1.qcow2
image: testimage1.qcow2
file format: qcow2
virtual size: 50G (53687091200 bytes)
disk size: 1.1G
cluster_size: 65536
Snapshot list:
ID TAG VM SIZE DATE VM
CLOCK 1 test 0 2016-06-22 22:41:44
00:00:00.000 Format specific information:
compat: 1.1
lazy refcounts: false
The backing file thing appeared to work on the 2nd one where I
specified raw as well.
# qemu-img create -f qcow2 -b testimage2.qcow2 backup-test2.qcow2
Formatting 'backup-test2.qcow2', fmt=qcow2 size=53687091200
backing_file='testimage2.qcow2' encryption=off cluster_size=65536
lazy_refcounts=off
# ls -lsh
backup-test2.qcow2 196K -rw-r--r-- 1 root root 193K Jun 22 22:48
backup-test2.qcow2
Though perhaps if I actually tried to add them and boot it up something
would not work?
Keelan
On Wed, 22 Jun 2016 09:58:29 +0100
"Richard W.M. Jones" <rjones(a)redhat.com> wrote:
> On Wed, Jun 22, 2016 at 12:40:40AM -0500,
> libvirt_users(a)skagitattic.com wrote:
> > So you are saying when I do the virt-sparsify its converting the
> > image from raw to [qcow2]?
>
> No. virt-sparsify will use the same input and output formats, unless
> you use the --convert option.
>
> > I studied the man page for virt-sparsify and tried again with
> > the flag "--format raw" (output2.cow2). This output files looks as
> > I first expected.
> >
> > # ls -lrh
> > total 4.7G
> > -rw-r--r-- 1 root root 790M Jun 14 22:36 output.qcow2
> > -rw-r--r-- 1 root root 51G Jun 21 18:34 output2.qcow2
> > -rw-r--r-- 1 root root 51G Jun 14 22:30 input.qcow2
> > -rw-r--r-- 1 root root 512M Jun 14 22:30 file.img
> > # du -sh *
> > 0 file.img
> > 2.4G input.qcow2
> > 1.6G output2.qcow2
> > 790M output.qcow2
> >
> > So I guess it was detecting the source wrong and doing a conversion?
> > (As it says "If this is not specified, then the input format is
> > used." in the man page)
> >
> > Also seems odd that it is 1.6G rather then the 790M of the output in
> > qcow2 format. Does qcow2 do some compression or something?
>
> I'm really confused about what you actually are doing.
>
Delete what you've done and start from the beginning. Describe
exactly how you created the guest. Use 'qemu-img info' to show the
format of the input file. Show precisely the virt-sparsify command
you are running. And use 'qemu-img info' on the output file too.
>
> > Is there any reason to use raw vs non-raw? From some reading
> > online it appears performance is better with raw files. Is there a
> > downside to using raw?
>
Raw has fewer features, especially no simple snapshotting or support
for backing files.
>
> Rich.
>
3:39 p.m.
On Wed, 2016-06-22 at 17:56 -0500, libvirt_users(a)skagitattic.com wrote:
If we try it again but specify raw its MUCH faster
root@testingbox: 09:26 PM # virt-sparsify testimage.qcow2
testimage2.qcow2 --tmp /bigtmp --format raw Input disk virtual size
= 53687091200 bytes (50.0G) Create overlay file in /bigtmp to
protect source disk ... Examine source disk ...
Copy to destination and make sparse ...
Sparsify operation completed with no errors. Before deleting the
old disk, carefully check that the target disk boots and works
correctly.
root@testingbox: 09:27 PM #
This time it takes up more space and reports real and apparent size
differently. It still reports as qcow2 with qemu-img.
# ls -slh testimage2.qcow2
1.7G -rw-r--r-- 1 root root 51G Jun 22 21:27 testimage2.qcow2
# qemu-img info testimage2.qcow2
image: testimage2.qcow2
file format: qcow2
virtual size: 50G (53687091200 bytes)
disk size: 1.7G
cluster_size: 65536
Format specific information:
compat: 1.1
lazy refcounts: true
The '--format' option is to specify the image format for the
input image. If you want the *output* image to be raw, you'll
have to use '--convert raw'.
-- Andrea Bolognani / Red Hat / Virtualization
3:52 p.m.
On Wed, Jun 22, 2016 at 05:56:09PM -0500, libvirt_users(a)skagitattic.com wrote:
> Delete what you've done and start from the beginning.
Describe
> exactly how you created the guest. Use 'qemu-img info' to show the
> format of the input file. Show precisely the virt-sparsify command
> you are running. And use 'qemu-img info' on the output file too.
Okay, here goes. Start off with an install
root@testingbox:~ # virt-install --name testimage --memory 512 -l
'http://ftp.nl.debian.org/debian/dists/jessie/main/installer-amd64/'
--disk size=50 --nographics -x "console=ttyS0"
Starting install...
Retrieving file
MANIFEST...
| 3.3 kB 00:00 ... Retrieving file
linux...
| 6.0 MB 00:03 ... Retrieving file
initrd.gz...
| 29 MB 00:09 ... Allocating
'testimage.qcow2'
| 50 GB 00:00 Creating
domain...
| 0 B 00:00 Connected to domain testimage Escape character is
[ 0.000000] Initializing cgroup subsys cpuset [ 0.000000]
Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup
subsys cpuacct [ 0.000000] Linux version 3.16.0-4-amd64
(debian-kernel(a)lists.debian.org) (gcc version 4.8.4 (Debian 4.8.4-1)
) #1 SMP Debian 3.16.7-ckt25-2 (2016-04-08)
.... Many more lines here and it turns interactive. Suffice it to
say I did a basic minimal install following the prompts. Nothing
special here.
This created an image that is 2.5G with a 50G apparent size and
identifies as qcow2 as shown below.
# ls -slh testimage.qcow2
2.5G -rw-r--r-- 1 root root 51G Jun 22 20:22 testimage.qcow2
# qemu-img info testimage.qcow2
image: testimage.qcow2
file format: qcow2
virtual size: 50G (53687091200 bytes)
disk size: 2.4G
cluster_size: 65536
Format specific information:
compat: 1.1
lazy refcounts: true
Now we sparsify (/bigtmp is on a larger partition then / to give
virt-sparsify enough room to work with so it does not complain).
root@testingbox: 08:59 PM # virt-sparsify testimage.qcow2
testimage1.qcow2 --tmp /bigtmp Input disk virtual size = 53687091200
bytes (50.0G) Create overlay file in /bigtmp to protect source
disk ... Examine source disk ...
100%
⟦▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒⟧
--:-- Fill free space in /dev/sda1 with zero ... 100%
⟦▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒⟧
00:00 Clearing Linux swap on /dev/sda5 ... 100%
⟦▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒⟧
--:-- Copy to destination and make sparse ...
Sparsify operation completed with no errors. Before deleting the
old disk, carefully check that the target disk boots and works
correctly.
root@testingbox: 09:04 PM #
This image reports still reports as qcow2 however the apparent and real
size is the same.
# ls -slh testimage1.qcow2
1.1G -rw-r--r-- 1 root root 1.1G Jun 22 21:04 testimage1.qcow2
# qemu-img info testimage1.qcow2
image: testimage1.qcow2
file format: qcow2
virtual size: 50G (53687091200 bytes)
disk size: 1.1G
cluster_size: 65536
Format specific information:
compat: 1.1
lazy refcounts: false
virt-sparsify saved about 1.3 GB of disk space (2.4G -> 1.1G).
If we try it again but specify raw its MUCH faster
root@testingbox: 09:26 PM # virt-sparsify testimage.qcow2
testimage2.qcow2 --tmp /bigtmp --format raw
This is incorrect usage. You've told virt-sparsify that the input
image is raw. But it's not raw, it's qcow2.
In any case, virt-sparsify opened the image as raw, couldn't make any
sense of it (it appears to virt-sparsify to be random data, not a disk
image), and so it cannot sparsify it properly. This is "quicker" only
because virt-sparsify didn't do any sparsification of filesystems,
because it couldn't see the filesystems.
This time it takes up more space and reports real and apparent size
differently. It still reports as qcow2 with qemu-img.
# ls -slh testimage2.qcow2
1.7G -rw-r--r-- 1 root root 51G Jun 22 21:27 testimage2.qcow2
# qemu-img info testimage2.qcow2
image: testimage2.qcow2
file format: qcow2
virtual size: 50G (53687091200 bytes)
disk size: 1.7G
Looks like virt-sparsify still found some clusters containing all
zeroes which it could trim, even though it didn't know how to read the
disk.
Anyway, the moral is, don't misinform virt-sparsify of the format
of your input image.
If what you really intended to do was to convert the format from qcow2
to raw, you should use '--convert raw' instead.
The only things I see different from the finished files is that the
raw
one uses a bit more space, the apparent size is not set to the
virtual size on the default sparsify one and the lazy refcounts is
different.
Is there something more detailed then "qemu-img info" that I could tell
the difference with?
qemu-img info is the correct tool to use.
> Raw has fewer features, especially no simple snapshotting or
support
> for backing files.
I wrote this thinking you were talking about actual raw format disks,
not about qcow2 but where you've misinformed virt-sparsify about the
format.
Is simple snapshotting using qemu-img snapshot or something else? I
tried making a snapshot with both test images and they both appeared to
work.
http://wiki.qemu.org/Documentation/CreateSnapshot
# qemu-img snapshot -c test testimage1.qcow2
# qemu-img snapshot -c test testimage2.qcow2
# qemu-img info testimage2.qcow2
image: testimage2.qcow2
file format: qcow2
virtual size: 50G (53687091200 bytes)
disk size: 1.7G
cluster_size: 65536
Snapshot list:
ID TAG VM SIZE DATE VM
CLOCK 1 test 0 2016-06-22 22:41:52
00:00:00.000 Format specific information:
compat: 1.1
lazy refcounts: true
This works because testimage2.qcow2 is still in qcow2 format.
Anyway, use the correct --format option.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines. Supports shell scripting,
bindings from many languages. http://libguestfs.org
11:57 p.m.
Hello,
Thanks for all the information, I think my last remaining question is
why the image created with virt-install shows the full apparent size
and the image from virt-sparsify does not? (The image testimage1.qcow2
created with default options).
As they are both qcow2 I would expect them both to have the same
behavior as far as apparent size goes. Or is there an option I am
missing in virt-sparsify that would keep the full apparent size? Or
perhaps one in virt-install to set apperent and real size the same?
On Thu, 23 Jun 2016 09:39:52 +0200
Andrea Bolognani <abologna(a)redhat.com> wrote:
The '--format' option is to specify the image format for the
input image. If you want the *output* image to be raw, you'll
have to use '--convert raw'.
On Thu, 23 Jun 2016 08:52:35 +0100
"Richard W.M. Jones" <rjones(a)redhat.com> wrote:
This is incorrect usage. You've told virt-sparsify that the
input
image is raw. But it's not raw, it's qcow2.
That makes sense, I won't try and tell it the input is raw anymore
when its not.
Keelan
On Thu, 23 Jun 2016 08:52:35 +0100
"Richard W.M. Jones" <rjones(a)redhat.com> wrote:
> On Wed, Jun 22, 2016 at 05:56:09PM -0500,
> libvirt_users(a)skagitattic.com wrote:
> > > Delete what you've done and start from the beginning. Describe
> > > exactly how you created the guest. Use 'qemu-img info' to show
> > > the format of the input file. Show precisely the virt-sparsify
> > > command you are running. And use 'qemu-img info' on the output
> > > file too.
> >
> > Okay, here goes. Start off with an install
> >
> > root@testingbox:~ # virt-install --name testimage --memory 512
> > -l
> > 'http://ftp.nl.debian.org/debian/dists/jessie/main/installer-amd64/'
> > --disk size=50 --nographics -x "console=ttyS0"
> >
> > Starting install...
> > Retrieving file
> > MANIFEST...
> > | 3.3 kB 00:00 ... Retrieving file
> > linux...
> > | 6.0 MB 00:03 ... Retrieving file
> > initrd.gz...
> > | 29 MB 00:09 ... Allocating
> > 'testimage.qcow2'
> > | 50 GB 00:00 Creating
> > domain...
> > | 0 B 00:00 Connected to domain testimage Escape
> > character is [ 0.000000] Initializing cgroup subsys cpuset
> > [ 0.000000] Initializing cgroup subsys cpu [ 0.000000]
> > Initializing cgroup subsys cpuacct [ 0.000000] Linux version
> > 3.16.0-4-amd64 (debian-kernel(a)lists.debian.org) (gcc version 4.8.4
> > (Debian 4.8.4-1) ) #1 SMP Debian 3.16.7-ckt25-2 (2016-04-08)
> >
> > .... Many more lines here and it turns interactive. Suffice it to
> > say I did a basic minimal install following the prompts. Nothing
> > special here.
> >
> > This created an image that is 2.5G with a 50G apparent size and
> > identifies as qcow2 as shown below.
> >
> > # ls -slh testimage.qcow2
> > 2.5G -rw-r--r-- 1 root root 51G Jun 22 20:22 testimage.qcow2
> >
> > # qemu-img info testimage.qcow2
> > image: testimage.qcow2
> > file format: qcow2
> > virtual size: 50G (53687091200 bytes)
> > disk size: 2.4G
> > cluster_size: 65536
> > Format specific information:
> > compat: 1.1
> > lazy refcounts: true
> >
> > Now we sparsify (/bigtmp is on a larger partition then / to give
> > virt-sparsify enough room to work with so it does not complain).
> >
> > root@testingbox: 08:59 PM # virt-sparsify testimage.qcow2
> > testimage1.qcow2 --tmp /bigtmp Input disk virtual size =
> > 53687091200 bytes (50.0G) Create overlay file in /bigtmp to protect
> > source disk ... Examine source disk ...
> > 100%
> >
⟦▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒⟧
> > --:-- Fill free space in /dev/sda1 with zero ... 100%
> >
⟦▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒⟧
> > 00:00 Clearing Linux swap on /dev/sda5 ... 100%
> >
⟦▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒⟧
> > --:-- Copy to destination and make sparse ...
> >
> > Sparsify operation completed with no errors. Before deleting
> > the old disk, carefully check that the target disk boots and works
> > correctly.
> > root@testingbox: 09:04 PM #
> >
> > This image reports still reports as qcow2 however the apparent and
> > real size is the same.
> >
> > # ls -slh testimage1.qcow2
> > 1.1G -rw-r--r-- 1 root root 1.1G Jun 22 21:04 testimage1.qcow2
> >
> > # qemu-img info testimage1.qcow2
> > image: testimage1.qcow2
> > file format: qcow2
> > virtual size: 50G (53687091200 bytes)
> > disk size: 1.1G
> > cluster_size: 65536
> > Format specific information:
> > compat: 1.1
> > lazy refcounts: false
>
> virt-sparsify saved about 1.3 GB of disk space (2.4G -> 1.1G).
>
> > If we try it again but specify raw its MUCH faster
> >
> > root@testingbox: 09:26 PM # virt-sparsify testimage.qcow2
> > testimage2.qcow2 --tmp /bigtmp --format raw
>
This is incorrect usage. You've told virt-sparsify that the
input
image is raw. But it's not raw, it's qcow2.
>
> In any case, virt-sparsify opened the image as raw, couldn't make any
> sense of it (it appears to virt-sparsify to be random data, not a disk
> image), and so it cannot sparsify it properly. This is "quicker" only
> because virt-sparsify didn't do any sparsification of filesystems,
> because it couldn't see the filesystems.
>
> > This time it takes up more space and reports real and apparent size
> > differently. It still reports as qcow2 with qemu-img.
> >
> > # ls -slh testimage2.qcow2
> > 1.7G -rw-r--r-- 1 root root 51G Jun 22 21:27 testimage2.qcow2
> > # qemu-img info testimage2.qcow2
> > image: testimage2.qcow2
> > file format: qcow2
> > virtual size: 50G (53687091200 bytes)
> > disk size: 1.7G
>
> Looks like virt-sparsify still found some clusters containing all
> zeroes which it could trim, even though it didn't know how to read the
> disk.
>
> Anyway, the moral is, don't misinform virt-sparsify of the format
> of your input image.
>
> If what you really intended to do was to convert the format from qcow2
> to raw, you should use '--convert raw' instead.
>
> > The only things I see different from the finished files is that the
> > raw one uses a bit more space, the apparent size is not set to the
> > virtual size on the default sparsify one and the lazy refcounts is
> > different.
> >
> > Is there something more detailed then "qemu-img info" that I could
> > tell the difference with?
>
> qemu-img info is the correct tool to use.
>
> > > Raw has fewer features, especially no simple snapshotting or
> > > support for backing files.
>
> I wrote this thinking you were talking about actual raw format disks,
> not about qcow2 but where you've misinformed virt-sparsify about the
> format.
>
> > Is simple snapshotting using qemu-img snapshot or something else? I
> > tried making a snapshot with both test images and they both
> > appeared to work.
>
> http://wiki.qemu.org/Documentation/CreateSnapshot
>
> > # qemu-img snapshot -c test testimage1.qcow2
> > # qemu-img snapshot -c test testimage2.qcow2
> > # qemu-img info testimage2.qcow2
> > image: testimage2.qcow2
> > file format: qcow2
> > virtual size: 50G (53687091200 bytes)
> > disk size: 1.7G
> > cluster_size: 65536
> > Snapshot list:
> > ID TAG VM SIZE DATE
> > VM CLOCK 1 test 0 2016-06-22 22:41:52
> > 00:00:00.000 Format specific information:
> > compat: 1.1
> > lazy refcounts: true
>
> This works because testimage2.qcow2 is still in qcow2 format.
>
> Anyway, use the correct --format option.
>
> Rich.
>
12:57 a.m.
On Thu, Jun 23, 2016 at 10:57:43AM -0500, libvirt_users(a)skagitattic.com wrote:
Thanks for all the information, I think my last remaining question
is
why the image created with virt-install shows the full apparent size
and the image from virt-sparsify does not? (The image testimage1.qcow2
created with default options).
I don't really know what you mean by "full apparent size". If you are
talking about the output of "ls -l", then don't -- that number is
unrelated to the virtual size or the space consumed on disk. Always
use "qemu-img info".
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW
1:07 a.m.
On Thu, 23 Jun 2016 17:57:53 +0100
"Richard W.M. Jones" <rjones(a)redhat.com> wrote:
On Thu, Jun 23, 2016 at 10:57:43AM -0500,
libvirt_users(a)skagitattic.com wrote:
> Thanks for all the information, I think my last remaining question
> is why the image created with virt-install shows the full apparent
> size and the image from virt-sparsify does not? (The image
> testimage1.qcow2 created with default options).
I don't really know what you mean by "full apparent size". If you are
talking about the output of "ls -l", then don't -- that number is
unrelated to the virtual size or the space consumed on disk. Always
use "qemu-img info".
So I think I figured out what's different. If I do
preallocation=metadata I end up with the apparent size the same as the
initial install.
# virt-sparsify testimage.qcow2 testimage3.qcow2 --tmp /bigtmp -o
preallocation=metadata Input disk virtual size = 53687091200 bytes
(50.0G) Create overlay file in /bigtmp to protect source disk ...
Examine source disk ...
Fill free space in /dev/sda1 with zero ...
100%
00:00 Clearing Linux swap on /dev/sda5 ... Copy to destination and
make sparse ...
Sparsify operation completed with no errors. Before deleting the
old disk, carefully check that the target disk boots and works
correctly.
# ls -lsh testimage.qcow2 testimage1.qcow2 testimage3.qcow2
1.1G -rw-r--r-- 1 root root 1.1G Jun 22 22:41 testimage1.qcow2
1.1G -rw-r--r-- 1 root root 51G Jun 23 16:20 testimage3.qcow2
2.5G -rw-r--r-- 1 root root 51G Jun 22 20:22 testimage.qcow2
I don't know how to tell what the image's pre-allocated state is other
then checking for apparent size as "qemu-img info" does not see a
difference (other then the fact I played around with snapshotting on
the testimage1).
# qemu-img info testimage1.qcow2
image: testimage1.qcow2
file format: qcow2
virtual size: 50G (53687091200 bytes)
disk size: 1.1G
cluster_size: 65536
Snapshot list:
ID TAG VM SIZE DATE VM
CLOCK 1 test 0 2016-06-22 22:41:44
00:00:00.000 Format specific information:
compat: 1.1
lazy refcounts: false
# qemu-img info
testimage3.qcow2 image: testimage3.qcow2
file format: qcow2
virtual size: 50G (53687091200 bytes)
disk size: 1.0G
cluster_size: 65536
Format specific information:
compat: 1.1
lazy refcounts: false
I will read up some more on preallocation though and hopefully find all
the answers I need.
Thanks for your help looking at this,
Keelan
Rich.
3055
days inactive
3063
days old
9 comments
3 participants
participants (3)
-
Andrea Bolognani -
libvirt_users＠skagitattic.com -
Richard W.M. Jones