4:31 a.m.
Dear libvirt expert,
we find a blocker issue in our project. Would you please offer us some feedback?
Our project have a requirement: restore as much as xen snapshot at same time. we have
done the following test(our environment is cpu E52620, 256G memory, SSD hard disk. The
xen snapshot is Win7 OS with 1G memory):
1) Save 40 xen snapshots to disk, then restore these snapshot at sametime, we found
the restore speed is too slow, and the bottleneck is the IO speed.
2) we save all the 40 xen snapshot to ramdisk, then we got a good performance, but
the space is too big, about 40G, it is too expensive .
Our question is :
1) each our xen snapshot is almost same except IP address(Win7 OS with 1G memory),
does there have some way to reduce the 40 snapshots space?
2) Do you have some suggestion to improve the performance of restore as much as xen
snapshot at same time?
Best regards, wish your reply.
8:01 p.m.
On 11/13/2017 03:31 AM, Chenjia (C) wrote:
Dear libvirt expert,
we find a blocker issue in our project. Would you please offer us some feedback?
Our project have a requirement: restore as much as xen snapshot at same time.
we have done the following test(our environment is cpu E52620, 256G memory,
SSD hard disk. The xen snapshot is Win7 OS with 1G memory):
1)Save 40 xen snapshots to disk, then restore these snapshot at sametime, we
found the restore speed is too slow, and the bottleneck is the IO speed.
You'll have to define "snapshots", since the libvirt Xen drivers do not
support
any of the virDomainSnapshot* APIs. By snapshot do you mean saving domain state
to a file with e.g. 'virsh save ...'?
2)we save all the 40 xen snapshot to ramdisk, then we got a good performance,
but the space is too big, about 40G, it is too expensive .
Our question is :
1)each our xen snapshot is almost same except IP address(Win7 OS with 1G
memory), does there have some way to reduce the 40 snapshots space?
Maybe I'm being dense, but it is still not clear to me what you mean by
"snapshot", or how your snapshots are created and restored.
Regards,
Jim
8:34 p.m.
New subject: [libvirt] 答复: Help：Can libvirt restore several xen snapshots more faster at same time?
Dear libvirt expert:
Thanks for your reply.
May be our description is not suitable, in last message,'snapshot' means the
domain state file which 'virsh save ' generate.
Our project detailed steps are as follows:
1) create 40 xen guestOS by 'vrish create xen*.xml'
2) save 40 guest OS to domain state file by 'virsh save' cmd
3) Start multiple processes, each process cycle to do the following job:
a)virsh restore the domain state file
b)do same job in guest OS in 20 seconds
c)virsh destroy the guest OS
We need higher performance on the project , so our question is these as last message:
1) each our xen state file is almost same except IP address(Win7 OS with 1G memory), does
there have some way to reduce the 40 state file space?
2) Do you have some suggestion to improve the performance of restore as much as xen
state file at same time?
Best regards, wish for your reply.
-----邮件原件-----
发件人: Jim Fehlig [mailto:jfehlig@suse.com]
发送时间: 2017年11月14日 10:01
收件人: Chenjia (C) <chenjia09(a)huawei.com>; libvir-list(a)redhat.com
抄送: HUANG SHENGQIANG <huang.shengqiang(a)huawei.com>; Yaoshaomin
<yaoshaomin(a)huawei.com>
主题: Re: [libvirt] Help：Can libvirt restore several xen snapshots more faster at same
time?
On 11/13/2017 03:31 AM, Chenjia (C) wrote:
Dear libvirt expert,
we find a blocker issue in our project. Would you please offer us some feedback?
Our project have a requirement: restore as much as xen snapshot at same time.
we have done the following test(our environment is cpu E52620, 256G
memory, SSD hard disk. The xen snapshot is Win7 OS with 1G memory):
1)Save 40 xen snapshots to disk, then restore these snapshot at
sametime, we found the restore speed is too slow, and the bottleneck is the IO speed.
You'll have to define "snapshots", since the libvirt Xen drivers do not
support any of the virDomainSnapshot* APIs. By snapshot do you mean saving domain state to
a file with e.g. 'virsh save ...'?
2)we save all the 40 xen snapshot to ramdisk, then we got a good
performance, but the space is too big, about 40G, it is too expensive .
Our question is :
1)each our xen snapshot is almost same except IP address(Win7 OS with
1G memory), does there have some way to reduce the 40 snapshots space?
Maybe I'm being dense, but it is still not clear to me what you mean by
"snapshot", or how your snapshots are created and restored.
Regards,
Jim
9:22 a.m.
New subject: [libvirt] 答复: Help：Can libvirt restore several xen snapshots more faster at same time?
On 11/14/2017 03:34 AM, Chenjia (C) wrote:
Dear libvirt expert:
Thanks for your reply.
May be our description is not suitable, in last message,'snapshot' means the
domain state file which 'virsh save ' generate.
Our project detailed steps are as follows:
1) create 40 xen guestOS by 'vrish create xen*.xml'
2) save 40 guest OS to domain state file by 'virsh save' cmd
3) Start multiple processes, each process cycle to do the following job:
a)virsh restore the domain state file
'virsh restore' has --bypass-cache option which basically enables
O_DIRECT. That can save you couple of seconds.
b)do same job in guest OS in 20 seconds
c)virsh destroy the guest OS
We need higher performance on the project , so our question is these as last message:
1) each our xen state file is almost same except IP address(Win7 OS with 1G memory),
does there have some way to reduce the 40 state file space?
I don't think so. I mean, I don't know about Xen that much, but for
instance in QEMU, the saved file contains memory for the guest (among
with the internal state of hypervisor). In general, no guest have the
same content of the memory. Also, you only *think* that the guests are
the same. There is a lot of subtle differences (e.g. internal kernel
state, RNG, network stack, stack randomization, etc.). So having some
shared base image is not really a way to go. But for saving some disk
space you can enable compression for the saved images (possibly not
implemented in Xen driver).
2) Do you have some suggestion to improve the performance of restore
as much as xen state file at same time?
See my replies above. But I'll let Jim reply as he knows more about Xen
than me.
Michal
3:36 a.m.
New subject: [libvirt] 答复: 答复: Help：Can libvirt restore several xen snapshots more faster at same time?
Dear libvirt expert:
Thanks for your reply.
if we use ' virsh restore file --bypass-cache', we got the error message:
'unsupported flags (0x1) in function libxlDomainRestoreFlags'.
By the way , May be our goal is not make it clear to you, add
argument:'--bypass-cache' means ' avoid file system cache when
restoring', it means if we restore one state file for the second time , it will be
more slower than normal , because IO is slower than cache, am I right? In our project ,
our goal is restore more faster, if we add this argument ,we will get a even worse result,
and this is not our goal.
Best regards from Beijing.
-----邮件原件-----
发件人: Michal Privoznik [mailto:mprivozn@redhat.com]
发送时间: 2017年11月14日 23:22
收件人: Chenjia (C) <chenjia09(a)huawei.com>; Jim Fehlig <jfehlig(a)suse.com>;
libvir-list(a)redhat.com
抄送: HUANG SHENGQIANG <huang.shengqiang(a)huawei.com>; Yaoshaomin
<yaoshaomin(a)huawei.com>
主题: Re: [libvirt] 答复: Help：Can libvirt restore several xen snapshots more faster at same
time?
On 11/14/2017 03:34 AM, Chenjia (C) wrote:
Dear libvirt expert:
Thanks for your reply.
May be our description is not suitable, in last message,'snapshot' means the
domain state file which 'virsh save ' generate.
Our project detailed steps are as follows:
1) create 40 xen guestOS by 'vrish create xen*.xml'
2) save 40 guest OS to domain state file by 'virsh save' cmd
3) Start multiple processes, each process cycle to do the following job:
a)virsh restore the domain state file
'virsh restore' has --bypass-cache option which basically enables O_DIRECT. That
can save you couple of seconds.
b)do same job in guest OS in 20 seconds
c)virsh destroy the guest OS
We need higher performance on the project , so our question is these as last message:
1) each our xen state file is almost same except IP address(Win7 OS with 1G memory),
does there have some way to reduce the 40 state file space?
I don't think so. I mean, I don't know about Xen that much, but for instance in
QEMU, the saved file contains memory for the guest (among with the internal state of
hypervisor). In general, no guest have the same content of the memory. Also, you only
*think* that the guests are the same. There is a lot of subtle differences (e.g. internal
kernel state, RNG, network stack, stack randomization, etc.). So having some shared base
image is not really a way to go. But for saving some disk space you can enable compression
for the saved images (possibly not implemented in Xen driver).
2) Do you have some suggestion to improve the performance of restore
as much as xen state file at same time?
See my replies above. But I'll let Jim reply as he knows more about Xen than me.
Michal
3:57 p.m.
New subject: [libvirt] 答复: 答复: Help：Can libvirt restore several xen snapshots more faster at same time?
On 11/15/2017 02:36 AM, Chenjia (C) wrote:
Dear libvirt expert:
Thanks for your reply.
if we use ' virsh restore file --bypass-cache', we got the error message:
'unsupported flags (0x1) in function libxlDomainRestoreFlags'.
--bypass-cache is not yet supported in the Xen driver. Should be easy to add,
patches welcome :-).
By the way , May be our goal is not make it clear to you, add
argument:'--bypass-cache' means ' avoid file system cache when
restoring', it means if we restore one state file for the second time , it will be
more slower than normal , because IO is slower than cache, am I right?
But you don't restore from the *same* file right? You restore VM1 from state-
file1, VM2 from state-file2, etc., correct? Restoring from the same state file
twice would result in 2 instances of the same VM. Actually, the libvirt Xen
driver does not even allow this
# virsh start test-hvm
Domain test-hvm started
# virsh save test-hvm /tmp/test-save.img
Domain test-hvm saved to /tmp/test-save.img
# virsh restore /tmp/test-save.img
Domain restored from /tmp/test-save.img
xen81:~ # virsh restore /tmp/test-save.img
error: Failed to restore domain from /tmp/test-save.img
error: Requested operation is not valid: domain 'test-hvm' is already active
Regards,
Jim
10:26 p.m.
New subject: [libvirt] 答复: 答复: 答复: Help：Can libvirt restore several xen snapshots more faster at same time?
Dear libvirt expert:
Thanks for your replay, we are not restore from the same state file twice, only one time.
we have 40 process, each process handle it's own VM and VM state file, and each
process will do following things cycle:
While(1)
{
Virsh restore state file
do something in VM in 20s
virsh destroy vm
}
so we have sure now: There is currently no way to reduce these 40 xen status files
space.
When we do the above things, we got a xen crash on xen-4.7.0, so we try to use the
xen-4.8.2 and libvirt 2.5.0, and the crash problem is no longer exist, but we got a new
problem:
we add the <interface> in config fie of VM:
<interface type='bridge'>
<mac address='18:c5:8a:15:c1:39'/>
<source bridge='virbr0'/>
</interface>
When we use 'virsh create virsh_0.xml', we can't create the vm, and got
follow error:
2017-11-28 12:36:56.329+0000: 17489: error : libxlDomainStart:1280 : internal error:
libxenlight failed to create new domain 'aap_0'
error: Failed to create domain from virsh_0.xml
error: internal error: libxenlight failed to create new domain 'aap_0'
and in the /var/log/libvirt/libxl/libxl-driver.log there have a log:
2017-11-28 20:36:53 CST libxl: error: libxl_exec.c:118:libxl_report_child_exitstatus:
/etc/xen/scripts/vif-bridge online [17580] exited with error status 127
2017-11-28 20:36:53 CST libxl: error: libxl_create.c:1461:domcreate_attach_devices:
unable to add nic devices.
Could you plz help us how to process this problem? Thank you !
-----邮件原件-----
发件人: Jim Fehlig [mailto:jfehlig@suse.com]
发送时间: 2017年11月17日 5:58
收件人: Chenjia (C) <chenjia09(a)huawei.com>; Michal Privoznik
<mprivozn(a)redhat.com>; libvir-list(a)redhat.com
抄送: HUANG SHENGQIANG <huang.shengqiang(a)huawei.com>; Yaoshaomin
<yaoshaomin(a)huawei.com>
主题: Re: 答复: [libvirt] 答复: Help：Can libvirt restore several xen snapshots more faster at
same time?
On 11/15/2017 02:36 AM, Chenjia (C) wrote:
Dear libvirt expert:
Thanks for your reply.
if we use ' virsh restore file --bypass-cache', we got the error message:
'unsupported flags (0x1) in function libxlDomainRestoreFlags'.
--bypass-cache is not yet supported in the Xen driver. Should be easy to add, patches
welcome :-).
By the way , May be our goal is not make it clear to you, add
argument:'--bypass-cache' means ' avoid file system cache when
restoring', it means if we restore one state file for the second time , it will be
more slower than normal , because IO is slower than cache, am I right?
But you don't restore from the *same* file right? You restore VM1 from state- file1,
VM2 from state-file2, etc., correct? Restoring from the same state file twice would result
in 2 instances of the same VM. Actually, the libvirt Xen driver does not even allow this
# virsh start test-hvm
Domain test-hvm started
# virsh save test-hvm /tmp/test-save.img
Domain test-hvm saved to /tmp/test-save.img # virsh restore /tmp/test-save.img Domain
restored from /tmp/test-save.img xen81:~ # virsh restore /tmp/test-save.img
error: Failed to restore domain from /tmp/test-save.img
error: Requested operation is not valid: domain 'test-hvm' is already active
Regards,
Jim
12:41 p.m.
New subject: [libvirt] 答复: 答复: 答复: Help：Can libvirt restore several xen snapshots more faster at same time?
On 11/27/2017 09:26 PM, Chenjia (C) wrote:
Dear libvirt expert:
Thanks for your replay, we are not restore from the same state file twice, only one
time. we have 40 process, each process handle it's own VM and VM state file, and each
process will do following things cycle:
While(1)
{
Virsh restore state file
do something in VM in 20s
virsh destroy vm
}
so we have sure now: There is currently no way to reduce these 40 xen status files
space.
When we do the above things, we got a xen crash on xen-4.7.0, so we try to use the
xen-4.8.2 and libvirt 2.5.0, and the crash problem is no longer exist, but we got a new
problem:
we add the <interface> in config fie of VM:
<interface type='bridge'>
<mac address='18:c5:8a:15:c1:39'/>
<source bridge='virbr0'/>
</interface>
When we use 'virsh create virsh_0.xml', we can't create the vm, and got
follow error:
2017-11-28 12:36:56.329+0000: 17489: error : libxlDomainStart:1280 : internal error:
libxenlight failed to create new domain 'aap_0'
error: Failed to create domain from virsh_0.xml
error: internal error: libxenlight failed to create new domain 'aap_0'
and in the /var/log/libvirt/libxl/libxl-driver.log there have a log:
2017-11-28 20:36:53 CST libxl: error: libxl_exec.c:118:libxl_report_child_exitstatus:
/etc/xen/scripts/vif-bridge online [17580] exited with error status 127
You need to determine why Xen's vif-bridge script failed. syslog should contain
info about that.
Regards,
Jim
3:43 p.m.
New subject: [libvirt] 答复: Help：Can libvirt restore several xen snapshots more faster at same time?
On 11/14/2017 08:22 AM, Michal Privoznik wrote:
On 11/14/2017 03:34 AM, Chenjia (C) wrote:
> Dear libvirt expert:
> Thanks for your reply.
> May be our description is not suitable, in last message,'snapshot' means the
domain state file which 'virsh save ' generate.
>
> Our project detailed steps are as follows:
> 1) create 40 xen guestOS by 'vrish create xen*.xml'
> 2) save 40 guest OS to domain state file by 'virsh save' cmd
> 3) Start multiple processes, each process cycle to do the following job:
> a)virsh restore the domain state file
'virsh restore' has --bypass-cache option which basically enables
O_DIRECT. That can save you couple of seconds.
As Chenjia already discovered, the Xen driver does not support that option,
although should be easy to provide.
> b)do same job in guest OS in 20 seconds
> c)virsh destroy the guest OS
>
> We need higher performance on the project , so our question is these as last
message:
> 1) each our xen state file is almost same except IP address(Win7 OS with 1G memory),
does there have some way to reduce the 40 state file space?
I don't think so. I mean, I don't know about Xen that much, but for
instance in QEMU, the saved file contains memory for the guest (among
with the internal state of hypervisor). In general, no guest have the
same content of the memory. Also, you only *think* that the guests are
the same. There is a lot of subtle differences (e.g. internal kernel
state, RNG, network stack, stack randomization, etc.). So having some
shared base image is not really a way to go.
The same applies for Xen. There is currently no way to share any content from
the saved state files.
But for saving some disk
space you can enable compression for the saved images (possibly not
implemented in Xen driver).
Yep, not yet implemented in the Xen driver.
> 2) Do you have some suggestion to improve the performance of
restore as much as xen state file at same time?
See my replies above. But I'll let Jim reply as he knows more about Xen
than me.
Xen is the same as qemu wrt save/restore. I'm not aware of any way to identify a
page about to be restored is already in memory. We'd need a KSM-type service
that could manage save and restore, writing shared pages to a common image and
only restoring those once. But again, I'm not aware of such service, which seems
non-trivial to create.
Regards,
Jim
2551
days inactive
2566
days old
8 comments
3 participants
participants (3)
-
Chenjia (C) -
Jim Fehlig -
Michal Privoznik