[libvirt] Help：Can libvirt restore several xen snapshots more faster at same time?
Dear libvirt expert, we find a blocker issue in our project. Would you please offer us some feedback? Our project have a requirement: restore as much as xen snapshot at same time. we have done the following test(our environment is cpu E52620, 256G memory, SSD hard disk. The xen snapshot is Win7 OS with 1G memory): 1) Save 40 xen snapshots to disk, then restore these snapshot at sametime, we found the restore speed is too slow, and the bottleneck is the IO speed. 2) we save all the 40 xen snapshot to ramdisk, then we got a good performance, but the space is too big, about 40G, it is too expensive . Our question is : 1) each our xen snapshot is almost same except IP address(Win7 OS with 1G memory), does there have some way to reduce the 40 snapshots space? 2) Do you have some suggestion to improve the performance of restore as much as xen snapshot at same time? Best regards, wish your reply.
On 11/13/2017 03:31 AM, Chenjia (C) wrote:
Dear libvirt expert,
we find a blocker issue in our project. Would you please offer us some feedback?
Our project have a requirement: restore as much as xen snapshot at same time. we have done the following test(our environment is cpu E52620, 256G memory, SSD hard disk. The xen snapshot is Win7 OS with 1G memory):
1)Save 40 xen snapshots to disk, then restore these snapshot at sametime, we found the restore speed is too slow, and the bottleneck is the IO speed.
You'll have to define "snapshots", since the libvirt Xen drivers do not support any of the virDomainSnapshot* APIs. By snapshot do you mean saving domain state to a file with e.g. 'virsh save ...'?
2)we save all the 40 xen snapshot to ramdisk, then we got a good performance, but the space is too big, about 40G, it is too expensive .
Our question is :
1)each our xen snapshot is almost same except IP address(Win7 OS with 1G memory), does there have some way to reduce the 40 snapshots space?
Maybe I'm being dense, but it is still not clear to me what you mean by "snapshot", or how your snapshots are created and restored. Regards, Jim
Dear libvirt expert: Thanks for your reply. May be our description is not suitable, in last message,'snapshot' means the domain state file which 'virsh save ' generate. Our project detailed steps are as follows: 1) create 40 xen guestOS by 'vrish create xen*.xml' 2) save 40 guest OS to domain state file by 'virsh save' cmd 3) Start multiple processes, each process cycle to do the following job: a)virsh restore the domain state file b)do same job in guest OS in 20 seconds c)virsh destroy the guest OS We need higher performance on the project , so our question is these as last message: 1) each our xen state file is almost same except IP address(Win7 OS with 1G memory), does there have some way to reduce the 40 state file space? 2) Do you have some suggestion to improve the performance of restore as much as xen state file at same time? Best regards, wish for your reply. -----邮件原件----- 发件人: Jim Fehlig [mailto:jfehlig@suse.com] 发送时间: 2017年11月14日 10:01 收件人: Chenjia (C) <chenjia09@huawei.com>; libvir-list@redhat.com 抄送: HUANG SHENGQIANG <huang.shengqiang@huawei.com>; Yaoshaomin <yaoshaomin@huawei.com> 主题: Re: [libvirt] Help：Can libvirt restore several xen snapshots more faster at same time? On 11/13/2017 03:31 AM, Chenjia (C) wrote:
Dear libvirt expert,
we find a blocker issue in our project. Would you please offer us some feedback?
Our project have a requirement: restore as much as xen snapshot at same time. we have done the following test(our environment is cpu E52620, 256G memory, SSD hard disk. The xen snapshot is Win7 OS with 1G memory):
1)Save 40 xen snapshots to disk, then restore these snapshot at sametime, we found the restore speed is too slow, and the bottleneck is the IO speed.
You'll have to define "snapshots", since the libvirt Xen drivers do not support any of the virDomainSnapshot* APIs. By snapshot do you mean saving domain state to a file with e.g. 'virsh save ...'?
2)we save all the 40 xen snapshot to ramdisk, then we got a good performance, but the space is too big, about 40G, it is too expensive .
Our question is :
1)each our xen snapshot is almost same except IP address(Win7 OS with 1G memory), does there have some way to reduce the 40 snapshots space?
Maybe I'm being dense, but it is still not clear to me what you mean by "snapshot", or how your snapshots are created and restored. Regards, Jim
On 11/14/2017 03:34 AM, Chenjia (C) wrote:
Dear libvirt expert: Thanks for your reply. May be our description is not suitable, in last message,'snapshot' means the domain state file which 'virsh save ' generate.
Our project detailed steps are as follows: 1) create 40 xen guestOS by 'vrish create xen*.xml' 2) save 40 guest OS to domain state file by 'virsh save' cmd 3) Start multiple processes, each process cycle to do the following job: a)virsh restore the domain state file
'virsh restore' has --bypass-cache option which basically enables O_DIRECT. That can save you couple of seconds.
b)do same job in guest OS in 20 seconds c)virsh destroy the guest OS
We need higher performance on the project , so our question is these as last message: 1) each our xen state file is almost same except IP address(Win7 OS with 1G memory), does there have some way to reduce the 40 state file space?
I don't think so. I mean, I don't know about Xen that much, but for instance in QEMU, the saved file contains memory for the guest (among with the internal state of hypervisor). In general, no guest have the same content of the memory. Also, you only *think* that the guests are the same. There is a lot of subtle differences (e.g. internal kernel state, RNG, network stack, stack randomization, etc.). So having some shared base image is not really a way to go. But for saving some disk space you can enable compression for the saved images (possibly not implemented in Xen driver).
2) Do you have some suggestion to improve the performance of restore as much as xen state file at same time?
See my replies above. But I'll let Jim reply as he knows more about Xen than me. Michal
Dear libvirt expert: Thanks for your reply. if we use ' virsh restore file --bypass-cache', we got the error message: 'unsupported flags (0x1) in function libxlDomainRestoreFlags'. By the way , May be our goal is not make it clear to you, add argument:'--bypass-cache' means ' avoid file system cache when restoring', it means if we restore one state file for the second time , it will be more slower than normal , because IO is slower than cache, am I right? In our project , our goal is restore more faster, if we add this argument ,we will get a even worse result, and this is not our goal. Best regards from Beijing. -----邮件原件----- 发件人: Michal Privoznik [mailto:mprivozn@redhat.com] 发送时间: 2017年11月14日 23:22 收件人: Chenjia (C) <chenjia09@huawei.com>; Jim Fehlig <jfehlig@suse.com>; libvir-list@redhat.com 抄送: HUANG SHENGQIANG <huang.shengqiang@huawei.com>; Yaoshaomin <yaoshaomin@huawei.com> 主题: Re: [libvirt] 答复: Help：Can libvirt restore several xen snapshots more faster at same time? On 11/14/2017 03:34 AM, Chenjia (C) wrote:
Dear libvirt expert: Thanks for your reply. May be our description is not suitable, in last message,'snapshot' means the domain state file which 'virsh save ' generate.
Our project detailed steps are as follows: 1) create 40 xen guestOS by 'vrish create xen*.xml' 2) save 40 guest OS to domain state file by 'virsh save' cmd 3) Start multiple processes, each process cycle to do the following job: a)virsh restore the domain state file
'virsh restore' has --bypass-cache option which basically enables O_DIRECT. That can save you couple of seconds.
b)do same job in guest OS in 20 seconds c)virsh destroy the guest OS
We need higher performance on the project , so our question is these as last message: 1) each our xen state file is almost same except IP address(Win7 OS with 1G memory), does there have some way to reduce the 40 state file space?
I don't think so. I mean, I don't know about Xen that much, but for instance in QEMU, the saved file contains memory for the guest (among with the internal state of hypervisor). In general, no guest have the same content of the memory. Also, you only *think* that the guests are the same. There is a lot of subtle differences (e.g. internal kernel state, RNG, network stack, stack randomization, etc.). So having some shared base image is not really a way to go. But for saving some disk space you can enable compression for the saved images (possibly not implemented in Xen driver).
2) Do you have some suggestion to improve the performance of restore as much as xen state file at same time?
See my replies above. But I'll let Jim reply as he knows more about Xen than me. Michal
On 11/15/2017 02:36 AM, Chenjia (C) wrote:
Dear libvirt expert: Thanks for your reply. if we use ' virsh restore file --bypass-cache', we got the error message: 'unsupported flags (0x1) in function libxlDomainRestoreFlags'.
--bypass-cache is not yet supported in the Xen driver. Should be easy to add, patches welcome :-).
By the way , May be our goal is not make it clear to you, add argument:'--bypass-cache' means ' avoid file system cache when restoring', it means if we restore one state file for the second time , it will be more slower than normal , because IO is slower than cache, am I right?
But you don't restore from the *same* file right? You restore VM1 from state- file1, VM2 from state-file2, etc., correct? Restoring from the same state file twice would result in 2 instances of the same VM. Actually, the libvirt Xen driver does not even allow this # virsh start test-hvm Domain test-hvm started # virsh save test-hvm /tmp/test-save.img Domain test-hvm saved to /tmp/test-save.img # virsh restore /tmp/test-save.img Domain restored from /tmp/test-save.img xen81:~ # virsh restore /tmp/test-save.img error: Failed to restore domain from /tmp/test-save.img error: Requested operation is not valid: domain 'test-hvm' is already active Regards, Jim
Dear libvirt expert: Thanks for your replay, we are not restore from the same state file twice, only one time. we have 40 process, each process handle it's own VM and VM state file, and each process will do following things cycle: While(1) { Virsh restore state file do something in VM in 20s virsh destroy vm } so we have sure now: There is currently no way to reduce these 40 xen status files space. When we do the above things, we got a xen crash on xen-4.7.0, so we try to use the xen-4.8.2 and libvirt 2.5.0, and the crash problem is no longer exist, but we got a new problem: we add the <interface> in config fie of VM: <interface type='bridge'> <mac address='18:c5:8a:15:c1:39'/> <source bridge='virbr0'/> </interface> When we use 'virsh create virsh_0.xml', we can't create the vm, and got follow error: 2017-11-28 12:36:56.329+0000: 17489: error : libxlDomainStart:1280 : internal error: libxenlight failed to create new domain 'aap_0' error: Failed to create domain from virsh_0.xml error: internal error: libxenlight failed to create new domain 'aap_0' and in the /var/log/libvirt/libxl/libxl-driver.log there have a log: 2017-11-28 20:36:53 CST libxl: error: libxl_exec.c:118:libxl_report_child_exitstatus: /etc/xen/scripts/vif-bridge online [17580] exited with error status 127 2017-11-28 20:36:53 CST libxl: error: libxl_create.c:1461:domcreate_attach_devices: unable to add nic devices. Could you plz help us how to process this problem? Thank you ! -----邮件原件----- 发件人: Jim Fehlig [mailto:jfehlig@suse.com] 发送时间: 2017年11月17日 5:58 收件人: Chenjia (C) <chenjia09@huawei.com>; Michal Privoznik <mprivozn@redhat.com>; libvir-list@redhat.com 抄送: HUANG SHENGQIANG <huang.shengqiang@huawei.com>; Yaoshaomin <yaoshaomin@huawei.com> 主题: Re: 答复: [libvirt] 答复: Help：Can libvirt restore several xen snapshots more faster at same time? On 11/15/2017 02:36 AM, Chenjia (C) wrote:
Dear libvirt expert: Thanks for your reply. if we use ' virsh restore file --bypass-cache', we got the error message: 'unsupported flags (0x1) in function libxlDomainRestoreFlags'.
--bypass-cache is not yet supported in the Xen driver. Should be easy to add, patches welcome :-).
By the way , May be our goal is not make it clear to you, add argument:'--bypass-cache' means ' avoid file system cache when restoring', it means if we restore one state file for the second time , it will be more slower than normal , because IO is slower than cache, am I right?
But you don't restore from the *same* file right? You restore VM1 from state- file1, VM2 from state-file2, etc., correct? Restoring from the same state file twice would result in 2 instances of the same VM. Actually, the libvirt Xen driver does not even allow this # virsh start test-hvm Domain test-hvm started # virsh save test-hvm /tmp/test-save.img Domain test-hvm saved to /tmp/test-save.img # virsh restore /tmp/test-save.img Domain restored from /tmp/test-save.img xen81:~ # virsh restore /tmp/test-save.img error: Failed to restore domain from /tmp/test-save.img error: Requested operation is not valid: domain 'test-hvm' is already active Regards, Jim
On 11/27/2017 09:26 PM, Chenjia (C) wrote:
Dear libvirt expert: Thanks for your replay, we are not restore from the same state file twice, only one time. we have 40 process, each process handle it's own VM and VM state file, and each process will do following things cycle: While(1) { Virsh restore state file do something in VM in 20s virsh destroy vm } so we have sure now: There is currently no way to reduce these 40 xen status files space.
When we do the above things, we got a xen crash on xen-4.7.0, so we try to use the xen-4.8.2 and libvirt 2.5.0, and the crash problem is no longer exist, but we got a new problem: we add the <interface> in config fie of VM: <interface type='bridge'> <mac address='18:c5:8a:15:c1:39'/> <source bridge='virbr0'/> </interface> When we use 'virsh create virsh_0.xml', we can't create the vm, and got follow error: 2017-11-28 12:36:56.329+0000: 17489: error : libxlDomainStart:1280 : internal error: libxenlight failed to create new domain 'aap_0' error: Failed to create domain from virsh_0.xml error: internal error: libxenlight failed to create new domain 'aap_0' and in the /var/log/libvirt/libxl/libxl-driver.log there have a log: 2017-11-28 20:36:53 CST libxl: error: libxl_exec.c:118:libxl_report_child_exitstatus: /etc/xen/scripts/vif-bridge online [17580] exited with error status 127
You need to determine why Xen's vif-bridge script failed. syslog should contain info about that. Regards, Jim
On 11/14/2017 08:22 AM, Michal Privoznik wrote:
On 11/14/2017 03:34 AM, Chenjia (C) wrote:
Dear libvirt expert: Thanks for your reply. May be our description is not suitable, in last message,'snapshot' means the domain state file which 'virsh save ' generate.
Our project detailed steps are as follows: 1) create 40 xen guestOS by 'vrish create xen*.xml' 2) save 40 guest OS to domain state file by 'virsh save' cmd 3) Start multiple processes, each process cycle to do the following job: a)virsh restore the domain state file
'virsh restore' has --bypass-cache option which basically enables O_DIRECT. That can save you couple of seconds.
As Chenjia already discovered, the Xen driver does not support that option, although should be easy to provide.
b)do same job in guest OS in 20 seconds c)virsh destroy the guest OS
We need higher performance on the project , so our question is these as last message: 1) each our xen state file is almost same except IP address(Win7 OS with 1G memory), does there have some way to reduce the 40 state file space?
I don't think so. I mean, I don't know about Xen that much, but for instance in QEMU, the saved file contains memory for the guest (among with the internal state of hypervisor). In general, no guest have the same content of the memory. Also, you only *think* that the guests are the same. There is a lot of subtle differences (e.g. internal kernel state, RNG, network stack, stack randomization, etc.). So having some shared base image is not really a way to go.
The same applies for Xen. There is currently no way to share any content from the saved state files.
But for saving some disk space you can enable compression for the saved images (possibly not implemented in Xen driver).
Yep, not yet implemented in the Xen driver.
2) Do you have some suggestion to improve the performance of restore as much as xen state file at same time?
See my replies above. But I'll let Jim reply as he knows more about Xen than me.
Xen is the same as qemu wrt save/restore. I'm not aware of any way to identify a page about to be restored is already in memory. We'd need a KSM-type service that could manage save and restore, writing shared pages to a common image and only restoring those once. But again, I'm not aware of such service, which seems non-trivial to create. Regards, Jim
participants (3)
-
Chenjia (C) -
Jim Fehlig -
Michal Privoznik