CIMTest run performance
Hello all, I had been trying to set up cimtest to work on my machine for a couple of days without success. So yesterday I started to dig the code and found that every command is sent via ssh, even if the test is supposed to run locally. The reason of cimtest not working was exactly that did not have the ssh keys set up properly. By fixing that configuration, I was able to make the test work as expected. :) But I was still intrigued with the fact that every command is sent via ssh, thus taking cimtest a very very long time to complete. So I hacked a simple patch to try to workaround this issue: ===== diff -r d6951b8799a8 lib/VirtLib/utils.py --- a/lib/VirtLib/utils.py Wed May 18 13:49:05 2011 -0400 +++ b/lib/VirtLib/utils.py Fri May 27 07:42:23 2011 -0700 @@ -33,7 +33,8 @@ def run_remote(ip, cmd): - cmd = 'ssh %s -i %s root@%s "%s"' % (SSH_PARMS, SSH_KEY, ip, cmd) + if ip not in ["localhost", "127.0.0.1"]: + cmd = 'ssh %s -i %s root@%s "%s"' % (SSH_PARMS, SSH_KEY, ip, cmd) return commands.getstatusoutput(cmd) def copy_remote(ip, local, remote='/tmp'): ===== As you can see, it is a quick check if 'ip' is neither localhost nor 127.0.0.1, which only in that case the command is sent via ssh. Note that this patch triggers some other issues, which I hope to be able to fix soon. But there is already a *huge increase* in the performance. I have used the most simpler and probably not the most correct measure: $ time && cimtest && time With the following results: Without patch: Fri May 27 06:46:28 PDT 2011 Fri May 27 07:16:17 PDT 2011 With patch: Fri May 27 07:29:06 PDT 2011 Fri May 27 07:34:50 PDT 2011 From 30 minutes down to 6! _BUT_, I can't really tell if it is actually intended that everything runs with ssh. Taking a look on the test results, it is noticeable that cimtest behaves in a different way, as follows. Without path: FAIL : 18 XFAIL : 5 SKIP : 6 PASS : 158 ----------------- Total : 187 ================================================= FAIL Test Summary: ComputerSystem - 04_defineStartVS.py: FAIL ComputerSystemMigrationJobIndication - 01_csmig_ind_for_offline_mig.py: FAIL ElementSettingData - 01_forward.py: FAIL HostSystem - 02_hostsystem_to_rasd.py: FAIL HostSystem - 04_hs_to_EAPF.py: FAIL KVMRedirectionSAP - 02_ipv6_support.py: FAIL Profile - 02_profile_to_elec.py: FAIL Profile - 04_verify_libvirt_cim_slp_profiles.py: FAIL SystemDevice - 01_forward.py: FAIL VirtualSystemManagementService - 13_refconfig_additional_devs.py: FAIL VirtualSystemManagementService - 19_definenetwork_ers.py: FAIL VirtualSystemManagementService - 20_verify_vnc_password.py: FAIL VirtualSystemMigrationService - 06_remote_live_migration.py: FAIL VirtualSystemMigrationService - 07_remote_offline_migration.py: FAIL VirtualSystemMigrationService - 08_remote_restart_resume_migration.py: FAIL VirtualSystemSettingDataComponent - 01_forward.py: FAIL VirtualSystemSettingDataComponent - 02_reverse.py: FAIL VSSD - 04_vssd_to_rasd.py: FAIL ================================================= XFAIL Test Summary: ComputerSystem - 32_start_reboot.py: XFAIL SwitchService - 01_enum.py: XFAIL VirtualSystemManagementService - 22_addmulti_brg_interface.py: XFAIL VirtualSystemManagementService - 28_definesystem_with_vsi_profile.py: XFAIL VirtualSystemManagementService - 30_dynamic_disk_mod.py: XFAIL ================================================= SKIP Test Summary: ComputerSystem - 02_nosystems.py: SKIP LogicalDisk - 02_nodevs.py: SKIP VirtualSystemMigrationService - 01_migratable_host.py: SKIP VirtualSystemMigrationService - 02_host_migrate_type.py: SKIP VirtualSystemMigrationService - 05_migratable_host_errs.py: SKIP VSSD - 02_bootldr.py: SKIP With patch: FAIL : 15 XFAIL : 5 SKIP : 10 PASS : 157 ----------------- Total : 187 ================================================= FAIL Test Summary: ComputerSystem - 04_defineStartVS.py: FAIL ComputerSystem - 41_cs_to_settingdefinestate.py: FAIL ElementSettingData - 01_forward.py: FAIL HostSystem - 02_hostsystem_to_rasd.py: FAIL HostSystem - 04_hs_to_EAPF.py: FAIL KVMRedirectionSAP - 02_ipv6_support.py: FAIL Profile - 02_profile_to_elec.py: FAIL Profile - 04_verify_libvirt_cim_slp_profiles.py: FAIL SystemDevice - 01_forward.py: FAIL VirtualSystemManagementService - 06_addresource.py: FAIL VirtualSystemManagementService - 13_refconfig_additional_devs.py: FAIL VirtualSystemManagementService - 20_verify_vnc_password.py: FAIL VirtualSystemSettingDataComponent - 01_forward.py: FAIL VirtualSystemSettingDataComponent - 02_reverse.py: FAIL VSSD - 04_vssd_to_rasd.py: FAIL ================================================= XFAIL Test Summary: ComputerSystem - 32_start_reboot.py: XFAIL SwitchService - 01_enum.py: XFAIL VirtualSystemManagementService - 22_addmulti_brg_interface.py: XFAIL VirtualSystemManagementService - 28_definesystem_with_vsi_profile.py: XFAIL VirtualSystemManagementService - 30_dynamic_disk_mod.py: XFAIL ================================================= SKIP Test Summary: ComputerSystem - 02_nosystems.py: SKIP ComputerSystemMigrationJobIndication - 01_csmig_ind_for_offline_mig.py: SKIP LogicalDisk - 02_nodevs.py: SKIP VirtualSystemMigrationService - 01_migratable_host.py: SKIP VirtualSystemMigrationService - 02_host_migrate_type.py: SKIP VirtualSystemMigrationService - 05_migratable_host_errs.py: SKIP VirtualSystemMigrationService - 06_remote_live_migration.py: SKIP VirtualSystemMigrationService - 07_remote_offline_migration.py: SKIP VirtualSystemMigrationService - 08_remote_restart_resume_migration.py: SKIP VSSD - 02_bootldr.py: SKIP I would really appreciate comments from the experts on this issue. IMHO, it is something really worthy and if there are any tests that are required to run remotely, I could work on a proper fix to try to achieve the same results. Best regards, Etrunko -- Eduardo de Barros Lima Software Engineer, Open Virtualization Linux Technology Center - IBM/Brazil eblima@br.ibm.com
On 05/27/2011 12:28 PM, Eduardo Lima (Etrunko) wrote:
Hello all,
I had been trying to set up cimtest to work on my machine for a couple of days without success. So yesterday I started to dig the code and found that every command is sent via ssh, even if the test is supposed to run locally.
The reason of cimtest not working was exactly that did not have the ssh keys set up properly. By fixing that configuration, I was able to make the test work as expected. :)
But I was still intrigued with the fact that every command is sent via ssh, thus taking cimtest a very very long time to complete. So I hacked a simple patch to try to workaround this issue:
[snip]
As you can see, it is a quick check if 'ip' is neither localhost nor 127.0.0.1, which only in that case the command is sent via ssh.
Note that this patch triggers some other issues, which I hope to be able to fix soon. But there is already a *huge increase* in the performance. I have used the most simpler and probably not the most correct measure:
So I have went somewhat deeper in my investigations and found most of the issues triggered by that patch. An updated version can be found attached to this message.
_BUT_, I can't really tell if it is actually intended that everything runs with ssh. Taking a look on the test results, it is noticeable that cimtest behaves in a different way, as follows.
Now the behavior of cimtest with the patch applied is much more similar to the version without the patch. 158 tests are labeled PASS, and only 3 that were FAIL without patch are now SKIP. Unfortunately I don't have access to the test machine at the moment, so I am not able to provide the detailed report, which I will do as soon as possible.
I would really appreciate comments from the experts on this issue. IMHO, it is something really worthy and if there are any tests that are required to run remotely, I could work on a proper fix to try to achieve the same results.
Still waiting for some light regarding this issue. Best Regards, -- Eduardo de Barros Lima Software Engineer, Open Virtualization Linux Technology Center - IBM/Brazil eblima@br.ibm.com
On 05/30/2011 02:35 PM, Eduardo Lima (Etrunko) wrote:
Now the behavior of cimtest with the patch applied is much more similar to the version without the patch. 158 tests are labeled PASS, and only 3 that were FAIL without patch are now SKIP. Unfortunately I don't have access to the test machine at the moment, so I am not able to provide the detailed report, which I will do as soon as possible.
I have found similar cases of commands running locally with unexpected behavior, so I went on and applied the same modifications applied for commands which involved awk to basically all other commands I could find. Some tests are still being skipped when running locally, but those tests fail when run via ssh. A special case is test VirtualSystemManagementService - 27_definesystem_macvtap_dev.py which fails only without the patch. Attached, you can find the latest version of the patch and reports of the test suite run both with and without the patch applied. -- Eduardo de Barros Lima Software Engineer, Open Virtualization Linux Technology Center - IBM/Brazil eblima@br.ibm.com
Functionally, I can't think of a single reason why the tests are always run remotely. I always run wbemcli and virsh commands locally, not through ssh, so I can't imagine why cimtest should be different. It is curious that the tests behave different with the patch. I would only expect the migration tests to behave different if, for example, the test checks the VM made it to the destination system. Rather than make this auto-detect, why not make it manually configurable in the near term so we can do more testing? On 05/30/2011 10:35 AM, Eduardo Lima (Etrunko) wrote:
On 05/27/2011 12:28 PM, Eduardo Lima (Etrunko) wrote:
Hello all,
I had been trying to set up cimtest to work on my machine for a couple of days without success. So yesterday I started to dig the code and found that every command is sent via ssh, even if the test is supposed to run locally.
The reason of cimtest not working was exactly that did not have the ssh keys set up properly. By fixing that configuration, I was able to make the test work as expected. :)
But I was still intrigued with the fact that every command is sent via ssh, thus taking cimtest a very very long time to complete. So I hacked a simple patch to try to workaround this issue:
[snip]
As you can see, it is a quick check if 'ip' is neither localhost nor 127.0.0.1, which only in that case the command is sent via ssh.
Note that this patch triggers some other issues, which I hope to be able to fix soon. But there is already a *huge increase* in the performance. I have used the most simpler and probably not the most correct measure:
So I have went somewhat deeper in my investigations and found most of the issues triggered by that patch. An updated version can be found attached to this message.
_BUT_, I can't really tell if it is actually intended that everything runs with ssh. Taking a look on the test results, it is noticeable that cimtest behaves in a different way, as follows.
Now the behavior of cimtest with the patch applied is much more similar to the version without the patch. 158 tests are labeled PASS, and only 3 that were FAIL without patch are now SKIP. Unfortunately I don't have access to the test machine at the moment, so I am not able to provide the detailed report, which I will do as soon as possible.
I would really appreciate comments from the experts on this issue. IMHO, it is something really worthy and if there are any tests that are required to run remotely, I could work on a proper fix to try to achieve the same results.
Still waiting for some light regarding this issue.
Best Regards,
_______________________________________________ Libvirt-cim mailing list Libvirt-cim@redhat.com https://www.redhat.com/mailman/listinfo/libvirt-cim
-- Chip Vincent Open Virtualization IBM Linux Technology Center cvincent@linux.vnet.ibm.com
On 05/31/2011 07:02 PM, Chip Vincent wrote:
Functionally, I can't think of a single reason why the tests are always run remotely. I always run wbemcli and virsh commands locally, not through ssh, so I can't imagine why cimtest should be different. It is curious that the tests behave different with the patch. I would only expect the migration tests to behave different if, for example, the test checks the VM made it to the destination system.
Rather than make this auto-detect, why not make it manually configurable in the near term so we can do more testing?
Well, actually it is already "configurable" if you specify the machine hostname instead of localhost, or 127.0.0.1 with -i option. -- Eduardo de Barros Lima Software Engineer, Open Virtualization Linux Technology Center - IBM/Brazil eblima@br.ibm.com
Yep, you're right. Seems like then the lack of a -i option would default to localhost using a local connection. I can't think of a good reason to have an option for forcing ssh when -i is localhost. On 06/01/2011 10:28 AM, Eduardo Lima (Etrunko) wrote:
On 05/31/2011 07:02 PM, Chip Vincent wrote:
Functionally, I can't think of a single reason why the tests are always run remotely. I always run wbemcli and virsh commands locally, not through ssh, so I can't imagine why cimtest should be different. It is curious that the tests behave different with the patch. I would only expect the migration tests to behave different if, for example, the test checks the VM made it to the destination system.
Rather than make this auto-detect, why not make it manually configurable in the near term so we can do more testing?
Well, actually it is already "configurable" if you specify the machine hostname instead of localhost, or 127.0.0.1 with -i option.
-- Chip Vincent Open Virtualization IBM Linux Technology Center cvincent@linux.vnet.ibm.com
participants (2)
-
Chip Vincent -
Eduardo Lima (Etrunko)