[libvirt] CPU type/flags when converting a physical machine to run on libvirt
I'm in the process of rewriting virt-p2v which is our program for converting physical machines to become virtual machines, running on top of libvirt + KVM. The physical machine has certain characteristics -- eg: - number of physical cores - amount of RAM - CPU type (eg. AMD Opteron, Intel Core i7) - CPU flags (eg. ACPI, SSE4) which it might make sense to reflect in the libvirt XML of the virtual machine we create. The old version of virt-p2v is pretty simplistic about this. It generates *only* an i386 or x86-64 VM, and the only flags it considers are 'apic', 'acpi', 'pae' and 'lm' [the latter to distinguish between 32 and 64 bit x86]. What should the new version do? Particular questions: - Should we try to reflect the CPU type of the physical machine in the virtual machine? eg. If it's an Opteron, we generate an Opteron target machine. (I believe the answer is *no*, because this is not live migration, and most guests can boot on any compatible CPU). - How can I ask libvirt to give me the best possible CPU, and not some baseline? Normally I use host-model, but I think that prevents migration. - What CPU flags should be reflected in the target libvirt XML? - Is it worth modelling CPU thread layout? (I suspect this will be a lot of work with the potential to break things rather than provide any measurable benefits.) - Is there anything else I haven't thought about? However the overriding rule is: - We *must not* end up with a target virtual machine which doesn't work! Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-p2v converts physical machines to virtual machines. Boot with a live CD or over the network (PXE) and turn machines into KVM guests. http://libguestfs.org/virt-v2v
CCing Cole, as he had to deal with libvirt and "host-cpu" before, and may have some comments about the bugs he has found. On Fri, May 02, 2014 at 01:07:05PM +0100, Richard W.M. Jones wrote:
I'm in the process of rewriting virt-p2v which is our program for converting physical machines to become virtual machines, running on top of libvirt + KVM.
The physical machine has certain characteristics -- eg:
- number of physical cores - amount of RAM - CPU type (eg. AMD Opteron, Intel Core i7) - CPU flags (eg. ACPI, SSE4)
which it might make sense to reflect in the libvirt XML of the virtual machine we create.
The old version of virt-p2v is pretty simplistic about this. It generates *only* an i386 or x86-64 VM, and the only flags it considers are 'apic', 'acpi', 'pae' and 'lm' [the latter to distinguish between 32 and 64 bit x86].
What should the new version do?
Particular questions:
- Should we try to reflect the CPU type of the physical machine in the virtual machine? eg. If it's an Opteron, we generate an Opteron target machine. (I believe the answer is *no*, because this is not live migration, and most guests can boot on any compatible CPU).
I see no reason to _not_ choose Opteron_Gx, if you know the host CPU is always going to be an Opteron_Gx. If the user simply plans to convert an existing physical machine to a single-VM machine and has no plans to ever migrate the VM, it makes sense to use "-cpu host" (but beware: this may uncover a few QEMU bugs). if the user plans to migrate the VM to a _similar_ host later, it makes sense to use an existing CPU model name that matches the host CPU (see "host-cpu-model" below). If the user plans to migrate the VM to a very different host later it makes sense to be more conservative and simply use the default CPU model. In other words: I don't know what's a good default because I don't know your use case very well.
- How can I ask libvirt to give me the best possible CPU, and not some baseline? Normally I use host-model, but I think that prevents migration.
The best possible CPU is "-cpu host" (host-passthrough in libvirt), but that doesn't allow migration. This may uncover QEMU bugs (but it is much better today than it was 1 or 2 years ago). The best possible CPU which allows migration is the one you get when explicitly asking libvirt to expand host-model (including baseline + flags). This is likely to uncover QEMU and libvirt bugs. A safer option is to use only the base CPU model (not the additional flags) provided by libvirt when asking about the host CPU model (I believe this is called "host-cpu-model" on virt-manager code).
- What CPU flags should be reflected in the target libvirt XML?
- Is it worth modelling CPU thread layout? (I suspect this will be a lot of work with the potential to break things rather than provide any measurable benefits.)
I wouldn't recommend this unless: 1) you know the VM will be kept running in the same host or on a similar host; 2) you pin the VCPUs/threads to corresponding host CPUs/threads. This may also uncover QEMU bugs, so I wouldn't do this by default unless the user explicitly asks for it.
- Is there anything else I haven't thought about?
In the future you may want to support multi-node NUMA VMs. This is similar to the multi-core/multi-thread case: it makes sense if you know the VM is going to run on a host with similar topology, and if you manually pin the guest nodes to the host nodes (something which is not possible yet, but should be possible in the near future).
However the overriding rule is:
- We *must not* end up with a target virtual machine which doesn't work!
Rich.
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-p2v converts physical machines to virtual machines. Boot with a live CD or over the network (PXE) and turn machines into KVM guests. http://libguestfs.org/virt-v2v
-- Eduardo
On Fri, May 02, 2014 at 03:53:31PM -0300, Eduardo Habkost wrote:
On Fri, May 02, 2014 at 01:07:05PM +0100, Richard W.M. Jones wrote:
- Should we try to reflect the CPU type of the physical machine in the virtual machine? eg. If it's an Opteron, we generate an Opteron target machine. (I believe the answer is *no*, because this is not live migration, and most guests can boot on any compatible CPU).
I see no reason to _not_ choose Opteron_Gx, if you know the host CPU is always going to be an Opteron_Gx.
Thanks Eduardo. I think I should clarify the use cases based on what you said here and below. It's almost never (probably *never*) the case that the converted guest would run on the same host as it originated from. The old and current virt-p2v programs would not let you do that (except with a lot of manual intervention). And no RHEL customer who uses virt-p2v is interested in that scenario anyway. They always want to migrate a physical machine to (eg) a pre-existing RHEV cluster and then recycle the original physical machine for something else. So I guess we never know the target CPU. What we do know in great detail is the current CPU that virt-p2v runs on (ie. the source CPU).
If the user simply plans to convert an existing physical machine to a single-VM machine and has no plans to ever migrate the VM, it makes sense to use "-cpu host" (but beware: this may uncover a few QEMU bugs).
if the user plans to migrate the VM to a _similar_ host later, it makes sense to use an existing CPU model name that matches the host CPU (see "host-cpu-model" below).
If the user plans to migrate the VM to a very different host later it makes sense to be more conservative and simply use the default CPU model.
In other words: I don't know what's a good default because I don't know your use case very well.
- How can I ask libvirt to give me the best possible CPU, and not some baseline? Normally I use host-model, but I think that prevents migration.
The best possible CPU is "-cpu host" (host-passthrough in libvirt), but that doesn't allow migration. This may uncover QEMU bugs (but it is much better today than it was 1 or 2 years ago).
The best possible CPU which allows migration is the one you get when explicitly asking libvirt to expand host-model (including baseline + flags). This is likely to uncover QEMU and libvirt bugs.
A safer option is to use only the base CPU model (not the additional flags) provided by libvirt when asking about the host CPU model (I believe this is called "host-cpu-model" on virt-manager code).
I think the issue I have is: If I get a baseline CPU, will it have features like SSE4? Really I want to be non-specific about the target CPU, in the libvirt XML. I don't want to exclude the target from having the best possible CPU features, but also I would like migration to work.
- What CPU flags should be reflected in the target libvirt XML?
- Is it worth modelling CPU thread layout? (I suspect this will be a lot of work with the potential to break things rather than provide any measurable benefits.)
I wouldn't recommend this unless: 1) you know the VM will be kept running in the same host or on a similar host; 2) you pin the VCPUs/threads to corresponding host CPUs/threads.
This may also uncover QEMU bugs, so I wouldn't do this by default unless the user explicitly asks for it.
OK.
- Is there anything else I haven't thought about?
In the future you may want to support multi-node NUMA VMs. This is similar to the multi-core/multi-thread case: it makes sense if you know the VM is going to run on a host with similar topology, and if you manually pin the guest nodes to the host nodes (something which is not possible yet, but should be possible in the near future).
OK. I guess we can record the original NUMA topology. Thanks, Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-builder quickly builds VMs from scratch http://libguestfs.org/virt-builder.1.html
On Fri, May 02, 2014 at 08:10:47PM +0100, Richard W.M. Jones wrote:
On Fri, May 02, 2014 at 03:53:31PM -0300, Eduardo Habkost wrote:
On Fri, May 02, 2014 at 01:07:05PM +0100, Richard W.M. Jones wrote:
- Should we try to reflect the CPU type of the physical machine in the virtual machine? eg. If it's an Opteron, we generate an Opteron target machine. (I believe the answer is *no*, because this is not live migration, and most guests can boot on any compatible CPU).
I see no reason to _not_ choose Opteron_Gx, if you know the host CPU is always going to be an Opteron_Gx.
Thanks Eduardo. I think I should clarify the use cases based on what you said here and below.
It's almost never (probably *never*) the case that the converted guest would run on the same host as it originated from. The old and current virt-p2v programs would not let you do that (except with a lot of manual intervention). And no RHEL customer who uses virt-p2v is interested in that scenario anyway. They always want to migrate a physical machine to (eg) a pre-existing RHEV cluster and then recycle the original physical machine for something else.
So I guess we never know the target CPU. What we do know in great detail is the current CPU that virt-p2v runs on (ie. the source CPU).
That's interesting. So you have many reasons to be conservative by default, unless you (or the user) have additional information about the target machine/cluster where the VM is going to run. Is the p2v tool going to accept options similar to the virt-install options? Except for additional conversion steps, the process look very similar to the creation of a new VM: only the user (or management software using the tool/library) has enough information to decide how the new VM should really look like.
If the user simply plans to convert an existing physical machine to a single-VM machine and has no plans to ever migrate the VM, it makes sense to use "-cpu host" (but beware: this may uncover a few QEMU bugs).
if the user plans to migrate the VM to a _similar_ host later, it makes sense to use an existing CPU model name that matches the host CPU (see "host-cpu-model" below).
If the user plans to migrate the VM to a very different host later it makes sense to be more conservative and simply use the default CPU model.
In other words: I don't know what's a good default because I don't know your use case very well.
- How can I ask libvirt to give me the best possible CPU, and not some baseline? Normally I use host-model, but I think that prevents migration.
The best possible CPU is "-cpu host" (host-passthrough in libvirt), but that doesn't allow migration. This may uncover QEMU bugs (but it is much better today than it was 1 or 2 years ago).
The best possible CPU which allows migration is the one you get when explicitly asking libvirt to expand host-model (including baseline + flags). This is likely to uncover QEMU and libvirt bugs.
A safer option is to use only the base CPU model (not the additional flags) provided by libvirt when asking about the host CPU model (I believe this is called "host-cpu-model" on virt-manager code).
I think the issue I have is: If I get a baseline CPU, will it have features like SSE4? Really I want to be non-specific about the target CPU, in the libvirt XML. I don't want to exclude the target from having the best possible CPU features, but also I would like migration to work.
It should, and if you are not getting it, there are even more reasons to worry about the safety of using "baseline+features" instead of just "baseline" because it means something unexpected happened. It is not a bug, strictly speaking, but something we want to avoid. If you are getting "-cpu NotSoNiceCPU,+NiceFeature1,+NiceFeature2" as result from libvirt instead of "-cpu NiceCPU" (which would already contain NiceFeature1, NiceFeature2), please bug us (libvirt and QEMU developers) so we can address it.
- What CPU flags should be reflected in the target libvirt XML?
- Is it worth modelling CPU thread layout? (I suspect this will be a lot of work with the potential to break things rather than provide any measurable benefits.)
I wouldn't recommend this unless: 1) you know the VM will be kept running in the same host or on a similar host; 2) you pin the VCPUs/threads to corresponding host CPUs/threads.
This may also uncover QEMU bugs, so I wouldn't do this by default unless the user explicitly asks for it.
OK.
- Is there anything else I haven't thought about?
In the future you may want to support multi-node NUMA VMs. This is similar to the multi-core/multi-thread case: it makes sense if you know the VM is going to run on a host with similar topology, and if you manually pin the guest nodes to the host nodes (something which is not possible yet, but should be possible in the near future).
OK. I guess we can record the original NUMA topology.
On both cases above (cores/sockets and NUMA), it doesn't make much sense to try to copy the original topology unless you want to fine-tune VCPU and memory pinning to maximize performance. So considering you don't know much about the new host machine, I don't see a reason to try to copy the original machine topology by default. -- Eduardo
On Fri, May 02, 2014 at 05:17:42PM -0300, Eduardo Habkost wrote:
That's interesting. So you have many reasons to be conservative by default, unless you (or the user) have additional information about the target machine/cluster where the VM is going to run.
Is the p2v tool going to accept options similar to the virt-install options? Except for additional conversion steps, the process look very similar to the creation of a new VM: only the user (or management software using the tool/library) has enough information to decide how the new VM should really look like.
At the moment, virt-p2v has some input from the user. The user may override the amount of RAM or number of vCPUs on the target (by default, they assume the same values as the physical machine). They could also change these values after conversion. What complicates this is that we have had in the past some users who use the tool to convert hundreds or thousands of physical machines. Of course they want to automate this as far as possible, and want to avoid custom tweaks of each VM. Thanks for the other advice. It sounds like we should not be too clever here, and just go with libvirt defaults. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org
participants (2)
-
Eduardo Habkost -
Richard W.M. Jones