Re: [libvirt] [PATCH RESEND v1 1/2] i386: Add Intel Processor Trace feature support
CCing libvirt developers. On Mon, Jan 15, 2018 at 10:33:35AM +0100, Paolo Bonzini wrote:
On 15/01/2018 08:19, Kang, Luwei wrote:
If you are forwarding host info directly to the guest, the feature is not migration-safe. The new feature needs to be added to feature_word_info[FEAT_7_0_EBX].unmigratable_flags.
Hi, Thanks for you review. I want to support Intel PT live migration and patch [2/2] to do this. I don't understand why need to add this feature in feature_word_info[FEAT_7_0_EBX].unmigratable_flags first to disable live migration.
Hi Luwei,
the issue is that different hosts can have different CPUID flags. You don't have a way to set the CPUID flags from the "-cpu" command line, and it's not clear what values will be there in the various Ice Lake SKUs.
I am not sure if there is a mechanism to allow live migration only for "-cpu host", but it cannot be supported for any other "-cpu" model.
IIRC, we don't have any mechanism to actually prevent migration if an unmigratable flag is enabled. We just avoid enabling them by accident on "-cpu host". This case is slightly more problematic, however: the new feature is actually migratable (under very controlled circumstances) because of patch 2/2, but it is not migration-safe[1]. This means libvirt shouldn't include it in "host-model" expansion (which uses the query-cpu-model-expansion QMP command) until we make the feature migration-safe. For QEMU, this means the feature shouldn't be returned by "query-cpu-model-expansion type=static model=max" (but it can be returned by "query-cpu-model-expansion type=full model=max"). Jiri, it looks like libvirt uses type=full on query-cpu-model-expansion on x86. It needs to use type=static[2], or it will have no way to find out if a feature is migration-safe or not. This case is similar to "pmu", which is not migration-safe but enabled by -cpu host by default. But "pmu" is less problematic because it is already skipped by query-cpu-model-expansion type=static. --- [1] "migration-safe" is defined in the documentation for CpuDefinitionInfo on the QAPI schema: # @migration-safe: whether a CPU definition can be safely used for # migration in combination with a QEMU compatibility machine # when migrating between different QMU versions and between # hosts with different sets of (hardware or software) # capabilities. If not provided, information is not available # and callers should not assume the CPU definition to be # migration-safe. (since 2.8) [2] It looks like libvirt uses type=full because it wants to get all QOM property aliases returned. In this case, one solution for libvirt is to use: static_expansion = query_cpu_model_expansion(type=static, model) all_props = query_cpu_model_expansion(type=full, static_expansion) -- Eduardo
On Mon, Jan 15, 2018 at 12:04:55 -0200, Eduardo Habkost wrote:
CCing libvirt developers. ... This case is slightly more problematic, however: the new feature is actually migratable (under very controlled circumstances) because of patch 2/2, but it is not migration-safe[1]. This means libvirt shouldn't include it in "host-model" expansion (which uses the query-cpu-model-expansion QMP command) until we make the feature migration-safe.
For QEMU, this means the feature shouldn't be returned by "query-cpu-model-expansion type=static model=max" (but it can be returned by "query-cpu-model-expansion type=full model=max").
Jiri, it looks like libvirt uses type=full on query-cpu-model-expansion on x86. It needs to use type=static[2], or it will have no way to find out if a feature is migration-safe or not. ... [2] It looks like libvirt uses type=full because it wants to get all QOM property aliases returned. In this case, one solution for libvirt is to use:
static_expansion = query_cpu_model_expansion(type=static, model) all_props = query_cpu_model_expansion(type=full, static_expansion)
This is exactly what libvirt is doing (with model = "host") ever since query-cpu-model-expansion support was implemented for x86. Jirka
On Mon, Jan 15, 2018 at 03:25:18PM +0100, Jiri Denemark wrote:
On Mon, Jan 15, 2018 at 12:04:55 -0200, Eduardo Habkost wrote:
CCing libvirt developers. ... This case is slightly more problematic, however: the new feature is actually migratable (under very controlled circumstances) because of patch 2/2, but it is not migration-safe[1]. This means libvirt shouldn't include it in "host-model" expansion (which uses the query-cpu-model-expansion QMP command) until we make the feature migration-safe.
For QEMU, this means the feature shouldn't be returned by "query-cpu-model-expansion type=static model=max" (but it can be returned by "query-cpu-model-expansion type=full model=max").
Jiri, it looks like libvirt uses type=full on query-cpu-model-expansion on x86. It needs to use type=static[2], or it will have no way to find out if a feature is migration-safe or not. ... [2] It looks like libvirt uses type=full because it wants to get all QOM property aliases returned. In this case, one solution for libvirt is to use:
static_expansion = query_cpu_model_expansion(type=static, model) all_props = query_cpu_model_expansion(type=full, static_expansion)
This is exactly what libvirt is doing (with model = "host") ever since query-cpu-model-expansion support was implemented for x86.
Oh, now I see that the x86 code uses QEMU_MONITOR_CPU_MODEL_EXPANSION_STATIC_FULL and not just QEMU_MONITOR_CPU_MODEL_EXPANSION_FULL. Nice! -- Eduardo
On Mon, Jan 15, 2018 at 12:04:55 -0200, Eduardo Habkost wrote:
CCing libvirt developers. ... This case is slightly more problematic, however: the new feature is actually migratable (under very controlled circumstances) because of patch 2/2, but it is not migration-safe[1]. This means libvirt shouldn't include it in "host-model" expansion (which uses the query-cpu-model-expansion QMP command) until we make the feature migration-safe.
For QEMU, this means the feature shouldn't be returned by "query-cpu-model-expansion type=static model=max" (but it can be returned by "query-cpu-model-expansion type=full model=max").
Jiri, it looks like libvirt uses type=full on query-cpu-model-expansion on x86. It needs to use type=static[2], or it will have no way to find out if a feature is migration-safe or not. ... [2] It looks like libvirt uses type=full because it wants to get all QOM property aliases returned. In this case, one solution for libvirt is to use:
static_expansion = query_cpu_model_expansion(type=static, model) all_props = query_cpu_model_expansion(type=full, static_expansion)
This is exactly what libvirt is doing (with model = "host") ever since query-cpu-model-expansion support was implemented for x86.
Oh, now I see that the x86 code uses QEMU_MONITOR_CPU_MODEL_EXPANSION_STATIC_FULL and not just QEMU_MONITOR_CPU_MODEL_EXPANSION_FULL. Nice!
So, I need to add Intel PT feature in "X86CPUDefinition builtin_x86_defs[]" so that we can get this CPUID in specific CPU model not only "-cpu host". Is that right? Intel PT is first supported in Intel Core M and 5th generation Intel Core processors that are based on the Intel micro-architecture code name Broadwell but Intel PT use EPT is first supported in Ice Lake. Intel PT virtualization depend on PT use EPT. I will add Intel PT to "Broadwell" CPU model and later to make sure a "Broadwell" guest can use Intel PT if the host is Ice Lake. Thanks, Luwei Kang
On Tue, Jan 16, 2018 at 06:10:17AM +0000, Kang, Luwei wrote:
On Mon, Jan 15, 2018 at 12:04:55 -0200, Eduardo Habkost wrote:
CCing libvirt developers. ... This case is slightly more problematic, however: the new feature is actually migratable (under very controlled circumstances) because of patch 2/2, but it is not migration-safe[1]. This means libvirt shouldn't include it in "host-model" expansion (which uses the query-cpu-model-expansion QMP command) until we make the feature migration-safe.
For QEMU, this means the feature shouldn't be returned by "query-cpu-model-expansion type=static model=max" (but it can be returned by "query-cpu-model-expansion type=full model=max").
Jiri, it looks like libvirt uses type=full on query-cpu-model-expansion on x86. It needs to use type=static[2], or it will have no way to find out if a feature is migration-safe or not. ... [2] It looks like libvirt uses type=full because it wants to get all QOM property aliases returned. In this case, one solution for libvirt is to use:
static_expansion = query_cpu_model_expansion(type=static, model) all_props = query_cpu_model_expansion(type=full, static_expansion)
This is exactly what libvirt is doing (with model = "host") ever since query-cpu-model-expansion support was implemented for x86.
Oh, now I see that the x86 code uses QEMU_MONITOR_CPU_MODEL_EXPANSION_STATIC_FULL and not just QEMU_MONITOR_CPU_MODEL_EXPANSION_FULL. Nice!
So, I need to add Intel PT feature in "X86CPUDefinition builtin_x86_defs[]" so that we can get this CPUID in specific CPU model not only "-cpu host". Is that right?
The problem is that you won't be able to add intel-pt to any CPU model unless the feature is made migration-safe (by not calling kvm_arch_get_supported_cpuid() from cpu_x86_cpuid()). What's missing here is to either: (a) make cpu_x86_cpuid() return host-independent data (it can be constant, or it can be configurable on the command-line); or (b) add a mechanism to skip intel-pt from "query-cpu-model-expansion type=static". Probably (a) is easier to implement, and it also makes the feature more useful (by making it migration-safe).
Intel PT is first supported in Intel Core M and 5th generation Intel Core processors that are based on the Intel micro-architecture code name Broadwell but Intel PT use EPT is first supported in Ice Lake. Intel PT virtualization depend on PT use EPT. I will add Intel PT to "Broadwell" CPU model and later to make sure a "Broadwell" guest can use Intel PT if the host is Ice Lake.
The "if the host is Ice Lake" part is problematic. On migration-safe CPU models (all of them except "max" and "host"), the data seen on CPUID can't depend on the host at all. It should depend only on the machine-type + command-line options. -- Eduardo
On Mon, Jan 15, 2018 at 12:04:55 -0200, Eduardo Habkost wrote:
CCing libvirt developers. ... This case is slightly more problematic, however: the new feature is actually migratable (under very controlled circumstances) because of patch 2/2, but it is not migration-safe[1]. This means libvirt shouldn't include it in "host-model" expansion (which uses the query-cpu-model-expansion QMP command) until we make the feature migration-safe.
For QEMU, this means the feature shouldn't be returned by "query-cpu-model-expansion type=static model=max" (but it can be returned by "query-cpu-model-expansion type=full model=max").
Jiri, it looks like libvirt uses type=full on query-cpu-model-expansion on x86. It needs to use type=static[2], or it will have no way to find out if a feature is migration-safe or not. ... [2] It looks like libvirt uses type=full because it wants to get all QOM property aliases returned. In this case, one solution for libvirt is to use:
static_expansion = query_cpu_model_expansion(type=static, model) all_props = query_cpu_model_expansion(type=full, static_expansion)
This is exactly what libvirt is doing (with model = "host") ever since query-cpu-model-expansion support was implemented for x86.
Oh, now I see that the x86 code uses QEMU_MONITOR_CPU_MODEL_EXPANSION_STATIC_FULL and not just QEMU_MONITOR_CPU_MODEL_EXPANSION_FULL. Nice!
So, I need to add Intel PT feature in "X86CPUDefinition builtin_x86_defs[]" so that we can get this CPUID in specific CPU model not only "-cpu host". Is that right?
The problem is that you won't be able to add intel-pt to any CPU model unless the feature is made migration-safe (by not calling kvm_arch_get_supported_cpuid() from cpu_x86_cpuid()).
Hi Eduardo, Have some question need you help clear. What is "migration-safe" feature mean? I find all the feature which included in CPU model (builtin_x86_defs[]) will make "xcc->migration_safe = true;" in x86_cpu_cpudef_class_init(). If create a Skylake guest on Skylake HW and live migrate this guest to another machine with old HW(e.g Haswell). Can we think the new feature or cpu model(Skylake guest) which only supported in Skylake HW is safe?
What's missing here is to either: (a) make cpu_x86_cpuid() return host-independent data (it can be constant, or it can be configurable on the command-line); or (b) add a mechanism to skip intel-pt from "query-cpu-model-expansion type=static".
==> it can be constant: I think it shouldn't be constant because Intel PT virtualization can only supported in Ice Lake hardware now. Intel PT cpuid info would be mask off on old platform. ==> or it can be configurable on the command-line: Because of I didn't add this feature in any CPU model. We only can enable this feature by "-cpu Skylake-Server,+intel-pt" at present. What about add a new cpu model name "Icelake" and add PT in this. Is that would be migration safe? Thanks, Luwei Kang
Probably (a) is easier to implement, and it also makes the feature more useful (by making it migration-safe).
Intel PT is first supported in Intel Core M and 5th generation Intel Core processors that are based on the Intel micro-architecture code name Broadwell but Intel PT use EPT is first supported in Ice Lake. Intel PT virtualization depend on PT use EPT. I will add Intel PT to "Broadwell" CPU model and later to make sure a "Broadwell" guest can use Intel PT if the host is Ice Lake.
The "if the host is Ice Lake" part is problematic. On migration-safe CPU models (all of them except "max" and "host"), the data seen on CPUID can't depend on the host at all. It should depend only on the machine-type + command-line options.
-- Eduardo
On Wed, Jan 17, 2018 at 10:32:56AM +0000, Kang, Luwei wrote:
On Mon, Jan 15, 2018 at 12:04:55 -0200, Eduardo Habkost wrote:
CCing libvirt developers. ... This case is slightly more problematic, however: the new feature is actually migratable (under very controlled circumstances) because of patch 2/2, but it is not migration-safe[1]. This means libvirt shouldn't include it in "host-model" expansion (which uses the query-cpu-model-expansion QMP command) until we make the feature migration-safe.
For QEMU, this means the feature shouldn't be returned by "query-cpu-model-expansion type=static model=max" (but it can be returned by "query-cpu-model-expansion type=full model=max").
Jiri, it looks like libvirt uses type=full on query-cpu-model-expansion on x86. It needs to use type=static[2], or it will have no way to find out if a feature is migration-safe or not. ... [2] It looks like libvirt uses type=full because it wants to get all QOM property aliases returned. In this case, one solution for libvirt is to use:
static_expansion = query_cpu_model_expansion(type=static, model) all_props = query_cpu_model_expansion(type=full, static_expansion)
This is exactly what libvirt is doing (with model = "host") ever since query-cpu-model-expansion support was implemented for x86.
Oh, now I see that the x86 code uses QEMU_MONITOR_CPU_MODEL_EXPANSION_STATIC_FULL and not just QEMU_MONITOR_CPU_MODEL_EXPANSION_FULL. Nice!
So, I need to add Intel PT feature in "X86CPUDefinition builtin_x86_defs[]" so that we can get this CPUID in specific CPU model not only "-cpu host". Is that right?
The problem is that you won't be able to add intel-pt to any CPU model unless the feature is made migration-safe (by not calling kvm_arch_get_supported_cpuid() from cpu_x86_cpuid()).
Hi Eduardo, Have some question need you help clear. What is "migration-safe" feature mean? I find all the feature which included in CPU model (builtin_x86_defs[]) will make "xcc->migration_safe = true;" in x86_cpu_cpudef_class_init(). If create a Skylake guest on Skylake HW and live migrate this guest to another machine with old HW(e.g Haswell). Can we think the new feature or cpu model(Skylake guest) which only supported in Skylake HW is safe?
I mean matching the requirements so we can say the feature is migration-safe, that means exposing the same CPUID data to the guest on any host, if the machine-type + command-line is the same. The data on CPUID[7] is OK (it changes according to the command-line options only), but the data exposed on CPUID[14h] on this patch is not migration-safe (it changes depending on the host it is running).
What's missing here is to either: (a) make cpu_x86_cpuid() return host-independent data (it can be constant, or it can be configurable on the command-line); or (b) add a mechanism to skip intel-pt from "query-cpu-model-expansion type=static".
==> it can be constant: I think it shouldn't be constant because Intel PT virtualization can only supported in Ice Lake hardware now. Intel PT cpuid info would be mask off on old platform. ==> or it can be configurable on the command-line: Because of I didn't add this feature in any CPU model. We only can enable this feature by "-cpu Skylake-Server,+intel-pt" at present.
Note that I'm talking about CPUID[14h], not CPUID[7]. The CPUID[7] bits are safe because they are set according to the CPU model + command-line options only. The bits on CPUID[14h] change depending on the host and are not migration-safe. CPUID[7].EBX[bit 25] is just the (already configurable) bit that enables the migration-unsafe code for CPUID[14h].
What about add a new cpu model name "Icelake" and add PT in this. Is that would be migration safe?
It won't, because of the CPUID[14h] code that makes it unsafe to migrate between hosts with different Intel-PT capabilities (i.e. different data on CPUID[14h]).
Thanks, Luwei Kang
Probably (a) is easier to implement, and it also makes the feature more useful (by making it migration-safe).
Intel PT is first supported in Intel Core M and 5th generation Intel Core processors that are based on the Intel micro-architecture code name Broadwell but Intel PT use EPT is first supported in Ice Lake. Intel PT virtualization depend on PT use EPT. I will add Intel PT to "Broadwell" CPU model and later to make sure a "Broadwell" guest can use Intel PT if the host is Ice Lake.
The "if the host is Ice Lake" part is problematic. On migration-safe CPU models (all of them except "max" and "host"), the data seen on CPUID can't depend on the host at all. It should depend only on the machine-type + command-line options.
-- Eduardo
-- Eduardo
On Mon, Jan 15, 2018 at 12:04:55 -0200, Eduardo Habkost wrote: > CCing libvirt developers. ... > This case is slightly more problematic, however: the new > feature is actually migratable (under very controlled > circumstances) because of patch 2/2, but it is not > migration-safe[1]. This means libvirt shouldn't include it > in "host-model" expansion (which uses the > query-cpu-model-expansion QMP command) until we make the feature migration-safe. > > For QEMU, this means the feature shouldn't be returned by > "query-cpu-model-expansion type=static model=max" (but it > can be returned by "query-cpu-model-expansion type=full model=max"). > > Jiri, it looks like libvirt uses type=full on > query-cpu-model-expansion on x86. It needs to use > type=static[2], or it will have no way to find out if a > feature is migration-safe or not. ... > [2] It looks like libvirt uses type=full because it wants to get > all QOM property aliases returned. In this case, one > solution for libvirt is to use: > > static_expansion = query_cpu_model_expansion(type=static, model) > all_props = query_cpu_model_expansion(type=full, > static_expansion)
This is exactly what libvirt is doing (with model = "host") ever since query-cpu-model-expansion support was implemented for x86.
Oh, now I see that the x86 code uses QEMU_MONITOR_CPU_MODEL_EXPANSION_STATIC_FULL and not just QEMU_MONITOR_CPU_MODEL_EXPANSION_FULL. Nice!
So, I need to add Intel PT feature in "X86CPUDefinition builtin_x86_defs[]" so that we can get this CPUID in specific CPU model not only "-cpu host". Is that right?
The problem is that you won't be able to add intel-pt to any CPU model unless the feature is made migration-safe (by not calling kvm_arch_get_supported_cpuid() from cpu_x86_cpuid()).
Hi Eduardo, Have some question need you help clear. What is "migration-safe" feature mean? I find all the feature which included in CPU model (builtin_x86_defs[]) will make "xcc->migration_safe = true;" in x86_cpu_cpudef_class_init(). If create a Skylake guest on Skylake HW and live migrate this guest to another machine with old HW(e.g Haswell). Can we think the new feature or cpu model(Skylake guest) which only supported in Skylake HW is safe?
I mean matching the requirements so we can say the feature is migration-safe, that means exposing the same CPUID data to the guest on any host, if the machine-type + command-line is the same. The data on CPUID[7] is OK (it changes according to the command-line options only), but the data exposed on CPUID[14h] on this patch is not migration-safe (it changes depending on the host it is running).
Many thanks for your clarification. I think I have understood. But CPUID[14h] is depend on CPUID[7].EBX[bit25] and it would be zero if CPUID[7].EBX[bit25] is not set. Or what you concerned is make live migration on two different HW which all support Intel PT virtualization but have different CPUID[14h] result? This may need to think about. Thanks, Luwei Kang
What's missing here is to either: (a) make cpu_x86_cpuid() return host-independent data (it can be constant, or it can be configurable on the command-line); or (b) add a mechanism to skip intel-
pt from "query-cpu-model-expansion type=static".
==> it can be constant: I think it shouldn't be constant because Intel PT virtualization can only supported in Ice Lake hardware now. Intel PT cpuid info
==> or it can be configurable on the command-line: Because of I didn't add this feature in any CPU model. We only can enable this feature by "-cpu Skylake-Server,+intel-pt" at
would be mask off on old platform. present.
Note that I'm talking about CPUID[14h], not CPUID[7]. The CPUID[7] bits are safe because they are set according to the CPU model + command-line options only. The bits on CPUID[14h] change depending on the host and are not migration-safe. CPUID[7].EBX[bit 25] is just the (already configurable) bit that enables the migration-unsafe code for CPUID[14h].
What about add a new cpu model name "Icelake" and add PT in this. Is that would be migration safe?
It won't, because of the CPUID[14h] code that makes it unsafe to migrate between hosts with different Intel-PT capabilities (i.e. different data on CPUID[14h]).
Thanks, Luwei Kang
Probably (a) is easier to implement, and it also makes the feature more useful (by making it migration-safe).
Intel PT is first supported in Intel Core M and 5th generation Intel Core processors that are based on the Intel micro-architecture code name Broadwell but Intel PT use EPT is first supported in Ice Lake. Intel PT virtualization depend on PT use EPT. I will add Intel PT to "Broadwell" CPU model and later to make sure a "Broadwell" guest can use Intel PT if the host is Ice Lake.
The "if the host is Ice Lake" part is problematic. On migration-safe CPU models (all of them except "max" and "host"), the data seen on CPUID can't depend on the host at all. It
should depend only on the machine-type + command-line options.
-- Eduardo
-- Eduardo
On Thu, Jan 18, 2018 at 05:33:53AM +0000, Kang, Luwei wrote:
> On Mon, Jan 15, 2018 at 12:04:55 -0200, Eduardo Habkost wrote: > > CCing libvirt developers. > ... > > This case is slightly more problematic, however: the new > > feature is actually migratable (under very controlled > > circumstances) because of patch 2/2, but it is not > > migration-safe[1]. This means libvirt shouldn't include it > > in "host-model" expansion (which uses the > > query-cpu-model-expansion QMP command) until we make the feature migration-safe. > > > > For QEMU, this means the feature shouldn't be returned by > > "query-cpu-model-expansion type=static model=max" (but it > > can be returned by "query-cpu-model-expansion type=full model=max"). > > > > Jiri, it looks like libvirt uses type=full on > > query-cpu-model-expansion on x86. It needs to use > > type=static[2], or it will have no way to find out if a > > feature is migration-safe or not. > ... > > [2] It looks like libvirt uses type=full because it wants to get > > all QOM property aliases returned. In this case, one > > solution for libvirt is to use: > > > > static_expansion = query_cpu_model_expansion(type=static, model) > > all_props = query_cpu_model_expansion(type=full, > > static_expansion) > > This is exactly what libvirt is doing (with model = "host") > ever since query-cpu-model-expansion support was implemented for x86.
Oh, now I see that the x86 code uses QEMU_MONITOR_CPU_MODEL_EXPANSION_STATIC_FULL and not just QEMU_MONITOR_CPU_MODEL_EXPANSION_FULL. Nice!
So, I need to add Intel PT feature in "X86CPUDefinition builtin_x86_defs[]" so that we can get this CPUID in specific CPU model not only "-cpu host". Is that right?
The problem is that you won't be able to add intel-pt to any CPU model unless the feature is made migration-safe (by not calling kvm_arch_get_supported_cpuid() from cpu_x86_cpuid()).
Hi Eduardo, Have some question need you help clear. What is "migration-safe" feature mean? I find all the feature which included in CPU model (builtin_x86_defs[]) will make "xcc->migration_safe = true;" in x86_cpu_cpudef_class_init(). If create a Skylake guest on Skylake HW and live migrate this guest to another machine with old HW(e.g Haswell). Can we think the new feature or cpu model(Skylake guest) which only supported in Skylake HW is safe?
I mean matching the requirements so we can say the feature is migration-safe, that means exposing the same CPUID data to the guest on any host, if the machine-type + command-line is the same. The data on CPUID[7] is OK (it changes according to the command-line options only), but the data exposed on CPUID[14h] on this patch is not migration-safe (it changes depending on the host it is running).
Many thanks for your clarification. I think I have understood. But CPUID[14h] is depend on CPUID[7].EBX[bit25] and it would be zero if CPUID[7].EBX[bit25] is not set. Or what you concerned is make live migration on two different HW which all support Intel PT virtualization but have different CPUID[14h] result? This may need to think about.
Yes. If intel-pt is off, the results (CPUID[14h] all zeroes) are migration-safe. Setting intel-pt=on makes the command-line not migration-safe. This is OK, in principle (e.g. the "pmu" option is not migration-safe and behaves the same way). The only problem is that this patch would make query-cpu-model-expansion return intel-pt=on on type=static expansion. Probably the easiest way to fix that is to specifically exclude intel-pt on x86_cpu_static_props(). However, if there's a simple way to make it possible to migrate between hosts with different CPUID[14h] data, it would be even better. With the current KVM intel-pt implementation, what happens if the CPUID[14h] data seen by the guest doesn't match exactly the CPUID[14h] leaves from the host?
Thanks, Luwei Kang
What's missing here is to either: (a) make cpu_x86_cpuid() return host-independent data (it can be constant, or it can be configurable on the command-line); or (b) add a mechanism to skip intel-
pt from "query-cpu-model-expansion type=static".
==> it can be constant: I think it shouldn't be constant because Intel PT virtualization can only supported in Ice Lake hardware now. Intel PT cpuid info
==> or it can be configurable on the command-line: Because of I didn't add this feature in any CPU model. We only can enable this feature by "-cpu Skylake-Server,+intel-pt" at
would be mask off on old platform. present.
Note that I'm talking about CPUID[14h], not CPUID[7]. The CPUID[7] bits are safe because they are set according to the CPU model + command-line options only. The bits on CPUID[14h] change depending on the host and are not migration-safe. CPUID[7].EBX[bit 25] is just the (already configurable) bit that enables the migration-unsafe code for CPUID[14h].
What about add a new cpu model name "Icelake" and add PT in this. Is that would be migration safe?
It won't, because of the CPUID[14h] code that makes it unsafe to migrate between hosts with different Intel-PT capabilities (i.e. different data on CPUID[14h]).
Thanks, Luwei Kang
Probably (a) is easier to implement, and it also makes the feature more useful (by making it migration-safe).
Intel PT is first supported in Intel Core M and 5th generation Intel Core processors that are based on the Intel micro-architecture code name Broadwell but Intel PT use EPT is first supported in Ice Lake. Intel PT virtualization depend on PT use EPT. I will add Intel PT to "Broadwell" CPU model and later to make sure a "Broadwell" guest can use Intel PT if the host is Ice Lake.
The "if the host is Ice Lake" part is problematic. On migration-safe CPU models (all of them except "max" and "host"), the data seen on CPUID can't depend on the host at all. It
should depend only on the machine-type + command-line options.
-- Eduardo
-- Eduardo
-- Eduardo
On 18/01/2018 14:24, Eduardo Habkost wrote:
However, if there's a simple way to make it possible to migrate between hosts with different CPUID[14h] data, it would be even better. With the current KVM intel-pt implementation, what happens if the CPUID[14h] data seen by the guest doesn't match exactly the CPUID[14h] leaves from the host?
Some bits in there can be treated as CPU features (e.g. EBX bit 0 "CR3 filtering support"). Probably we should handle these in KVM right now. KVM needs to compute a mask of valid 1 bits for IA32_RTIT_CTL based on CPUID, and apply it when the MSR is written. It also needs to whitelist bits like we do for other feature words. These include: - CPUID[EAX=14h,ECX=0].EBX - CPUID[EAX=14h,ECX=0].ECX except bit 31 - CPUID[EAX=14h,ECX=1].EAX bits 16:31 (if CPUID[EAX=14h,ECX=0].EBX[3]=1) - CPUID[EAX=14h,ECX=1].EBX (if CPUID[EAX=14h,ECX=0].EBX[1]=1) Others, currently only CPUID[EAX=14h,ECX=0].ECX[31] must match, there is no way to emulate the "wrong" value. Others, currently only CPUID[EAX=14h,ECX=1].EAX[2:0] are numeric values, and it's possible to emulate a lower value than the one in the processor. CPUID[EAX=14h,ECX=0].EAX is the maximum subleaf. It should be (barring guest bugs) okay to always present leaf 1. Paolo
On Thu, Jan 18, 2018 at 02:39:57PM +0100, Paolo Bonzini wrote:
On 18/01/2018 14:24, Eduardo Habkost wrote:
However, if there's a simple way to make it possible to migrate between hosts with different CPUID[14h] data, it would be even better. With the current KVM intel-pt implementation, what happens if the CPUID[14h] data seen by the guest doesn't match exactly the CPUID[14h] leaves from the host?
Some bits in there can be treated as CPU features (e.g. EBX bit 0 "CR3 filtering support"). Probably we should handle these in KVM right now. KVM needs to compute a mask of valid 1 bits for IA32_RTIT_CTL based on CPUID, and apply it when the MSR is written.
Does this mean QEMU can't set CPUID values that won't match the host with the existing implementation, or this won't matter for well-behaved guests that don't try to set reserved bits on the MSRs?
It also needs to whitelist bits like we do for other feature words. These include:
- CPUID[EAX=14h,ECX=0].EBX
- CPUID[EAX=14h,ECX=0].ECX except bit 31
- CPUID[EAX=14h,ECX=1].EAX bits 16:31 (if CPUID[EAX=14h,ECX=0].EBX[3]=1)
- CPUID[EAX=14h,ECX=1].EBX (if CPUID[EAX=14h,ECX=0].EBX[1]=1)
What do you mean by whitelist?
Others, currently only CPUID[EAX=14h,ECX=0].ECX[31] must match, there is no way to emulate the "wrong" value.
In this case we could make it configurable but require the host and guest value to always match. This might be an obstacle to enabling intel-pt by default (because it could make VMs not migratable to newer hosts), but may allow the feature to be configured in a predictable way.
Others, currently only CPUID[EAX=14h,ECX=1].EAX[2:0] are numeric values, and it's possible to emulate a lower value than the one in the processor.
This could be handled by QEMU. There's no requirement that all GET_SUPPORTED_CPUID values should be validated by simple bit masking.
CPUID[EAX=14h,ECX=0].EAX is the maximum subleaf. It should be (barring guest bugs) okay to always present leaf 1.
Paolo
-- Eduardo
On 18/01/2018 15:37, Eduardo Habkost wrote:
On Thu, Jan 18, 2018 at 02:39:57PM +0100, Paolo Bonzini wrote:
On 18/01/2018 14:24, Eduardo Habkost wrote:
However, if there's a simple way to make it possible to migrate between hosts with different CPUID[14h] data, it would be even better. With the current KVM intel-pt implementation, what happens if the CPUID[14h] data seen by the guest doesn't match exactly the CPUID[14h] leaves from the host?
Some bits in there can be treated as CPU features (e.g. EBX bit 0 "CR3 filtering support"). Probably we should handle these in KVM right now. KVM needs to compute a mask of valid 1 bits for IA32_RTIT_CTL based on CPUID, and apply it when the MSR is written.
Does this mean QEMU can't set CPUID values that won't match the host with the existing implementation, or this won't matter for well-behaved guests that don't try to set reserved bits on the MSRs?
All the features could be handled exactly like regular feature bits. If QEMU sets them incorrectly and "enforce" is not used, bad things happen but it's the user's fault.
It also needs to whitelist bits like we do for other feature words. These include:
- CPUID[EAX=14h,ECX=0].EBX
- CPUID[EAX=14h,ECX=0].ECX except bit 31
- CPUID[EAX=14h,ECX=1].EAX bits 16:31 (if CPUID[EAX=14h,ECX=0].EBX[3]=1)
- CPUID[EAX=14h,ECX=1].EBX (if CPUID[EAX=14h,ECX=0].EBX[1]=1)
What do you mean by whitelist?
KVM needs to tell QEMU the bits it knows about.
Others, currently only CPUID[EAX=14h,ECX=0].ECX[31] must match, there is no way to emulate the "wrong" value.
In this case we could make it configurable but require the host and guest value to always match.
This might be an obstacle to enabling intel-pt by default (because it could make VMs not migratable to newer hosts), but may allow the feature to be configured in a predictable way.
Yeah, but consider that virtualized PT anyway would only be enabled on Ice Lake processors. It's a few years away anyway!
Others, currently only CPUID[EAX=14h,ECX=1].EAX[2:0] are numeric values, and it's possible to emulate a lower value than the one in the processor.
This could be handled by QEMU. There's no requirement that all GET_SUPPORTED_CPUID values should be validated by simple bit masking.
Good! Paolo
On Thu, Jan 18, 2018 at 03:44:49PM +0100, Paolo Bonzini wrote:
On 18/01/2018 15:37, Eduardo Habkost wrote:
On Thu, Jan 18, 2018 at 02:39:57PM +0100, Paolo Bonzini wrote:
On 18/01/2018 14:24, Eduardo Habkost wrote:
However, if there's a simple way to make it possible to migrate between hosts with different CPUID[14h] data, it would be even better. With the current KVM intel-pt implementation, what happens if the CPUID[14h] data seen by the guest doesn't match exactly the CPUID[14h] leaves from the host?
Some bits in there can be treated as CPU features (e.g. EBX bit 0 "CR3 filtering support"). Probably we should handle these in KVM right now. KVM needs to compute a mask of valid 1 bits for IA32_RTIT_CTL based on CPUID, and apply it when the MSR is written.
Does this mean QEMU can't set CPUID values that won't match the host with the existing implementation, or this won't matter for well-behaved guests that don't try to set reserved bits on the MSRs?
All the features could be handled exactly like regular feature bits. If QEMU sets them incorrectly and "enforce" is not used, bad things happen but it's the user's fault.
Oh, I mean setting the bit to 0 when it's 1 on the host (if it's 0 on the host, QEMU would never set it anyway). Is it safe to do it with the current KVM intel-pt implementation?
It also needs to whitelist bits like we do for other feature words. These include:
- CPUID[EAX=14h,ECX=0].EBX
- CPUID[EAX=14h,ECX=0].ECX except bit 31
- CPUID[EAX=14h,ECX=1].EAX bits 16:31 (if CPUID[EAX=14h,ECX=0].EBX[3]=1)
- CPUID[EAX=14h,ECX=1].EBX (if CPUID[EAX=14h,ECX=0].EBX[1]=1)
What do you mean by whitelist?
KVM needs to tell QEMU the bits it knows about.
So KVM isn't currently doing it on GET_SUPPORTED_CPUID? Oops.
Others, currently only CPUID[EAX=14h,ECX=0].ECX[31] must match, there is no way to emulate the "wrong" value.
In this case we could make it configurable but require the host and guest value to always match.
This might be an obstacle to enabling intel-pt by default (because it could make VMs not migratable to newer hosts), but may allow the feature to be configured in a predictable way.
Yeah, but consider that virtualized PT anyway would only be enabled on Ice Lake processors. It's a few years away anyway!
Others, currently only CPUID[EAX=14h,ECX=1].EAX[2:0] are numeric values, and it's possible to emulate a lower value than the one in the processor.
This could be handled by QEMU. There's no requirement that all GET_SUPPORTED_CPUID values should be validated by simple bit masking.
Good!
Paolo
-- Eduardo
On 18/01/2018 17:52, Eduardo Habkost wrote:
On Thu, Jan 18, 2018 at 03:44:49PM +0100, Paolo Bonzini wrote:
On 18/01/2018 15:37, Eduardo Habkost wrote:
On Thu, Jan 18, 2018 at 02:39:57PM +0100, Paolo Bonzini wrote:
On 18/01/2018 14:24, Eduardo Habkost wrote:
However, if there's a simple way to make it possible to migrate between hosts with different CPUID[14h] data, it would be even better. With the current KVM intel-pt implementation, what happens if the CPUID[14h] data seen by the guest doesn't match exactly the CPUID[14h] leaves from the host?
Some bits in there can be treated as CPU features (e.g. EBX bit 0 "CR3 filtering support"). Probably we should handle these in KVM right now. KVM needs to compute a mask of valid 1 bits for IA32_RTIT_CTL based on CPUID, and apply it when the MSR is written.
Does this mean QEMU can't set CPUID values that won't match the host with the existing implementation, or this won't matter for well-behaved guests that don't try to set reserved bits on the MSRs?
All the features could be handled exactly like regular feature bits. If QEMU sets them incorrectly and "enforce" is not used, bad things happen but it's the user's fault.
Oh, I mean setting the bit to 0 when it's 1 on the host (if it's 0 on the host, QEMU would never set it anyway). Is it safe to do it with the current KVM intel-pt implementation?
It's not, but it's (very) easy to fix. Paolo
It also needs to whitelist bits like we do for other feature words. These include:
- CPUID[EAX=14h,ECX=0].EBX
- CPUID[EAX=14h,ECX=0].ECX except bit 31
- CPUID[EAX=14h,ECX=1].EAX bits 16:31 (if CPUID[EAX=14h,ECX=0].EBX[3]=1)
- CPUID[EAX=14h,ECX=1].EBX (if CPUID[EAX=14h,ECX=0].EBX[1]=1)
What do you mean by whitelist?
KVM needs to tell QEMU the bits it knows about.
So KVM isn't currently doing it on GET_SUPPORTED_CPUID? Oops.
Others, currently only CPUID[EAX=14h,ECX=0].ECX[31] must match, there is no way to emulate the "wrong" value.
In this case we could make it configurable but require the host and guest value to always match.
This might be an obstacle to enabling intel-pt by default (because it could make VMs not migratable to newer hosts), but may allow the feature to be configured in a predictable way.
Yeah, but consider that virtualized PT anyway would only be enabled on Ice Lake processors. It's a few years away anyway!
Others, currently only CPUID[EAX=14h,ECX=1].EAX[2:0] are numeric values, and it's possible to emulate a lower value than the one in the processor.
This could be handled by QEMU. There's no requirement that all GET_SUPPORTED_CPUID values should be validated by simple bit masking.
Good!
Paolo
On Thu, Jan 18, 2018 at 03:44:49PM +0100, Paolo Bonzini wrote:
On 18/01/2018 15:37, Eduardo Habkost wrote:
On Thu, Jan 18, 2018 at 02:39:57PM +0100, Paolo Bonzini wrote:
On 18/01/2018 14:24, Eduardo Habkost wrote:
However, if there's a simple way to make it possible to migrate between hosts with different CPUID[14h] data, it would be even better. With the current KVM intel-pt implementation, what happens if the CPUID[14h] data seen by the guest doesn't match exactly the CPUID[14h] leaves from the host?
Some bits in there can be treated as CPU features (e.g. EBX bit 0 "CR3 filtering support"). Probably we should handle these in KVM right now. KVM needs to compute a mask of valid 1 bits for IA32_RTIT_CTL based on CPUID, and apply it when the MSR is written.
Does this mean QEMU can't set CPUID values that won't match the host with the existing implementation, or this won't matter for well-behaved guests that don't try to set reserved bits on the MSRs?
All the features could be handled exactly like regular feature bits. If QEMU sets them incorrectly and "enforce" is not used, bad things happen but it's the user's fault.
Oh, I mean setting the bit to 0 when it's 1 on the host (if it's 0 on the host, QEMU would never set it anyway). Is it safe to do it with the current KVM intel-pt implementation?
It's not, but it's (very) easy to fix.
Hi Paolo, Do you mean there need to add some check before setting IA32_RTIT_CTL MSR in KVM because some bits of this MSR is depend on the result of CPUID[14]. Any attempts to change these reserved bit should cause a #GP.
It also needs to whitelist bits like we do for other feature words. These include:
- CPUID[EAX=14h,ECX=0].EBX
- CPUID[EAX=14h,ECX=0].ECX except bit 31
- CPUID[EAX=14h,ECX=1].EAX bits 16:31 (if CPUID[EAX=14h,ECX=0].EBX[3]=1)
- CPUID[EAX=14h,ECX=1].EBX (if CPUID[EAX=14h,ECX=0].EBX[1]=1)
What do you mean by whitelist?
KVM needs to tell QEMU the bits it knows about.
I think kvm_arch_get_supported_cpuid() function can get the result of CPUID[14] from KVM. Is this the whitelist what you mentioned? Thanks, Luwei Kang
So KVM isn't currently doing it on GET_SUPPORTED_CPUID? Oops.
Others, currently only CPUID[EAX=14h,ECX=0].ECX[31] must match, there is no way to emulate the "wrong" value.
In this case we could make it configurable but require the host and guest value to always match.
This might be an obstacle to enabling intel-pt by default (because it could make VMs not migratable to newer hosts), but may allow the feature to be configured in a predictable way.
Yeah, but consider that virtualized PT anyway would only be enabled on Ice Lake processors. It's a few years away anyway!
Others, currently only CPUID[EAX=14h,ECX=1].EAX[2:0] are numeric values, and it's possible to emulate a lower value than the one in the processor.
This could be handled by QEMU. There's no requirement that all GET_SUPPORTED_CPUID values should be validated by simple bit masking.
Good!
Paolo
On 22/01/2018 11:36, Kang, Luwei wrote:
On Thu, Jan 18, 2018 at 03:44:49PM +0100, Paolo Bonzini wrote:
On 18/01/2018 15:37, Eduardo Habkost wrote:
On Thu, Jan 18, 2018 at 02:39:57PM +0100, Paolo Bonzini wrote:
On 18/01/2018 14:24, Eduardo Habkost wrote: > However, if there's a simple way to make it possible to migrate > between hosts with different CPUID[14h] data, it would be even > better. With the current KVM intel-pt implementation, what > happens if the CPUID[14h] data seen by the guest doesn't match > exactly the CPUID[14h] leaves from the host?
Some bits in there can be treated as CPU features (e.g. EBX bit 0 "CR3 filtering support"). Probably we should handle these in KVM right now. KVM needs to compute a mask of valid 1 bits for IA32_RTIT_CTL based on CPUID, and apply it when the MSR is written.
Does this mean QEMU can't set CPUID values that won't match the host with the existing implementation, or this won't matter for well-behaved guests that don't try to set reserved bits on the MSRs?
All the features could be handled exactly like regular feature bits. If QEMU sets them incorrectly and "enforce" is not used, bad things happen but it's the user's fault.
Oh, I mean setting the bit to 0 when it's 1 on the host (if it's 0 on the host, QEMU would never set it anyway). Is it safe to do it with the current KVM intel-pt implementation?
It's not, but it's (very) easy to fix.
Hi Paolo, Do you mean there need to add some check before setting IA32_RTIT_CTL MSR in KVM because some bits of this MSR is depend on the result of CPUID[14]. Any attempts to change these reserved bit should cause a #GP.
Yes, but the guest's CPUID[14] need not match the host. Likewise, the number of address range MSRs in the guest, from CPUID[EAX=14h,ECX=1].EAX[2:0], might be lower than in the host.
It also needs to whitelist bits like we do for other feature words. These include:
- CPUID[EAX=14h,ECX=0].EBX
- CPUID[EAX=14h,ECX=0].ECX except bit 31
- CPUID[EAX=14h,ECX=1].EAX bits 16:31 (if CPUID[EAX=14h,ECX=0].EBX[3]=1)
- CPUID[EAX=14h,ECX=1].EBX (if CPUID[EAX=14h,ECX=0].EBX[1]=1)
What do you mean by whitelist?
KVM needs to tell QEMU the bits it knows about.
I think kvm_arch_get_supported_cpuid() function can get the result of CPUID[14] from KVM. Is this the whitelist what you mentioned?
Whitelist means that KVM must not return all the bits from CPUID[14]; only those it knows about. Paolo
Thanks, Luwei Kang
So KVM isn't currently doing it on GET_SUPPORTED_CPUID? Oops.
Others, currently only CPUID[EAX=14h,ECX=0].ECX[31] must match, there is no way to emulate the "wrong" value.
In this case we could make it configurable but require the host and guest value to always match.
This might be an obstacle to enabling intel-pt by default (because it could make VMs not migratable to newer hosts), but may allow the feature to be configured in a predictable way.
Yeah, but consider that virtualized PT anyway would only be enabled on Ice Lake processors. It's a few years away anyway!
Others, currently only CPUID[EAX=14h,ECX=1].EAX[2:0] are numeric values, and it's possible to emulate a lower value than the one in the processor.
This could be handled by QEMU. There's no requirement that all GET_SUPPORTED_CPUID values should be validated by simple bit masking.
Good!
Paolo
On 18/01/2018 14:24, Eduardo Habkost wrote:
However, if there's a simple way to make it possible to migrate between hosts with different CPUID[14h] data, it would be even better. With the current KVM intel-pt implementation, what happens if the CPUID[14h] data seen by the guest doesn't match exactly the CPUID[14h] leaves from the host?
Some bits in there can be treated as CPU features (e.g. EBX bit 0 "CR3 filtering support"). Probably we should handle these in KVM right now. KVM needs to compute a mask of valid 1 bits for IA32_RTIT_CTL based on CPUID, and apply it when the MSR is written.
Does this mean QEMU can't set CPUID values that won't match the host with the existing implementation, or this won't matter for well-behaved guests that don't try to set reserved bits on the MSRs?
It also needs to whitelist bits like we do for other feature words. These include:
- CPUID[EAX=14h,ECX=0].EBX
- CPUID[EAX=14h,ECX=0].ECX except bit 31
- CPUID[EAX=14h,ECX=1].EAX bits 16:31 (if CPUID[EAX=14h,ECX=0].EBX[3]=1)
- CPUID[EAX=14h,ECX=1].EBX (if CPUID[EAX=14h,ECX=0].EBX[1]=1)
What do you mean by whitelist?
Others, currently only CPUID[EAX=14h,ECX=0].ECX[31] must match, there is no way to emulate the "wrong" value.
In this case we could make it configurable but require the host and guest value to always match.
This might be an obstacle to enabling intel-pt by default (because it could make VMs not migratable to newer hosts), but may allow the feature to be configured in a predictable way.
Others, currently only CPUID[EAX=14h,ECX=1].EAX[2:0] are numeric values, and it's possible to emulate a lower value than the one in the processor.
This could be handled by QEMU. There's no requirement that all GET_SUPPORTED_CPUID values should be validated by simple bit masking.
So, we can get a lower value on the numeric of CPUID[EAX=14h,ECX=1].EAX[2:0] between different host. How about other bits of CPUID[14] ? Can we do like this as well? Thanks, Luwei Kang
CPUID[EAX=14h,ECX=0].EAX is the maximum subleaf. It should be (barring guest bugs) okay to always present leaf 1.
Paolo
-- Eduardo
participants (4)
-
Eduardo Habkost -
Jiri Denemark -
Kang, Luwei -
Paolo Bonzini