[libvirt] Cpu Modeling
Hi Jiri & Eduardo, You might remember a discussion with David Hildenbrand of IBM on the Qemu mailing list regarding a new Qemu<->Libvirt interface for cpu modeling. I am picking up this work from David and I wanted to confirm that we are still on the same page as to the direction of that interface. For your reference: https://www.redhat.com/archives/libvir-list/2016-June/thread.html#01413 https://lists.gnu.org/archive/html/qemu-devel/2016-09/threads.html#00489 The first link is to the discussion you were directly involved in. The second link is to the final patch set posted to qemu-devel. The cover letter gives a good overview of the interface added to Qemu and the proposed use-case for Libvirt to use this new cpu modeling support. I'll paste in the most relevant section for your convenience: --------------------------------Libvirt usecase---------------------------- Testing for runability: - Simply try to start QEMU with KVM, compat machine, CPU model - Could be done using query-cpu-model-comparison in the future. Identifying host model, e.g. "virsh capabilities" - query-cpu-model-expansion on "host" with "-M none --enable-kvm" <cpu mode='host-model'>: - simply copy the identified host model <cpu mode='host-passthrough'>: - "-cpu host" "virsh cpu-baseline": - query-cpu-model-baseline on two models with "-M none" "virsh cpu-compare": - query-cpu-model-comparison on two models with "-M none" There might be some cenarios where libvirt wants to convert another CPU model to a static variant, this can be done using query-cpu-model-expansion --------------------------------------------------------------------------- Now that I've hopefully refreshed your memory :) I just want to make sure that you are still on-board with this plan. I realize that this new approach does things differently than Libvirt does today for other platforms. Especially x86_64. The big differences are as follows: 1. We will invoke qemu to gather the host cpu data used for virsh capabilities. Today this data seems to be collected directly from the host hardware for most (all?) architectures. 2. virsh cpu-models {arch} will also use a Qemu invocation to gather cpu model data. 3. Most architectures seem to use a "map" (xml file containing cpu model data that ships with Libvirt) to satisfy #1 and #2 above. Our new method does not use this map as it gets all of the data directly from Qemu. 4. virsh cpu-baseline and cpu-compare will now invoke qemu directly as well. Note: I'm not sure exactly how much all of this will apply just to s390 with other architectures moving over to the new interface if/when they want to. Or if we will want to change all architectures to this new interface at the same time. Any guidance? Thanks for your time and consideration. -- -- Jason J. Herne (jjherne@linux.vnet.ibm.com)
Hi! On Thu, Sep 22, 2016 at 02:47:36PM -0400, Jason J. Herne wrote:
Hi Jiri & Eduardo,
You might remember a discussion with David Hildenbrand of IBM on the Qemu mailing list regarding a new Qemu<->Libvirt interface for cpu modeling. I am picking up this work from David and I wanted to confirm that we are still on the same page as to the direction of that interface.
I am, but I would like to hear Jiri's feedback in case there's anything that is still lacking in the new interface. Additionanl comments below:
For your reference: https://www.redhat.com/archives/libvir-list/2016-June/thread.html#01413 https://lists.gnu.org/archive/html/qemu-devel/2016-09/threads.html#00489
The first link is to the discussion you were directly involved in. The second link is to the final patch set posted to qemu-devel. The cover letter gives a good overview of the interface added to Qemu and the proposed use-case for Libvirt to use this new cpu modeling support. I'll paste in the most relevant section for your convenience:
--------------------------------Libvirt usecase---------------------------- Testing for runability: - Simply try to start QEMU with KVM, compat machine, CPU model
The main problem here is for libvirt to get machine-friendly error information from QEMU in case the VM fails to start. But if libvirt avoids starting QEMU based on previously-collected runnability information, we should be OK.
- Could be done using query-cpu-model-comparison in the future.
I sent a series adding runnability information to query-cpu-definitions as well.
Identifying host model, e.g. "virsh capabilities" - query-cpu-model-expansion on "host" with "-M none --enable-kvm"
Correct.
<cpu mode='host-model'>: - simply copy the identified host model
<cpu mode='host-passthrough'>: - "-cpu host"
Correct.
"virsh cpu-baseline": - query-cpu-model-baseline on two models with "-M none"
"virsh cpu-compare": - query-cpu-model-comparison on two models with "-M none"
Correct.
There might be some cenarios where libvirt wants to convert another CPU model to a static variant, this can be done using query-cpu-model-expansion ---------------------------------------------------------------------------
Now that I've hopefully refreshed your memory :) I just want to make sure that you are still on-board with this plan. I realize that this new approach does things differently than Libvirt does today for other platforms. Especially x86_64. The big differences are as follows:
1. We will invoke qemu to gather the host cpu data used for virsh capabilities. Today this data seems to be collected directly from the host hardware for most (all?) architectures.
I believe libvirt developers want to change this, because the current approach is wrong for x86 as well.
2. virsh cpu-models {arch} will also use a Qemu invocation to gather cpu model data.
I believe libvirt can collect this data once, and cache it (like it does today).
3. Most architectures seem to use a "map" (xml file containing cpu model data that ships with Libvirt) to satisfy #1 and #2 above. Our new method does not use this map as it gets all of the data directly from Qemu.
I believe the goal is to make cpu_map.xml unnecessary on x86 as well.
4. virsh cpu-baseline and cpu-compare will now invoke qemu directly as well.
Note: I'm not sure exactly how much all of this will apply just to s390 with other architectures moving over to the new interface if/when they want to. Or if we will want to change all architectures to this new interface at the same time. Any guidance?
I am not sure about this part, because I don't know what are the existing use cases of cpu-baseline and cpu-compare. But I believe this shouldn't be a problem as long as the function is not called very often and/or results are cached by libvirt. -- Eduardo
On Thu, Sep 22, 2016 at 14:47:36 -0400, Jason J. Herne wrote:
Testing for runability: - Simply try to start QEMU with KVM, compat machine, CPU model
Yes, if the domain XML explicitly requests a specific CPU model. Additionally, we need to make sure a CPU model chosen by libvirt (host-model) is runnable, but that should be handled by query-cpu-model-expansion.
Identifying host model, e.g. "virsh capabilities" - query-cpu-model-expansion on "host" with "-M none --enable-kvm"
Right, except that it doesn't seem to work on x86_64 now. It's not critical for query-cpu-model-expansion, but do we have an interface we can use to check whether the new commands are supported by a QEMU binary? Trying and checking for {"error": {"class": "GenericError", "desc": "this feature or command is not currently supported"}} is not really possible. At least we'd need a specific error class.
<cpu mode='host-model'>: - simply copy the identified host model
Yes.
<cpu mode='host-passthrough'>: - "-cpu host"
Exactly. ...
1. We will invoke qemu to gather the host cpu data used for virsh capabilities. Today this data seems to be collected directly from the host hardware for most (all?) architectures.
Not really, virsh capabilities will still contain data gathered directly from the host hardware. But, we have virsh domcapabilities for displaying data tight to a specific QEMU binary. Since yesterday afternoon we have support for displaying CPU related data in the domain capabilities XML; see http://libvirt.org/formatdomaincaps.html#elementsCPU The host-model part of the XML will show the result of query-cpu-model-expansion on "host" model, or the result of querying the hardware directly if we can't ask QEMU for that (which is the current state).
2. virsh cpu-models {arch} will also use a Qemu invocation to gather cpu model data.
No, virsh cpu-models will just list CPU models supported by libvirt (or an empty list if libvirt supports all models supported by QEMU). The CPU model data from QEMU can be found in domain capabilities XML.
3. Most architectures seem to use a "map" (xml file containing cpu model data that ships with Libvirt) to satisfy #1 and #2 above. Our new method does not use this map as it gets all of the data directly from Qemu.
Even if we switch some CPU operations to work through QEMU, we may still need to use the cpu_map.xml file for some other operations, or other hypervisor driver. It all depends on what we need to do for a given architecture. For example, ARM does not use cpu_map.xml at all even now. Slightly related, I don't think we have a way to list CPU features supported by QEMU over QMP, do we? "-cpu ?" will show them all, but I couldn't find a QMP command that would give me the same list.
4. virsh cpu-baseline and cpu-compare will now invoke qemu directly as well.
Perhaps, but before we can do that, we'll probably need to come up with new APIs. It don't think it's critical, though.
Note: I'm not sure exactly how much all of this will apply just to s390 with other architectures moving over to the new interface if/when they want to. Or if we will want to change all architectures to this new interface at the same time.
Well, IIUC the new commands are not supported on any other architecture right now, are they? Anyway, I don't think we need to switch everything at the same time. If we have a way to detect what commands are supported for a specific QEMU binary, we can implement the code in libvirt and use when we can or falling back to the current way. Jirka
On Fri, Sep 23, 2016 at 02:05:59PM +0200, Jiri Denemark wrote:
On Thu, Sep 22, 2016 at 14:47:36 -0400, Jason J. Herne wrote:
Testing for runability: - Simply try to start QEMU with KVM, compat machine, CPU model
Yes, if the domain XML explicitly requests a specific CPU model. Additionally, we need to make sure a CPU model chosen by libvirt (host-model) is runnable, but that should be handled by query-cpu-model-expansion.
Identifying host model, e.g. "virsh capabilities" - query-cpu-model-expansion on "host" with "-M none --enable-kvm"
Right, except that it doesn't seem to work on x86_64 now.
Yeah, I still need to implement it.
It's not critical for query-cpu-model-expansion, but do we have an interface we can use to check whether the new commands are supported by a QEMU binary? Trying and checking for
{"error": {"class": "GenericError", "desc": "this feature or command is not currently supported"}}
is not really possible. At least we'd need a specific error class.
I believe query-qmp-schema and/or query-commands can be used for that. But you may still have the command returning the error above if it is simply not implemented in that architecture. I will start a qemu-devel thread to look for better solutions.
<cpu mode='host-model'>: - simply copy the identified host model
Yes.
<cpu mode='host-passthrough'>: - "-cpu host"
Exactly.
...
1. We will invoke qemu to gather the host cpu data used for virsh capabilities. Today this data seems to be collected directly from the host hardware for most (all?) architectures.
Not really, virsh capabilities will still contain data gathered directly from the host hardware. But, we have virsh domcapabilities for displaying data tight to a specific QEMU binary. Since yesterday afternoon we have support for displaying CPU related data in the domain capabilities XML; see http://libvirt.org/formatdomaincaps.html#elementsCPU
The host-model part of the XML will show the result of query-cpu-model-expansion on "host" model, or the result of querying the hardware directly if we can't ask QEMU for that (which is the current state).
2. virsh cpu-models {arch} will also use a Qemu invocation to gather cpu model data.
No, virsh cpu-models will just list CPU models supported by libvirt (or an empty list if libvirt supports all models supported by QEMU). The CPU model data from QEMU can be found in domain capabilities XML.
That's interesting to know. I was often confused about the interfaces that seem to be generic (not depend on a specific QEMU binary) but seemed to depend on information from the QEMU binary (virsh capabilities and virsh cpu-models). It looks like I was looking at the wrong commands.
3. Most architectures seem to use a "map" (xml file containing cpu model data that ships with Libvirt) to satisfy #1 and #2 above. Our new method does not use this map as it gets all of the data directly from Qemu.
Even if we switch some CPU operations to work through QEMU, we may still need to use the cpu_map.xml file for some other operations, or other hypervisor driver. It all depends on what we need to do for a given architecture. For example, ARM does not use cpu_map.xml at all even now.
Slightly related, I don't think we have a way to list CPU features supported by QEMU over QMP, do we? "-cpu ?" will show them all, but I couldn't find a QMP command that would give me the same list.
device-list-properties should return all the properties you can set. But I recommend that you don't make libvirt set any of the properties in the list unless: 1) you know what it does; or 2) it is returned by a query-cpu-model-expansion call.
4. virsh cpu-baseline and cpu-compare will now invoke qemu directly as well.
Perhaps, but before we can do that, we'll probably need to come up with new APIs. It don't think it's critical, though.
Note: I'm not sure exactly how much all of this will apply just to s390 with other architectures moving over to the new interface if/when they want to. Or if we will want to change all architectures to this new interface at the same time.
Well, IIUC the new commands are not supported on any other architecture right now, are they? Anyway, I don't think we need to switch everything at the same time. If we have a way to detect what commands are supported for a specific QEMU binary, we can implement the code in libvirt and use when we can or falling back to the current way.
Right now the commands are available only on s390x. I plan to have them implemented in x86 before QEMU 2.8 is out. -- Eduardo
On Fri, Sep 23, 2016 at 15:23:45 -0300, Eduardo Habkost wrote:
On Fri, Sep 23, 2016 at 02:05:59PM +0200, Jiri Denemark wrote:
It's not critical for query-cpu-model-expansion, but do we have an interface we can use to check whether the new commands are supported by a QEMU binary? Trying and checking for
{"error": {"class": "GenericError", "desc": "this feature or command is not currently supported"}}
is not really possible. At least we'd need a specific error class.
I believe query-qmp-schema and/or query-commands can be used for that.
But you may still have the command returning the error above if it is simply not implemented in that architecture. I will start a qemu-devel thread to look for better solutions.
Yes, that's my concern. ...
No, virsh cpu-models will just list CPU models supported by libvirt (or an empty list if libvirt supports all models supported by QEMU). The CPU model data from QEMU can be found in domain capabilities XML.
That's interesting to know. I was often confused about the interfaces that seem to be generic (not depend on a specific QEMU binary) but seemed to depend on information from the QEMU binary (virsh capabilities and virsh cpu-models). It looks like I was looking at the wrong commands.
Those are just old interfaces. We may add newer versions for some of them that would take more input parameters, but I don't feel a strong need for it right now. And the domain capabilities XML gained support for showing CPU related stuff only recently (it will be included in the upcoming 2.3.0 release), so it's not like you had a choice where to look at :-) ...
Slightly related, I don't think we have a way to list CPU features supported by QEMU over QMP, do we? "-cpu ?" will show them all, but I couldn't find a QMP command that would give me the same list.
device-list-properties should return all the properties you can set. But I recommend that you don't make libvirt set any of the properties in the list unless: 1) you know what it does; or 2) it is returned by a query-cpu-model-expansion call.
The use case was for pre-checking whether all CPU features specified in domain XML are supported by current QEMU binary. What parameters would I need to use for device-list-properties? And would it work with -machine none? ...
Well, IIUC the new commands are not supported on any other architecture right now, are they? Anyway, I don't think we need to switch everything at the same time. If we have a way to detect what commands are supported for a specific QEMU binary, we can implement the code in libvirt and use when we can or falling back to the current way.
Right now the commands are available only on s390x. I plan to have them implemented in x86 before QEMU 2.8 is out.
That would be nice. Jirka
On Thu, Sep 29, 2016 at 04:21:07PM +0200, Jiri Denemark wrote: [...]
Slightly related, I don't think we have a way to list CPU features supported by QEMU over QMP, do we? "-cpu ?" will show them all, but I couldn't find a QMP command that would give me the same list.
device-list-properties should return all the properties you can set. But I recommend that you don't make libvirt set any of the properties in the list unless: 1) you know what it does; or 2) it is returned by a query-cpu-model-expansion call.
The use case was for pre-checking whether all CPU features specified in domain XML are supported by current QEMU binary. What parameters would I need to use for device-list-properties? And would it work with -machine none?
You just need the right QOM type name (in x86 it is "<model>-x86_64-cpu" or "<model>-i386-cpu"). I will send a patch that includes a "qom-type" field in CpuDefinitionInfo (query-cpu-definitions) so you know what's the right QOM type name to query. I think device-list-properties won't work with an abstract base type like "x86_64-cpu" or "i386-cpu". But I think it's better query the right class name for the CPU model you plan to use, anyway (this way architecture code will be able to have different CPU models supporting different sets of options, if necessary). -- Eduardo
On Thu, Sep 29, 2016 at 15:51:58 -0300, Eduardo Habkost wrote:
On Thu, Sep 29, 2016 at 04:21:07PM +0200, Jiri Denemark wrote: [...]
Slightly related, I don't think we have a way to list CPU features supported by QEMU over QMP, do we? "-cpu ?" will show them all, but I couldn't find a QMP command that would give me the same list.
device-list-properties should return all the properties you can set. But I recommend that you don't make libvirt set any of the properties in the list unless: 1) you know what it does; or 2) it is returned by a query-cpu-model-expansion call.
The use case was for pre-checking whether all CPU features specified in domain XML are supported by current QEMU binary. What parameters would I need to use for device-list-properties? And would it work with -machine none?
You just need the right QOM type name (in x86 it is "<model>-x86_64-cpu" or "<model>-i386-cpu"). I will send a patch that includes a "qom-type" field in CpuDefinitionInfo (query-cpu-definitions) so you know what's the right QOM type name to query.
Hmm, it doesn't seem to work (I tried both -machine none and -machine pc): (QEMU) device-list-properties typename=SandyBridge-x86_64-cpu {"error": {"class": "GenericError", "desc": "Can't list properties of device 'SandyBridge-x86_64-cpu'"}} Anyway, is the list of properties expected to depend on the CPU model? If not, wouldn't a simple query-cpu-properties returning a list of properties QEMU understands be good enough? The list would be pretty similar to what -cpu ? returns. It would help us provide a better error in case a user asks for a CPU features that is not supported by their QEMU binary, and it would also help us translate the result of query-cpu-model-expansion; as you probably know, we use older spellings for some CPU features (e.g., pclmulqdq vs. pclmuldq).
I think device-list-properties won't work with an abstract base type like "x86_64-cpu" or "i386-cpu".
Right: (QEMU) device-list-properties typename=x86_64-cpu {"error": {"class": "GenericError", "desc": "Parameter 'name' expects non-abstract device type"}}
But I think it's better query the right class name for the CPU model you plan to use, anyway (this way architecture code will be able to have different CPU models supporting different sets of options, if necessary).
I see, this make sense. Although a generic query-cpu-properties would still be useful for giving us the translation table between new and old features names. Jirka
On Fri, Sep 30, 2016 at 09:00:15AM +0200, Jiri Denemark wrote:
On Thu, Sep 29, 2016 at 15:51:58 -0300, Eduardo Habkost wrote:
On Thu, Sep 29, 2016 at 04:21:07PM +0200, Jiri Denemark wrote: [...]
Slightly related, I don't think we have a way to list CPU features supported by QEMU over QMP, do we? "-cpu ?" will show them all, but I couldn't find a QMP command that would give me the same list.
device-list-properties should return all the properties you can set. But I recommend that you don't make libvirt set any of the properties in the list unless: 1) you know what it does; or 2) it is returned by a query-cpu-model-expansion call.
The use case was for pre-checking whether all CPU features specified in domain XML are supported by current QEMU binary. What parameters would I need to use for device-list-properties? And would it work with -machine none?
You just need the right QOM type name (in x86 it is "<model>-x86_64-cpu" or "<model>-i386-cpu"). I will send a patch that includes a "qom-type" field in CpuDefinitionInfo (query-cpu-definitions) so you know what's the right QOM type name to query.
Hmm, it doesn't seem to work (I tried both -machine none and -machine pc):
(QEMU) device-list-properties typename=SandyBridge-x86_64-cpu {"error": {"class": "GenericError", "desc": "Can't list properties of device 'SandyBridge-x86_64-cpu'"}}
Oops, we need to remove cannot_destroy_with_object_finalize_yet from the CPU classes, and then this will work.
Anyway, is the list of properties expected to depend on the CPU model? If not, wouldn't a simple query-cpu-properties returning a list of properties QEMU understands be good enough? The list would be pretty similar to what -cpu ? returns. It would help us provide a better error in case a user asks for a CPU features that is not supported by their QEMU binary, and it would also help us translate the result of query-cpu-model-expansion; as you probably know, we use older spellings for some CPU features (e.g., pclmulqdq vs. pclmuldq).
I see. I will check if I can fit this information inside query-command-line-options.
I think device-list-properties won't work with an abstract base type like "x86_64-cpu" or "i386-cpu".
Right:
(QEMU) device-list-properties typename=x86_64-cpu {"error": {"class": "GenericError", "desc": "Parameter 'name' expects non-abstract device type"}}
But I think it's better query the right class name for the CPU model you plan to use, anyway (this way architecture code will be able to have different CPU models supporting different sets of options, if necessary).
I see, this make sense. Although a generic query-cpu-properties would still be useful for giving us the translation table between new and old features names.
If we add the aliases for old names like you sugested on qemu-devel, the old names should be available in the QOM queries, too. If we fix device-list-properties to work with abstract types and add the old names to the QOM property list as aliases (both things we should do eventually, anyway), then a specific query-cpu-properties or query-command-line-options support may be unnecessary. -- Eduardo
On 09/23/2016 08:05 AM, Jiri Denemark wrote:
On Thu, Sep 22, 2016 at 14:47:36 -0400, Jason J. Herne wrote: ...
1. We will invoke qemu to gather the host cpu data used for virsh capabilities. Today this data seems to be collected directly from the host hardware for most (all?) architectures.
Not really, virsh capabilities will still contain data gathered directly from the host hardware. But, we have virsh domcapabilities for displaying data tight to a specific QEMU binary. Since yesterday afternoon we have support for displaying CPU related data in the domain capabilities XML; see http://libvirt.org/formatdomaincaps.html#elementsCPU
The host-model part of the XML will show the result of query-cpu-model-expansion on "host" model, or the result of querying the hardware directly if we can't ask QEMU for that (which is the current state).
So the expectation here is that virsh capabilities" reports actual host cpu data. So for S390, we can leave our implementation of this "as-is" and not report any features here, I'm guessing. And the <cpu> section from virsh domcapabilities will contain the Qemu specific supported cpu modeling info. As you stated <mode name='host-model'> will contain the name (and possibly feature details?) of the model returned by qmp query-model-expansion on 'host'. Furthermore, the <mode name='custom'> section will list all supported model names, as indicated by qmp query-cpu-definitions. Do I have it right?
2. virsh cpu-models {arch} will also use a Qemu invocation to gather cpu model data.
No, virsh cpu-models will just list CPU models supported by libvirt
So, in other words we just spit out whatever models Libvirt managed to grab, and cache, from a call to qmp query-cpu-definitions? Or am I missing something?
(or an empty list if libvirt supports all models supported by QEMU).
Can you clarify? It seems odd that an empty list would be returned here. I thought the point was to list all supported cpu models. For x86 today I imagine it is the list found in cpu_map.xml. For s390, this will be the list we get back from qmp query-cpu-definitions, which as you mention below, is found in the domain capabilities XML.
The CPU model data from QEMU can be found in domain capabilities XML.
3. Most architectures seem to use a "map" (xml file containing cpu model data that ships with Libvirt) to satisfy #1 and #2 above. Our new method does not use this map as it gets all of the data directly from Qemu.
Even if we switch some CPU operations to work through QEMU, we may still need to use the cpu_map.xml file for some other operations, or other hypervisor driver. It all depends on what we need to do for a given architecture. For example, ARM does not use cpu_map.xml at all even now.
Slightly related, I don't think we have a way to list CPU features supported by QEMU over QMP, do we? "-cpu ?" will show them all, but I couldn't find a QMP command that would give me the same list.
query-model-expansion will return all features of a given model name. Model names can be enumerated via query-cpu-definitions.
4. virsh cpu-baseline and cpu-compare will now invoke qemu directly as well.
Perhaps, but before we can do that, we'll probably need to come up with new APIs. It don't think it's critical, though.
Do you mind elaborating on this a bit? Which APIs do we want to look at here? Are you talking about new commands/calls to replace cpu-baseline and cpu-compare? Thanks again for your time. -- -- Jason J. Herne (jjherne@linux.vnet.ibm.com)
Polite Ping? :) On 09/23/2016 03:51 PM, Jason J. Herne wrote:
On 09/23/2016 08:05 AM, Jiri Denemark wrote:
On Thu, Sep 22, 2016 at 14:47:36 -0400, Jason J. Herne wrote: ...
1. We will invoke qemu to gather the host cpu data used for virsh capabilities. Today this data seems to be collected directly from the host hardware for most (all?) architectures.
Not really, virsh capabilities will still contain data gathered directly from the host hardware. But, we have virsh domcapabilities for displaying data tight to a specific QEMU binary. Since yesterday afternoon we have support for displaying CPU related data in the domain capabilities XML; see http://libvirt.org/formatdomaincaps.html#elementsCPU
The host-model part of the XML will show the result of query-cpu-model-expansion on "host" model, or the result of querying the hardware directly if we can't ask QEMU for that (which is the current state).
So the expectation here is that virsh capabilities" reports actual host cpu data. So for S390, we can leave our implementation of this "as-is" and not report any features here, I'm guessing. And the <cpu> section from virsh domcapabilities will contain the Qemu specific supported cpu modeling info. As you stated <mode name='host-model'> will contain the name (and possibly feature details?) of the model returned by qmp query-model-expansion on 'host'. Furthermore, the <mode name='custom'> section will list all supported model names, as indicated by qmp query-cpu-definitions. Do I have it right?
2. virsh cpu-models {arch} will also use a Qemu invocation to gather cpu model data.
No, virsh cpu-models will just list CPU models supported by libvirt
So, in other words we just spit out whatever models Libvirt managed to grab, and cache, from a call to qmp query-cpu-definitions? Or am I missing something?
(or an empty list if libvirt supports all models supported by QEMU).
Can you clarify? It seems odd that an empty list would be returned here. I thought the point was to list all supported cpu models. For x86 today I imagine it is the list found in cpu_map.xml. For s390, this will be the list we get back from qmp query-cpu-definitions, which as you mention below, is found in the domain capabilities XML.
The CPU model data from QEMU can be found in domain capabilities XML.
3. Most architectures seem to use a "map" (xml file containing cpu model data that ships with Libvirt) to satisfy #1 and #2 above. Our new method does not use this map as it gets all of the data directly from Qemu.
Even if we switch some CPU operations to work through QEMU, we may still need to use the cpu_map.xml file for some other operations, or other hypervisor driver. It all depends on what we need to do for a given architecture. For example, ARM does not use cpu_map.xml at all even now.
Slightly related, I don't think we have a way to list CPU features supported by QEMU over QMP, do we? "-cpu ?" will show them all, but I couldn't find a QMP command that would give me the same list.
query-model-expansion will return all features of a given model name. Model names can be enumerated via query-cpu-definitions.
4. virsh cpu-baseline and cpu-compare will now invoke qemu directly as well.
Perhaps, but before we can do that, we'll probably need to come up with new APIs. It don't think it's critical, though.
Do you mind elaborating on this a bit? Which APIs do we want to look at here? Are you talking about new commands/calls to replace cpu-baseline and cpu-compare?
Thanks again for your time.
-- -- Jason J. Herne (jjherne@linux.vnet.ibm.com)
On Fri, Sep 23, 2016 at 15:51:12 -0400, Jason J. Herne wrote:
On 09/23/2016 08:05 AM, Jiri Denemark wrote:
The host-model part of the XML will show the result of query-cpu-model-expansion on "host" model, or the result of querying the hardware directly if we can't ask QEMU for that (which is the current state).
So the expectation here is that virsh capabilities" reports actual host cpu data. So for S390, we can leave our implementation of this "as-is" and not report any features here, I'm guessing.
Yes.
And the <cpu> section from virsh domcapabilities will contain the Qemu specific supported cpu modeling info. As you stated <mode name='host-model'> will contain the name (and possibly feature details?) of the model returned by qmp query-model-expansion on 'host'.
Right, the host-model part should basically contain the CPU configuration which libvirt will use if a user asks for host-model. For example (on x86_64, since I have no experience with s390), the following XML <mode name='host-model' supported='yes'> <model fallback='forbid'>Skylake-Client</model> <vendor>Intel</vendor> <feature policy='require' name='vmx'/> <feature policy='require' name='tsc_adjust'/> </mode> would result in "-cpu Skylake-Client,+vmx,+tsc_adjust" QEMU command line.
Furthermore, the <mode name='custom'> section will list all supported model names, as indicated by qmp query-cpu-definitions. Do I have it right?
Mostly, the list of supported CPUs is filtered by libvirt so on some architectures, the <mode name='custom'> section will list fewer models than what you can get from QEMU.
2. virsh cpu-models {arch} will also use a Qemu invocation to gather cpu model data.
No, virsh cpu-models will just list CPU models supported by libvirt
So, in other words we just spit out whatever models Libvirt managed to grab, and cache, from a call to qmp query-cpu-definitions?
No. CPU models supported by libvirt are listed in cpu_map.xml. So we have two basic sets of CPU models: - A: models returned query-cpu-definitions - B: models found in cpu_map.xml Various interfaces report different sets: - virsh cpu-models reports B - <mode name='custom'> section of domain capabilities XML reports A ∩ B
(or an empty list if libvirt supports all models supported by QEMU).
Can you clarify?
This is a special case of B being empty, which means libvirt does not filter CPU models and any model supported by QEMU can be used with libvirt. In this case, <mode name='custom'> section of domain capabilities XML will contain A. But virsh cpu-models will report an empty list. ...
4. virsh cpu-baseline and cpu-compare will now invoke qemu directly as well.
Perhaps, but before we can do that, we'll probably need to come up with new APIs. It don't think it's critical, though.
Do you mind elaborating on this a bit? Which APIs do we want to look at here? Are you talking about new commands/calls to replace cpu-baseline and cpu-compare?
The virConnectCompareCPU, virConnectBaselineCPU don't have enough parameters to be really useful. We'd need them to take more data (such as emulator binary, domain type, and machine type) into account when computing the results. But since we now provide CPU related data in domain capabilities, I don't think it's critical to create new public APIs for comparing CPUs. I think we may focus on the internals first. Jirka
On 09/29/2016 11:15 AM, Jiri Denemark wrote:
On Fri, Sep 23, 2016 at 15:51:12 -0400, Jason J. Herne wrote:
On 09/23/2016 08:05 AM, Jiri Denemark wrote:
The host-model part of the XML will show the result of query-cpu-model-expansion on "host" model, or the result of querying the hardware directly if we can't ask QEMU for that (which is the current state).
So the expectation here is that virsh capabilities" reports actual host cpu data. So for S390, we can leave our implementation of this "as-is" and not report any features here, I'm guessing.
Yes.
And the <cpu> section from virsh domcapabilities will contain the Qemu specific supported cpu modeling info. As you stated <mode name='host-model'> will contain the name (and possibly feature details?) of the model returned by qmp query-model-expansion on 'host'.
Right, the host-model part should basically contain the CPU configuration which libvirt will use if a user asks for host-model. For example (on x86_64, since I have no experience with s390), the following XML
<mode name='host-model' supported='yes'> <model fallback='forbid'>Skylake-Client</model> <vendor>Intel</vendor> <feature policy='require' name='vmx'/> <feature policy='require' name='tsc_adjust'/> </mode>
would result in "-cpu Skylake-Client,+vmx,+tsc_adjust" QEMU command line.
Jiri, In order to properly obtain the host cpu model data for virsh capabilities on s390 we will need to run a Qemu process. There is no precedent for doing this in a cpu connection driver today. I can imagine two ways to do this: 1. The s390 cpu connection driver (src/cpu/cpu_s390.c) can just privately make use of the default Qemu binary and the appropriate qmp calls can be made to get the model name and possibly features. 2. We can extend the existing interface somehow so all archs could make use of a hypervisor instance for nodeData() and decode() operations. Perhaps a new set of callbacks? nodeDataWithHypervisor(), decodeWithHypervisor() ? We'd have to modify the generic versions of these functions to try one set of interfaces, then the other in case the first set is not supported for a given arch. Do either of these sound sane to you, or am I way off on this? I'm trying to get the host cpu model for virCapsPtr caps, as it is filled in via virQEMUCapsInit --> virQEMUCapsInitCPU. And subsequently referenced for the output of virsh capabilities and copied into qemuCaps for reference by virsh domcapabilities. -- -- Jason J. Herne (jjherne@linux.vnet.ibm.com)
On Wed, Oct 12, 2016 at 13:55:49 -0400, Jason J. Herne wrote:
In order to properly obtain the host cpu model data for virsh capabilities on s390 we will need to run a Qemu process. There is no precedent for doing this in a cpu connection driver today.
There are two reasons for this: 1) it was not required so far 2) it doesn't belong to the CPU driver
1. The s390 cpu connection driver (src/cpu/cpu_s390.c) can just privately make use of the default Qemu binary and the appropriate qmp calls can be made to get the model name and possibly features.
No, the CPU driver is supposed to do generic stuff independently on any hypervisor.
2. We can extend the existing interface somehow so all archs could make use of a hypervisor instance for nodeData() and decode() operations. Perhaps a new set of callbacks? nodeDataWithHypervisor(), decodeWithHypervisor() ?
Why? The hypervisor code that calls CPU driver APIs can just decide to do something else for certain architectures.
I'm trying to get the host cpu model for virCapsPtr caps, as it is filled in via virQEMUCapsInit --> virQEMUCapsInitCPU. And subsequently referenced for the output of virsh capabilities and copied into qemuCaps for reference by virsh domcapabilities.
Anyway, host CPU capabilities in virCapsPtr is not supposed to contain the CPU description queried from QEMU. If we have no generic code to detect host CPU ourselves, we should just keep host CPU model empty. The host CPU model queried from QEMU should be placed in virQEMUCapsPtr. When probing QEMU, we cache data we get from QEMU and the host CPU model is supposed to be computed virQEMUCapsInitHostCPUModel from the cached data. (Currently, the function copies the model from virCapsPtr, but that's only because we don't query QEMU for the right stuff yet.) Jirka
CCing Christian
On Thu, Sep 22, 2016 at 14:47:36 -0400, Jason J. Herne wrote:
Testing for runability: - Simply try to start QEMU with KVM, compat machine, CPU model
Yes, if the domain XML explicitly requests a specific CPU model. Additionally, we need to make sure a CPU model chosen by libvirt (host-model) is runnable, but that should be handled by query-cpu-model-expansion.
Identifying host model, e.g. "virsh capabilities" - query-cpu-model-expansion on "host" with "-M none --enable-kvm"
Right, except that it doesn't seem to work on x86_64 now.
It's not critical for query-cpu-model-expansion, but do we have an interface we can use to check whether the new commands are supported by a QEMU binary? Trying and checking for
{"error": {"class": "GenericError", "desc": "this feature or command is not currently supported"}}
is not really possible. At least we'd need a specific error class.
<cpu mode='host-model'>: - simply copy the identified host model
Yes.
<cpu mode='host-passthrough'>: - "-cpu host"
Exactly.
...
1. We will invoke qemu to gather the host cpu data used for virsh capabilities. Today this data seems to be collected directly from the host hardware for most (all?) architectures.
Not really, virsh capabilities will still contain data gathered directly from the host hardware. But, we have virsh domcapabilities for displaying data tight to a specific QEMU binary. Since yesterday afternoon we have support for displaying CPU related data in the domain capabilities XML; see http://libvirt.org/formatdomaincaps.html#elementsCPU
The host-model part of the XML will show the result of query-cpu-model-expansion on "host" model, or the result of querying the hardware directly if we can't ask QEMU for that (which is the current state).
2. virsh cpu-models {arch} will also use a Qemu invocation to gather cpu model data.
No, virsh cpu-models will just list CPU models supported by libvirt (or an empty list if libvirt supports all models supported by QEMU). The CPU model data from QEMU can be found in domain capabilities XML.
3. Most architectures seem to use a "map" (xml file containing cpu model data that ships with Libvirt) to satisfy #1 and #2 above. Our new method does not use this map as it gets all of the data directly from Qemu.
Even if we switch some CPU operations to work through QEMU, we may still need to use the cpu_map.xml file for some other operations, or other hypervisor driver. It all depends on what we need to do for a given architecture. For example, ARM does not use cpu_map.xml at all even now.
Slightly related, I don't think we have a way to list CPU features supported by QEMU over QMP, do we? "-cpu ?" will show them all, but I couldn't find a QMP command that would give me the same list.
4. virsh cpu-baseline and cpu-compare will now invoke qemu directly as well.
Perhaps, but before we can do that, we'll probably need to come up with new APIs. It don't think it's critical, though.
Note: I'm not sure exactly how much all of this will apply just to s390 with other architectures moving over to the new interface if/when they want to. Or if we will want to change all architectures to this new interface at the same time.
Well, IIUC the new commands are not supported on any other architecture right now, are they? Anyway, I don't think we need to switch everything at the same time. If we have a way to detect what commands are supported for a specific QEMU binary, we can implement the code in libvirt and use when we can or falling back to the current way.
Jirka
David
participants (4)
-
David Hildenbrand -
Eduardo Habkost -
Jason J. Herne -
Jiri Denemark