[libvirt] [RFC] Add support for QEMU's l3-cache CPU property
Hi, Apparently, reporting a level 3 cache on a virtual CPU can dramatically increase performance in some use cases [1]. The interesting part is that l3-cache=on does not provide the real CPU cache data, it's just making it up. Anyway, we should be able to enable this via libvirt. And since there is another property which enables real CPU cache data to be passed to a guest, I suggest the following /domain/cpu/cache element equivalent to l3-cache=on: <cache level='3' mode='emulate'/> If we need to add support for passing the real CPU cache data, we can do that with <cache level='3' mode='passthrough'/> or we can even make the level attribute optional and support <cache mode='passthrough'/> Missing cache element means default behaviour of the hypervisor and we can eventually add <cache mode='disable'/> to turn off passing any CPU cache info to the guest. But I think we should now focus only on <cache level='3' mode='emulate'/> and leaving the rest for the future when we actually need it. This is how a more complete example would look like: <cpu mode='custom' match='exact'> <model>Broadwell</model> <cache level='3' mode='emulate'/> </cpu> And libvirt would translate it into -cpu Broadwell,l3-cache=on. Do you have any thoughts about the XML schema or naming? Thanks, Jirka [1] https://patchwork.kernel.org/patch/9308401/
On Tue, Apr 18, 2017 at 04:22:36PM +0200, Jiri Denemark wrote:
Hi,
Apparently, reporting a level 3 cache on a virtual CPU can dramatically increase performance in some use cases [1]. The interesting part is that l3-cache=on does not provide the real CPU cache data, it's just making it up. Anyway, we should be able to enable this via libvirt. And since there is another property which enables real CPU cache data to be passed to a guest, I suggest the following /domain/cpu/cache element equivalent to l3-cache=on:
<cache level='3' mode='emulate'/>
If we need to add support for passing the real CPU cache data, we can do that with
<cache level='3' mode='passthrough'/>
or we can even make the level attribute optional and support
<cache mode='passthrough'/>
Missing cache element means default behaviour of the hypervisor and we can eventually add <cache mode='disable'/> to turn off passing any CPU cache info to the guest.
But I think we should now focus only on <cache level='3' mode='emulate'/> and leaving the rest for the future when we actually need it.
This is how a more complete example would look like:
<cpu mode='custom' match='exact'> <model>Broadwell</model> <cache level='3' mode='emulate'/> </cpu>
And libvirt would translate it into -cpu Broadwell,l3-cache=on.
Do you have any thoughts about the XML schema or naming?
The second QEMU property 'host-cache-info' causes the guest to see the host cache topology. This affects L1, L2 and L3 caches all at once. We could allow use of '<cache>' without specifying a level. ie <cache mode="passthrough"/> to indicate passthrough of L1,L2 & L3 cache all together, mapping to host-cache-info=on. and the <cache level=3 mode=emulate> mapping to the l3-cache=on. These two elements would need to be mutually exclusive. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Thu, Apr 20, 2017 at 10:02:38 +0100, Daniel P. Berrange wrote:
On Tue, Apr 18, 2017 at 04:22:36PM +0200, Jiri Denemark wrote:
Hi,
Apparently, reporting a level 3 cache on a virtual CPU can dramatically increase performance in some use cases [1]. The interesting part is that l3-cache=on does not provide the real CPU cache data, it's just making it up. Anyway, we should be able to enable this via libvirt. And since there is another property which enables real CPU cache data to be passed to a guest, I suggest the following /domain/cpu/cache element equivalent to l3-cache=on:
<cache level='3' mode='emulate'/>
If we need to add support for passing the real CPU cache data, we can do that with
<cache level='3' mode='passthrough'/>
or we can even make the level attribute optional and support
<cache mode='passthrough'/>
Missing cache element means default behaviour of the hypervisor and we can eventually add <cache mode='disable'/> to turn off passing any CPU cache info to the guest.
But I think we should now focus only on <cache level='3' mode='emulate'/> and leaving the rest for the future when we actually need it.
This is how a more complete example would look like:
<cpu mode='custom' match='exact'> <model>Broadwell</model> <cache level='3' mode='emulate'/> </cpu>
And libvirt would translate it into -cpu Broadwell,l3-cache=on.
Do you have any thoughts about the XML schema or naming?
The second QEMU property 'host-cache-info' causes the guest to see the host cache topology. This affects L1, L2 and L3 caches all at once.
We could allow use of '<cache>' without specifying a level. ie
<cache mode="passthrough"/>
Yes, this is what I suggested a few lines above. However, it seems host-cache-info is only supported for -cpu host.
to indicate passthrough of L1,L2 & L3 cache all together, mapping to host-cache-info=on. and the <cache level=3 mode=emulate> mapping to the l3-cache=on.
These two elements would need to be mutually exclusive.
Mixing them could allow host cache info to be passed through with specific level-N cache data overridden by additional <cache level='N'/> element. QEMU does not complain of both host-cache-info and l3-cache are enabled. But anyway, making them mutually exclusive is probably the safest option. However, what would happen if <cpu mode='host-passthrough'> <cache level='3' mode='emulate'/> </cpu> is specified in domain XML? -cpu host,l3-cache=on or -cpu host,host-cache-info=off,l3-cache=on And similarly, I guess <cache mode="passthrough"/> should just unconditionally disable l3-cache if we want to be sure they are always mutually exclusive. Jirka
On Thu, Apr 20, 2017 at 11:53:21AM +0200, Jiri Denemark wrote:
On Thu, Apr 20, 2017 at 10:02:38 +0100, Daniel P. Berrange wrote:
On Tue, Apr 18, 2017 at 04:22:36PM +0200, Jiri Denemark wrote:
Hi,
Apparently, reporting a level 3 cache on a virtual CPU can dramatically increase performance in some use cases [1]. The interesting part is that l3-cache=on does not provide the real CPU cache data, it's just making it up. Anyway, we should be able to enable this via libvirt. And since there is another property which enables real CPU cache data to be passed to a guest, I suggest the following /domain/cpu/cache element equivalent to l3-cache=on:
<cache level='3' mode='emulate'/>
If we need to add support for passing the real CPU cache data, we can do that with
<cache level='3' mode='passthrough'/>
or we can even make the level attribute optional and support
<cache mode='passthrough'/>
Missing cache element means default behaviour of the hypervisor and we can eventually add <cache mode='disable'/> to turn off passing any CPU cache info to the guest.
But I think we should now focus only on <cache level='3' mode='emulate'/> and leaving the rest for the future when we actually need it.
This is how a more complete example would look like:
<cpu mode='custom' match='exact'> <model>Broadwell</model> <cache level='3' mode='emulate'/> </cpu>
And libvirt would translate it into -cpu Broadwell,l3-cache=on.
Do you have any thoughts about the XML schema or naming?
The second QEMU property 'host-cache-info' causes the guest to see the host cache topology. This affects L1, L2 and L3 caches all at once.
We could allow use of '<cache>' without specifying a level. ie
<cache mode="passthrough"/>
Yes, this is what I suggested a few lines above. However, it seems host-cache-info is only supported for -cpu host.
I don't think there's any dependancy on -cpu host actually. Semantically host-cache-info=on, only makes sense if the sockets/core/threads topology of the guest, matches the host. What other CPUID bits you choose is tangential.
to indicate passthrough of L1,L2 & L3 cache all together, mapping to host-cache-info=on. and the <cache level=3 mode=emulate> mapping to the l3-cache=on.
These two elements would need to be mutually exclusive.
Mixing them could allow host cache info to be passed through with specific level-N cache data overridden by additional <cache level='N'/> element. QEMU does not complain of both host-cache-info and l3-cache are enabled. But anyway, making them mutually exclusive is probably the
QEMU /should/ complain - the code completely ignores l3-cache=on if you gave host-cache-info=on.
safest option. However, what would happen if
<cpu mode='host-passthrough'> <cache level='3' mode='emulate'/> </cpu>
is specified in domain XML?
-cpu host,l3-cache=on
or
-cpu host,host-cache-info=off,l3-cache=on
host-cache-info default to off, but having libvirt explicitly give host-cache-info=off would be the paranoid safe approach.
And similarly, I guess <cache mode="passthrough"/> should just unconditionally disable l3-cache if we want to be sure they are always mutually exclusive.
Yes, l3-cache defaults to on for new machine types, so for safety sake we should explicitly give l3-cache=off, even though QEMU would ignore it (for now). Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Thu, Apr 20, 2017 at 11:07:27 +0100, Daniel P. Berrange wrote:
On Thu, Apr 20, 2017 at 11:53:21AM +0200, Jiri Denemark wrote:
On Thu, Apr 20, 2017 at 10:02:38 +0100, Daniel P. Berrange wrote:
On Tue, Apr 18, 2017 at 04:22:36PM +0200, Jiri Denemark wrote:
Hi,
Apparently, reporting a level 3 cache on a virtual CPU can dramatically increase performance in some use cases [1]. The interesting part is that l3-cache=on does not provide the real CPU cache data, it's just making it up. Anyway, we should be able to enable this via libvirt. And since there is another property which enables real CPU cache data to be passed to a guest, I suggest the following /domain/cpu/cache element equivalent to l3-cache=on:
<cache level='3' mode='emulate'/>
If we need to add support for passing the real CPU cache data, we can do that with
<cache level='3' mode='passthrough'/>
or we can even make the level attribute optional and support
<cache mode='passthrough'/>
Missing cache element means default behaviour of the hypervisor and we can eventually add <cache mode='disable'/> to turn off passing any CPU cache info to the guest.
But I think we should now focus only on <cache level='3' mode='emulate'/> and leaving the rest for the future when we actually need it.
This is how a more complete example would look like:
<cpu mode='custom' match='exact'> <model>Broadwell</model> <cache level='3' mode='emulate'/> </cpu>
And libvirt would translate it into -cpu Broadwell,l3-cache=on.
Do you have any thoughts about the XML schema or naming?
The second QEMU property 'host-cache-info' causes the guest to see the host cache topology. This affects L1, L2 and L3 caches all at once.
We could allow use of '<cache>' without specifying a level. ie
<cache mode="passthrough"/>
Yes, this is what I suggested a few lines above. However, it seems host-cache-info is only supported for -cpu host.
I don't think there's any dependancy on -cpu host actually.
Semantically host-cache-info=on, only makes sense if the sockets/core/threads topology of the guest, matches the host. What other CPUID bits you choose is tangential.
QEMU currently supports host-cache-info only for -cpu host. "Property '.host-cache-info' not found" if you try with any other CPU model. And device-list-properties QMP command confirms that, only host-x86_64-cpu type lists host-cache-info property. We'd probably need to check whether the property is available for the specific CPU model :( Jirka
On Thu, Apr 20, 2017 at 02:44:00PM +0200, Jiri Denemark wrote:
On Thu, Apr 20, 2017 at 11:07:27 +0100, Daniel P. Berrange wrote:
On Thu, Apr 20, 2017 at 11:53:21AM +0200, Jiri Denemark wrote:
On Thu, Apr 20, 2017 at 10:02:38 +0100, Daniel P. Berrange wrote:
On Tue, Apr 18, 2017 at 04:22:36PM +0200, Jiri Denemark wrote:
Hi,
Apparently, reporting a level 3 cache on a virtual CPU can dramatically increase performance in some use cases [1]. The interesting part is that l3-cache=on does not provide the real CPU cache data, it's just making it up. Anyway, we should be able to enable this via libvirt. And since there is another property which enables real CPU cache data to be passed to a guest, I suggest the following /domain/cpu/cache element equivalent to l3-cache=on:
<cache level='3' mode='emulate'/>
If we need to add support for passing the real CPU cache data, we can do that with
<cache level='3' mode='passthrough'/>
or we can even make the level attribute optional and support
<cache mode='passthrough'/>
Missing cache element means default behaviour of the hypervisor and we can eventually add <cache mode='disable'/> to turn off passing any CPU cache info to the guest.
But I think we should now focus only on <cache level='3' mode='emulate'/> and leaving the rest for the future when we actually need it.
This is how a more complete example would look like:
<cpu mode='custom' match='exact'> <model>Broadwell</model> <cache level='3' mode='emulate'/> </cpu>
And libvirt would translate it into -cpu Broadwell,l3-cache=on.
Do you have any thoughts about the XML schema or naming?
The second QEMU property 'host-cache-info' causes the guest to see the host cache topology. This affects L1, L2 and L3 caches all at once.
We could allow use of '<cache>' without specifying a level. ie
<cache mode="passthrough"/>
Yes, this is what I suggested a few lines above. However, it seems host-cache-info is only supported for -cpu host.
I don't think there's any dependancy on -cpu host actually.
Semantically host-cache-info=on, only makes sense if the sockets/core/threads topology of the guest, matches the host. What other CPUID bits you choose is tangential.
QEMU currently supports host-cache-info only for -cpu host. "Property '.host-cache-info' not found" if you try with any other CPU model. And device-list-properties QMP command confirms that, only host-x86_64-cpu type lists host-cache-info property. We'd probably need to check whether the property is available for the specific CPU model :(
Ah, I misread the code. The code which handles the property exists in the generic code paths, but the property itself is only registered for host model (and the new "max" model). Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
participants (2)
-
Daniel P. Berrange -
Jiri Denemark