1:15 a.m.
Hello Libvirt,
With more and more ARM servers on the market and increasing amount of
users, we got reported alot that Libvirt cannot provide as much function on
ARM platform as on other platforms(mainly x86), some basic founctions are
missing, such as showing cpu model details and features using ``virsh
capabilities``, comparing cpus etc.
So I would like to propose to extend the capability of ARM cpu driver.
Since I'm quite new here, I searched through the mailing list and did not
find much information on this, so I would like to know that will this be
something that I could work on?
BR,
Zhenyu Zheng
6:46 a.m.
On Wed, 2020-03-04 at 15:15 +0800, Zhenyu Zheng wrote:
Hello Libvirt,
With more and more ARM servers on the market and increasing amount of
users, we got reported alot that Libvirt cannot provide as much
function on ARM platform as on other platforms(mainly x86), some basic
founctions are missing, such as showing cpu model details and features
using ``virsh capabilities``, comparing cpus etc.
So I would like to propose to extend the capability of ARM cpu driver.
Since I'm quite new here, I searched through the mailing list and did
not find much information on this, so I would like to know that will
this be something that I could work on?
In general terms, any gaps between what is possible on x86 and what
is possible on ARM are considered bugs and anyone interested in
addressing them is very much welcome!
On the topic of CPUs specifically, the limitations have been known
for a long time but there are significant obstacles to solving them:
a) QEMU does not have named models for ARM CPUs at this point,
which makes comparing different guest CPUs basically impossible;
b) the Linux kernel does not report information about eg. CPU
frequencly consistently;
c) historically, ARM server vendors have included incorrect or
incomplete information in their products' ACPI and DMI tables.
b) is, I believe, mostly a consequence of c); the latter is also the
reason why libvirt can't really trust this information even if they
happen to be available on a particular machine.
Addressing a) is a long-term to do item for QEMU developers, and it
has seen some progress lately; b) can only be addressed by hardware
vendors.
--
Andrea Bolognani / Red Hat / Virtualization
3:23 a.m.
Hi Andrea Bolognani!
Thank you very much for the reply and guide, they are very helpful. And we
will treat
as this as a long-term task to try to enrich ARM capabilities, both in
libvirt and QEMU
an other related projects.
As for the missing features, maybe I can do it one by one, start with
simple ones,
for example, the simplest feature: "using virsh capabilities" to show host
cpu model,
vendor and features is missing, this is quite simple and currently asked by
our users.
I have a workable draft code that reads and parses /proc/cpuinfo for these
informations,
will thar work? or Libvirt prefers reads them directly from registers like
currently done
in x86 driver?
Thanks again,
Kevin Zheng
On Wed, Mar 4, 2020 at 8:46 PM Andrea Bolognani <abologna(a)redhat.com> wrote:
On Wed, 2020-03-04 at 15:15 +0800, Zhenyu Zheng wrote:
> Hello Libvirt,
>
> With more and more ARM servers on the market and increasing amount of
> users, we got reported alot that Libvirt cannot provide as much
> function on ARM platform as on other platforms(mainly x86), some basic
> founctions are missing, such as showing cpu model details and features
> using ``virsh capabilities``, comparing cpus etc.
>
> So I would like to propose to extend the capability of ARM cpu driver.
> Since I'm quite new here, I searched through the mailing list and did
> not find much information on this, so I would like to know that will
> this be something that I could work on?
In general terms, any gaps between what is possible on x86 and what
is possible on ARM are considered bugs and anyone interested in
addressing them is very much welcome!
On the topic of CPUs specifically, the limitations have been known
for a long time but there are significant obstacles to solving them:
a) QEMU does not have named models for ARM CPUs at this point,
which makes comparing different guest CPUs basically impossible;
b) the Linux kernel does not report information about eg. CPU
frequencly consistently;
c) historically, ARM server vendors have included incorrect or
incomplete information in their products' ACPI and DMI tables.
b) is, I believe, mostly a consequence of c); the latter is also the
reason why libvirt can't really trust this information even if they
happen to be available on a particular machine.
Addressing a) is a long-term to do item for QEMU developers, and it
has seen some progress lately; b) can only be addressed by hardware
vendors.
--
Andrea Bolognani / Red Hat / Virtualization
11:48 a.m.
On Thu, 2020-03-05 at 17:23 +0800, Zhenyu Zheng wrote:
Hi Andrea Bolognani!
Thank you very much for the reply and guide, they are very helpful.
And we will treat as this as a long-term task to try to enrich ARM
capabilities, both in libvirt and QEMU an other related projects.
That sounds good!
As for the missing features, maybe I can do it one by one, start
with
simple ones, for example, the simplest feature: "using virsh
capabilities" to show host cpu model, vendor and features is missing,
this is quite simple and currently asked by our users. I have a
workable draft code that reads and parses /proc/cpuinfo for these
informations, will thar work? or Libvirt prefers reads them directly
from registers like currently done in x86 driver?
Parsing /proc/cpuinfo is fine - all architectures, including x86,
currently do that for at least some of the information.
The problem is that on many ARM machines the file doesn't contain a
whole lot of information, at least not in a form that's suitable for
end users: for example, on this machine that I have access to, all
it contains is
processor : 0
BogoMIPS : 200.00
Features : fp asimd evtstrm aes pmull sha1 sha2 crc32 cpuid
CPU implementer : 0x43
CPU architecture: 8
CPU variant : 0x1
CPU part : 0x0a1
CPU revision : 1
which is definitely not as pretty as "Skylake-Client" or something :)
Another problem is that "virsh capabilities" on x86 will usually
report a CPU model that can then be used as a guest CPU[1], but that's
not really the case for ARM where CPU models are not as well defined
and QEMU definitely doesn't know about all of them.
Same goes with features: on x86 you can add or remove pretty much
any CPU feature from any CPU model, but on ARM QEMU only supports
toggling a small number of CPU features, and even the few CPU models
it knows about are hardly representative of what you'd find in the
average ARM server at this point.
[1] Even though you're not supposed to rely on that, and "virsh
domcapabilities" is much better if that's what you're after.
--
Andrea Bolognani / Red Hat / Virtualization
7:51 p.m.
Thanks alot for the reply!
The problem is that on many ARM machines the file doesn't contain a
whole lot of information, at least not in a form that's suitable
for
end users: for example, on this machine that I have access to, all
it contains is
processor : 0
BogoMIPS : 200.00
Features : fp asimd evtstrm aes pmull sha1 sha2 crc32 cpuid
CPU implementer : 0x43
CPU architecture: 8
CPU variant : 0x1
CPU part : 0x0a1
CPU revision : 1
which is definitely not as pretty as "Skylake-Client" or something :)
As for this, these actually have some mapping rules:
https://developer.arm.com/docs/ddi0595/f/aarch64-system-registers/midr_el1
what we will do is traslate these into human readable infomation, it
has already been done in the ``util-linux/lscpu`` tool:
https://github.com/karelzak/util-linux/blob/master/sys-utils/lscpu-arm.c
we can just do the same in our driver. You could try to download the latest
``util-linux`` in your arm server and have a try :)
Another problem is that "virsh capabilities" on x86 will usually
report a CPU model that can then be used as a guest CPU[1], but
that's
not really the case for ARM where CPU models are not as well defined
and QEMU definitely doesn't know about all of them.
I have try to run the command on1) x86 server 2) ARM server without patch
and
3) ARM server with my patch(please see attachments), the result seems pretty
much the same for this point, or maybe I missed your point here :)
Same goes with features: on x86 you can add or remove pretty much
any CPU feature from any CPU model, but on ARM QEMU only supports
toggling a small number of CPU features, and even the few CPU models
it knows about are hardly representative of what you'd find in the
average ARM server at this point.
So this will be another thing to work on in QEMU/Libvirt in long-term, we
will
look into this. Probably we can first display what we got from
``/proc/cpuinfo``
in the host field, this closes the gap between ARM and X86 a little bit, and
we will move the actual founctionality as you mentioned in the future.
P.S. my current patch is based on libvirt v5.5.0
Thanks again for the reply.
On Fri, Mar 6, 2020 at 1:48 AM Andrea Bolognani <abologna(a)redhat.com> wrote:
> On Thu, 2020-03-05 at 17:23 +0800, Zhenyu Zheng wrote:
> > Hi Andrea Bolognani!
> >
> > Thank you very much for the reply and guide, they are very helpful.
> > And we will treat as this as a long-term task to try to enrich ARM
> > capabilities, both in libvirt and QEMU an other related projects.
>
> That sounds good!
>
> > As for the missing features, maybe I can do it one by one, start with
> > simple ones, for example, the simplest feature: "using virsh
> > capabilities" to show host cpu model, vendor and features is missing,
> > this is quite simple and currently asked by our users. I have a
> > workable draft code that reads and parses /proc/cpuinfo for these
> > informations, will thar work? or Libvirt prefers reads them directly
> > from registers like currently done in x86 driver?
>
> Parsing /proc/cpuinfo is fine - all architectures, including x86,
> currently do that for at least some of the information.
>
> The problem is that on many ARM machines the file doesn't contain a
> whole lot of information, at least not in a form that's suitable for
> end users: for example, on this machine that I have access to, all
> it contains is
>
> processor : 0
> BogoMIPS : 200.00
> Features : fp asimd evtstrm aes pmull sha1 sha2 crc32 cpuid
> CPU implementer : 0x43
> CPU architecture: 8
> CPU variant : 0x1
> CPU part : 0x0a1
> CPU revision : 1
>
> which is definitely not as pretty as "Skylake-Client" or something :)
>
> Another problem is that "virsh capabilities" on x86 will usually
report a CPU model that can then be used as a guest CPU[1], but
that's
not really the case for ARM where CPU models are not as well defined
and QEMU definitely doesn't know about all of them.
>
> Same goes with features: on x86 you can add or remove pretty much
any CPU feature from any CPU model, but on ARM QEMU only supports
toggling a small number of CPU features, and even the few CPU models
it knows about are hardly representative of what you'd find in the
average ARM server at this point.
>
>
> [1] Even though you're not supposed to rely on that, and "virsh
> domcapabilities" is much better if that's what you're after.
> --
> Andrea Bolognani / Red Hat / Virtualization
>
>
10:29 a.m.
On Fri, 2020-03-06 at 09:51 +0800, Zhenyu Zheng wrote:
> The problem is that on many ARM machines the file doesn't
contain a
> whole lot of information, at least not in a form that's suitable for
> end users: for example, on this machine that I have access to, all
> it contains is
> processor : 0
> BogoMIPS : 200.00
> Features : fp asimd evtstrm aes pmull sha1 sha2 crc32 cpuid
> CPU implementer : 0x43
> CPU architecture: 8
> CPU variant : 0x1
> CPU part : 0x0a1
> CPU revision : 1
> which is definitely not as pretty as "Skylake-Client" or something :)
As for this, these actually have some mapping rules:
https://developer.arm.com/docs/ddi0595/f/aarch64-system-registers/midr_el1
what we will do is traslate these into human readable infomation, it
has already been done in the ``util-linux/lscpu`` tool:
https://github.com/karelzak/util-linux/blob/master/sys-utils/lscpu-arm.c
we can just do the same in our driver. You could try to download the latest
``util-linux`` in your arm server and have a try :)
I tried util-linux and lscpu reports reasonable information on at
least one (semi-recent) ARM machine I have access to. I have no idea
how well it would fare with older hardware.
Anyway, despite how impressive the results are, I'm pretty sure we
don't want to incorporate the same sort of translation table into
libvirt. Perhaps calling out to lscpu and parsing its output would
be a more acceptable solution...
> Another problem is that "virsh capabilities" on x86
will usually
> report a CPU model that can then be used as a guest CPU[1], but that's
> not really the case for ARM where CPU models are not as well defined
> and QEMU definitely doesn't know about all of them.
I have try to run the command on1) x86 server 2) ARM server without patch and
3) ARM server with my patch(please see attachments), the result seems pretty
much the same for this point, or maybe I missed your point here :)
The difference is that, on x86, you can mostly take the <host><cpu>
element from the capabilities XML and use it as <domain><cpu> in the
domain XML: in your case that would be
<domain>
<cpu>
<model>Broadwell-IBRS</model>
<topology sockets='1' cores='4' threads='2'/>
<feature name='invtsc'/>
<feature name='avx512cd'/>
<feature name='avx512f'/>
<feature name='mpx'/>
<feature name='ssbd'/>
<feature name='md-clear'/>
<feature name='abm'/>
<feature name='pdpe1gb'/>
<feature name='hypervisor'/>
<feature name='rdrand'/>
<feature name='f16c'/>
<feature name='osxsave'/>
<feature name='ht'/>
<feature name='ss'/>
<feature name='vme'/>
</cpu>
Again, this is not a good idea in general and you should rather look
at the domaincapabilities XML instead. So I don't think it's a
problem if the capabilities XML contains values that can't be used
for guests, but you have to ensure the same is not true when it comes
to the domcapabilities XML.
> Same goes with features: on x86 you can add or remove pretty
much
> any CPU feature from any CPU model, but on ARM QEMU only supports
> toggling a small number of CPU features, and even the few CPU models
> it knows about are hardly representative of what you'd find in the
> average ARM server at this point.
So this will be another thing to work on in QEMU/Libvirt in long-term, we will
look into this. Probably we can first display what we got from ``/proc/cpuinfo``
in the host field, this closes the gap between ARM and X86 a little bit, and
we will move the actual founctionality as you mentioned in the future.
As long as none of this leaks into the domcapabilities XML, I don't
see a problem in exposing more information about the host.
P.S. my current patch is based on libvirt v5.5.0
That's a pretty old version. Please make sure your patch applies on
top of the master branch, or it won't be possible to accept it into
libvirt!
--
Andrea Bolognani / Red Hat / Virtualization
9:06 p.m.
Hi Andrea,
Thanks again for the reply, I've already re-implement the codes on the
master branch last week and trying to get ready to post it in the
mailing list.
As for calling lscpu, it does show a very good textual information, I will
have a try using that as the source.
BR,
Zhenyu
On Mon, Mar 23, 2020 at 11:29 PM Andrea Bolognani <abologna(a)redhat.com>
wrote:
On Fri, 2020-03-06 at 09:51 +0800, Zhenyu Zheng wrote:
> > The problem is that on many ARM machines the file doesn't contain a
> > whole lot of information, at least not in a form that's suitable for
> > end users: for example, on this machine that I have access to, all
> > it contains is
> > processor : 0
> > BogoMIPS : 200.00
> > Features : fp asimd evtstrm aes pmull sha1 sha2 crc32 cpuid
> > CPU implementer : 0x43
> > CPU architecture: 8
> > CPU variant : 0x1
> > CPU part : 0x0a1
> > CPU revision : 1
> > which is definitely not as pretty as "Skylake-Client" or something
:)
>
> As for this, these actually have some mapping rules:
>
https://developer.arm.com/docs/ddi0595/f/aarch64-system-registers/midr_el1
> what we will do is traslate these into human readable infomation, it
> has already been done in the ``util-linux/lscpu`` tool:
> https://github.com/karelzak/util-linux/blob/master/sys-utils/lscpu-arm.c
> we can just do the same in our driver. You could try to download the
latest
> ``util-linux`` in your arm server and have a try :)
I tried util-linux and lscpu reports reasonable information on at
least one (semi-recent) ARM machine I have access to. I have no idea
how well it would fare with older hardware.
Anyway, despite how impressive the results are, I'm pretty sure we
don't want to incorporate the same sort of translation table into
libvirt. Perhaps calling out to lscpu and parsing its output would
be a more acceptable solution...
> > Another problem is that "virsh capabilities" on x86 will usually
> > report a CPU model that can then be used as a guest CPU[1], but that's
> > not really the case for ARM where CPU models are not as well defined
> > and QEMU definitely doesn't know about all of them.
>
> I have try to run the command on1) x86 server 2) ARM server without
patch and
> 3) ARM server with my patch(please see attachments), the result seems
pretty
> much the same for this point, or maybe I missed your point here :)
The difference is that, on x86, you can mostly take the <host><cpu>
element from the capabilities XML and use it as <domain><cpu> in the
domain XML: in your case that would be
<domain>
<cpu>
<model>Broadwell-IBRS</model>
<topology sockets='1' cores='4' threads='2'/>
<feature name='invtsc'/>
<feature name='avx512cd'/>
<feature name='avx512f'/>
<feature name='mpx'/>
<feature name='ssbd'/>
<feature name='md-clear'/>
<feature name='abm'/>
<feature name='pdpe1gb'/>
<feature name='hypervisor'/>
<feature name='rdrand'/>
<feature name='f16c'/>
<feature name='osxsave'/>
<feature name='ht'/>
<feature name='ss'/>
<feature name='vme'/>
</cpu>
Again, this is not a good idea in general and you should rather look
at the domaincapabilities XML instead. So I don't think it's a
problem if the capabilities XML contains values that can't be used
for guests, but you have to ensure the same is not true when it comes
to the domcapabilities XML.
> > Same goes with features: on x86 you can add or remove pretty much
> > any CPU feature from any CPU model, but on ARM QEMU only supports
> > toggling a small number of CPU features, and even the few CPU models
> > it knows about are hardly representative of what you'd find in the
> > average ARM server at this point.
>
> So this will be another thing to work on in QEMU/Libvirt in long-term,
we will
> look into this. Probably we can first display what we got from
``/proc/cpuinfo``
> in the host field, this closes the gap between ARM and X86 a little bit,
and
> we will move the actual founctionality as you mentioned in the future.
As long as none of this leaks into the domcapabilities XML, I don't
see a problem in exposing more information about the host.
> P.S. my current patch is based on libvirt v5.5.0
That's a pretty old version. Please make sure your patch applies on
top of the master branch, or it won't be possible to accept it into
libvirt!
--
Andrea Bolognani / Red Hat / Virtualization
1704
days inactive
1724
days old
6 comments
2 participants
participants (2)
-
Andrea Bolognani -
Zhenyu Zheng