On Mon, Dec 14, 2015 at 12:31:59PM +0100, Christoffer Dall wrote: > Hi all, > > I'm trying to figure out what the correct solution for libvirt support > for ARM KVM guests with GICv3 is. > > The challenge is that QEMU's virt machine model defaults to GICv2 unless > you specify an additional "-machine gic-version=host" parameter to QEMU, > in which case your VM gets the same GIC version as your host. > > Now, I'm told that this is inconvenient for libvirt, because it > typically does not pass any -machine options, is that correct?

If that is correct, then my guess is that that's because it's used to the qemu machine model automatically selecting appropriate defaults, and for pc, due to versioning, there are several machine model types to choose from. mach-virt hasn't started versioning yet. > > Further, there are two additional complications: First, some guest > images may not include GICv3 support. For Linux, this is relatively old > kernels (prior to v3.17), but other guests may only support GICv2. > > Second, some hardware platforms only support GICv2 guests, some only > support GICv3 guests, and some support both GICv2 and GICv3 guests > (those hosts with a GICv3 with the backwards compatibility support). > > It is unclear to me how libvirt users will relate to these constraints. > For example, in an OpenStack deployment, will it somehow be easy to tell > which nodes support which GIC version such that administrators can > choose compatible VM images, or how does that work? libvirt has the concept of capabilities that it can negotiate. I know this is used heavily with x86 cpu features that some guests depend on. I think a guest dependency on gic version could fit here. > > The end goal here is to determine if we need to add any functionality to > QEMU to better support libvirt, and what kind of features/glue need to > be added to libvirt for this to work. I've been starting to wonder if we should consider breaking mach-virt into multiple types. For example, one that focuses on big, enterprise-y type configs (only 64-bit guests, only virtio-pci, extending the size of main memory beyond 30G, for sure, and possibly even moving the PCIE_MMIO_HIGH mapping to extend beyond 511G. It'd even be good to remap the redistributor space to get more than 123 vcpus. Making new mach-virt types and versioning those types would probably avoid needing much libvirt changes, but teaching libvirt (if it doesn't know already) how to manage machine parameters to effectively create new types as needed, also sounds good to me. Thanks, drew -- libvir-list mailing list libvir-list(a)redhat.com https://www.redhat.com/mailman/listinfo/libvir-list

On 15 December 2015 at 09:36, Martin Kletzander <mkletzan(a)redhat.com&gt; wrote: > We do pass some options, for example, you can restrict the GIC to v2: > https://libvirt.org/formatdomain.html#elementsFeatures > > That could be modified to work for your purpose IIUC, right? What does that option do to the QEMU command line? The documentation doesn't seem to say.

thanks -- PMM

On 15 December 2015 at 14:12, Martin Kletzander <mkletzan(a)redhat.com&gt; wrote: > On Tue, Dec 15, 2015 at 09:41:02AM +0000, Peter Maydell wrote: >> >> On 15 December 2015 at 09:36, Martin Kletzander <mkletzan(a)redhat.com&gt; >> wrote: >>> >>> We do pass some options, for example, you can restrict the GIC to v2: >>> https://libvirt.org/formatdomain.html#elementsFeatures >>> >>> That could be modified to work for your purpose IIUC, right? >> >> >> What does that option do to the QEMU command line? The documentation >> doesn't seem to say. >> > > It appends gic-version= to the machine option, if bunch of conditions > are met -- it has to be ARM machine, it has to be something else than > version 2, and so on, more details are visible in the code and I won't > cover them here. I hope that's understandable. OK. So it should be sufficient just to have <gic version='host'/> which will then provide whatever VGIC the host machine can do with hardware acceleration.

On 15 December 2015 at 16:35, Andrew Jones <drjones(a)redhat.com&gt; wrote: > This is probably good for guests that happy with both. Guests that > need/want a specific choice will put their integer there, and then > we need a way to do a capabilities check before launching that guest > on an arbitrary host. OK, so how do we typically do that? I notice I have a 'kvm-ok' script on my machine which helpfully reports things like whether KVM is enabled, and it seems like it might be helpful to extend that to know a bit more about ARM hosts. But I'm guessing libvirt doesn't use that for its capability checking ?

On 15 December 2015 at 16:57, Andre Przywara <andre.przywara(a)arm.com&gt; wrote: > Even that wouldn't help us, I guess, as you cannot easily check for > GICv3/GICv2 compatibility with a _script_. Having access to ioctl's make > this pretty easy though: Just try to call KVM_CREATE_DEVICE with the > proper type and get -ENODEV if this one is not supported. This can be > done without any extra userland tool by just executing some ioctls on > /dev/kvm (from C or using some helper library). kvm-ok already runs a few external helper binaries for some things. (Also you can do ioctls from a script if it's a perl script ;-)) As you say the actual technical details of how to query for the host's current supported functionality are straightforward, so it's just a question of how libvirt is expecting that to be exposed to it.

thanks -- PMM

On 15 December 2015 at 16:35, Andrew Jones <drjones(a)redhat.com&gt; wrote: > This is probably good for guests that happy with both. Guests that > need/want a specific choice will put their integer there, and then > we need a way to do a capabilities check before launching that guest > on an arbitrary host. OK, so how do we typically do that? I notice I have a 'kvm-ok' script on my machine which helpfully reports things like whether KVM is enabled, and it seems like it might be helpful to extend that to know a bit more about ARM hosts. But I'm guessing libvirt doesn't use that for its capability checking ?

On Sun, Dec 20, 2015 at 04:49:20PM +0100, Kashyap Chamarthy wrote: > On Tue, Dec 15, 2015 at 04:42:13PM +0000, Peter Maydell wrote: > > On 15 December 2015 at 16:35, Andrew Jones <drjones(a)redhat.com&gt; wrote: > > > This is probably good for guests that happy with both. Guests that > > > need/want a specific choice will put their integer there, and then > > > we need a way to do a capabilities check before launching that guest > > > on an arbitrary host. > > > > OK, so how do we typically do that? I notice I have a 'kvm-ok' > > script on my machine which helpfully reports things like whether > > KVM is enabled, and it seems like it might be helpful to extend > > that to know a bit more about ARM hosts. But I'm guessing libvirt > > doesn't use that for its capability checking ? > > libvirt ships a tool called `virt-host-validate` that performs a bunch > of checks along with whether KVM is enabled or not in the BIOS: > > $ sudo virt-host-validate > QEMU: Checking for hardware virtualization : PASS > QEMU: Checking for device /dev/kvm : PASS > QEMU: Checking for device /dev/vhost-net : PASS > QEMU: Checking for device /dev/net/tun : PASS > LXC: Checking for Linux >= 2.6.26 : PASS > > The only ARM machine I have access to is an AArch64 one, running on it > results in: > > $ sudo virt-host-validate > QEMU: Checking for hardware virtualization : WARN (Only emulated CPUs are available, performance will be significantly limited) This check needs to be fixed to work on Arm hosts, or indeed any non-x86 hosts. It currently checks for 'svm' and 'vmx' flags in CPU.

On Mon, 2015-12-21 at 12:03 +0100, Christoffer Dall wrote: >  > So let me summarize the things we need to do: >  >  - patch libvirt to allow <gic version='host' /> >  - patch libvirt to properly probe for GICv2 and GICv3 support >  - patch libvirt to generally handle virt-host-validate on ARM >  > And that should be enough for other tools such as OpenStack with Nova to > probe and create the VMs it wishes. >  > Does that about capture it? Sounds about right. I can work on points 1) and 3) of your list on my own, but for point 2) we need QEMU to start probing the host for supported GIC versions and expose such information before libvirt can use it. Then of course higher level tools such as OpenStack will need to become aware of these new features in libvirt.

On Tue, 2016-01-05 at 13:08 +0100, Christoffer Dall wrote: > On Tue, Jan 05, 2016 at 12:37:02PM +0100, Andrea Bolognani wrote: > > On Mon, 2015-12-21 at 12:03 +0100, Christoffer Dall wrote: > > >   > > > So let me summarize the things we need to do: > > >   > > >   - patch libvirt to allow <gic version='host' /> > > >   - patch libvirt to properly probe for GICv2 and GICv3 support > > >   - patch libvirt to generally handle virt-host-validate on ARM > > >   > > > And that should be enough for other tools such as OpenStack with Nova to > > > probe and create the VMs it wishes. > > >   > > > Does that about capture it? > >  > > Sounds about right. > >  > > I can work on points 1) and 3) of your list on my own, but for point 2) > > we need QEMU to start probing the host for supported GIC versions and > > expose such information before libvirt can use it. Then of course higher > > level tools such as OpenStack will need to become aware of these new > > features in libvirt. > >  > I thought the virt-host-validate tool in libvirt should do the probing > directly? The tool is just a convenience for the administrator (and for higher level tools?), and it can't be where the validation happens for two reasons:   - it doesn't take any argument, so it can't possibly know whether     you're interested in running GICv2 or GICv3 guests

  - libvirt needs to check before running the guest anyway

Probing inside QEMU means libvirt can just ask it instead of implementing the checks itself, which means less code duplication; more importantly, this prevents the possibility of the checks ever going out of sync. > If I misunderstood, how would you like QEMU to report this? I'm not 100% sure about the best interface here (hopefully other people can weigh in) but I imagine there would be some sort of QMP command that lists the GIC versions supported by the QEMU binary, or some text in the output of 'qemu -help' that can be parsed to obtain such list.

> Should it > be the qemu binary directly or the script that Peter suggested? I think it should be the QEMU binary, especially since I assume QEMU itself will want to error out if an unsupported GIC version is requested by the user - there's no guarantee QEMU will be called by libvirt, so it will have to check on its own.

On Tue, 2016-01-05 at 14:47 +0100, Christoffer Dall wrote: > Why can libvirt not just run the tool and parse its output? That's what > I understood was done for other things, but perhaps I misunderstood. It's the other way around: the tool uses libvirt to implement its checks. Running a command and parsing its output makes sense when the binary (as is the case for QEMU) is developed by a third party; if you control both pieces, it's usually better to stick to C and avoid spawning extra processes.

I realized the information is already sort of exposed by QEMU: $ qemu-system-aarch64 -machine virt,? ... virt.gic-version=string (Set GIC version. Valid values are 2, 3 and host) ... Not the best format to parse, but the information is there. And we already have some code to extract such parameters from QEMU using QMP's query-command-line-options command![1]

On Tue, 2016-01-05 at 18:14 +0000, Peter Maydell wrote: > > > Why can libvirt not just run the tool and parse its output?  That's what > > > I understood was done for other things, but perhaps I misunderstood. > >  > > It's the other way around: the tool uses libvirt to implement its > > checks. Running a command and parsing its output makes sense when the > > binary (as is the case for QEMU) is developed by a third party; if > > you control both pieces, it's usually better to stick to C and avoid > > spawning extra processes. >  > That argument suggests libvirt should just go ahead and do the > ioctls to check for GICv2/v3 support, rather than spawning a > separate QEMU binary to do the check for it ? libvirt already has to spawn QEMU and run QMP commands to probe for dozens of other capabilities, so we have very robust code to do that. Most importantly, as I wrote in a previous mail, if the check is done by QEMU we remove the risk of it and libvirt ever disagreeing.

That's correct, having a QMP command that lists the values gic-version can have on the current host would be just great. If we had that, we could validate the GIC version chosen for a guest, and expose it in the capabilities XML so that higher-level tools can provide a list of choices to the user. Please note that this QMP command would have to work regardless of the machine type selected on QEMU's command line, because libvirt always runs a QEMU binary with '-M none' when probing its capabilities.

On Wed, Jan 06, 2016 at 01:30:16PM +0000, Peter Maydell wrote: > On 6 January 2016 at 12:49, Andrea Bolognani <abologna(a)redhat.com&gt; wrote: > > That's correct, having a QMP command that lists the values gic-version > > can have on the current host would be just great. > >  > > If we had that, we could validate the GIC version chosen for a guest, > > and expose it in the capabilities XML so that higher-level tools can > > provide a list of choices to the user. > >  > > Please note that this QMP command would have to work regardless of the > > machine type selected on QEMU's command line, because libvirt always > > runs a QEMU binary with '-M none' when probing its capabilities. >  > On the other hand, if you don't tell us the machine type you care > about then we can't tell you: >  (a) "this machine type doesn't support setting this property at all" >      (which applies to machines like vexpress-a15 which you can use with >      KVM on 32-bit hosts, and of course also to all the non-KVM models)

>  (b) "this machine type only supports GIC versions X and Y even if the >       host supports more" (this is currently only hypothetical, though, >       since we only have the property on 'virt'. it would only happen >       if in the future we needed something other than '2' or '3' or >       'host' I think.)

> If you use -M none then we can only tell you information you > could have got yourself by directly asking the kernel about it.

So where does this leave us?  Do we need a QMP command or not?   It sounds to me like we should implement one, and the answer depends on the machine option chosen, where -M none gives you the generic capabilities of the kernel and for other machines it tells you what's possible?

On Tue, Jan 12, 2016 at 04:48:27PM +0100, Andrea Bolognani wrote: > Would providing a QMP command that returns the list of GIC versions > available on the current host for the current machine type be > acceptable from QEMU's point of view? > Adding qemu-devel. Peter any opinion on this? If it sounds like a reasonable idea then either I or Wei can look into sending the patches.

On 6 January 2016 at 12:49, Andrea Bolognani <abologna(a)redhat.com&gt; wrote: > That's correct, having a QMP command that lists the values gic-version > can have on the current host would be just great. > > If we had that, we could validate the GIC version chosen for a guest, > and expose it in the capabilities XML so that higher-level tools can > provide a list of choices to the user. > > Please note that this QMP command would have to work regardless of the > machine type selected on QEMU's command line, because libvirt always > runs a QEMU binary with '-M none' when probing its capabilities. On the other hand, if you don't tell us the machine type you care about then we can't tell you: (a) "this machine type doesn't support setting this property at all" (which applies to machines like vexpress-a15 which you can use with KVM on 32-bit hosts, and of course also to all the non-KVM models)

(b) "this machine type only supports GIC versions X and Y even if the host supports more" (this is currently only hypothetical, though, since we only have the property on 'virt'. it would only happen if in the future we needed something other than '2' or '3' or 'host' I think.)

On Fri, Jan 22, 2016 at 02:44:32PM +0000, Daniel P. Berrange wrote: > On Wed, Jan 06, 2016 at 01:30:16PM +0000, Peter Maydell wrote: > > On 6 January 2016 at 12:49, Andrea Bolognani <abologna(a)redhat.com&gt; wrote: > > > That's correct, having a QMP command that lists the values gic-version > > > can have on the current host would be just great. > > > > > > If we had that, we could validate the GIC version chosen for a guest, > > > and expose it in the capabilities XML so that higher-level tools can > > > provide a list of choices to the user. > > > > > > Please note that this QMP command would have to work regardless of the > > > machine type selected on QEMU's command line, because libvirt always > > > runs a QEMU binary with '-M none' when probing its capabilities. > > > > On the other hand, if you don't tell us the machine type you care > > about then we can't tell you: > > (a) "this machine type doesn't support setting this property at all" > > (which applies to machines like vexpress-a15 which you can use with > > KVM on 32-bit hosts, and of course also to all the non-KVM models) > > We have just recently merged support for registering properties against > classes instead of object instances. There is also a proposed command > to allow querying the list of properties against a class > > https://lists.gnu.org/archive/html/qemu-devel/2016-01/msg04348.html > > So if we now update the machine types to register their properties against > the class instead of object, then we can query what properties are present > against each machine type, while still using '-M none'. > > > (b) "this machine type only supports GIC versions X and Y even if the > > host supports more" (this is currently only hypothetical, though, > > since we only have the property on 'virt'. it would only happen > > if in the future we needed something other than '2' or '3' or > > 'host' I think.) > > Our introspection support in QOM only allows us to say that a property > is a particular type (int / enum / str / whatever). We don't have any > way to expose info about what subset of possible values for a type are > permitted. So I don't see any near term way to inform apps that the > gic property accepts values x, y and but not z. > > IMHO, we shouldn't try to overthink this. Libvirt can query the host > to find out what GIC versions are supported and default to using the > most recent version, on the basis that people are likely to have a > matching QEMU. We can just rely on QEMU to report error if we pass > it a version it doesn't support and not try to pre-emptively check > before launch. The key is just getting the default right IMHO. > This sounds fine to me. However, not being familiar with the internals of libvirt I really can't just what the right implementation approach here is. As I understand we need to either: 1) Add a QMP command that lets you ask for -M virt, if GIC version X is supported 2) Just implement something in libvirt that checks what the kernel supports directly via the well-defined KVM interface and chooses the highest supported version per default. To me it sounds like we should just go ahead with (2) and document somehwere that if you get an error, you need to either manually change the gic version setting in your VM definition file or upgrade QEMU. Can someone with libvirt authority please advice whether (1) or (2) is what we need to do?

This actually doesn't matter for the v2 vs. v3 case. The gic-version property doesn't exist at all for v2-only QEMU. Although maybe the gic-version property should be reworked. Instead of gic-version=<highest-supported>, we could create one boolean property per version supported, i.e. gicv2, gicv3, gicv4...

I agree we should have a good argument to justify messing with it, but until we start versioning mach-virt, then I don't think anybody should depend on mach-virt command lines working everywhere. Every time we add a new property, and a user adds it to their command line, then they've just broken that command line for older QEMU. Yes, I know the upgrade path is a bit different, but still... Anyway, properties are changeable with versioned machines.

On Tue, Feb 02, 2016 at 01:59:26PM +0100, Andrew Jones wrote: > On Tue, Feb 02, 2016 at 12:10:10PM +0000, Daniel P. Berrange wrote: > > On Tue, Feb 02, 2016 at 12:49:33PM +0100, Christoffer Dall wrote: > > > On Fri, Jan 22, 2016 at 02:44:32PM +0000, Daniel P. Berrange wrote: > > > > On Wed, Jan 06, 2016 at 01:30:16PM +0000, Peter Maydell wrote: > > > > > On 6 January 2016 at 12:49, Andrea Bolognani <abologna(a)redhat.com&gt; wrote: > > > > > > That's correct, having a QMP command that lists the values gic-version > > > > > > can have on the current host would be just great. > > > > > > > > > > > > If we had that, we could validate the GIC version chosen for a guest, > > > > > > and expose it in the capabilities XML so that higher-level tools can > > > > > > provide a list of choices to the user. > > > > > > > > > > > > Please note that this QMP command would have to work regardless of the > > > > > > machine type selected on QEMU's command line, because libvirt always > > > > > > runs a QEMU binary with '-M none' when probing its capabilities. > > > > > > > > > > On the other hand, if you don't tell us the machine type you care > > > > > about then we can't tell you: > > > > > (a) "this machine type doesn't support setting this property at all" > > > > > (which applies to machines like vexpress-a15 which you can use with > > > > > KVM on 32-bit hosts, and of course also to all the non-KVM models) > > > > > > > > We have just recently merged support for registering properties against > > > > classes instead of object instances. There is also a proposed command > > > > to allow querying the list of properties against a class > > > > > > > > https://lists.gnu.org/archive/html/qemu-devel/2016-01/msg04348.html > > > > > > > > So if we now update the machine types to register their properties against > > > > the class instead of object, then we can query what properties are present > > > > against each machine type, while still using '-M none'. > > > > > > > > > (b) "this machine type only supports GIC versions X and Y even if the > > > > > host supports more" (this is currently only hypothetical, though, > > > > > since we only have the property on 'virt'. it would only happen > > > > > if in the future we needed something other than '2' or '3' or > > > > > 'host' I think.) > > > > > > > > Our introspection support in QOM only allows us to say that a property > > > > is a particular type (int / enum / str / whatever). We don't have any > > > > way to expose info about what subset of possible values for a type are > > > > permitted. So I don't see any near term way to inform apps that the > > > > gic property accepts values x, y and but not z. > > This actually doesn't matter for the v2 vs. v3 case. The gic-version property > doesn't exist at all for v2-only QEMU. Although maybe the gic-version property > should be reworked. Instead of gic-version=<highest-supported>, we could create > one boolean property per version supported, i.e. gicv2, gicv3, gicv4... > > > > > > > > > IMHO, we shouldn't try to overthink this. Libvirt can query the host > > > > to find out what GIC versions are supported and default to using the > > > > most recent version, on the basis that people are likely to have a > > > > matching QEMU. We can just rely on QEMU to report error if we pass > > > > it a version it doesn't support and not try to pre-emptively check > > > > before launch. The key is just getting the default right IMHO. > > > > > > > This sounds fine to me. > > > > > > However, not being familiar with the internals of libvirt I really can't > > > just what the right implementation approach here is. > > > > > > As I understand we need to either: > > > > > > 1) Add a QMP command that lets you ask for -M virt, if GIC version X > > > is supported > > > > > > 2) Just implement something in libvirt that checks what the kernel > > > supports directly via the well-defined KVM interface and chooses > > > the highest supported version per default. > > > > > > To me it sounds like we should just go ahead with (2) and document > > > somehwere that if you get an error, you need to either manually change > > > the gic version setting in your VM definition file or upgrade QEMU. > > > > > > Can someone with libvirt authority please advice whether (1) or (2) is > > > what we need to do? > > > > IMHO we should just go for 2. > > I'm not familiar enough with libvirt, nor the use of QMP, to really argue > one way or another, but I find it a bit strange that we'd prefer libvirt > to query two entities over one. And, why should the libvirt installed on > a particular host prefer gicv3 as the default, just because KVM supports > it, even when QEMU does not? I think the assumption here is that if you install a recent libvirt you also install a recent QEMU. You always have the risk of things not working if you have too old a QEMU, right?

> The default will fail every time until QEMU > is upgraded, which may not be necessary/desired. Finally, I thought we > were trying to get away from relying on QEMU's error messages to make any > sort of decisions. This I have no idea, if libvirt wants to go in the direction of doing anything humanly possible to avoid QEMU errors, then (2) is probably not the right approach. On the other hand, there's always the possibility that something fails, so you probably always have to deal with QEMU's errors. I don't think anyone here suggested that libvirt should try to run QEMU with something, catch an error, and then run QEMU with other options, if that's what you meant?

> > I don't know what else libvirt queries directly from KVM, but IMO, it > should be a long-term goal to stop doing it, not add to more. Besides > libvirt then properly selecting defaults that both KVM and QEMU support, > it would allow /dev/kvm to have QEMU-only selinux labels applied. > That's really for the libvirt community and maintainers to say, which direction they're going in.

As I've tried to indicate before, we don't mind doing some work here as a bunch of Linaro members care about this, but we just want to know from the libvirt people what they want...?

On 02/02/2016 07:05 AM, Christoffer Dall wrote: >> >> I'm not familiar enough with libvirt, nor the use of QMP, to really argue >> one way or another, but I find it a bit strange that we'd prefer libvirt >> to query two entities over one. And, why should the libvirt installed on >> a particular host prefer gicv3 as the default, just because KVM supports >> it, even when QEMU does not? > > I think the assumption here is that if you install a recent libvirt you > also install a recent QEMU. You always have the risk of things not > working if you have too old a QEMU, right? Libvirt exists for providing back-compat glue. The following combinations are supported: old libvirt, old qemu new libvirt, old qemu new libvirt, new qemu and it is only this combination that might require a libvirt upgrade to work correctly: old libvirt, new qemu

> > > Our introspection support in QOM only allows us to say

> > > is a particular type (int / enum / str / whatever). We don't have any > > > way to expose info

> > > permitted. So I don't see any near term way to inform apps

> > > gic property accepts values x, y and but not z.   This actually doesn't matter for the v2 vs. v3 case.

doesn't exist at all for v2-only QEMU. Although maybe the gic-version property should be

one boolean property per version supported, i.e.

> >   2) Just implement something in libvirt that checks what the kernel > >      supports directly via the well-defined KVM interface and chooses > >      the highest supported version per default.

I'm not familiar enough with libvirt, nor the use of QMP, to really argue one way or another, but I find it a bit strange that we'd prefer libvirt to query two entities over one. And, why should the libvirt installed on a particular host prefer gicv3 as the default, just because KVM supports it, even when QEMU does not? The default will fail every time until QEMU is upgraded, which may not be necessary/desired.

Finally, I thought we were trying to get away from relying on QEMU's error messages to make any sort of decisions.

I don't know what else libvirt queries directly from KVM, but IMO, it should be a long-term goal to stop doing it, not add to more. Besides libvirt then properly selecting defaults that both KVM and QEMU support, it would allow /dev/kvm to have QEMU-only selinux labels applied.

Hold on, so "gic-version=3" means "support all GIC versions up to 3", not "support GIC version 3 only"? I thought the latter.

> > > 2) Just implement something in libvirt that checks what the kernel > > > supports directly via the well-defined KVM interface and chooses > > > the highest supported version per default. [...] > I'm not familiar enough with libvirt, nor the use of QMP, to really argue > one way or another, but I find it a bit strange that we'd prefer libvirt > to query two entities over one. And, why should the libvirt installed on > a particular host prefer gicv3 as the default, just because KVM supports > it, even when QEMU does not? The default will fail every time until QEMU > is upgraded, which may not be necessary/desired. Shouldn't the default be "host", to mean "whatever the host supports", rather than a specific version based either on host or QEMU probing? That should work for every QEMU version, right?

On Tue, Dec 15, 2015 at 04:03:13PM +0000, Peter Maydell wrote: > On 15 December 2015 at 14:12, Martin Kletzander <mkletzan(a)redhat.com&gt; wrote: > > On Tue, Dec 15, 2015 at 09:41:02AM +0000, Peter Maydell wrote: > > > > > > On 15 December 2015 at 09:36, Martin Kletzander <mkletzan(a)redhat.com&gt; > > > wrote: > > > > > > > > We do pass some options, for example, you can restrict the GIC to v2: > > > > https://libvirt.org/formatdomain.html#elementsFeatures > > > > > > > > That could be modified to work for your purpose IIUC, right? > > > > > > > > > What does that option do to the QEMU command line? The documentation > > > doesn't seem to say. > > > > > > > It appends gic-version= to the machine option, if bunch of conditions > > are met -- it has to be ARM machine, it has to be something else than > > version 2, and so on, more details are visible in the code and I won't > > cover them here.  I hope that's understandable. > > OK. So it should be sufficient just to have >   <gic version='host'/> > which will then provide whatever VGIC the host machine can do with > hardware acceleration. This is probably good for guests that happy with both. Guests that need/want a specific choice will put their integer there, and then we need a way to do a capabilities check before launching that guest on an arbitrary host.

On Tue, 2015-12-15 at 10:35 -0600, Andrew Jones wrote: > On Tue, Dec 15, 2015 at 04:03:13PM +0000, Peter Maydell wrote: >> On 15 December 2015 at 14:12, Martin Kletzander <mkletzan(a)redhat.com&gt; wrote: >>> On Tue, Dec 15, 2015 at 09:41:02AM +0000, Peter Maydell wrote: >>>> >>>> On 15 December 2015 at 09:36, Martin Kletzander <mkletzan(a)redhat.com&gt; >>>> wrote: >>>>> >>>>> We do pass some options, for example, you can restrict the GIC to v2: >>>>> https://libvirt.org/formatdomain.html#elementsFeatures >>>>> >>>>> That could be modified to work for your purpose IIUC, right? >>>> >>>> >>>> What does that option do to the QEMU command line? The documentation >>>> doesn't seem to say. >>>> >>> >>> It appends gic-version= to the machine option, if bunch of conditions >>> are met -- it has to be ARM machine, it has to be something else than >>> version 2, and so on, more details are visible in the code and I won't >>> cover them here. I hope that's understandable. >> >> OK. So it should be sufficient just to have >> <gic version='host'/> >> which will then provide whatever VGIC the host machine can do with >> hardware acceleration. > > This is probably good for guests that happy with both. Guests that > need/want a specific choice will put their integer there, and then > we need a way to do a capabilities check before launching that guest > on an arbitrary host. That won't work at the moment since we're expecting the value to be a number, but extending it so that 'host' is accepted as well should be easy to do in a backwards-compatible way. The behaviour of the value '2' has been defined as "don't specify any GIC version" though, and I'm afraid we'll be unable to change that because it would affect existing guests. Martin can probably weigh in. Having a way for libvirt to know what values can be used on the specific machines it's running on it's another topic altogether - is there a good way to detect it? Is that based on the hardware, on the QEMU binary, on the machine type?

On Mon, Dec 14, 2015 at 10:47:26AM -0600, Andrew Jones wrote: >On Mon, Dec 14, 2015 at 12:31:59PM +0100, Christoffer Dall wrote: >>Hi all, >> >>I'm trying to figure out what the correct solution for libvirt support >>for ARM KVM guests with GICv3 is. >> >>The challenge is that QEMU's virt machine model defaults to GICv2 unless >>you specify an additional "-machine gic-version=host" parameter to QEMU, >>in which case your VM gets the same GIC version as your host. >> >>Now, I'm told that this is inconvenient for libvirt, because it >>typically does not pass any -machine options, is that correct? > We do pass some options, for example, you can restrict the GIC to v2: https://libvirt.org/formatdomain.html#elementsFeatures That could be modified to work for your purpose IIUC, right?

On Mon, Dec 14, 2015 at 12:31:59PM +0100, Christoffer Dall wrote: > Hi all, > > I'm trying to figure out what the correct solution for libvirt support > for ARM KVM guests with GICv3 is. > > The challenge is that QEMU's virt machine model defaults to GICv2 unless > you specify an additional "-machine gic-version=host" parameter to QEMU, > in which case your VM gets the same GIC version as your host. > > Now, I'm told that this is inconvenient for libvirt, because it > typically does not pass any -machine options, is that correct? If that is correct, then my guess is that that's because it's used to the qemu machine model automatically selecting appropriate defaults, and for pc, due to versioning, there are several machine model types to choose from. mach-virt hasn't started versioning yet. > > Further, there are two additional complications: First, some guest > images may not include GICv3 support. For Linux, this is relatively old > kernels (prior to v3.17), but other guests may only support GICv2. > > Second, some hardware platforms only support GICv2 guests, some only > support GICv3 guests, and some support both GICv2 and GICv3 guests > (those hosts with a GICv3 with the backwards compatibility support). > > It is unclear to me how libvirt users will relate to these constraints. > For example, in an OpenStack deployment, will it somehow be easy to tell > which nodes support which GIC version such that administrators can > choose compatible VM images, or how does that work? libvirt has the concept of capabilities that it can negotiate. I know this is used heavily with x86 cpu features that some guests depend on. I think a guest dependency on gic version could fit here.

> > The end goal here is to determine if we need to add any functionality to > QEMU to better support libvirt, and what kind of features/glue need to > be added to libvirt for this to work. I've been starting to wonder if we should consider breaking mach-virt into multiple types. For example, one that focuses on big, enterprise-y type configs (only 64-bit guests, only virtio-pci, extending the size of main memory beyond 30G, for sure, and possibly even moving the PCIE_MMIO_HIGH mapping to extend beyond 511G. It'd even be good to remap the redistributor space to get more than 123 vcpus.

Making new mach-virt types and versioning those types would probably avoid needing much libvirt changes, but teaching libvirt (if it doesn't know already) how to manage machine parameters to effectively create new types as needed, also sounds good to me.