[libvirt] ARM KVM GICv3 Support
Hi all, I'm trying to figure out what the correct solution for libvirt support for ARM KVM guests with GICv3 is. The challenge is that QEMU's virt machine model defaults to GICv2 unless you specify an additional "-machine gic-version=host" parameter to QEMU, in which case your VM gets the same GIC version as your host. Now, I'm told that this is inconvenient for libvirt, because it typically does not pass any -machine options, is that correct? Further, there are two additional complications: First, some guest images may not include GICv3 support. For Linux, this is relatively old kernels (prior to v3.17), but other guests may only support GICv2. Second, some hardware platforms only support GICv2 guests, some only support GICv3 guests, and some support both GICv2 and GICv3 guests (those hosts with a GICv3 with the backwards compatibility support). It is unclear to me how libvirt users will relate to these constraints. For example, in an OpenStack deployment, will it somehow be easy to tell which nodes support which GIC version such that administrators can choose compatible VM images, or how does that work? The end goal here is to determine if we need to add any functionality to QEMU to better support libvirt, and what kind of features/glue need to be added to libvirt for this to work. Much thanks for any feedback! -Christoffer
On Mon, Dec 14, 2015 at 12:31:59PM +0100, Christoffer Dall wrote:
Hi all,
I'm trying to figure out what the correct solution for libvirt support for ARM KVM guests with GICv3 is.
The challenge is that QEMU's virt machine model defaults to GICv2 unless you specify an additional "-machine gic-version=host" parameter to QEMU, in which case your VM gets the same GIC version as your host.
Now, I'm told that this is inconvenient for libvirt, because it typically does not pass any -machine options, is that correct?
If that is correct, then my guess is that that's because it's used to the qemu machine model automatically selecting appropriate defaults, and for pc, due to versioning, there are several machine model types to choose from. mach-virt hasn't started versioning yet.
Further, there are two additional complications: First, some guest images may not include GICv3 support. For Linux, this is relatively old kernels (prior to v3.17), but other guests may only support GICv2.
Second, some hardware platforms only support GICv2 guests, some only support GICv3 guests, and some support both GICv2 and GICv3 guests (those hosts with a GICv3 with the backwards compatibility support).
It is unclear to me how libvirt users will relate to these constraints. For example, in an OpenStack deployment, will it somehow be easy to tell which nodes support which GIC version such that administrators can choose compatible VM images, or how does that work?
libvirt has the concept of capabilities that it can negotiate. I know this is used heavily with x86 cpu features that some guests depend on. I think a guest dependency on gic version could fit here.
The end goal here is to determine if we need to add any functionality to QEMU to better support libvirt, and what kind of features/glue need to be added to libvirt for this to work.
I've been starting to wonder if we should consider breaking mach-virt into multiple types. For example, one that focuses on big, enterprise-y type configs (only 64-bit guests, only virtio-pci, extending the size of main memory beyond 30G, for sure, and possibly even moving the PCIE_MMIO_HIGH mapping to extend beyond 511G. It'd even be good to remap the redistributor space to get more than 123 vcpus. Making new mach-virt types and versioning those types would probably avoid needing much libvirt changes, but teaching libvirt (if it doesn't know already) how to manage machine parameters to effectively create new types as needed, also sounds good to me. Thanks, drew
On Mon, Dec 14, 2015 at 10:47:26AM -0600, Andrew Jones wrote:
On Mon, Dec 14, 2015 at 12:31:59PM +0100, Christoffer Dall wrote:
Hi all,
I'm trying to figure out what the correct solution for libvirt support for ARM KVM guests with GICv3 is.
The challenge is that QEMU's virt machine model defaults to GICv2 unless you specify an additional "-machine gic-version=host" parameter to QEMU, in which case your VM gets the same GIC version as your host.
Now, I'm told that this is inconvenient for libvirt, because it typically does not pass any -machine options, is that correct?
We do pass some options, for example, you can restrict the GIC to v2: https://libvirt.org/formatdomain.html#elementsFeatures That could be modified to work for your purpose IIUC, right?
If that is correct, then my guess is that that's because it's used to the qemu machine model automatically selecting appropriate defaults, and for pc, due to versioning, there are several machine model types to choose from. mach-virt hasn't started versioning yet.
Further, there are two additional complications: First, some guest images may not include GICv3 support. For Linux, this is relatively old kernels (prior to v3.17), but other guests may only support GICv2.
Second, some hardware platforms only support GICv2 guests, some only support GICv3 guests, and some support both GICv2 and GICv3 guests (those hosts with a GICv3 with the backwards compatibility support).
It is unclear to me how libvirt users will relate to these constraints. For example, in an OpenStack deployment, will it somehow be easy to tell which nodes support which GIC version such that administrators can choose compatible VM images, or how does that work?
libvirt has the concept of capabilities that it can negotiate. I know this is used heavily with x86 cpu features that some guests depend on. I think a guest dependency on gic version could fit here.
The end goal here is to determine if we need to add any functionality to QEMU to better support libvirt, and what kind of features/glue need to be added to libvirt for this to work.
I've been starting to wonder if we should consider breaking mach-virt into multiple types. For example, one that focuses on big, enterprise-y type configs (only 64-bit guests, only virtio-pci, extending the size of main memory beyond 30G, for sure, and possibly even moving the PCIE_MMIO_HIGH mapping to extend beyond 511G. It'd even be good to remap the redistributor space to get more than 123 vcpus.
Making new mach-virt types and versioning those types would probably avoid needing much libvirt changes, but teaching libvirt (if it doesn't know already) how to manage machine parameters to effectively create new types as needed, also sounds good to me.
Thanks, drew
-- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
On 15 December 2015 at 09:36, Martin Kletzander <mkletzan@redhat.com> wrote:
We do pass some options, for example, you can restrict the GIC to v2: https://libvirt.org/formatdomain.html#elementsFeatures
That could be modified to work for your purpose IIUC, right?
What does that option do to the QEMU command line? The documentation doesn't seem to say. thanks -- PMM
On Tue, Dec 15, 2015 at 09:41:02AM +0000, Peter Maydell wrote:
On 15 December 2015 at 09:36, Martin Kletzander <mkletzan@redhat.com> wrote:
We do pass some options, for example, you can restrict the GIC to v2: https://libvirt.org/formatdomain.html#elementsFeatures
That could be modified to work for your purpose IIUC, right?
What does that option do to the QEMU command line? The documentation doesn't seem to say.
It appends gic-version= to the machine option, if bunch of conditions are met -- it has to be ARM machine, it has to be something else than version 2, and so on, more details are visible in the code and I won't cover them here. I hope that's understandable.
thanks -- PMM
On 15 December 2015 at 14:12, Martin Kletzander <mkletzan@redhat.com> wrote:
On Tue, Dec 15, 2015 at 09:41:02AM +0000, Peter Maydell wrote:
On 15 December 2015 at 09:36, Martin Kletzander <mkletzan@redhat.com> wrote:
We do pass some options, for example, you can restrict the GIC to v2: https://libvirt.org/formatdomain.html#elementsFeatures
That could be modified to work for your purpose IIUC, right?
What does that option do to the QEMU command line? The documentation doesn't seem to say.
It appends gic-version= to the machine option, if bunch of conditions are met -- it has to be ARM machine, it has to be something else than version 2, and so on, more details are visible in the code and I won't cover them here. I hope that's understandable.
OK. So it should be sufficient just to have <gic version='host'/> which will then provide whatever VGIC the host machine can do with hardware acceleration. thanks -- PMM
On Tue, Dec 15, 2015 at 04:03:13PM +0000, Peter Maydell wrote:
On 15 December 2015 at 14:12, Martin Kletzander <mkletzan@redhat.com> wrote:
On Tue, Dec 15, 2015 at 09:41:02AM +0000, Peter Maydell wrote:
On 15 December 2015 at 09:36, Martin Kletzander <mkletzan@redhat.com> wrote:
We do pass some options, for example, you can restrict the GIC to v2: https://libvirt.org/formatdomain.html#elementsFeatures
That could be modified to work for your purpose IIUC, right?
What does that option do to the QEMU command line? The documentation doesn't seem to say.
It appends gic-version= to the machine option, if bunch of conditions are met -- it has to be ARM machine, it has to be something else than version 2, and so on, more details are visible in the code and I won't cover them here. I hope that's understandable.
OK. So it should be sufficient just to have <gic version='host'/> which will then provide whatever VGIC the host machine can do with hardware acceleration.
This is probably good for guests that happy with both. Guests that need/want a specific choice will put their integer there, and then we need a way to do a capabilities check before launching that guest on an arbitrary host. drew
On 15 December 2015 at 16:35, Andrew Jones <drjones@redhat.com> wrote:
This is probably good for guests that happy with both. Guests that need/want a specific choice will put their integer there, and then we need a way to do a capabilities check before launching that guest on an arbitrary host.
OK, so how do we typically do that? I notice I have a 'kvm-ok' script on my machine which helpfully reports things like whether KVM is enabled, and it seems like it might be helpful to extend that to know a bit more about ARM hosts. But I'm guessing libvirt doesn't use that for its capability checking ? thanks -- PMM
Hi, On 15/12/15 16:42, Peter Maydell wrote:
On 15 December 2015 at 16:35, Andrew Jones <drjones@redhat.com> wrote:
This is probably good for guests that happy with both. Guests that need/want a specific choice will put their integer there, and then we need a way to do a capabilities check before launching that guest on an arbitrary host.
OK, so how do we typically do that? I notice I have a 'kvm-ok' script on my machine which helpfully reports things like whether KVM is enabled, and it seems like it might be helpful to extend that to know a bit more about ARM hosts. But I'm guessing libvirt doesn't use that for its capability checking ?
Even that wouldn't help us, I guess, as you cannot easily check for GICv3/GICv2 compatibility with a _script_. Having access to ioctl's make this pretty easy though: Just try to call KVM_CREATE_DEVICE with the proper type and get -ENODEV if this one is not supported. This can be done without any extra userland tool by just executing some ioctls on /dev/kvm (from C or using some helper library). Does adding such a check to libvirt sounds feasible? Cheers, Andre.
On 15 December 2015 at 16:57, Andre Przywara <andre.przywara@arm.com> wrote:
Even that wouldn't help us, I guess, as you cannot easily check for GICv3/GICv2 compatibility with a _script_. Having access to ioctl's make this pretty easy though: Just try to call KVM_CREATE_DEVICE with the proper type and get -ENODEV if this one is not supported. This can be done without any extra userland tool by just executing some ioctls on /dev/kvm (from C or using some helper library).
kvm-ok already runs a few external helper binaries for some things. (Also you can do ioctls from a script if it's a perl script ;-)) As you say the actual technical details of how to query for the host's current supported functionality are straightforward, so it's just a question of how libvirt is expecting that to be exposed to it. thanks -- PMM
On Tue, Dec 15, 2015 at 05:01:39PM +0000, Peter Maydell wrote:
On 15 December 2015 at 16:57, Andre Przywara <andre.przywara@arm.com> wrote:
Even that wouldn't help us, I guess, as you cannot easily check for GICv3/GICv2 compatibility with a _script_. Having access to ioctl's make this pretty easy though: Just try to call KVM_CREATE_DEVICE with the proper type and get -ENODEV if this one is not supported. This can be done without any extra userland tool by just executing some ioctls on /dev/kvm (from C or using some helper library).
kvm-ok already runs a few external helper binaries for some things. (Also you can do ioctls from a script if it's a perl script ;-))
As you say the actual technical details of how to query for the host's current supported functionality are straightforward, so it's just a question of how libvirt is expecting that to be exposed to it.
We currently probe the host features as well using som ioctls on /dev/kvm, etc. There is no problem with adding any other probing. If qemu can report what it supports, that's good too. So I see it from the other side. It's straightforward to implement it in libvirt, so for me it's just a question of how exactly should we do that. What should we probe fr and also the outcome: what to (dis)allow for a domain.
thanks -- PMM
On Tue, Dec 15, 2015 at 04:42:13PM +0000, Peter Maydell wrote:
On 15 December 2015 at 16:35, Andrew Jones <drjones@redhat.com> wrote:
This is probably good for guests that happy with both. Guests that need/want a specific choice will put their integer there, and then we need a way to do a capabilities check before launching that guest on an arbitrary host.
OK, so how do we typically do that? I notice I have a 'kvm-ok' script on my machine which helpfully reports things like whether KVM is enabled, and it seems like it might be helpful to extend that to know a bit more about ARM hosts. But I'm guessing libvirt doesn't use that for its capability checking ?
I defer to our libvirt experts. I'm suspicious that we'll need libvirt changes though, as my experience with libvirt capabilities has always been for cpu features, not machine model features, and thus I'm not sure there's support for the later yet. Thanks, drew
On Tue, Dec 15, 2015 at 04:42:13PM +0000, Peter Maydell wrote:
On 15 December 2015 at 16:35, Andrew Jones <drjones@redhat.com> wrote:
This is probably good for guests that happy with both. Guests that need/want a specific choice will put their integer there, and then we need a way to do a capabilities check before launching that guest on an arbitrary host.
OK, so how do we typically do that? I notice I have a 'kvm-ok' script on my machine which helpfully reports things like whether KVM is enabled, and it seems like it might be helpful to extend that to know a bit more about ARM hosts. But I'm guessing libvirt doesn't use that for its capability checking ?
libvirt ships a tool called `virt-host-validate` that performs a bunch of checks along with whether KVM is enabled or not in the BIOS: $ sudo virt-host-validate QEMU: Checking for hardware virtualization : PASS QEMU: Checking for device /dev/kvm : PASS QEMU: Checking for device /dev/vhost-net : PASS QEMU: Checking for device /dev/net/tun : PASS LXC: Checking for Linux >= 2.6.26 : PASS The only ARM machine I have access to is an AArch64 one, running on it results in: $ sudo virt-host-validate QEMU: Checking for hardware virtualization : WARN (Only emulated CPUs are available, performance will be significantly limited) QEMU: Checking for device /dev/vhost-net : WARN (Load the 'vhost_net' module to improve performance of virtio networking) QEMU: Checking for device /dev/net/tun : PASS LXC: Checking for Linux >= 2.6.26 : PASS $ uname -r; rpm -q libvirt-client 4.1.0-0.rc4.git1.1.fc23.aarch64 libvirt-client-1.2.14-2.fc23.aarch64 -- /kashyap
On Sun, Dec 20, 2015 at 04:49:20PM +0100, Kashyap Chamarthy wrote:
On Tue, Dec 15, 2015 at 04:42:13PM +0000, Peter Maydell wrote:
On 15 December 2015 at 16:35, Andrew Jones <drjones@redhat.com> wrote:
This is probably good for guests that happy with both. Guests that need/want a specific choice will put their integer there, and then we need a way to do a capabilities check before launching that guest on an arbitrary host.
OK, so how do we typically do that? I notice I have a 'kvm-ok' script on my machine which helpfully reports things like whether KVM is enabled, and it seems like it might be helpful to extend that to know a bit more about ARM hosts. But I'm guessing libvirt doesn't use that for its capability checking ?
libvirt ships a tool called `virt-host-validate` that performs a bunch of checks along with whether KVM is enabled or not in the BIOS:
$ sudo virt-host-validate QEMU: Checking for hardware virtualization : PASS QEMU: Checking for device /dev/kvm : PASS QEMU: Checking for device /dev/vhost-net : PASS QEMU: Checking for device /dev/net/tun : PASS LXC: Checking for Linux >= 2.6.26 : PASS
The only ARM machine I have access to is an AArch64 one, running on it results in:
$ sudo virt-host-validate QEMU: Checking for hardware virtualization : WARN (Only emulated CPUs are available, performance will be significantly limited)
This check needs to be fixed to work on Arm hosts, or indeed any non-x86 hosts. It currently checks for 'svm' and 'vmx' flags in CPU. Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On Mon, Dec 21, 2015 at 10:15:13AM +0000, Daniel P. Berrange wrote:
On Sun, Dec 20, 2015 at 04:49:20PM +0100, Kashyap Chamarthy wrote:
On Tue, Dec 15, 2015 at 04:42:13PM +0000, Peter Maydell wrote:
On 15 December 2015 at 16:35, Andrew Jones <drjones@redhat.com> wrote:
This is probably good for guests that happy with both. Guests that need/want a specific choice will put their integer there, and then we need a way to do a capabilities check before launching that guest on an arbitrary host.
OK, so how do we typically do that? I notice I have a 'kvm-ok' script on my machine which helpfully reports things like whether KVM is enabled, and it seems like it might be helpful to extend that to know a bit more about ARM hosts. But I'm guessing libvirt doesn't use that for its capability checking ?
libvirt ships a tool called `virt-host-validate` that performs a bunch of checks along with whether KVM is enabled or not in the BIOS:
$ sudo virt-host-validate QEMU: Checking for hardware virtualization : PASS QEMU: Checking for device /dev/kvm : PASS QEMU: Checking for device /dev/vhost-net : PASS QEMU: Checking for device /dev/net/tun : PASS LXC: Checking for Linux >= 2.6.26 : PASS
The only ARM machine I have access to is an AArch64 one, running on it results in:
$ sudo virt-host-validate QEMU: Checking for hardware virtualization : WARN (Only emulated CPUs are available, performance will be significantly limited)
This check needs to be fixed to work on Arm hosts, or indeed any non-x86 hosts. It currently checks for 'svm' and 'vmx' flags in CPU.
So let me summarize the things we need to do: - patch libvirt to allow <gic version='host' /> - patch libvirt to properly probe for GICv2 and GICv3 support - patch libvirt to generally handle virt-host-validate on ARM And that should be enough for other tools such as OpenStack with Nova to probe and create the VMs it wishes. Does that about capture it? Thanks, -Christoffer
On Mon, 2015-12-21 at 12:03 +0100, Christoffer Dall wrote:
So let me summarize the things we need to do: - patch libvirt to allow <gic version='host' /> - patch libvirt to properly probe for GICv2 and GICv3 support - patch libvirt to generally handle virt-host-validate on ARM And that should be enough for other tools such as OpenStack with Nova to probe and create the VMs it wishes. Does that about capture it?
Sounds about right. I can work on points 1) and 3) of your list on my own, but for point 2) we need QEMU to start probing the host for supported GIC versions and expose such information before libvirt can use it. Then of course higher level tools such as OpenStack will need to become aware of these new features in libvirt. Cheers. -- Andrea Bolognani Software Engineer - Virtualization Team
On Tue, Jan 05, 2016 at 12:37:02PM +0100, Andrea Bolognani wrote:
On Mon, 2015-12-21 at 12:03 +0100, Christoffer Dall wrote:
So let me summarize the things we need to do: - patch libvirt to allow <gic version='host' /> - patch libvirt to properly probe for GICv2 and GICv3 support - patch libvirt to generally handle virt-host-validate on ARM And that should be enough for other tools such as OpenStack with Nova to probe and create the VMs it wishes. Does that about capture it?
Sounds about right.
I can work on points 1) and 3) of your list on my own, but for point 2) we need QEMU to start probing the host for supported GIC versions and expose such information before libvirt can use it. Then of course higher level tools such as OpenStack will need to become aware of these new features in libvirt.
I thought the virt-host-validate tool in libvirt should do the probing directly? If I misunderstood, how would you like QEMU to report this? Should it be the qemu binary directly or the script that Peter suggested? Thanks, -Christoffer
On Tue, 2016-01-05 at 13:08 +0100, Christoffer Dall wrote:
On Mon, 2015-12-21 at 12:03 +0100, Christoffer Dall wrote:
So let me summarize the things we need to do: - patch libvirt to allow <gic version='host' /> - patch libvirt to properly probe for GICv2 and GICv3 support - patch libvirt to generally handle virt-host-validate on ARM And that should be enough for other tools such as OpenStack with Nova to probe and create the VMs it wishes. Does that about capture it? Sounds about right. I can work on points 1) and 3) of your list on my own, but for point 2) we need QEMU to start probing the host for supported GIC versions and expose such information before libvirt can use it. Then of course higher level tools such as OpenStack will need to become aware of these new features in libvirt. I thought the virt-host-validate tool in libvirt should do the probing
On Tue, Jan 05, 2016 at 12:37:02PM +0100, Andrea Bolognani wrote: directly?
The tool is just a convenience for the administrator (and for higher level tools?), and it can't be where the validation happens for two reasons: - it doesn't take any argument, so it can't possibly know whether you're interested in running GICv2 or GICv3 guests - libvirt needs to check before running the guest anyway Probing inside QEMU means libvirt can just ask it instead of implementing the checks itself, which means less code duplication; more importantly, this prevents the possibility of the checks ever going out of sync.
If I misunderstood, how would you like QEMU to report this?
I'm not 100% sure about the best interface here (hopefully other people can weigh in) but I imagine there would be some sort of QMP command that lists the GIC versions supported by the QEMU binary, or some text in the output of 'qemu -help' that can be parsed to obtain such list.
Should it be the qemu binary directly or the script that Peter suggested?
I think it should be the QEMU binary, especially since I assume QEMU itself will want to error out if an unsupported GIC version is requested by the user - there's no guarantee QEMU will be called by libvirt, so it will have to check on its own. Cheers. -- Andrea Bolognani Software Engineer - Virtualization Team
On Tue, Jan 05, 2016 at 01:37:47PM +0100, Andrea Bolognani wrote:
On Tue, 2016-01-05 at 13:08 +0100, Christoffer Dall wrote:
On Mon, 2015-12-21 at 12:03 +0100, Christoffer Dall wrote:
So let me summarize the things we need to do: - patch libvirt to allow <gic version='host' /> - patch libvirt to properly probe for GICv2 and GICv3 support - patch libvirt to generally handle virt-host-validate on ARM And that should be enough for other tools such as OpenStack with Nova to probe and create the VMs it wishes. Does that about capture it? Sounds about right. I can work on points 1) and 3) of your list on my own, but for point 2) we need QEMU to start probing the host for supported GIC versions and expose such information before libvirt can use it. Then of course higher level tools such as OpenStack will need to become aware of these new features in libvirt. I thought the virt-host-validate tool in libvirt should do the probing
On Tue, Jan 05, 2016 at 12:37:02PM +0100, Andrea Bolognani wrote: directly?
The tool is just a convenience for the administrator (and for higher level tools?), and it can't be where the validation happens for two reasons:
- it doesn't take any argument, so it can't possibly know whether you're interested in running GICv2 or GICv3 guests
You don't need any input to report the capabilities of a host; either it supports only GICv2, or only GICv3, or both.
- libvirt needs to check before running the guest anyway
Why can libvirt not just run the tool and parse its output? That's what I understood was done for other things, but perhaps I misunderstood.
Probing inside QEMU means libvirt can just ask it instead of implementing the checks itself, which means less code duplication; more importantly, this prevents the possibility of the checks ever going out of sync.
If I misunderstood, how would you like QEMU to report this?
I'm not 100% sure about the best interface here (hopefully other people can weigh in) but I imagine there would be some sort of QMP command that lists the GIC versions supported by the QEMU binary, or some text in the output of 'qemu -help' that can be parsed to obtain such list.
ok, I don't think this is a big deal to implement, but I have no idea which interface should be used. Could you have a look at the libvirt side of things and figure out what is done for similar things on x86 and we can see if we can implement something like this for ARM in QEMU?
Should it be the qemu binary directly or the script that Peter suggested?
I think it should be the QEMU binary, especially since I assume QEMU itself will want to error out if an unsupported GIC version is requested by the user - there's no guarantee QEMU will be called by libvirt, so it will have to check on its own.
This is already done. If you try to create a VM which is not supported, the system exits with an error. Thanks, -Christoffer
On Tue, 2016-01-05 at 14:47 +0100, Christoffer Dall wrote:
The tool is just a convenience for the administrator (and for higher level tools?), and it can't be where the validation happens for two reasons: - it doesn't take any argument, so it can't possibly know whether you're interested in running GICv2 or GICv3 guests You don't need any input to report the capabilities of a host; either it supports only GICv2, or only GICv3, or both.
But virt-host-validate doesn't report the capabilities of the host, it just tells the user whether the host can be considered suitable for running guests using a certain virtualization technology... So I believe the check would be out of scope for the tool, as not supporting GICv2 is not a host configuration error. We already have facilities for reporting host capabilities (the output of 'virsh capabilities') and that's where higher level tools should look for GIC versions supported by the host.
- libvirt needs to check before running the guest anyway Why can libvirt not just run the tool and parse its output? That's what I understood was done for other things, but perhaps I misunderstood.
It's the other way around: the tool uses libvirt to implement its checks. Running a command and parsing its output makes sense when the binary (as is the case for QEMU) is developed by a third party; if you control both pieces, it's usually better to stick to C and avoid spawning extra processes.
If I misunderstood, how would you like QEMU to report this? I'm not 100% sure about the best interface here (hopefully other people can weigh in) but I imagine there would be some sort of QMP command that
Probing inside QEMU means libvirt can just ask it instead of implementing the checks itself, which means less code duplication; more importantly, this prevents the possibility of the checks ever going out of sync. lists the GIC versions supported by the QEMU binary, or some text in the output of 'qemu -help' that can be parsed to obtain such list. ok, I don't think this is a big deal to implement, but I have no idea which interface should be used. Could you have a look at the libvirt side of things and figure out what is done for similar things on x86 and we can see if we can implement something like this for ARM in QEMU?
I realized the information is already sort of exposed by QEMU: $ qemu-system-aarch64 -machine virt,? ... virt.gic-version=string (Set GIC version. Valid values are 2, 3 and host) ... Not the best format to parse, but the information is there. And we already have some code to extract such parameters from QEMU using QMP's query-command-line-options command![1] Problem is, QEMU doesn't seem to be reporting machine type-dependent parameters such as gic-version via QMP. Another problem is that, when we probe using QMP, we launch QEMU using '-machine none'. I'm not sure whether the former is a bug in QEMU or a design decision. I'd like to hear more opinions on this, but I think a possible solution would be for QMP to grow a query-gic-versions command that performs the appropriate probing regardless of the machine type, and for libvirt to start using it where available, falling back to accepting only 2, 3 and host as valid values for all versions of QEMU older than 2.4.50 as it currently does[2].
Should it be the qemu binary directly or the script that Peter suggested? I think it should be the QEMU binary, especially since I assume QEMU itself will want to error out if an unsupported GIC version is requested by the user - there's no guarantee QEMU will be called by libvirt, so it will have to check on its own. This is already done. If you try to create a VM which is not supported, the system exits with an error.
Looks to me like creation of the requested device is attempted and an error is reported on failure, which is not exactly the same as probing in advance and aborting early if the operation is know to be impossible. I might very well have missed something though, I'm far from being familiar with QEMU's code :) Cheers. [1] It currently ignores everything but the parameter name; extending it to look into the help text as well seems very doable [2] Well, it really accepts all numeric values and doesn't accept the string 'host', but that can be safely changed as other values never worked to begin with -- Andrea Bolognani Software Engineer - Virtualization Team
On 5 January 2016 at 17:58, Andrea Bolognani <abologna@redhat.com> wrote:
On Tue, 2016-01-05 at 14:47 +0100, Christoffer Dall wrote:
Why can libvirt not just run the tool and parse its output? That's what I understood was done for other things, but perhaps I misunderstood.
It's the other way around: the tool uses libvirt to implement its checks. Running a command and parsing its output makes sense when the binary (as is the case for QEMU) is developed by a third party; if you control both pieces, it's usually better to stick to C and avoid spawning extra processes.
That argument suggests libvirt should just go ahead and do the ioctls to check for GICv2/v3 support, rather than spawning a separate QEMU binary to do the check for it ?
I realized the information is already sort of exposed by QEMU:
$ qemu-system-aarch64 -machine virt,? ... virt.gic-version=string (Set GIC version. Valid values are 2, 3 and host) ...
Not the best format to parse, but the information is there. And we already have some code to extract such parameters from QEMU using QMP's query-command-line-options command![1]
Note that helpstring will tell you whether the qemu you have supports the machine parameter, but not whether the host CPU supports GICv2 or 3 (or both) -- it's just a fixed string. thanks -- PMM
On Tue, 2016-01-05 at 18:14 +0000, Peter Maydell wrote:
Why can libvirt not just run the tool and parse its output? That's what I understood was done for other things, but perhaps I misunderstood. It's the other way around: the tool uses libvirt to implement its checks. Running a command and parsing its output makes sense when the binary (as is the case for QEMU) is developed by a third party; if you control both pieces, it's usually better to stick to C and avoid spawning extra processes. That argument suggests libvirt should just go ahead and do the ioctls to check for GICv2/v3 support, rather than spawning a separate QEMU binary to do the check for it ?
libvirt already has to spawn QEMU and run QMP commands to probe for dozens of other capabilities, so we have very robust code to do that. Most importantly, as I wrote in a previous mail, if the check is done by QEMU we remove the risk of it and libvirt ever disagreeing.
I realized the information is already sort of exposed by QEMU: $ qemu-system-aarch64 -machine virt,? ... virt.gic-version=string (Set GIC version. Valid values are 2, 3 and host) ... Not the best format to parse, but the information is there. And we already have some code to extract such parameters from QEMU using QMP's query-command-line-options command![1] Note that helpstring will tell you whether the qemu you have supports the machine parameter, but not whether the host CPU supports GICv2 or 3 (or both) -- it's just a fixed string.
Right, thanks for pointing that out :) Cheers. -- Andrea Bolognani Software Engineer - Virtualization Team
On Tue, Jan 05, 2016 at 07:26:54PM +0100, Andrea Bolognani wrote:
On Tue, 2016-01-05 at 18:14 +0000, Peter Maydell wrote:
Why can libvirt not just run the tool and parse its output? That's what I understood was done for other things, but perhaps I misunderstood. It's the other way around: the tool uses libvirt to implement its checks. Running a command and parsing its output makes sense when the binary (as is the case for QEMU) is developed by a third party; if you control both pieces, it's usually better to stick to C and avoid spawning extra processes. That argument suggests libvirt should just go ahead and do the ioctls to check for GICv2/v3 support, rather than spawning a separate QEMU binary to do the check for it ?
libvirt already has to spawn QEMU and run QMP commands to probe for dozens of other capabilities, so we have very robust code to do that.
Most importantly, as I wrote in a previous mail, if the check is done by QEMU we remove the risk of it and libvirt ever disagreeing.
So do I understand correct that what you're looking for is a QMP command to query for host GIC capabilities? -Christoffer
On Wed, 2016-01-06 at 12:34 +0100, Christoffer Dall wrote:
libvirt already has to spawn QEMU and run QMP commands to probe for dozens of other capabilities, so we have very robust code to do that. Most importantly, as I wrote in a previous mail, if the check is done by QEMU we remove the risk of it and libvirt ever disagreeing. So do I understand correct that what you're looking for is a QMP command to query for host GIC capabilities?
That's correct, having a QMP command that lists the values gic-version can have on the current host would be just great. If we had that, we could validate the GIC version chosen for a guest, and expose it in the capabilities XML so that higher-level tools can provide a list of choices to the user. Please note that this QMP command would have to work regardless of the machine type selected on QEMU's command line, because libvirt always runs a QEMU binary with '-M none' when probing its capabilities. Cheers. -- Andrea Bolognani Software Engineer - Virtualization Team
On 6 January 2016 at 12:49, Andrea Bolognani <abologna@redhat.com> wrote:
That's correct, having a QMP command that lists the values gic-version can have on the current host would be just great.
If we had that, we could validate the GIC version chosen for a guest, and expose it in the capabilities XML so that higher-level tools can provide a list of choices to the user.
Please note that this QMP command would have to work regardless of the machine type selected on QEMU's command line, because libvirt always runs a QEMU binary with '-M none' when probing its capabilities.
On the other hand, if you don't tell us the machine type you care about then we can't tell you: (a) "this machine type doesn't support setting this property at all" (which applies to machines like vexpress-a15 which you can use with KVM on 32-bit hosts, and of course also to all the non-KVM models) (b) "this machine type only supports GIC versions X and Y even if the host supports more" (this is currently only hypothetical, though, since we only have the property on 'virt'. it would only happen if in the future we needed something other than '2' or '3' or 'host' I think.) If you use -M none then we can only tell you information you could have got yourself by directly asking the kernel about it. thanks -- PMM
On Wed, Jan 06, 2016 at 01:30:16PM +0000, Peter Maydell wrote:
On 6 January 2016 at 12:49, Andrea Bolognani <abologna@redhat.com> wrote:
That's correct, having a QMP command that lists the values gic-version can have on the current host would be just great.
If we had that, we could validate the GIC version chosen for a guest, and expose it in the capabilities XML so that higher-level tools can provide a list of choices to the user.
Please note that this QMP command would have to work regardless of the machine type selected on QEMU's command line, because libvirt always runs a QEMU binary with '-M none' when probing its capabilities.
On the other hand, if you don't tell us the machine type you care about then we can't tell you: (a) "this machine type doesn't support setting this property at all" (which applies to machines like vexpress-a15 which you can use with KVM on 32-bit hosts, and of course also to all the non-KVM models) (b) "this machine type only supports GIC versions X and Y even if the host supports more" (this is currently only hypothetical, though, since we only have the property on 'virt'. it would only happen if in the future we needed something other than '2' or '3' or 'host' I think.)
If you use -M none then we can only tell you information you could have got yourself by directly asking the kernel about it.
So where does this leave us? Do we need a QMP command or not? It sounds to me like we should implement one, and the answer depends on the machine option chosen, where -M none gives you the generic capabilities of the kernel and for other machines it tells you what's possible? Thanks, -Christoffer
On Sat, 2016-01-09 at 16:01 +0100, Christoffer Dall wrote:
On Wed, Jan 06, 2016 at 01:30:16PM +0000, Peter Maydell wrote:
On 6 January 2016 at 12:49, Andrea Bolognani <abologna@redhat.com> wrote:
That's correct, having a QMP command that lists the values gic-version can have on the current host would be just great. If we had that, we could validate the GIC version chosen for a guest, and expose it in the capabilities XML so that higher-level tools can provide a list of choices to the user. Please note that this QMP command would have to work regardless of the machine type selected on QEMU's command line, because libvirt always runs a QEMU binary with '-M none' when probing its capabilities. On the other hand, if you don't tell us the machine type you care about then we can't tell you: (a) "this machine type doesn't support setting this property at all" (which applies to machines like vexpress-a15 which you can use with KVM on 32-bit hosts, and of course also to all the non-KVM models)
We only allow specifying the GIC version for the 'virt' machine type, so this shouldn't be a problem.
(b) "this machine type only supports GIC versions X and Y even if the host supports more" (this is currently only hypothetical, though, since we only have the property on 'virt'. it would only happen if in the future we needed something other than '2' or '3' or 'host' I think.)
See below.
If you use -M none then we can only tell you information you could have got yourself by directly asking the kernel about it.
Sure. But the idea of having QEMU report this information, instead of probing for it directly, was to make sure that the whole stack agreed on a bunch of values that could be used, so that we could never find ourselves in the situation where eg. GICv4 is out, libvirt probes for it and believes it can be used, but since the QEMU binary on the system is old and it doesn't know about GICv4 it errors out.
So where does this leave us? Do we need a QMP command or not? It sounds to me like we should implement one, and the answer depends on the machine option chosen, where -M none gives you the generic capabilities of the kernel and for other machines it tells you what's possible?
libvirt probes QEMU for its capabilities using '-M none', but we should be able to extend its probing routines to look for features that are known to be machine type specific, such as gic-version, by specifying the appropriate machine type. Would providing a QMP command that returns the list of GIC versions available on the current host for the current machine type be acceptable from QEMU's point of view? Cheers. -- Andrea Bolognani Software Engineer - Virtualization Team
On Tue, Jan 12, 2016 at 04:48:27PM +0100, Andrea Bolognani wrote:
Would providing a QMP command that returns the list of GIC versions available on the current host for the current machine type be acceptable from QEMU's point of view?
Adding qemu-devel. Peter any opinion on this? If it sounds like a reasonable idea then either I or Wei can look into sending the patches. Thanks, drew
On 19 January 2016 at 16:46, Andrew Jones <drjones@redhat.com> wrote:
On Tue, Jan 12, 2016 at 04:48:27PM +0100, Andrea Bolognani wrote:
Would providing a QMP command that returns the list of GIC versions available on the current host for the current machine type be acceptable from QEMU's point of view?
Adding qemu-devel. Peter any opinion on this? If it sounds like a reasonable idea then either I or Wei can look into sending the patches.
I don't have a strong opinion, really. We should do whatever works for libvirt and fits with whatever we do currently (eg whatever we do at the moment for telling libvirt "yes this QEMU supports KVM", if we're not using some ancient silly way to do that). thanks -- PMM
On 01/19/16 17:53, Peter Maydell wrote:
On 19 January 2016 at 16:46, Andrew Jones <drjones@redhat.com> wrote:
On Tue, Jan 12, 2016 at 04:48:27PM +0100, Andrea Bolognani wrote:
Would providing a QMP command that returns the list of GIC versions available on the current host for the current machine type be acceptable from QEMU's point of view?
Adding qemu-devel. Peter any opinion on this? If it sounds like a reasonable idea then either I or Wei can look into sending the patches.
I don't have a strong opinion, really. We should do whatever works for libvirt and fits with whatever we do currently (eg whatever we do at the moment for telling libvirt "yes this QEMU supports KVM", if we're not using some ancient silly way to do that).
Would this kind of info fit the capabilities list that libvirtd fetches from QEMU at startup (as far as I recall)? Thanks Laszlo
On Wed, Jan 06, 2016 at 01:30:16PM +0000, Peter Maydell wrote:
On 6 January 2016 at 12:49, Andrea Bolognani <abologna@redhat.com> wrote:
That's correct, having a QMP command that lists the values gic-version can have on the current host would be just great.
If we had that, we could validate the GIC version chosen for a guest, and expose it in the capabilities XML so that higher-level tools can provide a list of choices to the user.
Please note that this QMP command would have to work regardless of the machine type selected on QEMU's command line, because libvirt always runs a QEMU binary with '-M none' when probing its capabilities.
On the other hand, if you don't tell us the machine type you care about then we can't tell you: (a) "this machine type doesn't support setting this property at all" (which applies to machines like vexpress-a15 which you can use with KVM on 32-bit hosts, and of course also to all the non-KVM models)
We have just recently merged support for registering properties against classes instead of object instances. There is also a proposed command to allow querying the list of properties against a class https://lists.gnu.org/archive/html/qemu-devel/2016-01/msg04348.html So if we now update the machine types to register their properties against the class instead of object, then we can query what properties are present against each machine type, while still using '-M none'.
(b) "this machine type only supports GIC versions X and Y even if the host supports more" (this is currently only hypothetical, though, since we only have the property on 'virt'. it would only happen if in the future we needed something other than '2' or '3' or 'host' I think.)
Our introspection support in QOM only allows us to say that a property is a particular type (int / enum / str / whatever). We don't have any way to expose info about what subset of possible values for a type are permitted. So I don't see any near term way to inform apps that the gic property accepts values x, y and but not z. IMHO, we shouldn't try to overthink this. Libvirt can query the host to find out what GIC versions are supported and default to using the most recent version, on the basis that people are likely to have a matching QEMU. We can just rely on QEMU to report error if we pass it a version it doesn't support and not try to pre-emptively check before launch. The key is just getting the default right IMHO. Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On Fri, Jan 22, 2016 at 02:44:32PM +0000, Daniel P. Berrange wrote:
On Wed, Jan 06, 2016 at 01:30:16PM +0000, Peter Maydell wrote:
On 6 January 2016 at 12:49, Andrea Bolognani <abologna@redhat.com> wrote:
That's correct, having a QMP command that lists the values gic-version can have on the current host would be just great.
If we had that, we could validate the GIC version chosen for a guest, and expose it in the capabilities XML so that higher-level tools can provide a list of choices to the user.
Please note that this QMP command would have to work regardless of the machine type selected on QEMU's command line, because libvirt always runs a QEMU binary with '-M none' when probing its capabilities.
On the other hand, if you don't tell us the machine type you care about then we can't tell you: (a) "this machine type doesn't support setting this property at all" (which applies to machines like vexpress-a15 which you can use with KVM on 32-bit hosts, and of course also to all the non-KVM models)
We have just recently merged support for registering properties against classes instead of object instances. There is also a proposed command to allow querying the list of properties against a class
https://lists.gnu.org/archive/html/qemu-devel/2016-01/msg04348.html
So if we now update the machine types to register their properties against the class instead of object, then we can query what properties are present against each machine type, while still using '-M none'.
(b) "this machine type only supports GIC versions X and Y even if the host supports more" (this is currently only hypothetical, though, since we only have the property on 'virt'. it would only happen if in the future we needed something other than '2' or '3' or 'host' I think.)
Our introspection support in QOM only allows us to say that a property is a particular type (int / enum / str / whatever). We don't have any way to expose info about what subset of possible values for a type are permitted. So I don't see any near term way to inform apps that the gic property accepts values x, y and but not z.
IMHO, we shouldn't try to overthink this. Libvirt can query the host to find out what GIC versions are supported and default to using the most recent version, on the basis that people are likely to have a matching QEMU. We can just rely on QEMU to report error if we pass it a version it doesn't support and not try to pre-emptively check before launch. The key is just getting the default right IMHO.
This sounds fine to me. However, not being familiar with the internals of libvirt I really can't just what the right implementation approach here is. As I understand we need to either: 1) Add a QMP command that lets you ask for -M virt, if GIC version X is supported 2) Just implement something in libvirt that checks what the kernel supports directly via the well-defined KVM interface and chooses the highest supported version per default. To me it sounds like we should just go ahead with (2) and document somehwere that if you get an error, you need to either manually change the gic version setting in your VM definition file or upgrade QEMU. Can someone with libvirt authority please advice whether (1) or (2) is what we need to do? Thanks, -Christoffer
On Tue, Feb 02, 2016 at 12:49:33PM +0100, Christoffer Dall wrote:
On Fri, Jan 22, 2016 at 02:44:32PM +0000, Daniel P. Berrange wrote:
On Wed, Jan 06, 2016 at 01:30:16PM +0000, Peter Maydell wrote:
On 6 January 2016 at 12:49, Andrea Bolognani <abologna@redhat.com> wrote:
That's correct, having a QMP command that lists the values gic-version can have on the current host would be just great.
If we had that, we could validate the GIC version chosen for a guest, and expose it in the capabilities XML so that higher-level tools can provide a list of choices to the user.
Please note that this QMP command would have to work regardless of the machine type selected on QEMU's command line, because libvirt always runs a QEMU binary with '-M none' when probing its capabilities.
On the other hand, if you don't tell us the machine type you care about then we can't tell you: (a) "this machine type doesn't support setting this property at all" (which applies to machines like vexpress-a15 which you can use with KVM on 32-bit hosts, and of course also to all the non-KVM models)
We have just recently merged support for registering properties against classes instead of object instances. There is also a proposed command to allow querying the list of properties against a class
https://lists.gnu.org/archive/html/qemu-devel/2016-01/msg04348.html
So if we now update the machine types to register their properties against the class instead of object, then we can query what properties are present against each machine type, while still using '-M none'.
(b) "this machine type only supports GIC versions X and Y even if the host supports more" (this is currently only hypothetical, though, since we only have the property on 'virt'. it would only happen if in the future we needed something other than '2' or '3' or 'host' I think.)
Our introspection support in QOM only allows us to say that a property is a particular type (int / enum / str / whatever). We don't have any way to expose info about what subset of possible values for a type are permitted. So I don't see any near term way to inform apps that the gic property accepts values x, y and but not z.
IMHO, we shouldn't try to overthink this. Libvirt can query the host to find out what GIC versions are supported and default to using the most recent version, on the basis that people are likely to have a matching QEMU. We can just rely on QEMU to report error if we pass it a version it doesn't support and not try to pre-emptively check before launch. The key is just getting the default right IMHO.
This sounds fine to me.
However, not being familiar with the internals of libvirt I really can't just what the right implementation approach here is.
As I understand we need to either:
1) Add a QMP command that lets you ask for -M virt, if GIC version X is supported
2) Just implement something in libvirt that checks what the kernel supports directly via the well-defined KVM interface and chooses the highest supported version per default.
To me it sounds like we should just go ahead with (2) and document somehwere that if you get an error, you need to either manually change the gic version setting in your VM definition file or upgrade QEMU.
Can someone with libvirt authority please advice whether (1) or (2) is what we need to do?
IMHO we should just go for 2. Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On Tue, Feb 02, 2016 at 12:10:10PM +0000, Daniel P. Berrange wrote:
On Tue, Feb 02, 2016 at 12:49:33PM +0100, Christoffer Dall wrote:
On Fri, Jan 22, 2016 at 02:44:32PM +0000, Daniel P. Berrange wrote:
On Wed, Jan 06, 2016 at 01:30:16PM +0000, Peter Maydell wrote:
On 6 January 2016 at 12:49, Andrea Bolognani <abologna@redhat.com> wrote:
That's correct, having a QMP command that lists the values gic-version can have on the current host would be just great.
If we had that, we could validate the GIC version chosen for a guest, and expose it in the capabilities XML so that higher-level tools can provide a list of choices to the user.
Please note that this QMP command would have to work regardless of the machine type selected on QEMU's command line, because libvirt always runs a QEMU binary with '-M none' when probing its capabilities.
On the other hand, if you don't tell us the machine type you care about then we can't tell you: (a) "this machine type doesn't support setting this property at all" (which applies to machines like vexpress-a15 which you can use with KVM on 32-bit hosts, and of course also to all the non-KVM models)
We have just recently merged support for registering properties against classes instead of object instances. There is also a proposed command to allow querying the list of properties against a class
https://lists.gnu.org/archive/html/qemu-devel/2016-01/msg04348.html
So if we now update the machine types to register their properties against the class instead of object, then we can query what properties are present against each machine type, while still using '-M none'.
(b) "this machine type only supports GIC versions X and Y even if the host supports more" (this is currently only hypothetical, though, since we only have the property on 'virt'. it would only happen if in the future we needed something other than '2' or '3' or 'host' I think.)
Our introspection support in QOM only allows us to say that a property is a particular type (int / enum / str / whatever). We don't have any way to expose info about what subset of possible values for a type are permitted. So I don't see any near term way to inform apps that the gic property accepts values x, y and but not z.
This actually doesn't matter for the v2 vs. v3 case. The gic-version property doesn't exist at all for v2-only QEMU. Although maybe the gic-version property should be reworked. Instead of gic-version=<highest-supported>, we could create one boolean property per version supported, i.e. gicv2, gicv3, gicv4...
IMHO, we shouldn't try to overthink this. Libvirt can query the host to find out what GIC versions are supported and default to using the most recent version, on the basis that people are likely to have a matching QEMU. We can just rely on QEMU to report error if we pass it a version it doesn't support and not try to pre-emptively check before launch. The key is just getting the default right IMHO.
This sounds fine to me.
However, not being familiar with the internals of libvirt I really can't just what the right implementation approach here is.
As I understand we need to either:
1) Add a QMP command that lets you ask for -M virt, if GIC version X is supported
2) Just implement something in libvirt that checks what the kernel supports directly via the well-defined KVM interface and chooses the highest supported version per default.
To me it sounds like we should just go ahead with (2) and document somehwere that if you get an error, you need to either manually change the gic version setting in your VM definition file or upgrade QEMU.
Can someone with libvirt authority please advice whether (1) or (2) is what we need to do?
IMHO we should just go for 2.
I'm not familiar enough with libvirt, nor the use of QMP, to really argue one way or another, but I find it a bit strange that we'd prefer libvirt to query two entities over one. And, why should the libvirt installed on a particular host prefer gicv3 as the default, just because KVM supports it, even when QEMU does not? The default will fail every time until QEMU is upgraded, which may not be necessary/desired. Finally, I thought we were trying to get away from relying on QEMU's error messages to make any sort of decisions. I don't know what else libvirt queries directly from KVM, but IMO, it should be a long-term goal to stop doing it, not add to more. Besides libvirt then properly selecting defaults that both KVM and QEMU support, it would allow /dev/kvm to have QEMU-only selinux labels applied. Thanks, drew
On 2 February 2016 at 12:59, Andrew Jones <drjones@redhat.com> wrote:
This actually doesn't matter for the v2 vs. v3 case. The gic-version property doesn't exist at all for v2-only QEMU. Although maybe the gic-version property should be reworked. Instead of gic-version=<highest-supported>, we could create one boolean property per version supported, i.e. gicv2, gicv3, gicv4...
QEMU 2.5 released with the gic-version property, so we can't just drop it now; that would break users' command lines. We'd need a strong argument for why it's worth adding new properties that duplicate the legacy syntax. thanks - PMM
On Tue, Feb 02, 2016 at 01:15:49PM +0000, Peter Maydell wrote:
On 2 February 2016 at 12:59, Andrew Jones <drjones@redhat.com> wrote:
This actually doesn't matter for the v2 vs. v3 case. The gic-version property doesn't exist at all for v2-only QEMU. Although maybe the gic-version property should be reworked. Instead of gic-version=<highest-supported>, we could create one boolean property per version supported, i.e. gicv2, gicv3, gicv4...
QEMU 2.5 released with the gic-version property, so we can't just drop it now; that would break users' command lines. We'd need a strong argument for why it's worth adding new properties that duplicate the legacy syntax.
I agree we should have a good argument to justify messing with it, but until we start versioning mach-virt, then I don't think anybody should depend on mach-virt command lines working everywhere. Every time we add a new property, and a user adds it to their command line, then they've just broken that command line for older QEMU. Yes, I know the upgrade path is a bit different, but still... Anyway, properties are changeable with versioned machines. drew
On 2 February 2016 at 14:04, Andrew Jones <drjones@redhat.com> wrote:
I agree we should have a good argument to justify messing with it, but until we start versioning mach-virt, then I don't think anybody should depend on mach-virt command lines working everywhere. Every time we add a new property, and a user adds it to their command line, then they've just broken that command line for older QEMU. Yes, I know the upgrade path is a bit different, but still... Anyway, properties are changeable with versioned machines.
Clearly command lines that use options added in newer QEMU won't work on older QEMU, but we do make a serious effort to continue to support older commandlines on newer QEMU, even if there is no versioned migration support. Upgrading QEMU and expecting your old configs to keep working is a reasonable user expectation. thanks -- PMM
On Tue, Feb 02, 2016 at 01:59:26PM +0100, Andrew Jones wrote:
On Tue, Feb 02, 2016 at 12:10:10PM +0000, Daniel P. Berrange wrote:
On Tue, Feb 02, 2016 at 12:49:33PM +0100, Christoffer Dall wrote:
On Fri, Jan 22, 2016 at 02:44:32PM +0000, Daniel P. Berrange wrote:
On Wed, Jan 06, 2016 at 01:30:16PM +0000, Peter Maydell wrote:
On 6 January 2016 at 12:49, Andrea Bolognani <abologna@redhat.com> wrote:
That's correct, having a QMP command that lists the values gic-version can have on the current host would be just great.
If we had that, we could validate the GIC version chosen for a guest, and expose it in the capabilities XML so that higher-level tools can provide a list of choices to the user.
Please note that this QMP command would have to work regardless of the machine type selected on QEMU's command line, because libvirt always runs a QEMU binary with '-M none' when probing its capabilities.
On the other hand, if you don't tell us the machine type you care about then we can't tell you: (a) "this machine type doesn't support setting this property at all" (which applies to machines like vexpress-a15 which you can use with KVM on 32-bit hosts, and of course also to all the non-KVM models)
We have just recently merged support for registering properties against classes instead of object instances. There is also a proposed command to allow querying the list of properties against a class
https://lists.gnu.org/archive/html/qemu-devel/2016-01/msg04348.html
So if we now update the machine types to register their properties against the class instead of object, then we can query what properties are present against each machine type, while still using '-M none'.
(b) "this machine type only supports GIC versions X and Y even if the host supports more" (this is currently only hypothetical, though, since we only have the property on 'virt'. it would only happen if in the future we needed something other than '2' or '3' or 'host' I think.)
Our introspection support in QOM only allows us to say that a property is a particular type (int / enum / str / whatever). We don't have any way to expose info about what subset of possible values for a type are permitted. So I don't see any near term way to inform apps that the gic property accepts values x, y and but not z.
This actually doesn't matter for the v2 vs. v3 case. The gic-version property doesn't exist at all for v2-only QEMU. Although maybe the gic-version property should be reworked. Instead of gic-version=<highest-supported>, we could create one boolean property per version supported, i.e. gicv2, gicv3, gicv4...
IMHO, we shouldn't try to overthink this. Libvirt can query the host to find out what GIC versions are supported and default to using the most recent version, on the basis that people are likely to have a matching QEMU. We can just rely on QEMU to report error if we pass it a version it doesn't support and not try to pre-emptively check before launch. The key is just getting the default right IMHO.
This sounds fine to me.
However, not being familiar with the internals of libvirt I really can't just what the right implementation approach here is.
As I understand we need to either:
1) Add a QMP command that lets you ask for -M virt, if GIC version X is supported
2) Just implement something in libvirt that checks what the kernel supports directly via the well-defined KVM interface and chooses the highest supported version per default.
To me it sounds like we should just go ahead with (2) and document somehwere that if you get an error, you need to either manually change the gic version setting in your VM definition file or upgrade QEMU.
Can someone with libvirt authority please advice whether (1) or (2) is what we need to do?
IMHO we should just go for 2.
I'm not familiar enough with libvirt, nor the use of QMP, to really argue one way or another, but I find it a bit strange that we'd prefer libvirt to query two entities over one. And, why should the libvirt installed on a particular host prefer gicv3 as the default, just because KVM supports it, even when QEMU does not?
I think the assumption here is that if you install a recent libvirt you also install a recent QEMU. You always have the risk of things not working if you have too old a QEMU, right?
The default will fail every time until QEMU is upgraded, which may not be necessary/desired. Finally, I thought we were trying to get away from relying on QEMU's error messages to make any sort of decisions.
This I have no idea, if libvirt wants to go in the direction of doing anything humanly possible to avoid QEMU errors, then (2) is probably not the right approach. On the other hand, there's always the possibility that something fails, so you probably always have to deal with QEMU's errors. I don't think anyone here suggested that libvirt should try to run QEMU with something, catch an error, and then run QEMU with other options, if that's what you meant?
I don't know what else libvirt queries directly from KVM, but IMO, it should be a long-term goal to stop doing it, not add to more. Besides libvirt then properly selecting defaults that both KVM and QEMU support, it would allow /dev/kvm to have QEMU-only selinux labels applied.
That's really for the libvirt community and maintainers to say, which direction they're going in. As I've tried to indicate before, we don't mind doing some work here as a bunch of Linaro members care about this, but we just want to know from the libvirt people what they want...? Thanks, -Christoffer
On Tue, Feb 02, 2016 at 03:05:28PM +0100, Christoffer Dall wrote:
On Tue, Feb 02, 2016 at 01:59:26PM +0100, Andrew Jones wrote:
On Tue, Feb 02, 2016 at 12:10:10PM +0000, Daniel P. Berrange wrote:
On Tue, Feb 02, 2016 at 12:49:33PM +0100, Christoffer Dall wrote:
On Fri, Jan 22, 2016 at 02:44:32PM +0000, Daniel P. Berrange wrote:
On Wed, Jan 06, 2016 at 01:30:16PM +0000, Peter Maydell wrote:
On 6 January 2016 at 12:49, Andrea Bolognani <abologna@redhat.com> wrote: > That's correct, having a QMP command that lists the values gic-version > can have on the current host would be just great. > > If we had that, we could validate the GIC version chosen for a guest, > and expose it in the capabilities XML so that higher-level tools can > provide a list of choices to the user. > > Please note that this QMP command would have to work regardless of the > machine type selected on QEMU's command line, because libvirt always > runs a QEMU binary with '-M none' when probing its capabilities.
On the other hand, if you don't tell us the machine type you care about then we can't tell you: (a) "this machine type doesn't support setting this property at all" (which applies to machines like vexpress-a15 which you can use with KVM on 32-bit hosts, and of course also to all the non-KVM models)
We have just recently merged support for registering properties against classes instead of object instances. There is also a proposed command to allow querying the list of properties against a class
https://lists.gnu.org/archive/html/qemu-devel/2016-01/msg04348.html
So if we now update the machine types to register their properties against the class instead of object, then we can query what properties are present against each machine type, while still using '-M none'.
(b) "this machine type only supports GIC versions X and Y even if the host supports more" (this is currently only hypothetical, though, since we only have the property on 'virt'. it would only happen if in the future we needed something other than '2' or '3' or 'host' I think.)
Our introspection support in QOM only allows us to say that a property is a particular type (int / enum / str / whatever). We don't have any way to expose info about what subset of possible values for a type are permitted. So I don't see any near term way to inform apps that the gic property accepts values x, y and but not z.
This actually doesn't matter for the v2 vs. v3 case. The gic-version property doesn't exist at all for v2-only QEMU. Although maybe the gic-version property should be reworked. Instead of gic-version=<highest-supported>, we could create one boolean property per version supported, i.e. gicv2, gicv3, gicv4...
IMHO, we shouldn't try to overthink this. Libvirt can query the host to find out what GIC versions are supported and default to using the most recent version, on the basis that people are likely to have a matching QEMU. We can just rely on QEMU to report error if we pass it a version it doesn't support and not try to pre-emptively check before launch. The key is just getting the default right IMHO.
This sounds fine to me.
However, not being familiar with the internals of libvirt I really can't just what the right implementation approach here is.
As I understand we need to either:
1) Add a QMP command that lets you ask for -M virt, if GIC version X is supported
2) Just implement something in libvirt that checks what the kernel supports directly via the well-defined KVM interface and chooses the highest supported version per default.
To me it sounds like we should just go ahead with (2) and document somehwere that if you get an error, you need to either manually change the gic version setting in your VM definition file or upgrade QEMU.
Can someone with libvirt authority please advice whether (1) or (2) is what we need to do?
IMHO we should just go for 2.
I'm not familiar enough with libvirt, nor the use of QMP, to really argue one way or another, but I find it a bit strange that we'd prefer libvirt to query two entities over one. And, why should the libvirt installed on a particular host prefer gicv3 as the default, just because KVM supports it, even when QEMU does not?
I think the assumption here is that if you install a recent libvirt you also install a recent QEMU. You always have the risk of things not working if you have too old a QEMU, right?
I'll grant that recent libvirt probably does imply recent QEMU. If we need to make assumptions for this solution, that's probably not a horrible one.
The default will fail every time until QEMU is upgraded, which may not be necessary/desired. Finally, I thought we were trying to get away from relying on QEMU's error messages to make any sort of decisions.
This I have no idea, if libvirt wants to go in the direction of doing anything humanly possible to avoid QEMU errors, then (2) is probably not the right approach.
On the other hand, there's always the possibility that something fails, so you probably always have to deal with QEMU's errors. I don't think anyone here suggested that libvirt should try to run QEMU with something, catch an error, and then run QEMU with other options, if that's what you meant?
I was thinking that, but now, after rereading, I see the proposal was just to propagate the error. So any error results in the same decision, which is just to quit, and that's reasonable.
I don't know what else libvirt queries directly from KVM, but IMO, it should be a long-term goal to stop doing it, not add to more. Besides libvirt then properly selecting defaults that both KVM and QEMU support, it would allow /dev/kvm to have QEMU-only selinux labels applied.
That's really for the libvirt community and maintainers to say, which direction they're going in.
Agreed. I'm just thinking out loud, perhaps too noisily...
As I've tried to indicate before, we don't mind doing some work here as a bunch of Linaro members care about this, but we just want to know from the libvirt people what they want...?
Yup. I look forward to more of them chiming in. I believe Andrea has been doing some experimenting. Thanks, drew
On 02/02/2016 07:05 AM, Christoffer Dall wrote:
I'm not familiar enough with libvirt, nor the use of QMP, to really argue one way or another, but I find it a bit strange that we'd prefer libvirt to query two entities over one. And, why should the libvirt installed on a particular host prefer gicv3 as the default, just because KVM supports it, even when QEMU does not?
I think the assumption here is that if you install a recent libvirt you also install a recent QEMU. You always have the risk of things not working if you have too old a QEMU, right?
Libvirt exists for providing back-compat glue. The following combinations are supported: old libvirt, old qemu new libvirt, old qemu new libvirt, new qemu and it is only this combination that might require a libvirt upgrade to work correctly: old libvirt, new qemu -- Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
On Tue, Feb 2, 2016 at 4:52 PM, Eric Blake <eblake@redhat.com> wrote:
On 02/02/2016 07:05 AM, Christoffer Dall wrote:
I'm not familiar enough with libvirt, nor the use of QMP, to really argue one way or another, but I find it a bit strange that we'd prefer libvirt to query two entities over one. And, why should the libvirt installed on a particular host prefer gicv3 as the default, just because KVM supports it, even when QEMU does not?
I think the assumption here is that if you install a recent libvirt you also install a recent QEMU. You always have the risk of things not working if you have too old a QEMU, right?
Libvirt exists for providing back-compat glue. The following combinations are supported:
old libvirt, old qemu new libvirt, old qemu new libvirt, new qemu
and it is only this combination that might require a libvirt upgrade to work correctly:
old libvirt, new qemu
ok, so that would mean we need to implement a QMP command to tell us which gic versions are supported for a given machine. Current possible responses are "2", "3" and "2,3" and we also need to add code to libvirt to try that QMP command, and if it doesn't exist, fall back to not specifying gic-version, using the old-qemu compatible default of providing a gicv2 to guests, and if the QMP command exists, use the newest gic-version. users can then always override this behavior by directly specifying a gic version "host", "2", or "3" in their xml file. any objections? -Christoffer
On Tue, 2016-02-02 at 16:58 +0100, Christoffer Dall wrote:
ok, so that would mean we need to implement a QMP command to tell us which gic versions are supported for a given machine. Current possible responses are "2", "3" and "2,3" and we also need to add code to libvirt to try that QMP command, and if it doesn't exist, fall back to not specifying gic-version, using the old-qemu compatible default of providing a gicv2 to guests, and if the QMP command exists, use the newest gic-version. users can then always override this behavior by directly specifying a gic version "host", "2", or "3" in their xml file. any objections?
Dan voiced his preference for probing the host GIC versions from libvirt and just passing that to QEMU dealing with any failure later on, but I think that was mostly to keep things simple and not because the QMP command approach was wrong? IOW Dan, if we went ahead with the QMP command approach, would you oppose it? Peter Xu has posted some RFC QEMU patches yesterday... Cheers. -- Andrea Bolognani Software Engineer - Virtualization Team
Our introspection support in QOM only allows us to say
is a particular type (int / enum / str / whatever). We don't have any way to expose info about what subset of possible values for a type are permitted. So I don't see any near term way to inform apps
On Tue, 2016-02-02 at 13:59 +0100, Andrew Jones wrote: that a property that the
gic property accepts values x, y and but not z. This actually doesn't matter for the v2 vs. v3 case. The gic-version property doesn't exist at all for v2-only QEMU. Although maybe the gic-version property should be reworked. Instead of gic-version=<highest-supported>, we could create one boolean property per version supported, i.e. gicv2, gicv3, gicv4...
Hold on, so "gic-version=3" means "support all GIC versions up to 3", not "support GIC version 3 only"? I thought the latter.
2) Just implement something in libvirt that checks what the kernel supports directly via the well-defined KVM interface and chooses the highest supported version per default. [...] I'm not familiar enough with libvirt, nor the use of QMP, to really argue one way or another, but I find it a bit strange that we'd prefer libvirt to query two entities over one. And, why should the libvirt installed on a particular host prefer gicv3 as the default, just because KVM supports it, even when QEMU does not? The default will fail every time until QEMU is upgraded, which may not be necessary/desired.
Shouldn't the default be "host", to mean "whatever the host supports", rather than a specific version based either on host or QEMU probing? That should work for every QEMU version, right?
Finally, I thought we were trying to get away from relying on QEMU's error messages to make any sort of decisions.
We wouldn't be looking at the error message and base our decision on that, we would just display it to the user if QEMU fails to run. That already happens for a bunch of other situations, so I don't think it's really a problem, especially because libvirt can't possibly be expected to catch every possible QEMU failure and sugar-coat it before reporting it to the user.
I don't know what else libvirt queries directly from KVM, but IMO, it should be a long-term goal to stop doing it, not add to more. Besides libvirt then properly selecting defaults that both KVM and QEMU support, it would allow /dev/kvm to have QEMU-only selinux labels applied.
One thing that comes to mind is the number of threads per subcores on ppc64 hosts, and I don't think that's the kind of information QEMU would provide via QMP. In the short term we definitely need libvirt to be able to pass "gic-version=host" to QEMU, and I'm working on a patch that enables that feature. As I've already voiced in this thread, my feeling is that the probing should happen in one place only (QEMU) and libvirt should merely query that information and report it back to the user, to avoid any possible disagreement. On the other hand, Dan has plenty more experience and also knowledge that spans the whole stack, so in general I trust his opinion :) One other way to handle this would be to simply report the GIC versions *libvirt* supports, and let the user pick either the default ("host", which should work anywhere) or a specific version, which might or might not actually be accepted by QEMU. I think there are other places in libvirt where this approach is used, even though is probably not the most user friendly... Cheers. -- Andrea Bolognani Software Engineer - Virtualization Team
Hello!
Shouldn't the default be "host", to mean "whatever the host supports", rather than a specific version based either on host or QEMU probing?
No, because: 1) Older qemu, which does not support this option, uses v2. 2) It also depends on whether KVM or TCG is used. Currently we can have v3 only with KVM, but when software emulation is implemented, it can be used in all cases, regardless of host KVM restrictions.
That should work for every QEMU version, right?
Wrong. See (1) above. Kind regards, Pavel Fedin Senior Engineer Samsung Electronics Research center Russia
On 2 February 2016 at 15:09, Andrea Bolognani <abologna@redhat.com> wrote:
Hold on, so "gic-version=3" means "support all GIC versions up to 3", not "support GIC version 3 only"? I thought the latter.
In the current implemented syntax, it means "expose a version 3 GIC to the guest". You can't simultaneously give the guest a GICv3 and a GICv2. Whether the host actually supports our doing this is the thing we can test at runtime (and currently handle by making vm creation fail if you asked for something the host can't do, just as we do if you say '-cpu cortex-a57' on a non-A57 host).
2) Just implement something in libvirt that checks what the kernel supports directly via the well-defined KVM interface and chooses the highest supported version per default. [...] I'm not familiar enough with libvirt, nor the use of QMP, to really argue one way or another, but I find it a bit strange that we'd prefer libvirt to query two entities over one. And, why should the libvirt installed on a particular host prefer gicv3 as the default, just because KVM supports it, even when QEMU does not? The default will fail every time until QEMU is upgraded, which may not be necessary/desired.
Shouldn't the default be "host", to mean "whatever the host supports", rather than a specific version based either on host or QEMU probing?
That should work for every QEMU version, right?
For the existing default, the horse has already bolted. The default (from QEMU's POV) is "2", because this retains command line compatibility with old command lines, which didn't specify because the option didn't exist and which assume the guest will be given a GICv2. (This also makes the handling of -gic-version match -cpu : if you don't specify you get a default which you might or might not want but which is the same thing it's always been, ie A15 and GICv2; if you want 'host' you must specify it; 'host' doesn't work for TCG, only KVM.) thanks -- PMM
On 01/05/2016 05:37 AM, Andrea Bolognani wrote:
If I misunderstood, how would you like QEMU to report this?
I'm not 100% sure about the best interface here (hopefully other people can weigh in) but I imagine there would be some sort of QMP command that lists the GIC versions supported by the QEMU binary, or some text in the output of 'qemu -help' that can be parsed to obtain such list.
It must be available in at least a QMP command; libvirt no longer parses qemu -help output. -- Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
On Tue, 2015-12-15 at 10:35 -0600, Andrew Jones wrote:
On Tue, Dec 15, 2015 at 04:03:13PM +0000, Peter Maydell wrote:
On 15 December 2015 at 14:12, Martin Kletzander <mkletzan@redhat.com> wrote:
On Tue, Dec 15, 2015 at 09:41:02AM +0000, Peter Maydell wrote:
On 15 December 2015 at 09:36, Martin Kletzander <mkletzan@redhat.com> wrote:
We do pass some options, for example, you can restrict the GIC to v2: https://libvirt.org/formatdomain.html#elementsFeatures
That could be modified to work for your purpose IIUC, right?
What does that option do to the QEMU command line? The documentation doesn't seem to say.
It appends gic-version= to the machine option, if bunch of conditions are met -- it has to be ARM machine, it has to be something else than version 2, and so on, more details are visible in the code and I won't cover them here. I hope that's understandable.
OK. So it should be sufficient just to have <gic version='host'/> which will then provide whatever VGIC the host machine can do with hardware acceleration.
This is probably good for guests that happy with both. Guests that need/want a specific choice will put their integer there, and then we need a way to do a capabilities check before launching that guest on an arbitrary host.
That won't work at the moment since we're expecting the value to be a number, but extending it so that 'host' is accepted as well should be easy to do in a backwards-compatible way. The behaviour of the value '2' has been defined as "don't specify any GIC version" though, and I'm afraid we'll be unable to change that because it would affect existing guests. Martin can probably weigh in. Having a way for libvirt to know what values can be used on the specific machines it's running on it's another topic altogether - is there a good way to detect it? Is that based on the hardware, on the QEMU binary, on the machine type? Cheers. -- Andrea Bolognani Software Engineer - Virtualization Team
On 15 December 2015 at 16:53, Andrea Bolognani <abologna@redhat.com> wrote:
The behaviour of the value '2' has been defined as "don't specify any GIC version" though, and I'm afraid we'll be unable to change that because it would affect existing guests. Martin can probably weigh in.
QEMU also treats "nothing specified" as meaning "2", at least for the virt board, so that part is fine. (The option is board specific anyway.) -- PMM
On 12/15/2015 11:53 AM, Andrea Bolognani wrote:
On Tue, 2015-12-15 at 10:35 -0600, Andrew Jones wrote:
On Tue, Dec 15, 2015 at 04:03:13PM +0000, Peter Maydell wrote:
On 15 December 2015 at 14:12, Martin Kletzander <mkletzan@redhat.com> wrote:
On Tue, Dec 15, 2015 at 09:41:02AM +0000, Peter Maydell wrote:
On 15 December 2015 at 09:36, Martin Kletzander <mkletzan@redhat.com> wrote:
We do pass some options, for example, you can restrict the GIC to v2: https://libvirt.org/formatdomain.html#elementsFeatures
That could be modified to work for your purpose IIUC, right?
What does that option do to the QEMU command line? The documentation doesn't seem to say.
It appends gic-version= to the machine option, if bunch of conditions are met -- it has to be ARM machine, it has to be something else than version 2, and so on, more details are visible in the code and I won't cover them here. I hope that's understandable.
OK. So it should be sufficient just to have <gic version='host'/> which will then provide whatever VGIC the host machine can do with hardware acceleration.
This is probably good for guests that happy with both. Guests that need/want a specific choice will put their integer there, and then we need a way to do a capabilities check before launching that guest on an arbitrary host.
That won't work at the moment since we're expecting the value to be a number, but extending it so that 'host' is accepted as well should be easy to do in a backwards-compatible way.
The behaviour of the value '2' has been defined as "don't specify any GIC version" though, and I'm afraid we'll be unable to change that because it would affect existing guests. Martin can probably weigh in.
Having a way for libvirt to know what values can be used on the specific machines it's running on it's another topic altogether - is there a good way to detect it? Is that based on the hardware, on the QEMU binary, on the machine type?
If one digs into the qemu_command code though they would find that if "<gic>" were supplied in the domain XML without the version='#', then we don't generate the command line option with any "gic*" string. qemuBuildMachineArgStr() if (def->features[VIR_DOMAIN_FEATURE_GIC] == VIR_TRISTATE_SWITCH_ON) { if (def->gic_version) { ... virBufferAsprintf(&buf, ",gic-version=%d", def->gic_version); } } Perhaps the "else" to (def->gic_version) could be "gic-version=host". Would that suffice? or be the right thing to do? John
On Tue, Dec 15, 2015 at 05:53:43PM +0100, Andrea Bolognani wrote:
On Tue, 2015-12-15 at 10:35 -0600, Andrew Jones wrote:
On Tue, Dec 15, 2015 at 04:03:13PM +0000, Peter Maydell wrote:
On 15 December 2015 at 14:12, Martin Kletzander <mkletzan@redhat.com> wrote:
On Tue, Dec 15, 2015 at 09:41:02AM +0000, Peter Maydell wrote:
On 15 December 2015 at 09:36, Martin Kletzander <mkletzan@redhat.com> wrote:
We do pass some options, for example, you can restrict the GIC to v2: https://libvirt.org/formatdomain.html#elementsFeatures
That could be modified to work for your purpose IIUC, right?
What does that option do to the QEMU command line? The documentation doesn't seem to say.
It appends gic-version= to the machine option, if bunch of conditions are met -- it has to be ARM machine, it has to be something else than version 2, and so on, more details are visible in the code and I won't cover them here. I hope that's understandable.
OK. So it should be sufficient just to have <gic version='host'/> which will then provide whatever VGIC the host machine can do with hardware acceleration.
This is probably good for guests that happy with both. Guests that need/want a specific choice will put their integer there, and then we need a way to do a capabilities check before launching that guest on an arbitrary host.
That won't work at the moment since we're expecting the value to be a number, but extending it so that 'host' is accepted as well should be easy to do in a backwards-compatible way.
The behaviour of the value '2' has been defined as "don't specify any GIC version" though, and I'm afraid we'll be unable to change that because it would affect existing guests. Martin can probably weigh in.
Having a way for libvirt to know what values can be used on the specific machines it's running on it's another topic altogether - is there a good way to detect it? Is that based on the hardware, on the QEMU binary, on the machine type?
My understanding of libvirt host feature probing is that it generally queries QEMU, rather than the host directly. QEMU using Andre's ioctl test, when asked with QEMU's command line using some '\?' type of input, seems like the typical approach to me. Thanks, drew
On Tue, Dec 15, 2015 at 10:36:32AM +0100, Martin Kletzander wrote:
On Mon, Dec 14, 2015 at 10:47:26AM -0600, Andrew Jones wrote:
On Mon, Dec 14, 2015 at 12:31:59PM +0100, Christoffer Dall wrote:
Hi all,
I'm trying to figure out what the correct solution for libvirt support for ARM KVM guests with GICv3 is.
The challenge is that QEMU's virt machine model defaults to GICv2 unless you specify an additional "-machine gic-version=host" parameter to QEMU, in which case your VM gets the same GIC version as your host.
Now, I'm told that this is inconvenient for libvirt, because it typically does not pass any -machine options, is that correct?
We do pass some options, for example, you can restrict the GIC to v2: https://libvirt.org/formatdomain.html#elementsFeatures
That could be modified to work for your purpose IIUC, right?
How do I figure out which values the version can take (v2, v3, host?) and which QEMU cmdline that translates into? (sorry for my libvirt n00b-ness here.) Thanks, -Christoffer
On Tue, 2015-12-15 at 11:11 +0100, Christoffer Dall wrote:
On Tue, Dec 15, 2015 at 10:36:32AM +0100, Martin Kletzander wrote:
On Mon, Dec 14, 2015 at 10:47:26AM -0600, Andrew Jones wrote:
On Mon, Dec 14, 2015 at 12:31:59PM +0100, Christoffer Dall wrote:
Hi all,
I'm trying to figure out what the correct solution for libvirt support for ARM KVM guests with GICv3 is.
The challenge is that QEMU's virt machine model defaults to GICv2 unless you specify an additional "-machine gic-version=host" parameter to QEMU, in which case your VM gets the same GIC version as your host.
Now, I'm told that this is inconvenient for libvirt, because it typically does not pass any -machine options, is that correct?
We do pass some options, for example, you can restrict the GIC to v2: https://libvirt.org/formatdomain.html#elementsFeatures
That could be modified to work for your purpose IIUC, right?
How do I figure out which values the version can take (v2, v3, host?) and which QEMU cmdline that translates into?
The parser will currently accept any numeric value, including eg. 1 and 42. The value 2 is special and is silently ignored for backwards compatibility; any other value will cause libvirt to add the gic-version option to the machine type. Cheers. -- Andrea Bolognani Software Engineer - Virtualization Team
On Mon, Dec 14, 2015 at 10:47:26AM -0600, Andrew Jones wrote:
On Mon, Dec 14, 2015 at 12:31:59PM +0100, Christoffer Dall wrote:
Hi all,
I'm trying to figure out what the correct solution for libvirt support for ARM KVM guests with GICv3 is.
The challenge is that QEMU's virt machine model defaults to GICv2 unless you specify an additional "-machine gic-version=host" parameter to QEMU, in which case your VM gets the same GIC version as your host.
Now, I'm told that this is inconvenient for libvirt, because it typically does not pass any -machine options, is that correct?
If that is correct, then my guess is that that's because it's used to the qemu machine model automatically selecting appropriate defaults, and for pc, due to versioning, there are several machine model types to choose from. mach-virt hasn't started versioning yet.
Further, there are two additional complications: First, some guest images may not include GICv3 support. For Linux, this is relatively old kernels (prior to v3.17), but other guests may only support GICv2.
Second, some hardware platforms only support GICv2 guests, some only support GICv3 guests, and some support both GICv2 and GICv3 guests (those hosts with a GICv3 with the backwards compatibility support).
It is unclear to me how libvirt users will relate to these constraints. For example, in an OpenStack deployment, will it somehow be easy to tell which nodes support which GIC version such that administrators can choose compatible VM images, or how does that work?
libvirt has the concept of capabilities that it can negotiate. I know this is used heavily with x86 cpu features that some guests depend on. I think a guest dependency on gic version could fit here.
Sounds fair to me, but I don't know what is required to actually implement/use this for GIC support on ARM?
The end goal here is to determine if we need to add any functionality to QEMU to better support libvirt, and what kind of features/glue need to be added to libvirt for this to work.
I've been starting to wonder if we should consider breaking mach-virt into multiple types. For example, one that focuses on big, enterprise-y type configs (only 64-bit guests, only virtio-pci, extending the size of main memory beyond 30G, for sure, and possibly even moving the PCIE_MMIO_HIGH mapping to extend beyond 511G. It'd even be good to remap the redistributor space to get more than 123 vcpus.
I kind of feel that we should wait until someone says they're trying to run VMs that they cannot, which we haven't heard about yet, AFAIK. My fear with the multiple mach-virt versions is a lot of duplicated code in QEMU or at least a much more complicated board definition. Peter and QEMU developers will have better and stronger opinions on this than me though.
Making new mach-virt types and versioning those types would probably avoid needing much libvirt changes, but teaching libvirt (if it doesn't know already) how to manage machine parameters to effectively create new types as needed, also sounds good to me.
It's not clear to me if adding a new mach-virt type solves the GICv3 question or if that is an othorgonal discussion. -Christoffer
Hello! Nobody ever CCed me, so i saw this topic only occasionally... Sorry for the late reply.
The challenge is that QEMU's virt machine model defaults to GICv2 unless you specify an additional "-machine gic-version=host" parameter to QEMU, in which case your VM gets the same GIC version as your host.
Or you can specify gic-version=3, and get v3. The idea behind this was to be backwards-compatible. In old qemu versions you don't specify anything, and get v2. With new qemu it's the same. The only problem here is that on some GICv3 hosts you can run only GICv3 KVM guests, because vGICv2 support is optional. With TCG (once GICv3 software emulation is implemented) you wouldn't have this limitation at all. Actually, i am also proposing kernel patches which enable to use software-emulated GIC with KVM (http://www.spinics.net/lists/kvm/msg124539.html, http://www.spinics.net/lists/kvm/msg124545.html), but got no review so far. I also have support in qemu for this. With these patches you would actually be able to run GICv2 guest on GICv3-only host, but with some performance penalty, just by adding kernel_irqchip=off machine option.
Now, I'm told that this is inconvenient for libvirt, because it typically does not pass any -machine options, is that correct?
It's incorrect: http://www.redhat.com/archives/libvir-list/2015-September/msg01067.html
Second, some hardware platforms only support GICv2 guests, some only support GICv3 guests, and some support both GICv2 and GICv3 guests (those hosts with a GICv3 with the backwards compatibility support).
It is unclear to me how libvirt users will relate to these constraints.
Actually, i see GICv3 and GICv2 "virt" actually as different machine types. Because in real world these would be different motherboards, even if we imagine that CPUs are the same. Actually, initially i proposed to simply add "virt-v3" (or "virt-gicv3", if you want) machine type. I believe this would resolve many things automatically (obviously in order to run a machine you need an OS new enough to support it, and all applications would just see the new machine choice). But Peter strongly disagreed with that and asked me to add an option to existing virt machine instead.
The end goal here is to determine if we need to add any functionality to QEMU to better support libvirt, and what kind of features/glue need to be added to libvirt for this to work.
I'd say that it's not libvirt's work now. We have support in the XML for this option, so now it's up to application to be able to specify this. For example, applications, once "virt" type is chosen, should be able to present one more control to choose between GIC versions. Just think of it as "machine sub-type". Kind regards, Pavel Fedin Expert Engineer Samsung Electronics Research center Russia
participants (12)
-
Andre Przywara -
Andrea Bolognani -
Andrew Jones -
Christoffer Dall -
Daniel P. Berrange -
Eric Blake -
John Ferlan -
Kashyap Chamarthy -
Laszlo Ersek -
Martin Kletzander -
Pavel Fedin -
Peter Maydell