[libvirt] [PATCH V2] Expose all CPU features in host definition
[V2 - Improve the error handling] I've opened BZ 697141 on this as I would consider it more a bug than a feature request. Anyway, to re-iterate my rationale from the BZ: The virConnectGetCapabilities API describes the host capabilities by returning an XML description that includes the CPU model name and a set of CPU features. The problem is that any features that are part of the CPU model are not explicitly listed, they are assumed to be part of the definition of that CPU model. This makes it extremely difficult for the caller of this API to check for the presence of a specific CPU feature, the caller would have to know what features are part of which CPU models, a very daunting task. This patch solves this problem by having the API return a model name, as it currently does, but it will also explicitly list all of the CPU features that are present. This would make it much easier for a caller of this API to check for specific features. Signed-off-by: Don Dugger <donald.d.dugger@intel.com> -- src/cpu/cpu_x86.c | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+) diff --git a/src/cpu/cpu_x86.c b/src/cpu/cpu_x86.c index 5d479c2..098ab05 100644 --- a/src/cpu/cpu_x86.c +++ b/src/cpu/cpu_x86.c @@ -1296,6 +1296,35 @@ x86GuestData(virCPUDefPtr host, return x86Compute(host, guest, data, message); } +static int +x86AddFeatures(virCPUDefPtr cpu, + struct x86_map *map) +{ + int ret; + const struct x86_model *candidate; + const struct x86_feature *feature = map->features; + + candidate = map->models; + while (candidate != NULL) { + if (STREQ(cpu->model, candidate->name)) + break; + candidate = candidate->next; + } + if (!candidate) { + virReportError(VIR_ERR_INTERNAL_ERROR, + "Odd, %s not a known CPU model\n", cpu->model); + return -1; + } + while (feature != NULL) { + if (x86DataIsSubset(candidate->data, feature->data)) { + if ((ret = virCPUDefAddFeature(cpu, feature->name, VIR_CPU_FEATURE_DISABLE)) < 0) + return ret; + } + feature = feature->next; + } + return 0; +} + static int x86Decode(virCPUDefPtr cpu, @@ -1383,6 +1412,8 @@ x86Decode(virCPUDefPtr cpu, goto out; } + if (x86AddFeatures(cpuModel, map) < 0) + goto out; cpu->model = cpuModel->model; cpu->vendor = cpuModel->vendor; cpu->nfeatures = cpuModel->nfeatures; -- 1.7.10.4
On Tue, May 28, 2013 at 16:11:57 -0600, Don Dugger wrote:
[V2 - Improve the error handling]
I've opened BZ 697141 on this as I would consider it more
I guess you meant BZ 967141.
a bug than a feature request. Anyway, to re-iterate my rationale from the BZ:
The virConnectGetCapabilities API describes the host capabilities by returning an XML description that includes the CPU model name and a set of CPU features. The problem is that any features that are part of the CPU model are not explicitly listed, they are assumed to be part of the definition of that CPU model. This makes it extremely difficult for the caller of this API to check for the presence of a specific CPU feature, the caller would have to know what features are part of which CPU models, a very daunting task.
This patch solves this problem by having the API return a model name, as it currently does, but it will also explicitly list all of the CPU features that are present. This would make it much easier for a caller of this API to check for specific features.
It's actually the desired behavior and not a bug. We went that way for several reasons. First, given how QEMU/KVM works, the fact that a CPU feature is detected by libvirt in host CPU and reported in capabilities XML does not mean that the feature will be available to guests. This is the reason why we have virConnectCompareCPU. You can create a guest CPU definition you would like to see in a guest and call virConnectCompareCPU on it to check if that guest CPU can be run on that particular host. And providing all features in capabilities XML would just make the XML larger with no practical benefit. In other words, as Martin already said, we don't really want to expand the CPU model in capabilities XML. However, implementing a new API that would take a CPU XML definition and return the exact set of CPU features (including those included in the CPU model) seems like a useful addition. Jirka
On Thu, May 30, 2013 at 01:48:32PM +0200, Jiri Denemark wrote:
On Tue, May 28, 2013 at 16:11:57 -0600, Don Dugger wrote:
[V2 - Improve the error handling]
I've opened BZ 697141 on this as I would consider it more
I guess you meant BZ 967141.
a bug than a feature request. Anyway, to re-iterate my rationale from the BZ:
The virConnectGetCapabilities API describes the host capabilities by returning an XML description that includes the CPU model name and a set of CPU features. The problem is that any features that are part of the CPU model are not explicitly listed, they are assumed to be part of the definition of that CPU model. This makes it extremely difficult for the caller of this API to check for the presence of a specific CPU feature, the caller would have to know what features are part of which CPU models, a very daunting task.
This patch solves this problem by having the API return a model name, as it currently does, but it will also explicitly list all of the CPU features that are present. This would make it much easier for a caller of this API to check for specific features.
It's actually the desired behavior and not a bug. We went that way for several reasons. First, given how QEMU/KVM works, the fact that a CPU feature is detected by libvirt in host CPU and reported in capabilities XML does not mean that the feature will be available to guests. This is the reason why we have virConnectCompareCPU. You can create a guest CPU definition you would like to see in a guest and call virConnectCompareCPU on it to check if that guest CPU can be run on that particular host. And providing all features in capabilities XML would just make the XML larger with no practical benefit.
The fact that host CPU features at not necessarily available to a guest has no impact on whether or not we report all features that the host supports, that issue remains in either case. Also, I created some test programs and, for test cases of CPU definitions that were identical, incompatible or a subset of the host the virConnectCompareCPU API worked the same with or without my patch to expose all host CPU features. So far the only argument I see against explicitly exposing all features is that it makes the XML description slightly larger. Given that the increased size exposes useful information I think that is a good trade off to make.
In other words, as Martin already said, we don't really want to expand the CPU model in capabilities XML. However, implementing a new API that would take a CPU XML definition and return the exact set of CPU features (including those included in the CPU model) seems like a useful addition.
Independent from the discussion of what the virConnectGetCapabilites API should return I agree, an API to decompose a specified CPU XML definition seems like a good idea, I'll work on creating a second patch for that.
Jirka
-- Don Dugger "Censeo Toto nos in Kansa esse decisse." - D. Gale n0ano@n0ano.com Ph: 303/443-3786
Jiri- Have you had a chance to look over this email? Barring any counter examples I still think my patch is appropriate. On Fri, May 31, 2013 at 01:04:17PM -0600, Don Dugger wrote:
On Thu, May 30, 2013 at 01:48:32PM +0200, Jiri Denemark wrote:
On Tue, May 28, 2013 at 16:11:57 -0600, Don Dugger wrote:
[V2 - Improve the error handling]
I've opened BZ 697141 on this as I would consider it more
I guess you meant BZ 967141.
a bug than a feature request. Anyway, to re-iterate my rationale from the BZ:
The virConnectGetCapabilities API describes the host capabilities by returning an XML description that includes the CPU model name and a set of CPU features. The problem is that any features that are part of the CPU model are not explicitly listed, they are assumed to be part of the definition of that CPU model. This makes it extremely difficult for the caller of this API to check for the presence of a specific CPU feature, the caller would have to know what features are part of which CPU models, a very daunting task.
This patch solves this problem by having the API return a model name, as it currently does, but it will also explicitly list all of the CPU features that are present. This would make it much easier for a caller of this API to check for specific features.
It's actually the desired behavior and not a bug. We went that way for several reasons. First, given how QEMU/KVM works, the fact that a CPU feature is detected by libvirt in host CPU and reported in capabilities XML does not mean that the feature will be available to guests. This is the reason why we have virConnectCompareCPU. You can create a guest CPU definition you would like to see in a guest and call virConnectCompareCPU on it to check if that guest CPU can be run on that particular host. And providing all features in capabilities XML would just make the XML larger with no practical benefit.
The fact that host CPU features at not necessarily available to a guest has no impact on whether or not we report all features that the host supports, that issue remains in either case. Also, I created some test programs and, for test cases of CPU definitions that were identical, incompatible or a subset of the host the virConnectCompareCPU API worked the same with or without my patch to expose all host CPU features.
So far the only argument I see against explicitly exposing all features is that it makes the XML description slightly larger. Given that the increased size exposes useful information I think that is a good trade off to make.
In other words, as Martin already said, we don't really want to expand the CPU model in capabilities XML. However, implementing a new API that would take a CPU XML definition and return the exact set of CPU features (including those included in the CPU model) seems like a useful addition.
Independent from the discussion of what the virConnectGetCapabilites API should return I agree, an API to decompose a specified CPU XML definition seems like a good idea, I'll work on creating a second patch for that.
Jirka
-- Don Dugger "Censeo Toto nos in Kansa esse decisse." - D. Gale n0ano@n0ano.com Ph: 303/443-3786
-- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
-- Don Dugger "Censeo Toto nos in Kansa esse decisse." - D. Gale n0ano@n0ano.com Ph: 303/443-3786
Hi Don, I'm sorry for the delay... On Fri, May 31, 2013 at 13:04:17 -0600, Don Dugger wrote:
On Thu, May 30, 2013 at 01:48:32PM +0200, Jiri Denemark wrote:
On Tue, May 28, 2013 at 16:11:57 -0600, Don Dugger wrote:
The virConnectGetCapabilities API describes the host capabilities by returning an XML description that includes the CPU model name and a set of CPU features. The problem is that any features that are part of the CPU model are not explicitly listed, they are assumed to be part of the definition of that CPU model. This makes it extremely difficult for the caller of this API to check for the presence of a specific CPU feature, the caller would have to know what features are part of which CPU models, a very daunting task.
This patch solves this problem by having the API return a model name, as it currently does, but it will also explicitly list all of the CPU features that are present. This would make it much easier for a caller of this API to check for specific features.
It's actually the desired behavior and not a bug. We went that way for several reasons. First, given how QEMU/KVM works, the fact that a CPU feature is detected by libvirt in host CPU and reported in capabilities XML does not mean that the feature will be available to guests. This is the reason why we have virConnectCompareCPU. You can create a guest CPU definition you would like to see in a guest and call virConnectCompareCPU on it to check if that guest CPU can be run on that particular host. And providing all features in capabilities XML would just make the XML larger with no practical benefit.
The fact that host CPU features at not necessarily available to a guest has no impact on whether or not we report all features that the host supports, that issue remains in either case.
I was just trying to say that it doesn't provide anything more than "it's supported by the host CPU", which gives mostly no value in the context of libvirt. Can you explain more what the use case is in which a virt client would need to know what specific feature are supported by host CPU? I feel like we should avoid people from being under the impression that they can actually use the CPU capabilities for checking whether a host can run guests that require specific CPU features. Moreover, there's one thing that makes this issue even more complicated. As the CPU model is the most useful part of host CPU capabilities (it should be the best CPU model supported by the host), I wanted to extend the probing code to select the right model even if it's missing some features that we expect to be supported by that model. In other words, if host CPU is, e.g., SandyBridge but has some basic feature disabled, we would detect it as the best model which did not have the disabled feature, which is not optimal. We'd ideally detect the CPU as SandyBridge with just the feature disabled. That is, another reason for having feature list relative to the CPU model. On the other hand, it might be hard or even impossible to do without breaking compatibility and perhaps even doubtful without involving emulator, which would make it impossible to do within capabilities XML. Jirka
On Fri, Jun 14, 2013 at 03:06:52PM +0200, Jiri Denemark wrote:
Hi Don,
I'm sorry for the delay...
NP.
On Fri, May 31, 2013 at 13:04:17 -0600, Don Dugger wrote:
On Thu, May 30, 2013 at 01:48:32PM +0200, Jiri Denemark wrote:
On Tue, May 28, 2013 at 16:11:57 -0600, Don Dugger wrote:
The virConnectGetCapabilities API describes the host capabilities by returning an XML description that includes the CPU model name and a set of CPU features. The problem is that any features that are part of the CPU model are not explicitly listed, they are assumed to be part of the definition of that CPU model. This makes it extremely difficult for the caller of this API to check for the presence of a specific CPU feature, the caller would have to know what features are part of which CPU models, a very daunting task.
This patch solves this problem by having the API return a model name, as it currently does, but it will also explicitly list all of the CPU features that are present. This would make it much easier for a caller of this API to check for specific features.
It's actually the desired behavior and not a bug. We went that way for several reasons. First, given how QEMU/KVM works, the fact that a CPU feature is detected by libvirt in host CPU and reported in capabilities XML does not mean that the feature will be available to guests. This is the reason why we have virConnectCompareCPU. You can create a guest CPU definition you would like to see in a guest and call virConnectCompareCPU on it to check if that guest CPU can be run on that particular host. And providing all features in capabilities XML would just make the XML larger with no practical benefit.
The fact that host CPU features at not necessarily available to a guest has no impact on whether or not we report all features that the host supports, that issue remains in either case.
I was just trying to say that it doesn't provide anything more than "it's supported by the host CPU", which gives mostly no value in the context of libvirt. Can you explain more what the use case is in which a virt client would need to know what specific feature are supported by host CPU? I feel like we should avoid people from being under the impression that they can actually use the CPU capabilities for checking whether a host can run guests that require specific CPU features.
The specific use case I'm trying to address is a cloud environment where, with hundreds of hosts, you might want to schedule an instance to a host that supports a particular HW acceleration, like AES/NI. I agree, what I `really` want is an API that shows the capabilities of a specific guest that could be created on the host but, absent that API, at least knowing that a host doesn't support the feature is more information than I can get right now. I'm willing to walk before I run and start from knowing what capabilities the `host` has (and create a new API to report on actual guests in the not too distant future). Independent from my use case I still think this API is wrong in that it is effectively obfuscating by omission. Given that there's no good way, given a model name, to know what features are in that model then, if the API is going to report anything, it should report all features. It can still give the model name, I completely support that, but I see nothing wrong with reporting the model name and, in addition, the list of all features supported by the host. It comes down to, other than a minor size issue in the resulting XML file, what's the advantage of reporting features relative to model? If we report all features then people that care about the model will still have that info and people that care about specific features will now have a usable list of features they can rely upon.
Moreover, there's one thing that makes this issue even more complicated. As the CPU model is the most useful part of host CPU capabilities (it should be the best CPU model supported by the host), I wanted to extend the probing code to select the right model even if it's missing some features that we expect to be supported by that model. In other words, if host CPU is, e.g., SandyBridge but has some basic feature disabled, we would detect it as the best model which did not have the disabled feature, which is not optimal. We'd ideally detect the CPU as SandyBridge with just the feature disabled. That is, another reason for having feature list relative to the CPU model. On the other hand, it might be hard or even impossible to do without breaking compatibility and perhaps even doubtful without involving emulator, which would make it impossible to do within capabilities XML.
If I understand you, you're proposing that you report the host as being a SandyBridge when it doesn't support all SandyBridge features? I don't like that idea as it seems really confusing. Taken to the extreme, this would mean you could take a Nehalem host and report it as a SanyBridge that lacks some features. I don't see the point of that.
Jirka
-- Don Dugger "Censeo Toto nos in Kansa esse decisse." - D. Gale n0ano@n0ano.com Ph: 303/443-3786
On Fri, Jun 14, 2013 at 12:32:35 -0600, Don Dugger wrote:
On Fri, Jun 14, 2013 at 03:06:52PM +0200, Jiri Denemark wrote:
I was just trying to say that it doesn't provide anything more than "it's supported by the host CPU", which gives mostly no value in the context of libvirt. Can you explain more what the use case is in which a virt client would need to know what specific feature are supported by host CPU? I feel like we should avoid people from being under the impression that they can actually use the CPU capabilities for checking whether a host can run guests that require specific CPU features.
The specific use case I'm trying to address is a cloud environment where, with hundreds of hosts, you might want to schedule an instance to a host that supports a particular HW acceleration, like AES/NI. I agree, what I `really` want is an API that shows the capabilities of a specific guest that could be created on the host but, absent that API, at least knowing that a host doesn't support the feature is more information than I can get right now.
Hmm, fair enough. Let's wait a few days for Daniel to return from vacation in case he wants to express his opinion here.
Moreover, there's one thing that makes this issue even more complicated. As the CPU model is the most useful part of host CPU capabilities (it should be the best CPU model supported by the host), I wanted to extend the probing code to select the right model even if it's missing some features that we expect to be supported by that model. In other words, if host CPU is, e.g., SandyBridge but has some basic feature disabled, we would detect it as the best model which did not have the disabled feature, which is not optimal. We'd ideally detect the CPU as SandyBridge with just the feature disabled. That is, another reason for having feature list relative to the CPU model. On the other hand, it might be hard or even impossible to do without breaking compatibility and perhaps even doubtful without involving emulator, which would make it impossible to do within capabilities XML.
If I understand you, you're proposing that you report the host as being a SandyBridge when it doesn't support all SandyBridge features? I don't like that idea as it seems really confusing. Taken to the extreme, this would mean you could take a Nehalem host and report it as a SanyBridge that lacks some features. I don't see the point of that.
Yes and no :-) I'd like to report the CPU as a SandyBridge if it really is a SandyBridge but lacks a feature that we require to be present for us to detect the CPU as a SandyBridge. There are (or at least were) features that can actually be disabled on the host, e.g., by BIOS and I remember that doing so may result in that particular feature to be disabled in CPUID. For example, if NX (not sure if that's the right example that is actually masked from CPUID) is disabled, than the CPU still is a SandyBridge but libvirt would detect it as something generic and old as NX is part of all modern CPU models libvirt supports. However, as I said, this could be hard or even impossible to do in a compatible manner. Jirka
On Mon, Jun 17, 2013 at 10:27:36AM +0200, Jiri Denemark wrote:
On Fri, Jun 14, 2013 at 12:32:35 -0600, Don Dugger wrote:
On Fri, Jun 14, 2013 at 03:06:52PM +0200, Jiri Denemark wrote:
I was just trying to say that it doesn't provide anything more than "it's supported by the host CPU", which gives mostly no value in the context of libvirt. Can you explain more what the use case is in which a virt client would need to know what specific feature are supported by host CPU? I feel like we should avoid people from being under the impression that they can actually use the CPU capabilities for checking whether a host can run guests that require specific CPU features.
The specific use case I'm trying to address is a cloud environment where, with hundreds of hosts, you might want to schedule an instance to a host that supports a particular HW acceleration, like AES/NI. I agree, what I `really` want is an API that shows the capabilities of a specific guest that could be created on the host but, absent that API, at least knowing that a host doesn't support the feature is more information than I can get right now.
Hmm, fair enough. Let's wait a few days for Daniel to return from vacation in case he wants to express his opinion here.
So, any progress here?
Moreover, there's one thing that makes this issue even more complicated. As the CPU model is the most useful part of host CPU capabilities (it should be the best CPU model supported by the host), I wanted to extend the probing code to select the right model even if it's missing some features that we expect to be supported by that model. In other words, if host CPU is, e.g., SandyBridge but has some basic feature disabled, we would detect it as the best model which did not have the disabled feature, which is not optimal. We'd ideally detect the CPU as SandyBridge with just the feature disabled. That is, another reason for having feature list relative to the CPU model. On the other hand, it might be hard or even impossible to do without breaking compatibility and perhaps even doubtful without involving emulator, which would make it impossible to do within capabilities XML.
If I understand you, you're proposing that you report the host as being a SandyBridge when it doesn't support all SandyBridge features? I don't like that idea as it seems really confusing. Taken to the extreme, this would mean you could take a Nehalem host and report it as a SanyBridge that lacks some features. I don't see the point of that.
Yes and no :-) I'd like to report the CPU as a SandyBridge if it really is a SandyBridge but lacks a feature that we require to be present for us to detect the CPU as a SandyBridge. There are (or at least were) features that can actually be disabled on the host, e.g., by BIOS and I remember that doing so may result in that particular feature to be disabled in CPUID. For example, if NX (not sure if that's the right example that is actually masked from CPUID) is disabled, than the CPU still is a SandyBridge but libvirt would detect it as something generic and old as NX is part of all modern CPU models libvirt supports. However, as I said, this could be hard or even impossible to do in a compatible manner.
Jirka
-- Don Dugger "Censeo Toto nos in Kansa esse decisse." - D. Gale n0ano@n0ano.com Ph: 303/443-3786
On Thu, Jun 27, 2013 at 10:35:58AM -0600, Don Dugger wrote:
On Mon, Jun 17, 2013 at 10:27:36AM +0200, Jiri Denemark wrote:
On Fri, Jun 14, 2013 at 12:32:35 -0600, Don Dugger wrote:
On Fri, Jun 14, 2013 at 03:06:52PM +0200, Jiri Denemark wrote:
I was just trying to say that it doesn't provide anything more than "it's supported by the host CPU", which gives mostly no value in the context of libvirt. Can you explain more what the use case is in which a virt client would need to know what specific feature are supported by host CPU? I feel like we should avoid people from being under the impression that they can actually use the CPU capabilities for checking whether a host can run guests that require specific CPU features.
The specific use case I'm trying to address is a cloud environment where, with hundreds of hosts, you might want to schedule an instance to a host that supports a particular HW acceleration, like AES/NI. I agree, what I `really` want is an API that shows the capabilities of a specific guest that could be created on the host but, absent that API, at least knowing that a host doesn't support the feature is more information than I can get right now.
Hmm, fair enough. Let's wait a few days for Daniel to return from vacation in case he wants to express his opinion here.
So, any progress here?
I believe the cloud use case above is approaching the problem in the wrong way. We designed our APIs such that apps should never need to write logic for comparing CPU features directly. If an application has a set of CPU features it requires from the host, then it should use a libvirt API to query whether those features are available, not try to write the CPU fetaure comparison logic itself. You can already pretty much do this with te virConnectCompareCPU() method, by passing an XML document which specifies the AES/NI feature flag that you want to check for support of. Then libvirt will tell you whether the host CPU can support it. It is entirely possible to make use of this capability as is in OpenStack. Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On Fri, Jun 28, 2013 at 10:24:48AM +0100, Daniel P. Berrange wrote:
On Thu, Jun 27, 2013 at 10:35:58AM -0600, Don Dugger wrote:
On Mon, Jun 17, 2013 at 10:27:36AM +0200, Jiri Denemark wrote:
On Fri, Jun 14, 2013 at 12:32:35 -0600, Don Dugger wrote:
On Fri, Jun 14, 2013 at 03:06:52PM +0200, Jiri Denemark wrote:
I was just trying to say that it doesn't provide anything more than "it's supported by the host CPU", which gives mostly no value in the context of libvirt. Can you explain more what the use case is in which a virt client would need to know what specific feature are supported by host CPU? I feel like we should avoid people from being under the impression that they can actually use the CPU capabilities for checking whether a host can run guests that require specific CPU features.
The specific use case I'm trying to address is a cloud environment where, with hundreds of hosts, you might want to schedule an instance to a host that supports a particular HW acceleration, like AES/NI. I agree, what I `really` want is an API that shows the capabilities of a specific guest that could be created on the host but, absent that API, at least knowing that a host doesn't support the feature is more information than I can get right now.
Hmm, fair enough. Let's wait a few days for Daniel to return from vacation in case he wants to express his opinion here.
So, any progress here?
I believe the cloud use case above is approaching the problem in the wrong way. We designed our APIs such that apps should never need to write logic for comparing CPU features directly. If an application has a set of CPU features it requires from the host, then it should use a libvirt API to query whether those features are available, not try to write the CPU fetaure comparison logic itself.
You can already pretty much do this with te virConnectCompareCPU() method, by passing an XML document which specifies the AES/NI feature flag that you want to check for support of. Then libvirt will tell you whether the host CPU can support it. It is entirely possible to make use of this capability as is in OpenStack.
I don't think this would work with the way scheduling in OpenStack works. The problem is that the OpenStack scheduler doesn't want to query each node in the system on every schedule request (with 100s if not 1,000s of compute nodes this would not be practical). Instead the scheduler maintains info about all of the compute nodes and, when a request comes in, the scheduler identifies the best compute node for the request and then causes the VM to be started on that node. Apriori the scheduler doesn't even know which CPU features users are interested in, that information only becomes available when a schedule request comes in so trying to do a `virConnectCompareCPU()' call at that point in time is too late. Having said all of that I still think that conceptually the current `virConnectGetCapabilities()' is wrong. If the API is going to list any CPU features is should list all of them. Any user of the API that cares about CPU features probably cares about all of the features and a user that doesn't care about CPU features will just ignore that part of the definition anyway.
Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
-- Don Dugger "Censeo Toto nos in Kansa esse decisse." - D. Gale n0ano@n0ano.com Ph: 303/443-3786
On Fri, Jun 28, 2013 at 11:51:32AM -0600, Don Dugger wrote:
On Fri, Jun 28, 2013 at 10:24:48AM +0100, Daniel P. Berrange wrote:
On Thu, Jun 27, 2013 at 10:35:58AM -0600, Don Dugger wrote:
On Mon, Jun 17, 2013 at 10:27:36AM +0200, Jiri Denemark wrote:
On Fri, Jun 14, 2013 at 12:32:35 -0600, Don Dugger wrote:
On Fri, Jun 14, 2013 at 03:06:52PM +0200, Jiri Denemark wrote:
I was just trying to say that it doesn't provide anything more than "it's supported by the host CPU", which gives mostly no value in the context of libvirt. Can you explain more what the use case is in which a virt client would need to know what specific feature are supported by host CPU? I feel like we should avoid people from being under the impression that they can actually use the CPU capabilities for checking whether a host can run guests that require specific CPU features.
The specific use case I'm trying to address is a cloud environment where, with hundreds of hosts, you might want to schedule an instance to a host that supports a particular HW acceleration, like AES/NI. I agree, what I `really` want is an API that shows the capabilities of a specific guest that could be created on the host but, absent that API, at least knowing that a host doesn't support the feature is more information than I can get right now.
Hmm, fair enough. Let's wait a few days for Daniel to return from vacation in case he wants to express his opinion here.
So, any progress here?
I believe the cloud use case above is approaching the problem in the wrong way. We designed our APIs such that apps should never need to write logic for comparing CPU features directly. If an application has a set of CPU features it requires from the host, then it should use a libvirt API to query whether those features are available, not try to write the CPU fetaure comparison logic itself.
You can already pretty much do this with te virConnectCompareCPU() method, by passing an XML document which specifies the AES/NI feature flag that you want to check for support of. Then libvirt will tell you whether the host CPU can support it. It is entirely possible to make use of this capability as is in OpenStack.
I don't think this would work with the way scheduling in OpenStack works. The problem is that the OpenStack scheduler doesn't want to query each node in the system on every schedule request (with 100s if not 1,000s of compute nodes this would not be practical). Instead the scheduler maintains info about all of the compute nodes and, when a request comes in, the scheduler identifies the best compute node for the request and then causes the VM to be started on that node. Apriori the scheduler doesn't even know which CPU features users are interested in, that information only becomes available when a schedule request comes in so trying to do a `virConnectCompareCPU()' call at that point in time is too late.
I think your model for user interaction is wrong here. I don't believe that OpenStack should be directly exposing the ability for a user to explicitly request individual CPU flags for individual VMs. This is too risky from a cloud administrator POV, because it could result in a user monopolizing a small subset of machines in the guest with particular features. Instead an administrator should be defining new flavours with specific CPU feature characteristics. The user can then choose the flavour with the CPU characteristics. In this way the system can know ahead of time what flavours are possible on what host, and do you don't need to query all nodes at scheduling time. Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On Fri, Jun 28, 2013 at 09:03:55PM +0100, Daniel P. Berrange wrote:
On Fri, Jun 28, 2013 at 11:51:32AM -0600, Don Dugger wrote:
On Fri, Jun 28, 2013 at 10:24:48AM +0100, Daniel P. Berrange wrote:
On Thu, Jun 27, 2013 at 10:35:58AM -0600, Don Dugger wrote:
On Mon, Jun 17, 2013 at 10:27:36AM +0200, Jiri Denemark wrote:
On Fri, Jun 14, 2013 at 12:32:35 -0600, Don Dugger wrote:
On Fri, Jun 14, 2013 at 03:06:52PM +0200, Jiri Denemark wrote: > I was just trying to say that it doesn't provide anything more than > "it's supported by the host CPU", which gives mostly no value in the > context of libvirt. Can you explain more what the use case is in which a > virt client would need to know what specific feature are supported by > host CPU? I feel like we should avoid people from being under the > impression that they can actually use the CPU capabilities for checking > whether a host can run guests that require specific CPU features.
The specific use case I'm trying to address is a cloud environment where, with hundreds of hosts, you might want to schedule an instance to a host that supports a particular HW acceleration, like AES/NI. I agree, what I `really` want is an API that shows the capabilities of a specific guest that could be created on the host but, absent that API, at least knowing that a host doesn't support the feature is more information than I can get right now.
Hmm, fair enough. Let's wait a few days for Daniel to return from vacation in case he wants to express his opinion here.
So, any progress here?
I believe the cloud use case above is approaching the problem in the wrong way. We designed our APIs such that apps should never need to write logic for comparing CPU features directly. If an application has a set of CPU features it requires from the host, then it should use a libvirt API to query whether those features are available, not try to write the CPU fetaure comparison logic itself.
You can already pretty much do this with te virConnectCompareCPU() method, by passing an XML document which specifies the AES/NI feature flag that you want to check for support of. Then libvirt will tell you whether the host CPU can support it. It is entirely possible to make use of this capability as is in OpenStack.
I don't think this would work with the way scheduling in OpenStack works. The problem is that the OpenStack scheduler doesn't want to query each node in the system on every schedule request (with 100s if not 1,000s of compute nodes this would not be practical). Instead the scheduler maintains info about all of the compute nodes and, when a request comes in, the scheduler identifies the best compute node for the request and then causes the VM to be started on that node. Apriori the scheduler doesn't even know which CPU features users are interested in, that information only becomes available when a schedule request comes in so trying to do a `virConnectCompareCPU()' call at that point in time is too late.
I think your model for user interaction is wrong here. I don't believe that OpenStack should be directly exposing the ability for a user to explicitly request individual CPU flags for individual VMs. This is too risky from a cloud administrator POV, because it could result in a user monopolizing a small subset of machines in the guest with particular features. Instead an administrator should be defining new flavours with specific CPU feature characteristics. The user can
That's the exact mechanism that is being proposed, the ability to define a flavor that specifies capabilities that are required. The issue is that the flavor is defined independently from the scheduler, it's only when a schedule request is made that the flavor is presented to the scheduler and then the scheduler needs to identify which of 1,000s of nodes can satisfy that flavor. There shouldn't be a problem with users monopolizing machines because the cloud administrator defines the flavors and can charge appropriate premium prices for specific flavors.
then choose the flavour with the CPU characteristics. In this way the system can know ahead of time what flavours are possible on what host, and do you don't need to query all nodes at scheduling time.
Note I am not proposing that we make a query at schedule time. The system is already setup to have the compute nodes send configuration info to the scheduler, all I want to do is sent complete info (e.g. all of the CPU features).
Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
-- Don Dugger "Censeo Toto nos in Kansa esse decisse." - D. Gale n0ano@n0ano.com Ph: 303/443-3786
On Fri, Jun 28, 2013 at 02:26:02PM -0600, Don Dugger wrote:
On Fri, Jun 28, 2013 at 09:03:55PM +0100, Daniel P. Berrange wrote:
On Fri, Jun 28, 2013 at 11:51:32AM -0600, Don Dugger wrote:
On Fri, Jun 28, 2013 at 10:24:48AM +0100, Daniel P. Berrange wrote:
On Thu, Jun 27, 2013 at 10:35:58AM -0600, Don Dugger wrote:
On Mon, Jun 17, 2013 at 10:27:36AM +0200, Jiri Denemark wrote:
On Fri, Jun 14, 2013 at 12:32:35 -0600, Don Dugger wrote: > On Fri, Jun 14, 2013 at 03:06:52PM +0200, Jiri Denemark wrote: > > I was just trying to say that it doesn't provide anything more than > > "it's supported by the host CPU", which gives mostly no value in the > > context of libvirt. Can you explain more what the use case is in which a > > virt client would need to know what specific feature are supported by > > host CPU? I feel like we should avoid people from being under the > > impression that they can actually use the CPU capabilities for checking > > whether a host can run guests that require specific CPU features. > > The specific use case I'm trying to address is a cloud environment where, > with hundreds of hosts, you might want to schedule an instance to a host > that supports a particular HW acceleration, like AES/NI. I agree, what > I `really` want is an API that shows the capabilities of a specific guest > that could be created on the host but, absent that API, at least knowing > that a host doesn't support the feature is more information than I can get > right now.
Hmm, fair enough. Let's wait a few days for Daniel to return from vacation in case he wants to express his opinion here.
So, any progress here?
I believe the cloud use case above is approaching the problem in the wrong way. We designed our APIs such that apps should never need to write logic for comparing CPU features directly. If an application has a set of CPU features it requires from the host, then it should use a libvirt API to query whether those features are available, not try to write the CPU fetaure comparison logic itself.
You can already pretty much do this with te virConnectCompareCPU() method, by passing an XML document which specifies the AES/NI feature flag that you want to check for support of. Then libvirt will tell you whether the host CPU can support it. It is entirely possible to make use of this capability as is in OpenStack.
I don't think this would work with the way scheduling in OpenStack works. The problem is that the OpenStack scheduler doesn't want to query each node in the system on every schedule request (with 100s if not 1,000s of compute nodes this would not be practical). Instead the scheduler maintains info about all of the compute nodes and, when a request comes in, the scheduler identifies the best compute node for the request and then causes the VM to be started on that node. Apriori the scheduler doesn't even know which CPU features users are interested in, that information only becomes available when a schedule request comes in so trying to do a `virConnectCompareCPU()' call at that point in time is too late.
I think your model for user interaction is wrong here. I don't believe that OpenStack should be directly exposing the ability for a user to explicitly request individual CPU flags for individual VMs. This is too risky from a cloud administrator POV, because it could result in a user monopolizing a small subset of machines in the guest with particular features. Instead an administrator should be defining new flavours with specific CPU feature characteristics. The user can
That's the exact mechanism that is being proposed, the ability to define a flavor that specifies capabilities that are required. The issue is that the flavor is defined independently from the scheduler, it's only when a schedule request is made that the flavor is presented to the scheduler and then the scheduler needs to identify which of 1,000s of nodes can satisfy that flavor.
Every 60 seconds or so, every node issues an update indicating what its capabilities are. In that update the nodes could do the CPU compatbility checks and then include the list of which flavours they are capable of executing in their capability update, so that it is then available to the schedular when needed
then choose the flavour with the CPU characteristics. In this way the system can know ahead of time what flavours are possible on what host, and do you don't need to query all nodes at scheduling time.
Note I am not proposing that we make a query at schedule time. The system is already setup to have the compute nodes send configuration info to the scheduler, all I want to do is sent complete info (e.g. all of the CPU features).
Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On Mon, Jul 01, 2013 at 09:44:52AM +0100, Daniel P. Berrange wrote:
On Fri, Jun 28, 2013 at 02:26:02PM -0600, Don Dugger wrote:
On Fri, Jun 28, 2013 at 09:03:55PM +0100, Daniel P. Berrange wrote:
On Fri, Jun 28, 2013 at 11:51:32AM -0600, Don Dugger wrote:
On Fri, Jun 28, 2013 at 10:24:48AM +0100, Daniel P. Berrange wrote:
On Thu, Jun 27, 2013 at 10:35:58AM -0600, Don Dugger wrote:
On Mon, Jun 17, 2013 at 10:27:36AM +0200, Jiri Denemark wrote: > On Fri, Jun 14, 2013 at 12:32:35 -0600, Don Dugger wrote: > > On Fri, Jun 14, 2013 at 03:06:52PM +0200, Jiri Denemark wrote: > > > I was just trying to say that it doesn't provide anything more than > > > "it's supported by the host CPU", which gives mostly no value in the > > > context of libvirt. Can you explain more what the use case is in which a > > > virt client would need to know what specific feature are supported by > > > host CPU? I feel like we should avoid people from being under the > > > impression that they can actually use the CPU capabilities for checking > > > whether a host can run guests that require specific CPU features. > > > > The specific use case I'm trying to address is a cloud environment where, > > with hundreds of hosts, you might want to schedule an instance to a host > > that supports a particular HW acceleration, like AES/NI. I agree, what > > I `really` want is an API that shows the capabilities of a specific guest > > that could be created on the host but, absent that API, at least knowing > > that a host doesn't support the feature is more information than I can get > > right now. > > Hmm, fair enough. Let's wait a few days for Daniel to return from > vacation in case he wants to express his opinion here.
So, any progress here?
I believe the cloud use case above is approaching the problem in the wrong way. We designed our APIs such that apps should never need to write logic for comparing CPU features directly. If an application has a set of CPU features it requires from the host, then it should use a libvirt API to query whether those features are available, not try to write the CPU fetaure comparison logic itself.
You can already pretty much do this with te virConnectCompareCPU() method, by passing an XML document which specifies the AES/NI feature flag that you want to check for support of. Then libvirt will tell you whether the host CPU can support it. It is entirely possible to make use of this capability as is in OpenStack.
I don't think this would work with the way scheduling in OpenStack works. The problem is that the OpenStack scheduler doesn't want to query each node in the system on every schedule request (with 100s if not 1,000s of compute nodes this would not be practical). Instead the scheduler maintains info about all of the compute nodes and, when a request comes in, the scheduler identifies the best compute node for the request and then causes the VM to be started on that node. Apriori the scheduler doesn't even know which CPU features users are interested in, that information only becomes available when a schedule request comes in so trying to do a `virConnectCompareCPU()' call at that point in time is too late.
I think your model for user interaction is wrong here. I don't believe that OpenStack should be directly exposing the ability for a user to explicitly request individual CPU flags for individual VMs. This is too risky from a cloud administrator POV, because it could result in a user monopolizing a small subset of machines in the guest with particular features. Instead an administrator should be defining new flavours with specific CPU feature characteristics. The user can
That's the exact mechanism that is being proposed, the ability to define a flavor that specifies capabilities that are required. The issue is that the flavor is defined independently from the scheduler, it's only when a schedule request is made that the flavor is presented to the scheduler and then the scheduler needs to identify which of 1,000s of nodes can satisfy that flavor.
Every 60 seconds or so, every node issues an update indicating what its capabilities are. In that update the nodes could do the CPU compatbility checks and then include the list of which flavours they are capable of executing in their capability update, so that it is then available to the schedular when needed
This doesn't work for multiple reasons. 1. Ultimately, I want to remove the periodic capability update completely. The better technique is to update compute node state when the state changes, periodic updates are just extra overhead. 2. There's no concept of which nodes support which flavors so this is a completely new infrastructure that would have to be added to the scheduler. 3. There's no easy way for the compute node to know which flavors it supports. It doesn't know which filters are enabled in the scheduler so it doesn't know which clauses of a flavor actually apply (ignoring that the compute node would now have to duplicate the filtering mechanism from the scheduler even if it knew which filters were enabled). The virConnectGetCapabilities already returns a list of CPU features, all my patch does is have it explicitly return a complete set of features, which I think is the right thing to do and certainly simplifies my specific use case.
then choose the flavour with the CPU characteristics. In this way the system can know ahead of time what flavours are possible on what host, and do you don't need to query all nodes at scheduling time.
Note I am not proposing that we make a query at schedule time. The system is already setup to have the compute nodes send configuration info to the scheduler, all I want to do is sent complete info (e.g. all of the CPU features).
Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
-- Don Dugger "Censeo Toto nos in Kansa esse decisse." - D. Gale n0ano@n0ano.com Ph: 303/443-3786
On Mon, Jul 01, 2013 at 09:43:58AM -0600, Don Dugger wrote:
On Mon, Jul 01, 2013 at 09:44:52AM +0100, Daniel P. Berrange wrote:
On Fri, Jun 28, 2013 at 02:26:02PM -0600, Don Dugger wrote:
On Fri, Jun 28, 2013 at 09:03:55PM +0100, Daniel P. Berrange wrote:
On Fri, Jun 28, 2013 at 11:51:32AM -0600, Don Dugger wrote:
On Fri, Jun 28, 2013 at 10:24:48AM +0100, Daniel P. Berrange wrote:
On Thu, Jun 27, 2013 at 10:35:58AM -0600, Don Dugger wrote: > On Mon, Jun 17, 2013 at 10:27:36AM +0200, Jiri Denemark wrote: > > On Fri, Jun 14, 2013 at 12:32:35 -0600, Don Dugger wrote: > > > On Fri, Jun 14, 2013 at 03:06:52PM +0200, Jiri Denemark wrote: > > > > I was just trying to say that it doesn't provide anything more than > > > > "it's supported by the host CPU", which gives mostly no value in the > > > > context of libvirt. Can you explain more what the use case is in which a > > > > virt client would need to know what specific feature are supported by > > > > host CPU? I feel like we should avoid people from being under the > > > > impression that they can actually use the CPU capabilities for checking > > > > whether a host can run guests that require specific CPU features. > > > > > > The specific use case I'm trying to address is a cloud environment where, > > > with hundreds of hosts, you might want to schedule an instance to a host > > > that supports a particular HW acceleration, like AES/NI. I agree, what > > > I `really` want is an API that shows the capabilities of a specific guest > > > that could be created on the host but, absent that API, at least knowing > > > that a host doesn't support the feature is more information than I can get > > > right now. > > > > Hmm, fair enough. Let's wait a few days for Daniel to return from > > vacation in case he wants to express his opinion here. > > So, any progress here?
I believe the cloud use case above is approaching the problem in the wrong way. We designed our APIs such that apps should never need to write logic for comparing CPU features directly. If an application has a set of CPU features it requires from the host, then it should use a libvirt API to query whether those features are available, not try to write the CPU fetaure comparison logic itself.
You can already pretty much do this with te virConnectCompareCPU() method, by passing an XML document which specifies the AES/NI feature flag that you want to check for support of. Then libvirt will tell you whether the host CPU can support it. It is entirely possible to make use of this capability as is in OpenStack.
I don't think this would work with the way scheduling in OpenStack works. The problem is that the OpenStack scheduler doesn't want to query each node in the system on every schedule request (with 100s if not 1,000s of compute nodes this would not be practical). Instead the scheduler maintains info about all of the compute nodes and, when a request comes in, the scheduler identifies the best compute node for the request and then causes the VM to be started on that node. Apriori the scheduler doesn't even know which CPU features users are interested in, that information only becomes available when a schedule request comes in so trying to do a `virConnectCompareCPU()' call at that point in time is too late.
I think your model for user interaction is wrong here. I don't believe that OpenStack should be directly exposing the ability for a user to explicitly request individual CPU flags for individual VMs. This is too risky from a cloud administrator POV, because it could result in a user monopolizing a small subset of machines in the guest with particular features. Instead an administrator should be defining new flavours with specific CPU feature characteristics. The user can
That's the exact mechanism that is being proposed, the ability to define a flavor that specifies capabilities that are required. The issue is that the flavor is defined independently from the scheduler, it's only when a schedule request is made that the flavor is presented to the scheduler and then the scheduler needs to identify which of 1,000s of nodes can satisfy that flavor.
Every 60 seconds or so, every node issues an update indicating what its capabilities are. In that update the nodes could do the CPU compatbility checks and then include the list of which flavours they are capable of executing in their capability update, so that it is then available to the schedular when needed
This doesn't work for multiple reasons.
1. Ultimately, I want to remove the periodic capability update completely. The better technique is to update compute node state when the state changes, periodic updates are just extra overhead. 2. There's no concept of which nodes support which flavors so this is a completely new infrastructure that would have to be added to the scheduler. 3. There's no easy way for the compute node to know which flavors it supports. It doesn't know which filters are enabled in the scheduler so it doesn't know which clauses of a flavor actually apply (ignoring that the compute node would now have to duplicate the filtering mechanism from the scheduler even if it knew which filters were enabled).
The virConnectGetCapabilities already returns a list of CPU features, all my patch does is have it explicitly return a complete set of features, which I think is the right thing to do and certainly simplifies my specific use case.
Had any more thoughts on these issues?
then choose the flavour with the CPU characteristics. In this way the system can know ahead of time what flavours are possible on what host, and do you don't need to query all nodes at scheduling time.
Note I am not proposing that we make a query at schedule time. The system is already setup to have the compute nodes send configuration info to the scheduler, all I want to do is sent complete info (e.g. all of the CPU features).
Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
-- Don Dugger "Censeo Toto nos in Kansa esse decisse." - D. Gale n0ano@n0ano.com Ph: 303/443-3786
-- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
-- Don Dugger "Censeo Toto nos in Kansa esse decisse." - D. Gale n0ano@n0ano.com Ph: 303/443-3786
On Mon, Jul 01, 2013 at 09:43:58AM -0600, Don Dugger wrote:
On Mon, Jul 01, 2013 at 09:44:52AM +0100, Daniel P. Berrange wrote:
On Fri, Jun 28, 2013 at 02:26:02PM -0600, Don Dugger wrote:
On Fri, Jun 28, 2013 at 09:03:55PM +0100, Daniel P. Berrange wrote:
On Fri, Jun 28, 2013 at 11:51:32AM -0600, Don Dugger wrote:
On Fri, Jun 28, 2013 at 10:24:48AM +0100, Daniel P. Berrange wrote:
On Thu, Jun 27, 2013 at 10:35:58AM -0600, Don Dugger wrote: > On Mon, Jun 17, 2013 at 10:27:36AM +0200, Jiri Denemark wrote: > > On Fri, Jun 14, 2013 at 12:32:35 -0600, Don Dugger wrote: > > > On Fri, Jun 14, 2013 at 03:06:52PM +0200, Jiri Denemark wrote: > > > > I was just trying to say that it doesn't provide anything more than > > > > "it's supported by the host CPU", which gives mostly no value in the > > > > context of libvirt. Can you explain more what the use case is in which a > > > > virt client would need to know what specific feature are supported by > > > > host CPU? I feel like we should avoid people from being under the > > > > impression that they can actually use the CPU capabilities for checking > > > > whether a host can run guests that require specific CPU features. > > > > > > The specific use case I'm trying to address is a cloud environment where, > > > with hundreds of hosts, you might want to schedule an instance to a host > > > that supports a particular HW acceleration, like AES/NI. I agree, what > > > I `really` want is an API that shows the capabilities of a specific guest > > > that could be created on the host but, absent that API, at least knowing > > > that a host doesn't support the feature is more information than I can get > > > right now. > > > > Hmm, fair enough. Let's wait a few days for Daniel to return from > > vacation in case he wants to express his opinion here. > > So, any progress here?
I believe the cloud use case above is approaching the problem in the wrong way. We designed our APIs such that apps should never need to write logic for comparing CPU features directly. If an application has a set of CPU features it requires from the host, then it should use a libvirt API to query whether those features are available, not try to write the CPU fetaure comparison logic itself.
You can already pretty much do this with te virConnectCompareCPU() method, by passing an XML document which specifies the AES/NI feature flag that you want to check for support of. Then libvirt will tell you whether the host CPU can support it. It is entirely possible to make use of this capability as is in OpenStack.
I don't think this would work with the way scheduling in OpenStack works. The problem is that the OpenStack scheduler doesn't want to query each node in the system on every schedule request (with 100s if not 1,000s of compute nodes this would not be practical). Instead the scheduler maintains info about all of the compute nodes and, when a request comes in, the scheduler identifies the best compute node for the request and then causes the VM to be started on that node. Apriori the scheduler doesn't even know which CPU features users are interested in, that information only becomes available when a schedule request comes in so trying to do a `virConnectCompareCPU()' call at that point in time is too late.
I think your model for user interaction is wrong here. I don't believe that OpenStack should be directly exposing the ability for a user to explicitly request individual CPU flags for individual VMs. This is too risky from a cloud administrator POV, because it could result in a user monopolizing a small subset of machines in the guest with particular features. Instead an administrator should be defining new flavours with specific CPU feature characteristics. The user can
That's the exact mechanism that is being proposed, the ability to define a flavor that specifies capabilities that are required. The issue is that the flavor is defined independently from the scheduler, it's only when a schedule request is made that the flavor is presented to the scheduler and then the scheduler needs to identify which of 1,000s of nodes can satisfy that flavor.
Every 60 seconds or so, every node issues an update indicating what its capabilities are. In that update the nodes could do the CPU compatbility checks and then include the list of which flavours they are capable of executing in their capability update, so that it is then available to the schedular when needed
This doesn't work for multiple reasons.
1. Ultimately, I want to remove the periodic capability update completely. The better technique is to update compute node state when the state changes, periodic updates are just extra overhead. 2. There's no concept of which nodes support which flavors so this is a completely new infrastructure that would have to be added to the scheduler. 3. There's no easy way for the compute node to know which flavors it supports. It doesn't know which filters are enabled in the scheduler so it doesn't know which clauses of a flavor actually apply (ignoring that the compute node would now have to duplicate the filtering mechanism from the scheduler even if it knew which filters were enabled).
The virConnectGetCapabilities already returns a list of CPU features, all my patch does is have it explicitly return a complete set of features, which I think is the right thing to do and certainly simplifies my specific use case.
So have you had any more thoughts on this? I'd still like to get my patch incoporated into the tree.
then choose the flavour with the CPU characteristics. In this way the system can know ahead of time what flavours are possible on what host, and do you don't need to query all nodes at scheduling time.
Note I am not proposing that we make a query at schedule time. The system is already setup to have the compute nodes send configuration info to the scheduler, all I want to do is sent complete info (e.g. all of the CPU features).
Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
-- Don Dugger "Censeo Toto nos in Kansa esse decisse." - D. Gale n0ano@n0ano.com Ph: 303/443-3786
-- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
-- Don Dugger "Censeo Toto nos in Kansa esse decisse." - D. Gale n0ano@n0ano.com Ph: 303/443-3786
On Sun, Jul 14, 2013 at 11:43:57PM -0600, Don Dugger wrote:
On Mon, Jul 01, 2013 at 09:43:58AM -0600, Don Dugger wrote:
1. Ultimately, I want to remove the periodic capability update completely. The better technique is to update compute node state when the state changes, periodic updates are just extra overhead.
It is easy enough for a node to refresh its list of flavours upon change too, so that's not really a blocker.
2. There's no concept of which nodes support which flavors so this is a completely new infrastructure that would have to be added to the scheduler.
Again that's not a problem. We're not aiming for the least work solution here, we want the one that makes most sense from a design pov even if that requires extra dev work.
3. There's no easy way for the compute node to know which flavors it supports. It doesn't know which filters are enabled in the scheduler so it doesn't know which clauses of a flavor actually apply (ignoring that the compute node would now have to duplicate the filtering mechanism from the scheduler even if it knew which filters were enabled).
I don't think that's an issue either. When the node filters the list of flavours, it is doing so based on criteria about what it is technically capable of supporting at a hardware level. When the schedular is filtering flavours it is doing so based on operational rules. The node doesn't need to take over the filtering that the scedular currently does. They are complementary filtering rules. Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
I think we're getting a little side tracked on how the OpenStack scheduler works (although I'll still respond to issues with my current use case). I still come back to the issue that I feel the API is not returning the proper information (ignoring how that info is used by the caller). Note that the API already returns a list of CPU features, it just hides many of those features behind a code name. We really should return the complete list of features for those interested in specific features, while still providing the code name for those interested in that shorthand. Note that I can think of another use case where the complete feature list makes sense. Imagine a dashboard that lists all platforms in a datacenter/cloud. You might want to have the dashboard filter its display to all platforms with Westmere CPUs, you could just as easily want to filter the display to all platfomrs with AES/NI capable CPUs. On Mon, Jul 15, 2013 at 11:05:07AM +0100, Daniel P. Berrange wrote:
On Sun, Jul 14, 2013 at 11:43:57PM -0600, Don Dugger wrote:
On Mon, Jul 01, 2013 at 09:43:58AM -0600, Don Dugger wrote:
1. Ultimately, I want to remove the periodic capability update completely. The better technique is to update compute node state when the state changes, periodic updates are just extra overhead.
It is easy enough for a node to refresh its list of flavours upon change too, so that's not really a blocker.
Since there is no notification given to anyone when a flavor is created or deleted this is a problem. Given the compute nodes don't know when flavors change they don't know when to do any sort of `refresh`.
2. There's no concept of which nodes support which flavors so this is a completely new infrastructure that would have to be added to the scheduler.
Again that's not a problem. We're not aiming for the least work solution here, we want the one that makes most sense from a design pov even if that requires extra dev work.
I'm trying to apply Occam's razor here. Having the API return a complete list of capabilities is the simplest solution. With that minor change there's no need to add a new infrastructure to track nodes vs. flavors.
3. There's no easy way for the compute node to know which flavors it supports. It doesn't know which filters are enabled in the scheduler so it doesn't know which clauses of a flavor actually apply (ignoring that the compute node would now have to duplicate the filtering mechanism from the scheduler even if it knew which filters were enabled).
I don't think that's an issue either. When the node filters the list of flavours, it is doing so based on criteria about what it is technically capable of supporting at a hardware level. When the schedular is filtering flavours it is doing so based on operational rules. The node doesn't need to take over the filtering that the scedular currently does. They are complementary filtering rules.
I'm not sure what you're trying to say here. All I'm trying to discover is what the node is `technically capable of supporting at a hardware level'. Given that info we can then create flavors that the sheduler can use to better utilize the hardware resources available. Without duplicating the filtering work that the scheduler is already doing there's no way for a compute node to know whether it supports a flavor or not.
Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
-- Don Dugger "Censeo Toto nos in Kansa esse decisse." - D. Gale n0ano@n0ano.com Ph: 303/443-3786
On Mon, Jul 15, 2013 at 01:45:56PM -0600, Don Dugger wrote: [snip] We've been going back & forth a while now, and I don't think we're ever going to agree on this matter. To move this forward, I'm prepared to accept an API addition that lets apps get the full list of CPU feature flags, provided that there is no change to the existing default behaviour of our APIs. eg it should be an opt-in decision IIUC, in essence you want a way to take the host CPU description (as generated in the XML for virConnectGetCapabilties), and expand this see the full set of features that are able to be presented to the guest. I think this could probably be done by adding a flag to the existing virConnectBaselineCPU API. eg something like capsxml = virConnectGetCapabilties() cpuxml = ...extract the <cpu> bit of capsxml... newxml = virConnectBaselineCPU(conn, &cpuxml, 1, VIR_CONNECT_BASELINE_CPU_EXPAND_FEATURES) Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On Thu, Jul 18, 2013 at 02:37:33PM +0100, Daniel P. Berrange wrote:
On Mon, Jul 15, 2013 at 01:45:56PM -0600, Don Dugger wrote: [snip]
We've been going back & forth a while now, and I don't think we're ever going to agree on this matter.
To move this forward, I'm prepared to accept an API addition that lets apps get the full list of CPU feature flags, provided that there is no change to the existing default behaviour of our APIs. eg it should be an opt-in decision
You took the words out of my mouth :-), I was actually thinking about proposing something along these lines. I agree, this is a good compromise and I'll work up a patch along the lines you propose.
IIUC, in essence you want a way to take the host CPU description (as generated in the XML for virConnectGetCapabilties), and expand this see the full set of features that are able to be presented to the guest. I think this could probably be done by adding a flag to the existing virConnectBaselineCPU API. eg something like
capsxml = virConnectGetCapabilties() cpuxml = ...extract the <cpu> bit of capsxml... newxml = virConnectBaselineCPU(conn, &cpuxml, 1, VIR_CONNECT_BASELINE_CPU_EXPAND_FEATURES)
Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
-- Don Dugger "Censeo Toto nos in Kansa esse decisse." - D. Gale n0ano@n0ano.com Ph: 303/443-3786
participants (3)
-
Daniel P. Berrange -
Don Dugger -
Jiri Denemark