Hi, A recent OpenStack nova commit make use of virNodeGetCPUMap to get the list of online cpu of a host. But this API is not implemented for the libvirt xen driver. The commit: Add handling for offlined CPUs to the nova libvirt driver. https://review.openstack.org/gitweb?p=openstack/nova.git;a=commitdiff;h=0...

Is there a need to use this under Xen? (Is it possible to have offline CPU?). What libxl API those provide this information, if it exist? I found libxl_get_online_cpus() but that not enough. They want a bitmap.

On Tue, Feb 24, 2015 at 12:46:44PM +0000, Daniel P. Berrange wrote: > On Tue, Feb 24, 2015 at 12:41:01PM +0000, Anthony PERARD wrote: > > Hi, > > > > A recent OpenStack nova commit make use of virNodeGetCPUMap to get the list > > of online cpu of a host. But this API is not implemented for the libvirt > > xen driver. > > > > The commit: > > Add handling for offlined CPUs to the nova libvirt driver. > > https://review.openstack.org/gitweb?p=openstack/nova.git;a=commitdiff;h=0... > > FWIW, this should not impact Xen based on my understanding. The code > path in question should only be used when Nova is setup todo NUMA > pinning support, and that is not supported with Xen in OpenStack, > only KVM. Did it actually cause failures for you, or are you simply > keeping track of all used APIs in Nova as a sanity check ? It prevent nova from starting. I do the setup with DevStack. The error: libvirtError: this function is not supported by the connection driver: virNodeGetCPUMap And a part of the traceback: File "/opt/stack/nova/nova/openstack/common/service.py", line 491, in run_service service.start() File "/opt/stack/nova/nova/service.py", line 181, in start self.manager.pre_start_hook() File "/opt/stack/nova/nova/compute/manager.py", line 1188, in pre_start_hook self.update_available_resource(nova.context.get_admin_context()) File "/opt/stack/nova/nova/compute/manager.py", line 6062, in update_available_resource rt.update_available_resource(context) File "/opt/stack/nova/nova/compute/resource_tracker.py", line 315, in update_available_resource resources = self.driver.get_available_resource(self.nodename) File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4896, in get_available_resource numa_topology = self._get_host_numa_topology() File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4749, in _get_host_numa_topology online_cpus = self._host.get_online_cpus() File "/opt/stack/nova/nova/virt/libvirt/host.py", line 599, in get_online_cpus (cpus, cpu_map, online) = self.get_connection().getCPUMap() I'll look into why nova is going through NUMA code paths then.

On Tue, Feb 24, 2015 at 01:22:19PM +0000, Daniel P. Berrange wrote: > On Tue, Feb 24, 2015 at 01:15:57PM +0000, Anthony PERARD wrote: > > On Tue, Feb 24, 2015 at 12:46:44PM +0000, Daniel P. Berrange wrote: > > > On Tue, Feb 24, 2015 at 12:41:01PM +0000, Anthony PERARD wrote: > > > > Hi, > > > > > > > > A recent OpenStack nova commit make use of virNodeGetCPUMap to get the list > > > > of online cpu of a host. But this API is not implemented for the libvirt > > > > xen driver. > > > > > > > > The commit: > > > > Add handling for offlined CPUs to the nova libvirt driver. > > > > https://review.openstack.org/gitweb?p=openstack/nova.git;a=commitdiff;h=0... > > > > > > FWIW, this should not impact Xen based on my understanding. The code > > > path in question should only be used when Nova is setup todo NUMA > > > pinning support, and that is not supported with Xen in OpenStack, > > > only KVM. Did it actually cause failures for you, or are you simply > > > keeping track of all used APIs in Nova as a sanity check ? > > > > It prevent nova from starting. I do the setup with DevStack. > > > > The error: > > libvirtError: this function is not supported by the connection driver: virNodeGetCPUMap > > > > And a part of the traceback: > > File "/opt/stack/nova/nova/openstack/common/service.py", line 491, in run_service > > service.start() > > File "/opt/stack/nova/nova/service.py", line 181, in start > > self.manager.pre_start_hook() > > File "/opt/stack/nova/nova/compute/manager.py", line 1188, in pre_start_hook > > self.update_available_resource(nova.context.get_admin_context()) > > File "/opt/stack/nova/nova/compute/manager.py", line 6062, in update_available_resource > > rt.update_available_resource(context) > > File "/opt/stack/nova/nova/compute/resource_tracker.py", line 315, in update_available_resource > > resources = self.driver.get_available_resource(self.nodename) > > File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4896, in get_available_resource > > numa_topology = self._get_host_numa_topology() > > File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4749, in _get_host_numa_topology > > online_cpus = self._host.get_online_cpus() > > File "/opt/stack/nova/nova/virt/libvirt/host.py", line 599, in get_online_cpus > > (cpus, cpu_map, online) = self.get_connection().getCPUMap() > > > > I'll look into why nova is going through NUMA code paths then. > > Oh damn, yes, I understand why now. Please file a bug against Nova for > this, as we must fix it as a high pripority. It was certainly not my > intention to break Xen when I approved this change Here is the bug report: https://bugs.launchpad.net/nova/+bug/1425115

Hi, A recent OpenStack nova commit make use of virNodeGetCPUMap to get the list of online cpu of a host. But this API is not implemented for the libvirt xen driver. The commit: Add handling for offlined CPUs to the nova libvirt driver. https://review.openstack.org/gitweb?p=openstack/nova.git;a=commitdiff;h=0... Is there a need to use this under Xen? (Is it possible to have offline CPU?).

What libxl API those provide this information, if it exist? I found libxl_get_online_cpus() but that not enough. They want a bitmap.

On Tue, 2015-02-24 at 13:10 +0000, Ian Campbell wrote: > On Tue, 2015-02-24 at 12:41 +0000, Anthony PERARD wrote: > > What libxl API those provide this information, if it exist? > > > > I found libxl_get_online_cpus() but that not enough. They want a > > bitmap. > > I think that is all which currently exists, at least at the libxl level, > you may need to add a new interface. > > It'd be worth looking into the various host numa interfaces -- perhaps > one of them indirectly exposes what you want? > Given Daniel's latest emails, I'm not sure this is useful but libxl_get_cpu_topology() should put LIBXL_CPUTOPOLOGY_INVALID_ENTRY in all the fields of the i-eth element of the array it returns, if the i-eth pcpu is offline (see the implementation of XEN_SYSCTL_topologyinfo in xen/common/sysctl.c). So, scanning that array and constructing the bitmap according to whether or not we find that marker on the various elements would be the way to go, I would say.

I've actually never tested this, i.e., I've never tried offlining a pcpu on the host. I'll give it a go as soon as I find 5 minutes, and let know if it works. Regards, Dario

FWIW, this code in openstack was only added for benefit of s390 architecture where apparently it is common to have hosts with CPUs offlined. Presumably you have to pay IBM for each extra CPU you turn online :)

On Wed, Feb 25, 2015 at 03:13:36PM +0000, Ian Campbell wrote: > On Wed, 2015-02-25 at 15:03 +0000, Daniel P. Berrange wrote: > > FWIW, this code in openstack was only added for benefit of s390 > > architecture where apparently it is common to have hosts with > > CPUs offlined. Presumably you have to pay IBM for each extra > > CPU you turn online :) > > Presumably :-) > > OOI, why does the code care which CPUs are online rather than just the > total number (IOW why a bitmap)? When doing strict CPU pinning,the openstack scheduler needs to have the list of all pCPUs available in the host. It then tries to place guests on pCPUs such that the guest does not span across NUMA nodes. To do this it needs to know which particular pCPUs in the host are available. So we need the full bitmap rather than just a total count.