[Libvir] Proposal for new Libvirt functions about virtual CPUs
1) Currently the NULL virDomainPtr (meaning Domain0) is only specified in the 3 virDomain*Memory functions (GetMax, SetMax, Set). The other virDomain* functions need the actual virDomainPtr for Domain0. What is the reason for this Memory-specific option? Is it applicable to virtual CPUs? 2) For "xm vcpu-list" equivalent function (suggested virDomainGetVcpus) I propose to return in a new structure all the information provided by Xen Daemon. I am trying the following addition in libvirt.h : typedef struct _virVcpuInfo virVcpuInfo; struct _virVcpuInfo { int number; /* virtual CPU number */ unsigned char state; /* 'r'=running, 'b'=blocked, 'p'=offline */ unsigned long long cpuTime; /* CPU time used, in nanoseconds */ int cpu; /* last real CPU number allocated */ char cpumap[128]; /* affinity map of real CPUs which can be allocated */ }; typedef virVcpuInfo *virVcpuInfoPtr; For cpumap, the xm command displays a string "x,y-z..." more compact than the list of all possible CPUs "x y y+1 y+2...z..." provided by xend. But the routines for this reformatting are not easy, so I propose to keep the xend format in virCpuInfo. What is your opinion? 3) For "xm vcpu-pin" equivalent function (suggested virDomainPinVcpu) I propose a CPU list argument in same format as virCpuInfo cpumap. Do you agree? Veuillez répondre à veillard@redhat.com Pour : michel.ponceau@bull.net cc : libvir-list@redhat.com Objet : Re: [Libvir] Proposal for a new Libvirt function: change number of virtual CPUs On Fri, Jun 23, 2006 at 05:48:40PM +0200, michel.ponceau@bull.net wrote:
For our administration, we need to dynamically reduce the number of virtual CPUs in each domain, in same way as Xen command "xm vcpu-set". It seems that no Libvirt function is currently provided for that purpose. I successfully tried some simple changes in libvirt.c an xend_internal.c, mainly the following: snprintf(buf, sizeof(buf), "%d", vcpus); return xend_op(domain->conn, domain->name, "op", "set_vcpus", "vcpus", buf, NULL); We suggest to add this new function "virDomainSetVcpus" to Libvirt.
Hum, let's see. We expose the number of virtual CPUs in the info structure so yes that's something which is part of what we expose. So that would be /** * virDomainSetVcpus: * @domain: a domain object or NULL * @nvcpus: the new number of virtual CPUs for this domain * * Dynamically change the number of virtual CPUs used by the domain. * Note that this call may fail if the underlying virtualization hypervisor * does not support it or if growing the number is arbitrary limited. * This function requires priviledged access to the hypervisor. * * Returns 0 in case of success and -1 in case of failure. */ int virDomainSetVcpus(virDomainPtr domain, unsigned int nvcpus) { }
I have not tried the hypervisor and xen store accesses for this function. Do you think they would be necessary?
It would be preferable to go though Xend first at least. I don't think xenstore is a good idea at this point. Now for the direct hypervisor, I'm uncertain, it could be a good idea to provide redundancy over xend, I assume it should not be hard to do that at the hypervisor level, the risk is introducing incoherence between what xend thinks and what the hypervisor atually implement :-)
I am also working on two other functions to reproduce "xm vcpu-pin" and "xm vcpu-list" from Libvirt.
vcpu-list might be a bit subtle as an API design level. Better not allocate the array in libvirt, but have the user pass an array. Also how much information would you extract per CPU, just on/off ? Daniel -- Daniel Veillard | Red Hat http://redhat.com/ veillard@redhat.com | libxml GNOME XML XSLT toolkit http://xmlsoft.org/ http://veillard.com/ | Rpmfind RPM search engine http://rpmfind.net/
On Mon, Jun 26, 2006 at 04:14:59PM +0200, michel.ponceau@bull.net wrote:
2) For "xm vcpu-list" equivalent function (suggested virDomainGetVcpus) I propose to return in a new structure all the information provided by Xen Daemon. I am trying the following addition in libvirt.h : typedef struct _virVcpuInfo virVcpuInfo; struct _virVcpuInfo { int number; /* virtual CPU number */ unsigned char state; /* 'r'=running, 'b'=blocked, 'p'=offline */ unsigned long long cpuTime; /* CPU time used, in nanoseconds */ int cpu; /* last real CPU number allocated */ char cpumap[128]; /* affinity map of real CPUs which can be allocated */ }; typedef virVcpuInfo *virVcpuInfoPtr;
For cpumap, the xm command displays a string "x,y-z..." more compact than the list of all possible CPUs "x y y+1 y+2...z..." provided by xend. But the routines for this reformatting are not easy, so I propose to keep the xend format in virCpuInfo. What is your opinion?
That is a rather unpleasent format to expose to applications - each application will be forced to write their own (buggy) parser to extract useful information from this string. It really needs to be sructured in a format which can be directly interpreted, without requiring parsing. Dan. -- |=- Red Hat, Engineering, Emerging Technologies, Boston. +1 978 392 2496 -=| |=- Perl modules: http://search.cpan.org/~danberr/ -=| |=- Projects: http://freshmeat.net/~danielpb/ -=| |=- GnuPG: 7D3B9505 F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 -=|
On Mon, Jun 26, 2006 at 03:22:15PM +0100, Daniel P. Berrange wrote:
On Mon, Jun 26, 2006 at 04:14:59PM +0200, michel.ponceau@bull.net wrote:
2) For "xm vcpu-list" equivalent function (suggested virDomainGetVcpus) I propose to return in a new structure all the information provided by Xen Daemon. I am trying the following addition in libvirt.h : typedef struct _virVcpuInfo virVcpuInfo; struct _virVcpuInfo { int number; /* virtual CPU number */ unsigned char state; /* 'r'=running, 'b'=blocked, 'p'=offline */ unsigned long long cpuTime; /* CPU time used, in nanoseconds */ int cpu; /* last real CPU number allocated */ char cpumap[128]; /* affinity map of real CPUs which can be allocated */ }; typedef virVcpuInfo *virVcpuInfoPtr;
For cpumap, the xm command displays a string "x,y-z..." more compact than the list of all possible CPUs "x y y+1 y+2...z..." provided by xend. But the routines for this reformatting are not easy, so I propose to keep the xend format in virCpuInfo. What is your opinion?
That is a rather unpleasent format to expose to applications - each application will be forced to write their own (buggy) parser to extract useful information from this string. It really needs to be sructured in a format which can be directly interpreted, without requiring parsing.
Hum, I missed that. Agreed, do not use a string, we should use an array. The virVcpuInfo structure will be allocated by the client, make sure it is extensible to whatever the need might be in the future. Say you have a box with 1024 CPU, ideally the client should be able to call libvirt and specify any combination of those. Also let's check what is the low level hypervisor interface there. Daniel -- Daniel Veillard | Red Hat http://redhat.com/ veillard@redhat.com | libxml GNOME XML XSLT toolkit http://xmlsoft.org/ http://veillard.com/ | Rpmfind RPM search engine http://rpmfind.net/
On Mon, Jun 26, 2006 at 04:14:59PM +0200, michel.ponceau@bull.net wrote:
1) Currently the NULL virDomainPtr (meaning Domain0) is only specified in the 3 virDomain*Memory functions (GetMax, SetMax, Set). The other virDomain* functions need the actual virDomainPtr for Domain0. What is the reason for this Memory-specific option?
I think it's problem. We should always require a virDomainPtr, I actually think the shortcut is not implemented. I wonder if it would be better to just remove it that would be cleaner, however when think about other kind of virtualization, where the main os would not be represented as a domain, but still with memory management, in that case that should be kept.
Is it applicable to virtual CPUs?
Hum, similary, what if the main OS CPU usage could be set up even if not mapped as a real domain, maybe this should be kept for the vCPU APIs too, yes, a bit messy but more generic.
2) For "xm vcpu-list" equivalent function (suggested virDomainGetVcpus) I propose to return in a new structure all the information provided by Xen Daemon. I am trying the following addition in libvirt.h :
Remember to change libvirt.h.in as libvirt.h is generated by configure.
typedef struct _virVcpuInfo virVcpuInfo; struct _virVcpuInfo { int number; /* virtual CPU number */
unsigned int rather
unsigned char state; /* 'r'=running, 'b'=blocked, 'p'=offline */
Not too fond of using a letter there. I would prefer a number and a clean enum
unsigned long long cpuTime; /* CPU time used, in nanoseconds */
int cpu; /* last real CPU number allocated */
That's a bit fishy to me, how do you expect to use that information ?
char cpumap[128]; /* affinity map of real CPUs which can be allocated */
so we know we will never have libvirt used for a virtualization platform which has no more than 256 physical CPU to offer ? Remember I don't want that API to be limited to Xen and to me if we put this in the API this must be future proof. I would at least grow the CPU map to 256 entries at the API level. Oh and use unsigned char to go 0-255.
};
one last comment about packing the structure: it may be intersting to reorganize fields in that arry to help, int, int, long long, char, char[] looks better to me than the current organization. Another note is about access, do you really want to get those just from xend or also from the hypervisor ? The hypervisor might be way cheaper to reach and more accurate. Hopefully that structure still make sense with the associated hypervisor informations could you look up the associated hypervisor call and double check ?
typedef virVcpuInfo *virVcpuInfoPtr;
For cpumap, the xm command displays a string "x,y-z..." more compact than the list of all possible CPUs "x y y+1 y+2...z..." provided by xend. But the routines for this reformatting are not easy, so I propose to keep the xend format in virCpuInfo. What is your opinion?
I never played with mapping (well I don't have anymore SMP box to play locally), could you explain, I'm not sure I follow :-) . If what xm does make sense, I don't see why introducing a new way is an improvement, but I can hardly juge myself.
3) For "xm vcpu-pin" equivalent function (suggested virDomainPinVcpu) I propose a CPU list argument in same format as virCpuInfo cpumap. Do you agree?
I have the same worries about the hard constraints put on the number of CPUs at the API level. Daniel -- Daniel Veillard | Red Hat http://redhat.com/ veillard@redhat.com | libxml GNOME XML XSLT toolkit http://xmlsoft.org/ http://veillard.com/ | Rpmfind RPM search engine http://rpmfind.net/
On Mon, Jun 26, 2006 at 10:43:24AM -0400, Daniel Veillard wrote:
On Mon, Jun 26, 2006 at 04:14:59PM +0200, michel.ponceau@bull.net wrote:
1) Currently the NULL virDomainPtr (meaning Domain0) is only specified in the 3 virDomain*Memory functions (GetMax, SetMax, Set). The other virDomain* functions need the actual virDomainPtr for Domain0. What is the reason for this Memory-specific option?
I think it's problem. We should always require a virDomainPtr, I actually think the shortcut is not implemented. I wonder if it would be better to just remove it that would be cleaner, however when think about other kind of virtualization, where the main os would not be represented as a domain, but still with memory management, in that case that should be kept.
Is it applicable to virtual CPUs?
Hum, similary, what if the main OS CPU usage could be set up even if not mapped as a real domain, maybe this should be kept for the vCPU APIs too, yes, a bit messy but more generic.
2) For "xm vcpu-list" equivalent function (suggested virDomainGetVcpus) I propose to return in a new structure all the information provided by Xen Daemon. I am trying the following addition in libvirt.h :
Remember to change libvirt.h.in as libvirt.h is generated by configure.
typedef struct _virVcpuInfo virVcpuInfo; struct _virVcpuInfo { int number; /* virtual CPU number */
unsigned int rather
unsigned char state; /* 'r'=running, 'b'=blocked, 'p'=offline */
Not too fond of using a letter there. I would prefer a number and a clean enum
unsigned long long cpuTime; /* CPU time used, in nanoseconds */
int cpu; /* last real CPU number allocated */
That's a bit fishy to me, how do you expect to use that information ?
I guess that's to allow the 'CPU' column in the following output to be generated # xm vcpu-list Name ID VCPU CPU State Time(s) CPU Affinity Domain-0 0 0 0 -b- 636.8 any cpu Domain-0 0 1 1 -b- 396.8 any cpu Domain-0 0 2 2 r-- 319.5 any cpu Domain-0 0 3 3 -b- 369.4 any cpu xperf1 1 0 3 -b- 545.6 any cpu xperf2 3 0 2 -b- 20.0 any cpu xperf2 3 1 3 -b- 18.2 any cpu xperf2 3 2 0 -b- 18.6 any cpu xperf2 3 3 1 -b- 17.8 any cpu
char cpumap[128]; /* affinity map of real CPUs which can be allocated */
so we know we will never have libvirt used for a virtualization platform which has no more than 256 physical CPU to offer ? Remember I don't want that API to be limited to Xen and to me if we put this in the API this must be future proof. I would at least grow the CPU map to 256 entries at the API level. Oh and use unsigned char to go 0-255.
I expect we really need a bit-set in that field, one bit per CPU that the VCPU can be scheduled on. One thought though, is do we want to mix up configuration data (VCPU/VPU affinity) with performance metrics (state, runtime, last cpu) ? Might it make more sense to have a single API for retrieving the entire VCPU <-> VPU mapping for all CPUs in a single call - basically return a 2-d matrix with VCPU in one axis and CPU in the other access, and the elements are basically a bit 1/0
Another note is about access, do you really want to get those just from xend or also from the hypervisor ? The hypervisor might be way cheaper to reach and more accurate. Hopefully that structure still make sense with the associated hypervisor informations could you look up the associated hypervisor call and double check ?
We definitely want to get this info via the hypervisor if its available, since the performance hit in making the multiple RPC calls will be far too great to make this usable otherwise.
3) For "xm vcpu-pin" equivalent function (suggested virDomainPinVcpu) I propose a CPU list argument in same format as virCpuInfo cpumap. Do you agree?
I have the same worries about the hard constraints put on the number of CPUs at the API level.
Whatever we do, should match up in its data structure with the API for retrieving CPU pining Dan. -- |=- Red Hat, Engineering, Emerging Technologies, Boston. +1 978 392 2496 -=| |=- Perl modules: http://search.cpan.org/~danberr/ -=| |=- Projects: http://freshmeat.net/~danielpb/ -=| |=- GnuPG: 7D3B9505 F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 -=|
participants (3)
-
Daniel P. Berrange -
Daniel Veillard -
michel.ponceau＠bull.net