9 a.m.
For our administration, we need the following actions, while concerned
domain is running:
1) Change the number of virtual CPUs.
2) Change the pinning (affinity) of a virtual CPU on real CPUs.
3) Get detailed information for each virtual CPU.
Currently there is no Libvirt function provided for that. We suggest to
add the following 3 functions (in libvirt.c):
/**
* virDomainSetVcpus:
* @domain: pointer to domain object, or NULL for Domain0
* @nvcpus: the new number of virtual CPUs for this domain
*
* Dynamically change the number of virtual CPUs used by the domain.
* Note that this call may fail if the underlying virtualization
hypervisor
* does not support it or if growing the number is arbitrary limited.
* This function requires priviledged access to the hypervisor.
*
* Returns 0 in case of success, -1 in case of failure.
*/
int virDomainSetVcpus(virDomainPtr domain, unsigned int nvcpus)
/**
* virDomainPinVcpu:
* @domain: pointer to domain object, or NULL for Domain0
* @vcpu: virtual CPU number
* @cpumap: pointer to a bit map of real CPUs (format in virVcpuInfo)
* @maplen: length of cpumap, in 8-bit bytes
*
* Dynamically change the real CPUs which can be allocated to a virtual
CPU.
* This function requires priviledged access to the hypervisor.
*
* Returns 0 in case of success, -1 in case of failure.
*/
int virDomainPinVcpu(virDomainPtr domain, unsigned int vcpu,
unsigned char *cpumap, int maplen)
/**
* virDomainGetVcpus:
* @domain: pointer to domain object, or NULL for Domain0
* @info: pointer to an array of virVcpuInfo structures
* @maxinfo: number of structures in info array
*
* Extract information about virtual CPUs of a domain, store it in info
array.
*
* Returns the number of info filled in case of success, -1 in case of
failure.
*/
int virDomainGetVcpus(virDomainPtr domain, virVcpuInfoPtr info, int
maxinfo)
with the following structure (in libvirt.h):
/**
* virVcpuInfo: structure for information about a virtual CPU in a domain.
*/
#define VIR_MAX_CPUS 256
typedef enum {
VIR_VCPU_OFFLINE = 0, /* the virtual CPU is offline */
VIR_VCPU_RUNNING = 1, /* the virtual CPU is running */
VIR_VCPU_BLOCKED = 2, /* the virtual CPU is blocked on resource
*/
} virVcpuState;
typedef struct _virVcpuInfo virVcpuInfo;
struct _virVcpuInfo {
unsigned int number; /* virtual CPU number */
int state; /* value from virVcpuState */
unsigned long long cpuTime; /* CPU time used, in nanoseconds */
int cpu; /* real CPU number, or -1 if offline */
unsigned char cpumap[VIR_MAX_CPUS/8]; /* Bit map of usable real CPUs.
Each bit set to 1 means that corresponding CPU is usable.
Bytes are stored in little-endian order: CPU0-7, 8-15...
In each byte, lowest CPU number is least significant bit
*/
};
typedef virVcpuInfo *virVcpuInfoPtr;
I have successfully tried those functions via Xen hypervisor, except for
the first (SetVcpus) where hypervisor operation DOM0_MAX_VCPUS fails
(maybe it is not possible on a running domain ?). That function was
successful via Xen daemon.
11:44 a.m.
New subject: [Libvir] Re: Proposal : add 3 functions to Libvirt API, for virtual CPUs
On Fri, Jun 30, 2006 at 04:00:45PM +0200, michel.ponceau(a)bull.net wrote:
For our administration, we need the following actions, while
concerned
domain is running:
1) Change the number of virtual CPUs.
2) Change the pinning (affinity) of a virtual CPU on real CPUs.
3) Get detailed information for each virtual CPU.
Currently there is no Libvirt function provided for that. We suggest to
add the following 3 functions (in libvirt.c):
/**
* virDomainSetVcpus:
* @domain: pointer to domain object, or NULL for Domain0
* @nvcpus: the new number of virtual CPUs for this domain
*
* Dynamically change the number of virtual CPUs used by the domain.
* Note that this call may fail if the underlying virtualization
hypervisor
* does not support it or if growing the number is arbitrary limited.
* This function requires priviledged access to the hypervisor.
*
* Returns 0 in case of success, -1 in case of failure.
*/
int virDomainSetVcpus(virDomainPtr domain, unsigned int nvcpus)
okay
/**
* virDomainPinVcpu:
* @domain: pointer to domain object, or NULL for Domain0
* @vcpu: virtual CPU number
* @cpumap: pointer to a bit map of real CPUs (format in virVcpuInfo)
* @maplen: length of cpumap, in 8-bit bytes
*
* Dynamically change the real CPUs which can be allocated to a virtual
CPU.
* This function requires priviledged access to the hypervisor.
*
* Returns 0 in case of success, -1 in case of failure.
*/
int virDomainPinVcpu(virDomainPtr domain, unsigned int vcpu,
unsigned char *cpumap, int maplen)
Can you explain more clearly what is the format of cpumap ? An example
would be welcome, and that would be needed for the doc and maybe testing.
What would happen if maplen is < or > than the number of CPU divided by 8 ?
/**
* virDomainGetVcpus:
* @domain: pointer to domain object, or NULL for Domain0
* @info: pointer to an array of virVcpuInfo structures
* @maxinfo: number of structures in info array
*
* Extract information about virtual CPUs of a domain, store it in info
array.
*
* Returns the number of info filled in case of success, -1 in case of
failure.
*/
int virDomainGetVcpus(virDomainPtr domain, virVcpuInfoPtr info, int
maxinfo)
Hum ... now the problem with that API entry point is that we 'burn' the
maximum 256 processors in the ABI, i.e. if we ever need to go past 256
client and servers need to be recompiled. Maybe this is not a real problem
in practice but that's annoying. Is there existing APIs doing this kind
of things (in POSIX for example), and what hard limit did they use ?
Maybe
int virDomainGetVcpusNr(virDomainPtr domain, int nr, virVcpuInfoPtr info,
int maxCPU);
Where the maxCPU is defined by the client as the number of real CPU
it defined in its virVcpuInfoPtr and then an iteration over the virtual
CPU defined in the domain is possible too.
Of course if the domain uses many virtual CPUs this would become expensive
but somehow I don't see that being the common use, I would rather guess
the domains created use a few CPUs even if instantiated on a very large machine.
This
with the following structure (in libvirt.h):
/**
* virVcpuInfo: structure for information about a virtual CPU in a domain.
*/
#define VIR_MAX_CPUS 256
Hum, there is already NUMA machines with more than 256 processors,
it's annoying to define an API limit when you know it is already breakable.
typedef enum {
VIR_VCPU_OFFLINE = 0, /* the virtual CPU is offline */
VIR_VCPU_RUNNING = 1, /* the virtual CPU is running */
VIR_VCPU_BLOCKED = 2, /* the virtual CPU is blocked on resource
*/
} virVcpuState;
typedef struct _virVcpuInfo virVcpuInfo;
struct _virVcpuInfo {
unsigned int number; /* virtual CPU number */
int state; /* value from virVcpuState */
unsigned long long cpuTime; /* CPU time used, in nanoseconds */
int cpu; /* real CPU number, or -1 if offline */
unsigned char cpumap[VIR_MAX_CPUS/8]; /* Bit map of usable real CPUs.
Each bit set to 1 means that corresponding CPU is usable.
Bytes are stored in little-endian order: CPU0-7, 8-15...
In each byte, lowest CPU number is least significant bit
*/
};
typedef virVcpuInfo *virVcpuInfoPtr;
Hum, maybe some accessors should be provided in the API, letting the client
code handle access and having to take care of indianness issues doesn't feel
very nice. Something like the FD_CLR/FD_ISSET/FD_SET/FD_ZERO equivalents
when using the select() POSIx call.
I have successfully tried those functions via Xen hypervisor, except
for
the first (SetVcpus) where hypervisor operation DOM0_MAX_VCPUS fails
(maybe it is not possible on a running domain ?). That function was
successful via Xen daemon.
Maybe that operation requires more than one hypervisor call to actually enable
the processors. The simpler would be to look at the code in xend about what's
needed there, maybe the kernel need to be made aware of that and I would expect
this to be a xenstore operation.
At least we know we can fallback to xend if the hypervisor call doesn't work
directly.
I don't know if virDomainGetVcpus should really be mutated into
virDomainGetVcpusNr, I would certainly like to be able to keep that API
extensible for more than 256 CPUs. Maybe I'm just too cautious there,
I would really like feedback from others on this issue !
In the meantime sending the current set of patches you developped could
allow to look closely at the 2 calls that I feel okay with.
thanks and sorry it took so long !
Daniel
--
Daniel Veillard | Red Hat http://redhat.com/
veillard(a)redhat.com | libxml GNOME XML XSLT toolkit http://xmlsoft.org/
http://veillard.com/ | Rpmfind RPM search engine http://rpmfind.net/
12:27 p.m.
New subject: [Libvir] Re: Proposal : add 3 functions to Libvirt API, for virtual CPUs
On Mon, Jul 10, 2006 at 12:44:34PM -0400, Daniel Veillard wrote:
> /**
> * virDomainGetVcpus:
> * @domain: pointer to domain object, or NULL for Domain0
> * @info: pointer to an array of virVcpuInfo structures
> * @maxinfo: number of structures in info array
> *
> * Extract information about virtual CPUs of a domain, store it in info
> array.
> *
> * Returns the number of info filled in case of success, -1 in case of
> failure.
> */
> int virDomainGetVcpus(virDomainPtr domain, virVcpuInfoPtr info, int
> maxinfo)
Hum ... now the problem with that API entry point is that we 'burn' the
maximum 256 processors in the ABI, i.e. if we ever need to go past 256
client and servers need to be recompiled. Maybe this is not a real problem
in practice but that's annoying. Is there existing APIs doing this kind
of things (in POSIX for example), and what hard limit did they use ?
Maybe
int virDomainGetVcpusNr(virDomainPtr domain, int nr, virVcpuInfoPtr info,
int maxCPU);
Where the maxCPU is defined by the client as the number of real CPU
it defined in its virVcpuInfoPtr and then an iteration over the virtual
CPU defined in the domain is possible too.
Of course if the domain uses many virtual CPUs this would become expensive
but somehow I don't see that being the common use, I would rather guess
the domains created use a few CPUs even if instantiated on a very large machine.
This
> with the following structure (in libvirt.h):
> /**
> * virVcpuInfo: structure for information about a virtual CPU in a domain.
> */
> #define VIR_MAX_CPUS 256
Hum, there is already NUMA machines with more than 256 processors,
it's annoying to define an API limit when you know it is already breakable.
Yeah, if we define such a hard limit in the ABI, then I can pretty much
guarentee this will come back to bite us. As Daniel says, there are already
machines in existance with huge number of CPUS, so I can easily see us hitting
that limit within a pretty short time period - particularly now that Xen has
been ported to IA64 - cf HP Superdome machines with 128 physical CPUs - add
in dual core & hyperthreading to that....
Regards,
Dan
--
|=- Red Hat, Engineering, Emerging Technologies, Boston. +1 978 392 2496 -=|
|=- Perl modules: http://search.cpan.org/~danberr/ -=|
|=- Projects: http://freshmeat.net/~danielpb/ -=|
|=- GnuPG: 7D3B9505 F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 -=|
6523
days inactive
6533
days old
2 comments
3 participants
participants (3)
-
Daniel P. Berrange -
Daniel Veillard -
michel.ponceau＠bull.net