[Libvir] Here the patch for virtual CPU functions.
The patch (vcpus.diff) is joined with this mail. To apply the patch, untar the libvirt-0.1.3.tar.gz archive (if not already done). Then in libvirt-0.1.3 directory, do ./configure --prefix=/usr if not already done, and after this: patch -p1 < vcpus.diff make make install From the initial patch of Michel Ponceau (not delivered), I've just modify the virDomainGetVcpus function and added some useful macro for bit manipulation in cpu bit maps. TODO: These vcpus functions permits to dynamically change the vcpu/cpu relation after the domain has been started but nothing has been done regarding static cpus attribution in the XML configuration file of a domain. Philippe Berthault. --- libvirt-0.1.3/include/libvirt/libvirt.h 2006-07-11 18:05:07.000000000 +0200 +++ libvirt-0.1.3.with_vcpus/include/libvirt/libvirt.h 2006-08-02 16:42:51.000000000 +0200 @@ -289,6 +289,106 @@ const char **names, int maxnames); int virDomainCreate (virDomainPtr domain); +/////////////////////////////////////////////////////////////////////////// + +/** + * virVcpuInfo: structure for information about a virtual CPU in a domain. + */ + +typedef enum { + VIR_VCPU_OFFLINE = 0, /* the virtual CPU is offline */ + VIR_VCPU_RUNNING = 1, /* the virtual CPU is running */ + VIR_VCPU_BLOCKED = 2, /* the virtual CPU is blocked on resource */ +} virVcpuState; + +typedef struct _virVcpuInfo virVcpuInfo; +struct _virVcpuInfo { + unsigned int number; /* virtual CPU number */ + int state; /* value from virVcpuState */ + unsigned long long cpuTime; /* CPU time used, in nanoseconds */ + int cpu; /* real CPU number, or -1 if offline */ +}; +typedef virVcpuInfo *virVcpuInfoPtr; + +int virDomainSetVcpus (virDomainPtr domain, + unsigned int nvcpus); + +int virDomainPinVcpu (virDomainPtr domain, + unsigned int vcpu, + unsigned char *cpumap, + int maplen); + +/** + * USE_CPU: + * @cpumap: pointer to a bit map of real CPUs (in 8-bit bytes) (IN/OUT) + * @cpu: the physical CPU number + * + * This macro is to be used in conjonction with virDomainPinVcpu() API. + * USE_CPU macro set the bit (CPU usable) of the related cpu in cpumap. + */ + +#define USE_CPU(cpumap,cpu) (cpumap[(cpu)/8] |= (1<<((cpu)%8))) + +/** + * UNUSE_CPU: + * @cpumap: pointer to a bit map of real CPUs (in 8-bit bytes) (IN/OUT) + * @cpu: the physical CPU number + * + * This macro is to be used in conjonction with virDomainPinVcpu() API. + * USE_CPU macro reset the bit (CPU not usable) of the related cpu in cpumap. + */ + +#define UNUSE_CPU(cpumap,cpu) (cpumap[(cpu)/8] &= ~(1<<((cpu)%8))) + +int virDomainGetVcpus (virDomainPtr domain, + virVcpuInfoPtr info, + int maxinfo, + unsigned char *cpumaps, + int maplen); + +/** + * CPU_USABLE: + * @cpumaps: pointer to an array of cpumap (in 8-bit bytes) (IN) + * @maplen: the length (in bytes) of one cpumap + * @vcpu: the virtual CPU number + * @cpu: the physical CPU number + * + * This macro is to be used in conjonction with virDomainGetVcpus() API. + * CPU_USABLE macro returns a non zero value (true) if the cpu + * is usable by the vcpu, and 0 otherwise. + */ + +#define CPU_USABLE(cpumaps,maplen,vcpu,cpu) \ + (cpumaps[((vcpu)*(maplen))+((cpu)/8)] & (1<<((cpu)%8))) + +/** + * COPY_CPUMAP: + * @cpumaps: pointer to an array of cpumap (in 8-bit bytes) (IN) + * @maplen: the length (in bytes) of one cpumap + * @vcpu: the virtual CPU number + * @cpumap: pointer to a cpumap (in 8-bit bytes) (OUT) + * This cpumap must be previously allocated by the caller (ie: malloc(maplen)) + * + * This macro is to be used in conjonction with virDomainGetVcpus() and + * virDomainPinVcpu() APIs. COPY_CPUMAP macro extract the cpumap of the specified + * vcpu from cpumaps array and copy it into cpumap to be used later by virDomainPinVcpu() API. + */ +#define COPY_CPUMAP(cpumaps,maplen,vcpu,cpumap) \ + memcpy(cpumap, &(cpumaps[(vcpu)*(maplen)]), (maplen)) + + +/** + * GET_CPUMAP: + * @cpumaps: pointer to an array of cpumap (in 8-bit bytes) (IN) + * @maplen: the length (in bytes) of one cpumap + * @vcpu: the virtual CPU number + * + * This macro is to be used in conjonction with virDomainGetVcpus() and + * virDomainPinVcpu() APIs. GET_CPUMAP macro returns a pointer to the cpumap + * of the specified vcpu from cpumaps array. + */ +#define GET_CPUMAP(cpumaps,maplen,vcpu) &(cpumaps[(vcpu)*(maplen)]) + #ifdef __cplusplus } --- libvirt-0.1.3/src/xen_internal.h 2006-07-03 17:09:24.000000000 +0200 +++ libvirt-0.1.3.with_vcpus/src/xen_internal.h 2006-08-02 16:53:42.000000000 +0200 @@ -44,6 +44,18 @@ unsigned long memory); int xenHypervisorCheckID (virConnectPtr conn, int id); +int xenHypervisorSetVcpus (virDomainPtr domain, + unsigned int nvcpus); +int xenHypervisorPinVcpu (virDomainPtr domain, + unsigned int vcpu, + unsigned char *cpumap, + int maplen); +int xenHypervisorGetVcpus (virDomainPtr domain, + virVcpuInfoPtr info, + int maxinfo, + unsigned char *cpumaps, + int maplen); + #ifdef __cplusplus } #endif --- libvirt-0.1.3/src/xen_internal.c 2006-07-03 17:09:24.000000000 +0200 +++ libvirt-0.1.3.with_vcpus/src/xen_internal.c 2006-08-03 13:52:28.000000000 +0200 @@ -18,7 +18,7 @@ #include <fcntl.h> #include <sys/mman.h> #include <sys/ioctl.h> - +#include <limits.h> #include <stdint.h> /* required for dom0_getdomaininfo_t */ @@ -838,4 +838,138 @@ return (0); } +/////////////////////////////////////////////////////////////////////////////// + +/** + * xenHypervisorSetVcpus: + * @domain: pointer to domain object + * @nvcpus: the new number of virtual CPUs for this domain + * + * Dynamically change the number of virtual CPUs used by the domain. + * + * Returns 0 in case of success, -1 in case of failure. + */ + +int +xenHypervisorSetVcpus(virDomainPtr domain, unsigned int nvcpus) +{ + dom0_op_t op; + + if ((domain == NULL) || (domain->conn == NULL) || (domain->conn->handle < 0) + || (nvcpus < 1)) + return (-1); + op.cmd = DOM0_MAX_VCPUS; + op.u.max_vcpus.domain = (domid_t) domain->handle; + op.u.max_vcpus.max = nvcpus; + if (xenHypervisorDoOp(domain->conn->handle, &op) < 0) + return (-1); + return 0; +} + +/** + * xenHypervisorPinVcpu: + * @domain: pointer to domain object + * @vcpu: virtual CPU number + * @cpumap: pointer to a bit map of real CPUs (in 8-bit bytes) + * @maplen: length of cpumap in bytes + * + * Dynamically change the real CPUs which can be allocated to a virtual CPU. + * + * Returns 0 in case of success, -1 in case of failure. + */ + +int +xenHypervisorPinVcpu(virDomainPtr domain, unsigned int vcpu, + unsigned char *cpumap, int maplen) +{ + dom0_op_t op; + uint64_t *pm = (uint64_t *)&op.u.setvcpuaffinity.cpumap; + int j; + + if ((domain == NULL) || (domain->conn == NULL) || (domain->conn->handle < 0) + || (cpumap == NULL) || (maplen < 1) || (maplen > (int)sizeof(cpumap_t)) + || (sizeof(cpumap_t) & 7)) + return (-1); + op.cmd = DOM0_SETVCPUAFFINITY; + op.u.setvcpuaffinity.domain = (domid_t) domain->handle; + op.u.setvcpuaffinity.vcpu = vcpu; + memset(pm, 0, sizeof(cpumap_t)); + for (j = 0; j < maplen; j++) + *(pm + (j / 8)) |= cpumap[j] << (8 * (j & 7)); + if (xenHypervisorDoOp(domain->conn->handle, &op) < 0) + return (-1); + return 0; +} +/** + * virDomainGetVcpus: + * @domain: pointer to domain object, or NULL for Domain0 + * @info: pointer to an array of virVcpuInfo structures (OUT) + * @maxinfo: number of structures in info array + * @cpumaps: pointer to an bit map of real CPUs for all vcpus of this domain (in 8-bit bytes) (OUT) + * If cpumaps is NULL, then no cupmap information is returned by the API. + * It's assumed there is <maxinfo> cpumap in cpumaps array. + * The memory allocated to cpumaps must be (maxinfo * maplen) bytes + * (ie: calloc(maxinfo, maplen)). + * One cpumap inside cpumaps has the format described in virDomainPinVcpu() API. + * @maplen: number of bytes in one cpumap, from 1 up to size of CPU map in + * underlying virtualization system (Xen...). + * + * Extract information about virtual CPUs of domain, store it in info array + * and also in cpumaps if this pointer is'nt NULL. + * + * Returns the number of info filled in case of success, -1 in case of failure. + */ +int +xenHypervisorGetVcpus(virDomainPtr domain, virVcpuInfoPtr info, int maxinfo, + unsigned char *cpumaps, int maplen) +{ + dom0_op_t op; + uint64_t *pm = (uint64_t *)&op.u.getvcpuinfo.cpumap; + virVcpuInfoPtr ipt; + int nbinfo, mapl, i; + unsigned char *cpumap; + int vcpu, cpu; + + if ((domain == NULL) || (domain->conn == NULL) || (domain->conn->handle < 0) + || (info == NULL) || (maxinfo < 1) + || (sizeof(cpumap_t) & 7)) + return (-1); + if (cpumaps != NULL && maplen < 1) + return -1; + + /* first get the number of virtual CPUs in this domain */ + op.cmd = DOM0_GETDOMAININFO; + op.u.getdomaininfo.domain = (domid_t) domain->handle; + if (xenHypervisorDoOp(domain->conn->handle, &op) < 0) + return (-1); + nbinfo = (int)op.u.getdomaininfo.max_vcpu_id + 1; + if (nbinfo > maxinfo) nbinfo = maxinfo; + + if (cpumaps != NULL) + memset(cpumaps, 0, maxinfo * maplen); + + op.cmd = DOM0_GETVCPUINFO; + for (i=0, ipt=info; i < nbinfo; i++, ipt++) { + vcpu = op.u.getvcpuinfo.vcpu = i; + if (xenHypervisorDoOp(domain->conn->handle, &op) < 0) + return (-1); + ipt->number = i; + if (op.u.getvcpuinfo.online) { + if (op.u.getvcpuinfo.running) ipt->state = VIR_VCPU_RUNNING; + if (op.u.getvcpuinfo.blocked) ipt->state = VIR_VCPU_BLOCKED; + } + else ipt->state = VIR_VCPU_OFFLINE; + ipt->cpuTime = op.u.getvcpuinfo.cpu_time; + ipt->cpu = op.u.getvcpuinfo.online ? (int)op.u.getvcpuinfo.cpu : -1; + if (cpumaps != NULL && vcpu >= 0 && vcpu < maxinfo) { + cpumap = (unsigned char *)GET_CPUMAP(cpumaps, maplen, vcpu); + mapl = (maplen > (int)sizeof(cpumap_t)) ? (int)sizeof(cpumap_t) : maplen; + for (cpu = 0; cpu < (mapl * CHAR_BIT); cpu++) { + if (*pm & ((uint64_t)1<<cpu)) + USE_CPU(cpumap, cpu); + } + } + } + return nbinfo; +} --- libvirt-0.1.3/src/libvirt.c 2006-07-03 17:09:24.000000000 +0200 +++ libvirt-0.1.3.with_vcpus/src/libvirt.c 2006-08-02 15:30:29.000000000 +0200 @@ -1648,4 +1648,126 @@ return(-1); } +/////////////////////////////////////////////////////////////////////////// +/** + * virDomainSetVcpus: + * @domain: pointer to domain object, or NULL for Domain0 + * @nvcpus: the new number of virtual CPUs for this domain + * + * Dynamically change the number of virtual CPUs used by the domain. + * Note that this call may fail if the underlying virtualization hypervisor + * does not support it or if growing the number is arbitrary limited. + * This function requires priviledged access to the hypervisor. + * + * Returns 0 in case of success, -1 in case of failure. + */ + +int +virDomainSetVcpus(virDomainPtr domain, unsigned int nvcpus) +{ + if (domain == NULL) { + TODO + return (-1); + } + if (!VIR_IS_CONNECTED_DOMAIN(domain)) { + virLibDomainError(domain, VIR_ERR_INVALID_DOMAIN, __FUNCTION__); + return (-1); + } + if (domain->conn->flags & VIR_CONNECT_RO) + return (-1); + if (nvcpus < 1) { + virLibDomainError(domain, VIR_ERR_INVALID_ARG, __FUNCTION__); + return (-1); + } +// if (xenHypervisorSetVcpus(domain, nvcpus) == 0) +// return 0; + return xenDaemonDomainSetVcpus(domain, nvcpus); +} + +/** + * virDomainPinVcpu: + * @domain: pointer to domain object, or NULL for Domain0 + * @vcpu: virtual CPU number + * @cpumap: pointer to a bit map of real CPUs (in 8-bit bytes) (IN) + * Each bit set to 1 means that corresponding CPU is usable. + * Bytes are stored in little-endian order: CPU0-7, 8-15... + * In each byte, lowest CPU number is least significant bit. + * @maplen: number of bytes in cpumap, from 1 up to size of CPU map in + * underlying virtualization system (Xen...). + * If maplen < size, missing bytes are set to zero. + * If maplen > size, failure code is returned. + * + * Dynamically change the real CPUs which can be allocated to a virtual CPU. + * This function requires priviledged access to the hypervisor. + * + * Returns 0 in case of success, -1 in case of failure. + */ +int +virDomainPinVcpu(virDomainPtr domain, unsigned int vcpu, + unsigned char *cpumap, int maplen) +{ + if (domain == NULL) { + TODO + return (-1); + } + if (!VIR_IS_CONNECTED_DOMAIN(domain)) { + virLibDomainError(domain, VIR_ERR_INVALID_DOMAIN, __FUNCTION__); + return (-1); + } + if (domain->conn->flags & VIR_CONNECT_RO) + return (-1); + if ((vcpu < 1) || (cpumap == NULL) || (maplen < 1)) { + virLibDomainError(domain, VIR_ERR_INVALID_ARG, __FUNCTION__); + return (-1); + } + if (xenHypervisorPinVcpu(domain, vcpu, cpumap, maplen) == 0) + return 0; + return (-1); //xenDaemonDomainPinVcpu(domain, vcpu, cpumap, maplen); +} + +/** + * virDomainGetVcpus: + * @domain: pointer to domain object, or NULL for Domain0 + * @info: pointer to an array of virVcpuInfo structures (OUT) + * @maxinfo: number of structures in info array + * @cpumaps: pointer to an bit map of real CPUs for all vcpus of this domain (in 8-bit bytes) (OUT) + * If cpumaps is NULL, then no cupmap information is returned by the API. + * It's assumed there is <maxinfo> cpumap in cpumaps array. + * The memory allocated to cpumaps must be (maxinfo * maplen) bytes + * (ie: calloc(maxinfo, maplen)). + * One cpumap inside cpumaps has the format described in virDomainPinVcpu() API. + * @maplen: number of bytes in one cpumap, from 1 up to size of CPU map in + * underlying virtualization system (Xen...). + * + * Extract information about virtual CPUs of domain, store it in info array + * and also in cpumaps if this pointer is'nt NULL. + * + * Returns the number of info filled in case of success, -1 in case of failure. + */ +int +virDomainGetVcpus(virDomainPtr domain, virVcpuInfoPtr info, int maxinfo, + unsigned char *cpumaps, int maplen) +{ + int ret; + + if (domain == NULL) { + TODO + return (-1); + } + if (!VIR_IS_CONNECTED_DOMAIN(domain)) { + virLibDomainError(domain, VIR_ERR_INVALID_DOMAIN, __FUNCTION__); + return (-1); + } + if ((info == NULL) || (maxinfo < 1)) { + virLibDomainError(domain, VIR_ERR_INVALID_ARG, __FUNCTION__); + return (-1); + } + if (cpumaps != NULL && maplen < 1) { + virLibDomainError(domain, VIR_ERR_INVALID_ARG, __FUNCTION__); + return (-1); + } + ret = xenHypervisorGetVcpus(domain, info, maxinfo, cpumaps, maplen); + if (ret != -1) return ret; + return xenDaemonDomainGetVcpus(domain, info, maxinfo, cpumaps, maplen); +} --- libvirt-0.1.3/src/xend_internal.h 2006-07-10 12:24:05.000000000 +0200 +++ libvirt-0.1.3.with_vcpus/src/xend_internal.h 2006-08-02 16:54:27.000000000 +0200 @@ -631,6 +631,18 @@ unsigned long xenDaemonDomainGetMaxMemory(virDomainPtr domain); char **xenDaemonListDomainsOld(virConnectPtr xend); +int xenDaemonDomainSetVcpus (virDomainPtr domain, + int vcpus); +int xenDaemonDomainPinVcpu (virDomainPtr domain, + unsigned int vcpu, + unsigned char *cpumap, + int maplen); +int xenDaemonDomainGetVcpus (virDomainPtr domain, + virVcpuInfoPtr info, + int maxinfo, + unsigned char *cpumaps, + int maplen); + #ifdef __cplusplus } #endif --- libvirt-0.1.3/src/libvirt_sym.version 2006-05-29 18:19:30.000000000 +0200 +++ libvirt-0.1.3.with_vcpus/src/libvirt_sym.version 2006-08-02 15:26:56.000000000 +0200 @@ -47,5 +47,9 @@ virConnResetLastError; virDefaultErrorFunc; virNodeGetInfo; + + virDomainSetVcpus; + virDomainPinVcpu; + virDomainGetVcpus; local: *; }; --- libvirt-0.1.3/src/xend_internal.c 2006-07-10 16:41:54.000000000 +0200 +++ libvirt-0.1.3.with_vcpus/src/xend_internal.c 2006-08-02 16:48:24.000000000 +0200 @@ -2429,6 +2429,156 @@ free(name); return (NULL); } +/////////////////////////////////////////////////////////////////////////// + +/** + * xenDaemonDomainSetVcpus: + * @domain: pointer to domain object + * @nvcpus: the new number of virtual CPUs for this domain + * + * Dynamically change the number of virtual CPUs used by the domain. + * + * Returns 0 for success; -1 (with errno) on error + */ +int +xenDaemonDomainSetVcpus(virDomainPtr domain, int vcpus) +{ + char buf[16]; + + if ((domain == NULL) || (domain->conn == NULL) || (domain->name == NULL) + || (vcpus < 1)) { + virXendError((domain ? domain->conn : NULL), VIR_ERR_INVALID_ARG, + __FUNCTION__); + return (-1); + } + snprintf(buf, sizeof(buf), "%d", vcpus); + return xend_op(domain->conn, domain->name, "op", "set_vcpus", "vcpus", + buf, NULL); +} + +/** + * xenDaemonDomainPinCpu: + * @domain: pointer to domain object + * @vcpu: virtual CPU number + * @cpumap: pointer to a bit map of real CPUs (in 8-bit bytes) + * @maplen: length of cpumap in bytes + * + * Dynamically change the real CPUs which can be allocated to a virtual CPU. + * + * Returns 0 for success; -1 (with errno) on error + */ +int +xenDaemonDomainPinVcpu(virDomainPtr domain, unsigned int vcpu, + unsigned char *cpumap, int maplen) +{ + char buf[16], mapstr[sizeof(cpumap_t) * 64] = "["; + int i, j; + + if ((domain == NULL) || (domain->conn == NULL) || (domain->name == NULL) + || (cpumap == NULL) || (maplen < 1) || (maplen > (int)sizeof(cpumap_t))) { + virXendError((domain ? domain->conn : NULL), VIR_ERR_INVALID_ARG, + __FUNCTION__); + return (-1); + } + + /* from bit map, build character string of mapped CPU numbers */ + for (i = 0; i < maplen; i++) for (j = 0; j < 8; j++) + if (cpumap[i] & (1 << j)) { + sprintf(buf, "%d,", (8 * i) + j); + strcat(mapstr, buf); + } + mapstr[strlen(mapstr) - 1] = ']'; + snprintf(buf, sizeof(buf), "%d", vcpu); + return xend_op(domain->conn, domain->name, "op", "pincpu", "vcpu", buf, + "cpumap", mapstr, NULL); +} + +/** + * virDomainGetVcpus: + * @domain: pointer to domain object, or NULL for Domain0 + * @info: pointer to an array of virVcpuInfo structures (OUT) + * @maxinfo: number of structures in info array + * @cpumaps: pointer to an bit map of real CPUs for all vcpus of this domain (in 8-bit bytes) (OUT) + * If cpumaps is NULL, then no cupmap information is returned by the API. + * It's assumed there is <maxinfo> cpumap in cpumaps array. + * The memory allocated to cpumaps must be (maxinfo * maplen) bytes + * (ie: calloc(maxinfo, maplen)). + * One cpumap inside cpumaps has the format described in virDomainPinVcpu() API. + * @maplen: number of bytes in one cpumap, from 1 up to size of CPU map in + * underlying virtualization system (Xen...). + * + * Extract information about virtual CPUs of domain, store it in info array + * and also in cpumaps if this pointer is'nt NULL. + * + * Returns the number of info filled in case of success, -1 in case of failure. + */ +int +xenDaemonDomainGetVcpus(virDomainPtr domain, virVcpuInfoPtr info, int maxinfo, + unsigned char *cpumaps, int maplen) +{ + struct sexpr *root, *s, *t; + virVcpuInfoPtr ipt = info; + int nbinfo = 0, oln; + unsigned char *cpumap; + int vcpu, cpu; + + if ((domain == NULL) || (domain->conn == NULL) || (domain->name == NULL) + || (info == NULL) || (maxinfo < 1)) { + virXendError((domain ? domain->conn : NULL), VIR_ERR_INVALID_ARG, + __FUNCTION__); + return (-1); + } + if (cpumaps != NULL && maplen < 1) { + virXendError((domain ? domain->conn : NULL), VIR_ERR_INVALID_ARG, + __FUNCTION__); + return (-1); + } + root = sexpr_get(domain->conn, "/xend/domain/%s?op=vcpuinfo", domain->name); + if (root == NULL) + return (-1); + + if (cpumaps != NULL) + memset(cpumaps, 0, maxinfo * maplen); + + /* scan the sexprs from "(vcpu (number x)...)" and get parameter values */ + for (s = root; s->kind == SEXPR_CONS; s = s->cdr) + if ((s->car->kind == SEXPR_CONS) && + (s->car->car->kind == SEXPR_VALUE) && + !strcmp(s->car->car->value, "vcpu")) { + t = s->car; + vcpu = ipt->number = sexpr_int(t, "vcpu/number"); + if (oln = sexpr_int(t, "vcpu/online")) { + if (sexpr_int(t, "vcpu/running")) ipt->state = VIR_VCPU_RUNNING; + if (sexpr_int(t, "vcpu/blocked")) ipt->state = VIR_VCPU_BLOCKED; + } + else ipt->state = VIR_VCPU_OFFLINE; + ipt->cpuTime = sexpr_float(t, "vcpu/cpu_time") * 1000000000; + ipt->cpu = oln ? sexpr_int(t, "vcpu/cpu") : -1; + + if (cpumaps != NULL && vcpu >= 0 && vcpu < maxinfo) { + cpumap = (unsigned char *)GET_CPUMAP(cpumaps, maplen, vcpu); + /* get sexpr from "(cpumap (x y z...))" and convert values to bitmap */ + for (t = t->cdr; t->kind == SEXPR_CONS; t = t->cdr) + if ((t->car->kind == SEXPR_CONS) && + (t->car->car->kind == SEXPR_VALUE) && + !strcmp(t->car->car->value, "cpumap") && + (t->car->cdr->kind == SEXPR_CONS)) { + for (t = t->car->cdr->car; t->kind == SEXPR_CONS; t = t->cdr) + if (t->car->kind == SEXPR_VALUE) { + cpu = strtol(t->car->value, NULL, 0); + if (cpu >= 0) + USE_CPU(cpumap, cpu); + } + break; + } + } + + if (++nbinfo == maxinfo) break; + ipt++; + } + sexpr_free(root); + return nbinfo; +} /** * xenDaemonLookupByUUID:
On Thu, Aug 03, 2006 at 03:32:46PM +0200, Philippe Berthault wrote:
The patch (vcpus.diff) is joined with this mail.
To apply the patch, untar the libvirt-0.1.3.tar.gz archive (if not already done). Then in libvirt-0.1.3 directory, do ./configure --prefix=/usr if not already done, and after this: patch -p1 < vcpus.diff make make install
From the initial patch of Michel Ponceau (not delivered), I've just modify the virDomainGetVcpus function and added some useful macro for bit manipulation in cpu bit maps.
Excellent, thanks, I got it and reviewed it. I had a few things to change: - discard // based comments to stick just to /* - apply the header patch to libvirt.h.in not to the generated header - shorten some comments to fit in 80 characters wide - renamed the macros adding the VIR_ prefix to avoid potential name collision with other headers - Added TODOs in the front-end functions, as they should go through the driver API not calling Xen specific function directly If I remember correctly some points need to be sorted out there, any function with side effect should probably go though xend only when talking to a Xen virtualization - some cosmetic formatting changes After those I think it was in a shape to be commited, so it's in but there is a few thing left to be done, see below:
TODO: These vcpus functions permits to dynamically change the vcpu/cpu relation after the domain has been started but nothing has been done regarding static cpus attribution in the XML configuration file of a domain.
Yes there is a few things left: - as you said the impact on XML format (I'm not sure how much should be extracted though) - there is the driver cleanup, the VCPU entry points should be added to the interface, the front-end function should call the drivers and some tweaking might be needed in the Xen case to select when or when not to use direct hypervisor calls if available. - the doc and formal XML API need to be generated (make rebuild in doc) - in turn once the XML API is there, see how this affects Python binding generation, we will probably need to write specific glue to interface Python and make nicer functions there (using list instead of bitmaps possibly) - add tests - then release 0.1.4 Ahum, it's just a few things :-) I will probably take on the driver fixup first myself. So thanks again, and don't forget to update your CVS checkout, Daniel -- Daniel Veillard | Red Hat http://redhat.com/ veillard@redhat.com | libxml GNOME XML XSLT toolkit http://xmlsoft.org/ http://veillard.com/ | Rpmfind RPM search engine http://rpmfind.net/
On Fri, Aug 04, 2006 at 05:45:41AM -0400, Daniel Veillard wrote:
On Thu, Aug 03, 2006 at 03:32:46PM +0200, Philippe Berthault wrote:
The patch (vcpus.diff) is joined with this mail.
To apply the patch, untar the libvirt-0.1.3.tar.gz archive (if not already done). Then in libvirt-0.1.3 directory, do ./configure --prefix=/usr if not already done, and after this: patch -p1 < vcpus.diff make make install
From the initial patch of Michel Ponceau (not delivered), I've just modify the virDomainGetVcpus function and added some useful macro for bit manipulation in cpu bit maps.
Excellent, thanks, I got it and reviewed it. I had a few things to change: - discard // based comments to stick just to /* - apply the header patch to libvirt.h.in not to the generated header - shorten some comments to fit in 80 characters wide - renamed the macros adding the VIR_ prefix to avoid potential name collision with other headers - Added TODOs in the front-end functions, as they should go through the driver API not calling Xen specific function directly If I remember correctly some points need to be sorted out there, any function with side effect should probably go though xend only when talking to a Xen virtualization - some cosmetic formatting changes
I've just reviewed the code you committed & it looks pretty sane to me. I'd definitely want to hook up the driver API asap - it should be pretty easy todo since the logic is already split out into appropriate backend files. I'd really like to test it before we releease 0.1.4 but can't just yet - see my point below..
After those I think it was in a shape to be commited, so it's in but there is a few thing left to be done, see below:
TODO: These vcpus functions permits to dynamically change the vcpu/cpu relation after the domain has been started but nothing has been done regarding static cpus attribution in the XML configuration file of a domain.
Yes there is a few things left: - as you said the impact on XML format (I'm not sure how much should be extracted though)
Whether we need the vcpu mapping in the XML really depends on whether its useful to persist this stuff, or whether its something you can just define at runtime as required. VCPU mappings really get into the realm of runtime policy which I'm not convinced really wants/needs to be part of the basic XML document data. For example, when talking in terms of policy I don't think you'd directly store info in terms of 'map vcpu-1 to host cpu 3'. Instead you'd probably want to express it as 'give domain X a dedicated fixed CPU', or 'let domain X share/float 3 cpus' with other domains. The policy/management app would then decide which particular VCPU<->Host CPU mappings to implement when starting up the domain. Such a decision would have to take into account existing VCPU pinning from other domains already running. So really IMHO we don't need to rush into adding VCPU mapping to the XML. Better to wait and see if there are apps being created which explicitly need this info persisted in the XML.
- there is the driver cleanup, the VCPU entry points should be added to the interface, the front-end function should call the drivers and some tweaking might be needed in the Xen case to select when or when not to use direct hypervisor calls if available. - the doc and formal XML API need to be generated (make rebuild in doc) - in turn once the XML API is there, see how this affects Python binding generation, we will probably need to write specific glue to interface Python and make nicer functions there (using list instead of bitmaps possibly) - add tests
One more: - expose the new APIs via virsh. Michael originally suggsted pretty much following the style of 'xm vcpu-pin' and 'vcpu-list'. Seems like a fairly reasonable pattern to follow.
- then release 0.1.4
Ahum, it's just a few things :-) I will probably take on the driver fixup first myself.
Regards, Dan. -- |=- Red Hat, Engineering, Emerging Technologies, Boston. +1 978 392 2496 -=| |=- Perl modules: http://search.cpan.org/~danberr/ -=| |=- Projects: http://freshmeat.net/~danielpb/ -=| |=- GnuPG: 7D3B9505 F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 -=|
On Fri, Aug 04, 2006 at 06:55:17PM +0100, Daniel P. Berrange wrote:
On Fri, Aug 04, 2006 at 05:45:41AM -0400, Daniel Veillard wrote:
On Thu, Aug 03, 2006 at 03:32:46PM +0200, Philippe Berthault wrote: I had a few things to change: - discard // based comments to stick just to /* - apply the header patch to libvirt.h.in not to the generated header - shorten some comments to fit in 80 characters wide - renamed the macros adding the VIR_ prefix to avoid potential name collision with other headers
I added 3 more convenience macros: VIR_NODEINFO_MAXCPUS(nodeinfo) VIR_CPU_MAPLEN(cpu) VIR_CPU_FULL_MAPLEN(vcpu, cpu) To hide the maths from developers. I also bug-fixed the pin cpu method which mistakenly checked for vcpu < 1 instead of vcpu < 0 when validating args. See atached patch for these changes
One more:
- expose the new APIs via virsh. Michael originally suggsted pretty much following the style of 'xm vcpu-pin' and 'vcpu-list'. Seems like a fairly reasonable pattern to follow.
See the attached patch which adds these commands. eg: # ~berrange/usr/bin/virsh vcpuinfo Demo03 VCPU: 0 CPU: 1 State: blocked CPU time: 13.4s CPU Affinity: y- VCPU: 1 CPU: 0 State: blocked CPU time: 5.0s CPU Affinity: -y # ~berrange/usr/bin/virsh vcpupin Demo03 1 0,1 # ~berrange/usr/bin/virsh vcpuinfo Demo03 VCPU: 0 CPU: 1 State: blocked CPU time: 13.4s CPU Affinity: -y VCPU: 1 CPU: 1 State: blocked CPU time: 5.0s CPU Affinity: yy Dan. -- |=- Red Hat, Engineering, Emerging Technologies, Boston. +1 978 392 2496 -=| |=- Perl modules: http://search.cpan.org/~danberr/ -=| |=- Projects: http://freshmeat.net/~danielpb/ -=| |=- GnuPG: 7D3B9505 F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 -=|
On Sat, Aug 05, 2006 at 01:27:43AM +0100, Daniel P. Berrange wrote:
I added 3 more convenience macros:
VIR_NODEINFO_MAXCPUS(nodeinfo) VIR_CPU_MAPLEN(cpu) VIR_CPU_FULL_MAPLEN(vcpu, cpu)
good, yes
To hide the maths from developers.
I also bug-fixed the pin cpu method which mistakenly checked for vcpu < 1 instead of vcpu < 0 when validating args.
good catch !
See atached patch for these changes
Looks sane to me, please commit :-)
One more:
- expose the new APIs via virsh. Michael originally suggsted pretty much following the style of 'xm vcpu-pin' and 'vcpu-list'. Seems like a fairly reasonable pattern to follow.
See the attached patch which adds these commands. eg:
very cool :-), Yay ! Daniel -- Daniel Veillard | Red Hat http://redhat.com/ veillard@redhat.com | libxml GNOME XML XSLT toolkit http://xmlsoft.org/ http://veillard.com/ | Rpmfind RPM search engine http://rpmfind.net/
Daniel P. Berrange a écrit :
One more:
- expose the new APIs via virsh. Michael originally suggsted pretty much following the style of 'xm vcpu-pin' and 'vcpu-list'. Seems like a fairly reasonable pattern to follow.
See the attached patch which adds these commands. eg:
# ~berrange/usr/bin/virsh vcpuinfo Demo03 VCPU: 0 CPU: 1 State: blocked CPU time: 13.4s CPU Affinity: y-
VCPU: 1 CPU: 0 State: blocked CPU time: 5.0s CPU Affinity: -y
I've applied your patch on my system (RHEL-AS4, update 2) with Xen 3.0.2-2. The compilation of virsh is OK but the execution is failed. The virsh vcpuinfo command returns all vcpus of the specified domain (the list is OK) but after the last vcpu of the list, I've the following message: *** glibc detected *** free(): invalid next size (fast): 0x08051dd8 *** Aborted With the domain-0, the virsh vcpuinfo is OK. I haven't investigated with the debugger but perhaps you have an idea of what is the problem ? Thanks, Philippe Berthault.
On Mon, Aug 07, 2006 at 11:55:22AM +0200, Philippe Berthault wrote:
Daniel P. Berrange a écrit :
One more:
- expose the new APIs via virsh. Michael originally suggsted pretty much following the style of 'xm vcpu-pin' and 'vcpu-list'. Seems like a fairly reasonable pattern to follow.
See the attached patch which adds these commands. eg:
# ~berrange/usr/bin/virsh vcpuinfo Demo03 VCPU: 0 CPU: 1 State: blocked CPU time: 13.4s CPU Affinity: y-
VCPU: 1 CPU: 0 State: blocked CPU time: 5.0s CPU Affinity: -y
I've applied your patch on my system (RHEL-AS4, update 2) with Xen 3.0.2-2. The compilation of virsh is OK but the execution is failed. The virsh vcpuinfo command returns all vcpus of the specified domain (the list is OK) but after the last vcpu of the list, I've the following message:
*** glibc detected *** free(): invalid next size (fast): 0x08051dd8 *** Aborted
With the domain-0, the virsh vcpuinfo is OK. I haven't investigated with the debugger but perhaps you have an idea of what is the problem ?
Hum, valgrind should be able to pinpoint where the error occurs exactly Daniel -- Daniel Veillard | Red Hat http://redhat.com/ veillard@redhat.com | libxml GNOME XML XSLT toolkit http://xmlsoft.org/ http://veillard.com/ | Rpmfind RPM search engine http://rpmfind.net/
On Mon, Aug 07, 2006 at 06:04:16AM -0400, Daniel Veillard wrote:
On Mon, Aug 07, 2006 at 11:55:22AM +0200, Philippe Berthault wrote:
I've applied your patch on my system (RHEL-AS4, update 2) with Xen 3.0.2-2. The compilation of virsh is OK but the execution is failed. The virsh vcpuinfo command returns all vcpus of the specified domain (the list is OK) but after the last vcpu of the list, I've the following message:
*** glibc detected *** free(): invalid next size (fast): 0x08051dd8 *** Aborted
With the domain-0, the virsh vcpuinfo is OK. I haven't investigated with the debugger but perhaps you have an idea of what is the problem ?
Hum, valgrind should be able to pinpoint where the error occurs exactly
I've found the problem - in the last parameter to virDomainGetVcpuInfo I thought the cpumaplen parameter was spposed to be the size of the entire cpumap, when it is only supposed to be the size of one dimension of it. So, inside virDomainGetVcpuInfo it then did a memset on the cpumap using cpumaplen * nrVirtCpu, which obviously overflowed if nrVirtCpu was greater than one. I'll fix this before committing. Regards, Dan. -- |=- Red Hat, Engineering, Emerging Technologies, Boston. +1 978 392 2496 -=| |=- Perl modules: http://search.cpan.org/~danberr/ -=| |=- Projects: http://freshmeat.net/~danielpb/ -=| |=- GnuPG: 7D3B9505 F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 -=|
On Mon, Aug 07, 2006 at 01:46:19PM +0100, Daniel P. Berrange wrote:
I've found the problem - in the last parameter to virDomainGetVcpuInfo I thought the cpumaplen parameter was spposed to be the size of the entire cpumap, when it is only supposed to be the size of one dimension of it. So, inside virDomainGetVcpuInfo it then did a memset on the cpumap using cpumaplen * nrVirtCpu, which obviously overflowed if nrVirtCpu was greater than one. I'll fix this before committing.
The patch is now committed. Dan. -- |=- Red Hat, Engineering, Emerging Technologies, Boston. +1 978 392 2496 -=| |=- Perl modules: http://search.cpan.org/~danberr/ -=| |=- Projects: http://freshmeat.net/~danielpb/ -=| |=- GnuPG: 7D3B9505 F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 -=|
Daniel P. Berrange a écrit :
TODO: These vcpus functions permits to dynamically change the vcpu/cpu relation after the domain has been started but nothing has been done regarding static cpus attribution in the XML configuration file of a domain.
Yes there is a few things left: - as you said the impact on XML format (I'm not sure how much should be extracted though)
Whether we need the vcpu mapping in the XML really depends on whether its useful to persist this stuff, or whether its something you can just define at runtime as required. VCPU mappings really get into the realm of runtime policy which I'm not convinced really wants/needs to be part of the basic XML document data.
For example, when talking in terms of policy I don't think you'd directly store info in terms of 'map vcpu-1 to host cpu 3'. Instead you'd probably want to express it as 'give domain X a dedicated fixed CPU', or 'let domain X share/float 3 cpus' with other domains. The policy/management app would then decide which particular VCPU<->Host CPU mappings to implement when starting up the domain. Such a decision would have to take into account existing VCPU pinning from other domains already running.
So really IMHO we don't need to rush into adding VCPU mapping to the XML. Better to wait and see if there are apps being created which explicitly need this info persisted in the XML.
Yes, I'm agree. It's better to wait and see the next Xen release with NUMA support and the new "Credit" scheduler because the cpu mapping policy depend narrowly on these features.
participants (3)
-
Daniel P. Berrange -
Daniel Veillard -
Philippe Berthault