2017-03-30 18:47 GMT+02:00 Dawid Zamirski <dzamirski@datto.com>:

This struct is to be used to carry all the information necessary to issue wsman requests for given WMI class. Those will be defined by the generator code (as lists) so that they are handy for the driver code to "extract" needed info depending on which hyper-v we're connected to. For example:

hypervWmiClassInfoListPtr Msvm_ComputerSystem_WmiInfo = { .count = 2 { { .name = "Msvm_ComputerSystem", .version = "v1", .rootUri = "http://asdf.com", ... }, { .name = "Msvm_ComputerSystem", .version = "v2", .rootUri = "http://asdf.com/v2", ... }, } };

Then the driver code will grab either "v1" or "v2" to pass info wsman API, depending on hypervPrivate->wmiVersion value. --- src/hyperv/hyperv_wmi.c | 6 ++---- src/hyperv/hyperv_wmi_classes.h | 22 ++++++++++++++++++++++ 2 files changed, 24 insertions(+), 4 deletions(-)

diff --git a/src/hyperv/hyperv_wmi.c b/src/hyperv/hyperv_wmi.c index 947101f..309edac 100644 --- a/src/hyperv/hyperv_wmi.c +++ b/src/hyperv/hyperv_wmi.c @@ -36,16 +36,14 @@

#define WS_SERIALIZER_FREE_MEM_WORKS 0

-#define ROOT_CIMV2 \ +# define ROOT_CIMV2 \ "http://schemas.microsoft.com/wbem/wsman/1/wmi/root/cimv2/*"

-#define ROOT_VIRTUALIZATION \ +# define ROOT_VIRTUALIZATION \ "http://schemas.microsoft.com/wbem/wsman/1/wmi/root/virtualization/*"

#define VIR_FROM_THIS VIR_FROM_HYPERV

- - int hypervVerifyResponse(WsManClient *client, WsXmlDocH response, const char *detail)

Unnecessary whitespace change. Also this will not pass make syntax-check. I'll drop this complete hunk before pushing it. -- Matthias Bolte http://photron.blogspot.com

On Sat, 2017-04-01 at 14:27 +0200, Matthias Bolte wrote:
> 2017-03-30 18:47 GMT+02:00 Dawid Zamirski <dzamirski@datto.com>:
> > This struct is to be passed to enumerate-and-pull wsman request (to
> > run
> > "Select" queries) and provides the hypervWmiClassInfoListPtr
> > instance
> > from which we can extract the version specific info using the new
> > hypervGetWmiClassInfo function (currently unused)
> > ---
> >  src/hyperv/hyperv_wmi.c | 35 +++++++++++++++++++++++++++++++++++
> >  src/hyperv/hyperv_wmi.h |  8 ++++++++
> >  2 files changed, 43 insertions(+)
> > 
> > diff --git a/src/hyperv/hyperv_wmi.c b/src/hyperv/hyperv_wmi.c
> > index 309edac..42ce507 100644
> > --- a/src/hyperv/hyperv_wmi.c
> > +++ b/src/hyperv/hyperv_wmi.c
> > @@ -44,6 +44,41 @@
> > 
> >  #define VIR_FROM_THIS VIR_FROM_HYPERV
> > 
> > +static int
> > +hypervGetWmiClassInfo(hypervPrivate *priv,
> > hypervWmiClassInfoListPtr list,
> > +                      hypervWmiClassInfoPtr *info)
> > +{
> > +    const char *version = "v2";
> > +    size_t i;
> > +
> > +    if (list->count == 0) {
> > +        virReportError(VIR_ERR_INTERNAL_ERROR, "%s",
> > +                       _("The WMI class info list is empty"));
> > +        return -1;
> > +    }
> > +
> > +    if (list->count == 1) {
> > +        *info = list->objs[0];
> > +        return 0;
> 
> Here you assume that if there is only one variation of a WMI class
> that it is the v1 variation, and you don't do any version check. At
> the moment this is fine, because your assumption holds. But this is
> not future proof. If v2-only WMI classes are added then this check
> will happily return a v2 WMI class info for a v1 connection.
> 
> 

Yep, that's a problem that needs to be addressed. The reason why I had
it like this was because there are WMI classes that apply to both e.g
Win32_* or CIM_* classes - the only ones currently "versioned" are
Msvm_* classes while the rest is assumed to be "shared". So this code
was assuming if there's just one version then it's "shared" but yeah I
did not think about the case you pointed out. 

To address this, I'm going to set version to NULL in the generator for
"shared" classes using the following rule:
* there's only one version specified in .input file
* it does not have explicitly defined version

So if there's a class that's v2-only without v1 equivalent it needs to
be "versioned" in the .input file, e.g "v2/Msvm_VirtualEthernetSwitch,
and vice-versa

This should cover the following cases:
* v1 and v2 classes (list->count > 1, match version == priv->version)
* v2-only or v1-only classes (list->count == 1, match version == priv-
>version)
* "shared" classes (list->count == 1, match version == NULL)

Regards,
Dawid

This patch updates the code generator that outputs C headers and code for WMI classes. It has been updated to handle multiple versions (or namespaces) of the same class which were introduced with Hyperv 2012+ --- changes in v2: * make it pass "make syntax-check" src/hyperv/hyperv_wmi_generator.py | 407 ++++++++++++++++++++++++++----------- 1 file changed, 283 insertions(+), 124 deletions(-) mode change 100755 => 100644 src/hyperv/hyperv_wmi_generator.py diff --git a/src/hyperv/hyperv_wmi_generator.py b/src/hyperv/hyperv_wmi_generator.py old mode 100755 new mode 100644 index 8c62882..21934fd --- a/src/hyperv/hyperv_wmi_generator.py +++ b/src/hyperv/hyperv_wmi_generator.py @@ -1,153 +1,308 @@ #!/usr/bin/env python -# -# hyperv_wmi_generator.py: generates most of the WMI type mapping code -# -# Copyright (C) 2011 Matthias Bolte <matthias.bolte@googlemail.com> -# -# This library is free software; you can redistribute it and/or -# modify it under the terms of the GNU Lesser General Public -# License as published by the Free Software Foundation; either -# version 2.1 of the License, or (at your option) any later version. -# -# This library is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# Lesser General Public License for more details. -# -# You should have received a copy of the GNU Lesser General Public -# License along with this library. If not, see -# <http://www.gnu.org/licenses/>. -# - import sys import os import os.path +separator = "/*" + ("*" * 50) + "*\n" +wmi_version_separator = "/" +wmi_classes_by_name = {} + +class WmiClass: + """Represents WMI class and provides methods to generate C code. + + This class holds one or more instances of WmiClassVersion because with the + Windows 2012 release, Microsoft introduced "v2" version of Msvm_* family of + classes that need different URI for making wsman requests and also have + some additional/changed properties (though many of the properies are the + same as in "v1". Therefore, this class makes sure that C code is generated + for each of them while avoiding name conflics, identifies common members, + and defined *_WmiInfo structs holding info about each version so the driver + code can make the right choices based on which Hyper-v host it's connected + to. + """ + + def __init__(self, name, versions = []): + self.name = name + self.versions = versions + self.common = None -separator = "/* " + ("* " * 37) + "*\n" + def prepare(self): + """Prepares the class for code generation + Makes sure that "versioned" classes are sorted by version, identfies + common properies and ensures that they are aligned by name and + type in each version + """ + # sort vesioned classes by version in case input file did not have them + # in order + self.versions = sorted(self.versions, key=lambda cls: cls.version) + # if there's more than one verion make sure first one has name suffixed + # because we'll generate "common" memeber and will the "base" name + if len(self.versions) > 1: + first = self.versions[0] + first.name = "%s_%s" % (first.name, first.version) -class Class: - def __init__(self, name, properties): - self.name = name - self.properties = properties + # finally, identify common members in all versions and make sure they + # are in the same order - to ensure C struc member alignment + self._align_property_members() + + + def generate_classes_header(self): + """Generate C header code and return it as string + Declares: + <class_name>_Data - used as one of hypervObject->data members + <class_name>_TypeInfo - used as wsman XmlSerializerInfo + <class_name> - "inherits" hypervObject struct + """ - def generate_header(self): name_upper = self.name.upper() header = separator header += " * %s\n" % self.name header += " */\n" header += "\n" - header += "int hypervGet%sList(hypervPrivate *priv, virBufferPtr query, %s **list);\n" \ - % (self.name.replace("_", ""), self.name) - header += "\n" + header += "#define %s_CLASSNAME \\\n" % name_upper + header += " \"%s\"\n" % self.name header += "\n" + header += "#define %s_WQL_SELECT \\\n" % name_upper + header += " \"SELECT * FROM %s \"\n" % self.name header += "\n" + header += "extern hypervWmiClassInfoListPtr %s_WmiInfo;\n\n" % self.name + + header += self._declare_data_structs() + header += self._declare_hypervObject_struct() return header + def generate_classes_source(self): + """Returns a C code string defining wsman data structs + + Defines: + <class_name>_Data structs + <class_name>_WmiInfo - list holding metadata (e.g. request URIs) for + each known version of WMI class. + """ + + source = separator + source += " * %s\n" % self.name + source += " */\n" + + for cls in self.versions: + source += "SER_START_ITEMS(%s_Data)\n" % cls.name + + for property in cls.properties: + source += property.generate_classes_source(cls.name) + + source += "SER_END_ITEMS(%s_Data);\n\n" % cls.name + + + source += self._define_WmiInfo_struct() + source += "\n\n" + + return source + + def generate_classes_typedef(self): - typedef = "typedef struct _%s_Data %s_Data;\n" % (self.name, self.name) - typedef += "typedef struct _%s %s;\n" % (self.name, self.name) + """Returns C string for typdefs""" + + typedef = "typedef struct _%s %s;\n" % (self.name, self.name) + + if self.common is not None: + typedef += "typedef struct _%s_Data %s_Data;\n" % (self.name, self.name) + + for cls in self.versions: + typedef += "typedef struct _%s_Data %s_Data;\n" % (cls.name, cls.name) return typedef - def generate_classes_header(self): - name_upper = self.name.upper() - header = separator - header += " * %s\n" % self.name - header += " */\n" - header += "\n" - header += "#define %s_RESOURCE_URI \\\n" % name_upper + def _declare_data_structs(self): + """Returns string C code declaring data structs. - if self.name.startswith("Win32_") or self.name.startswith("CIM_"): - header += " \"http://schemas.microsoft.com/wbem/wsman/1/wmi/root/cimv2/%s\"\n" % self.name - else: - header += " \"http://schemas.microsoft.com/wbem/wsman/1/wmi/root/virtualization/%s\"\n" % self.name + The *_Data structs are members of hypervObject data union. Each one has + corresponding *_TypeInfo that is used for wsman unserialization of + response XML into the *_Data structs. If there's a "common" member, it + won't have corresponding *_TypeInfo becuase this is a special case only + used to provide a common "view" of v1, v2 etc members + """ - header += "\n" - header += "#define %s_CLASSNAME \\\n" % name_upper - header += " \"%s\"\n" % self.name - header += "\n" - header += "#define %s_WQL_SELECT \\\n" % name_upper - header += " \"select * from %s \"\n" % self.name - header += "\n" - header += "struct _%s_Data {\n" % self.name + header = "" + if self.common is not None: + header += "struct _%s_Data {\n" % self.name + for property in self.common: + header += property.generate_classes_header() + header += "};\n\n" - for property in self.properties: - header += property.generate_classes_header() + # Declare actual data struct for each versions + for cls in self.versions: + header += "#define %s_RESOURCE_URI \\\n" % cls.name.upper() + header += " \"%s\"\n" % cls.uri_info.resourceUri + header += "\n" + header += "struct _%s_Data {\n" % cls.name + for property in cls.properties: + header += property.generate_classes_header() + header += "};\n\n" + header += "SER_DECLARE_TYPE(%s_Data);\n" % cls.name - header += "};\n" - header += "\n" - header += "SER_DECLARE_TYPE(%s_Data);\n" % self.name - header += "\n" + return header + + + def _declare_hypervObject_struct(self): + """Return string for C code declaring hypervObject instance""" + + header = "\n/* must match hypervObject */\n" header += "struct _%s {\n" % self.name - header += " XmlSerializerInfo *serializerInfo;\n" - header += " %s_Data *data;\n" % self.name + header += " union {\n" + + # if there's common use it as "common" else first and only version is + # the "common" member + if self.common is not None: + header += " %s_Data *common;\n" % self.name + else: + header += " %s_Data *common;\n" % self.versions[0].name + + for cls in self.versions: + header += " %s_Data *%s;\n" % (cls.name, cls.version) + + header += " } data;\n" + header += " hypervWmiClassInfoPtr info;\n" header += " %s *next;\n" % self.name header += "};\n" - header += "\n" - header += "\n" - header += "\n" + + header += "\n\n\n" return header - def generate_source(self): - name_upper = self.name.upper() + def _define_WmiInfo_struct(self): + """Return string for C code defining *_WmiInfo struct - source = separator - source += " * %s\n" % self.name - source += " */\n" - source += "\n" - source += "int\n" - source += "hypervGet%sList(hypervPrivate *priv, virBufferPtr query, %s **list)\n" \ - % (self.name.replace("_", ""), self.name) - source += "{\n" - - if self.name.startswith("Win32_") or self.name.startswith("CIM_"): - source += " return hypervEnumAndPull(priv, query, ROOT_CIMV2,\n" - else: - source += " return hypervEnumAndPull(priv, query, ROOT_VIRTUALIZATION,\n" + Those structs hold info with meta-data needed to make wsman requests for + each version of WMI class + """ + + source = "hypervWmiClassInfoListPtr %s_WmiInfo = &(hypervWmiClassInfoList) {\n" % self.name + source += " .count = %d,\n" % len(self.versions) + source += " .objs = (hypervWmiClassInfoPtr []) {\n" + + for cls in self.versions: + source += " &(hypervWmiClassInfo) {\n" + source += " .name = %s_CLASSNAME,\n" % self.name.upper() + source += " .version = \"%s\",\n" % cls.version + source += " .rootUri = %s,\n" % cls.uri_info.rootUri + source += " .resourceUri = %s_RESOURCE_URI,\n" % cls.name.upper() + source += " .serializerInfo = %s_Data_TypeInfo\n" % cls.name + source += " },\n" - source += " %s_Data_TypeInfo,\n" % self.name - source += " %s_RESOURCE_URI,\n" % name_upper - source += " %s_CLASSNAME,\n" % name_upper - source += " (hypervObject **)list);\n" - source += "}\n" - source += "\n" - source += "\n" - source += "\n" + source += " }\n" + source += "};\n" return source - def generate_classes_source(self): - name_upper = self.name.upper() + def _align_property_members(self): + """Identifies common properties in all class versions. - source = separator - source += " * %s\n" % self.name - source += " */\n" - source += "\n" - source += "SER_START_ITEMS(%s_Data)\n" % self.name + Makes sure that properties in all versions are ordered with common + members first and that they are in the same order. This makes the + generated C structs memory aligned and safe to access via the "common" + struct that "shares" members with v1, v2 etc. + """ - for property in self.properties: - source += property.generate_classes_source(self.name) + num_classes = len(self.versions) + common = {} + property_info = {} - source += "SER_END_ITEMS(%s_Data);\n" % self.name - source += "\n" - source += "\n" - source += "\n" + if num_classes < 2: + return + + # count property occurences in all class versions + for cls in self.versions: + for prop in cls.properties: + # consdered same if matches by name AND type + key = "%s_%s" % (prop.name, prop.type) + + if key in property_info: + property_info[key][1] += 1 + else: + property_info[key] = [prop, 1] + + # isolate those that are common for all and keep track of their postions + pos = 0 + for key in property_info: + info = property_info[key] + # exists in all class versions + if info[1] == num_classes: + common[info[0].name] = [info[0], pos] + pos += 1 + + # alter each versions's property list so that common members are first + # and in the same order as in the common dictionary + total = len(common) + for cls in self.versions: + index = 0 + count = len(cls.properties) + + while index < count: + prop = cls.properties[index] + + # it's a "common" proptery + if prop.name in common: + pos = common[prop.name][1] + + # move to the same position as in "common" dictionary + if index != pos: + tmp = cls.properties[pos] + cls.properties[pos] = prop + cls.properties[index] = tmp + else: + index += 1 + else: + index += 1 + + # finally, get common properties as list sorted by position in dictionary + tmp = sorted(common.values(), key=lambda x: x[1]) + self.common = [] + for x in tmp: + self.common.append(x[0]) + + + +class ClassUriInfo: + """Prepares URI information needed for wsman requests.""" + + def __init__(self, wmi_name, version): + self.rootUri = "ROOT_CIMV2" + self.resourceUri = None + baseUri = "http://schemas.microsoft.com/wbem/wsman/1/wmi/root/cimv2" + + if wmi_name.startswith("Msvm_"): + baseUri = "http://schemas.microsoft.com/wbem/wsman/1/wmi/root/virtualization" + self.rootUri = "ROOT_VIRTUALIZATION" + + if version == "v2": + baseUri += "/v2" + self.rootUri = "ROOT_VIRTUALIZATION_V2" + + self.resourceUri = "%s/%s" % (baseUri, wmi_name) + + + +class WmiClassVersion: + """Represents specific verssion of WMI class.""" + + def __init__(self, name, version, properties, uri_info): + self.name = name + self.version = version + self.properties = properties + self.uri_info = uri_info - return source class Property: @@ -155,9 +310,13 @@ class Property: "string" : "STR", "datetime" : "STR", "int8" : "INT8", + "sint8" : "INT8", "int16" : "INT16", + "sint16" : "INT16", "int32" : "INT32", + "sint32" : "INT32", "int64" : "INT64", + "sint64" : "INT64", "uint8" : "UINT8", "uint16" : "UINT16", "uint32" : "UINT32", @@ -189,8 +348,6 @@ class Property: return " SER_NS_%s(%s_RESOURCE_URI, \"%s\", 1),\n" \ % (Property.typemap[self.type], class_name.upper(), self.name) - - def open_and_print(filename): if filename.startswith("./"): print " GEN " + filename[2:] @@ -217,8 +374,15 @@ def parse_class(block): assert header_items[0] == "class" name = header_items[1] - properties = [] + version = "v1" + wmi_name = name + ns_separator = name.find(wmi_version_separator) + + if ns_separator != -1: + version = name[:ns_separator] + wmi_name = name[ns_separator + 1:] + name = "%s_%s" % (wmi_name, version) for line in block[1:]: # expected format: <type> <name> @@ -236,7 +400,13 @@ def parse_class(block): properties.append(Property(type=items[0], name=items[1], is_array=is_array)) - return Class(name=name, properties=properties) + cls = WmiClassVersion(name=name, version=version, properties=properties, + uri_info=ClassUriInfo(wmi_name, version)) + + if wmi_name in wmi_classes_by_name: + wmi_classes_by_name[wmi_name].versions.append(cls) + else: + wmi_classes_by_name[wmi_name] = WmiClass(wmi_name, [cls]) @@ -248,15 +418,13 @@ def main(): input_filename = os.path.join(os.getcwd(), "hyperv_wmi_generator.input") output_dirname = os.getcwd() - header = open_and_print(os.path.join(output_dirname, "hyperv_wmi.generated.h")) - source = open_and_print(os.path.join(output_dirname, "hyperv_wmi.generated.c")) + classes_typedef = open_and_print(os.path.join(output_dirname, "hyperv_wmi_classes.generated.typedef")) classes_header = open_and_print(os.path.join(output_dirname, "hyperv_wmi_classes.generated.h")) classes_source = open_and_print(os.path.join(output_dirname, "hyperv_wmi_classes.generated.c")) - # parse input file + number = 0 - classes_by_name = {} block = None for line in file(input_filename, "rb").readlines(): @@ -268,7 +436,7 @@ def main(): line = line.lstrip().rstrip() if len(line) < 1: - continue + continue if line.startswith("class"): if block is not None: @@ -279,31 +447,22 @@ def main(): if block is not None: if line == "end": if block[0][1].startswith("class"): - cls = parse_class(block) - classes_by_name[cls.name] = cls + parse_class(block) block = None else: block.append((number, line)) - # write output files - notice = "/* Generated by hyperv_wmi_generator.py */\n\n\n\n" - - header.write(notice) - source.write(notice) - classes_typedef.write(notice) - classes_header.write(notice) - classes_source.write(notice) - - names = classes_by_name.keys() + names = wmi_classes_by_name.keys() names.sort() for name in names: - header.write(classes_by_name[name].generate_header()) - source.write(classes_by_name[name].generate_source()) - classes_typedef.write(classes_by_name[name].generate_classes_typedef()) - classes_header.write(classes_by_name[name].generate_classes_header()) - classes_source.write(classes_by_name[name].generate_classes_source()) + cls = wmi_classes_by_name[name] + cls.prepare() + + classes_typedef.write(cls.generate_classes_typedef()) + classes_header.write(cls.generate_classes_header()) + classes_source.write(cls.generate_classes_source()) -- 2.9.3

On Thu, 2017-03-30 at 16:07 -0400, Neal Gompa wrote:

On Thu, Mar 30, 2017 at 2:00 PM, Dawid Zamirski <dzamirski@datto.com>

...

b/src/hyperv/hyperv_wmi_generator.py old mode 100755 new mode 100644

Why are you changing the mode of the file?

...

index 8c62882..21934fd --- a/src/hyperv/hyperv_wmi_generator.py +++ b/src/hyperv/hyperv_wmi_generator.py @@ -1,153 +1,308 @@  #!/usr/bin/env python

-# -# hyperv_wmi_generator.py: generates most of the WMI type mapping code -# -# Copyright (C) 2011 Matthias Bolte <matthias.bolte@googlemail.com

...

-# -# This library is free software; you can redistribute it and/or -# modify it under the terms of the GNU Lesser General Public -# License as published by the Free Software Foundation; either -# version 2.1 of the License, or (at your option) any later version. -# -# This library is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU -# Lesser General Public License for more details. -# -# You should have received a copy of the GNU Lesser General Public -# License along with this library.  If not, see -# <http://www.gnu.org/licenses/>. -# -

Please put this back. It should not have been deleted.

Other than these two issues, this patch looks okay to me.

Whoops, must have fat-fingered something in vim :-) Neither I remember changing the mode of the file.

This patch updates the code generator that outputs C headers and code for WMI classes. It has been updated to handle multiple versions (or namespaces) of the same class which were introduced with Hyperv 2012+ --- changes in v3: * do not change file mode * restore accidentally deleted copyright notice src/hyperv/hyperv_wmi_generator.py | 387 +++++++++++++++++++++++++++---------- 1 file changed, 283 insertions(+), 104 deletions(-) diff --git a/src/hyperv/hyperv_wmi_generator.py b/src/hyperv/hyperv_wmi_generator.py index 8c62882..ccad04d 100755 --- a/src/hyperv/hyperv_wmi_generator.py +++ b/src/hyperv/hyperv_wmi_generator.py @@ -24,130 +24,305 @@ import sys import os import os.path +separator = "/*" + ("*" * 50) + "*\n" +wmi_version_separator = "/" +wmi_classes_by_name = {} + +class WmiClass: + """Represents WMI class and provides methods to generate C code. + + This class holds one or more instances of WmiClassVersion because with the + Windows 2012 release, Microsoft introduced "v2" version of Msvm_* family of + classes that need different URI for making wsman requests and also have + some additional/changed properties (though many of the properies are the + same as in "v1". Therefore, this class makes sure that C code is generated + for each of them while avoiding name conflics, identifies common members, + and defined *_WmiInfo structs holding info about each version so the driver + code can make the right choices based on which Hyper-v host it's connected + to. + """ + + def __init__(self, name, versions = []): + self.name = name + self.versions = versions + self.common = None -separator = "/* " + ("* " * 37) + "*\n" + def prepare(self): + """Prepares the class for code generation + Makes sure that "versioned" classes are sorted by version, identfies + common properies and ensures that they are aligned by name and + type in each version + """ + # sort vesioned classes by version in case input file did not have them + # in order + self.versions = sorted(self.versions, key=lambda cls: cls.version) + # if there's more than one verion make sure first one has name suffixed + # because we'll generate "common" memeber and will the "base" name + if len(self.versions) > 1: + first = self.versions[0] + first.name = "%s_%s" % (first.name, first.version) -class Class: - def __init__(self, name, properties): - self.name = name - self.properties = properties + # finally, identify common members in all versions and make sure they + # are in the same order - to ensure C struc member alignment + self._align_property_members() - def generate_header(self): + def generate_classes_header(self): + """Generate C header code and return it as string + + Declares: + <class_name>_Data - used as one of hypervObject->data members + <class_name>_TypeInfo - used as wsman XmlSerializerInfo + <class_name> - "inherits" hypervObject struct + """ + name_upper = self.name.upper() header = separator header += " * %s\n" % self.name header += " */\n" header += "\n" - header += "int hypervGet%sList(hypervPrivate *priv, virBufferPtr query, %s **list);\n" \ - % (self.name.replace("_", ""), self.name) - header += "\n" + header += "#define %s_CLASSNAME \\\n" % name_upper + header += " \"%s\"\n" % self.name header += "\n" + header += "#define %s_WQL_SELECT \\\n" % name_upper + header += " \"SELECT * FROM %s \"\n" % self.name header += "\n" + header += "extern hypervWmiClassInfoListPtr %s_WmiInfo;\n\n" % self.name + + header += self._declare_data_structs() + header += self._declare_hypervObject_struct() return header + def generate_classes_source(self): + """Returns a C code string defining wsman data structs + + Defines: + <class_name>_Data structs + <class_name>_WmiInfo - list holding metadata (e.g. request URIs) for + each known version of WMI class. + """ + + source = separator + source += " * %s\n" % self.name + source += " */\n" + + for cls in self.versions: + source += "SER_START_ITEMS(%s_Data)\n" % cls.name + + for property in cls.properties: + source += property.generate_classes_source(cls.name) + + source += "SER_END_ITEMS(%s_Data);\n\n" % cls.name + + + source += self._define_WmiInfo_struct() + source += "\n\n" + + return source + + def generate_classes_typedef(self): - typedef = "typedef struct _%s_Data %s_Data;\n" % (self.name, self.name) - typedef += "typedef struct _%s %s;\n" % (self.name, self.name) + """Returns C string for typdefs""" + + typedef = "typedef struct _%s %s;\n" % (self.name, self.name) + + if self.common is not None: + typedef += "typedef struct _%s_Data %s_Data;\n" % (self.name, self.name) + + for cls in self.versions: + typedef += "typedef struct _%s_Data %s_Data;\n" % (cls.name, cls.name) return typedef - def generate_classes_header(self): - name_upper = self.name.upper() - header = separator - header += " * %s\n" % self.name - header += " */\n" - header += "\n" - header += "#define %s_RESOURCE_URI \\\n" % name_upper + def _declare_data_structs(self): + """Returns string C code declaring data structs. - if self.name.startswith("Win32_") or self.name.startswith("CIM_"): - header += " \"http://schemas.microsoft.com/wbem/wsman/1/wmi/root/cimv2/%s\"\n" % self.name - else: - header += " \"http://schemas.microsoft.com/wbem/wsman/1/wmi/root/virtualization/%s\"\n" % self.name + The *_Data structs are members of hypervObject data union. Each one has + corresponding *_TypeInfo that is used for wsman unserialization of + response XML into the *_Data structs. If there's a "common" member, it + won't have corresponding *_TypeInfo becuase this is a special case only + used to provide a common "view" of v1, v2 etc members + """ - header += "\n" - header += "#define %s_CLASSNAME \\\n" % name_upper - header += " \"%s\"\n" % self.name - header += "\n" - header += "#define %s_WQL_SELECT \\\n" % name_upper - header += " \"select * from %s \"\n" % self.name - header += "\n" - header += "struct _%s_Data {\n" % self.name + header = "" + if self.common is not None: + header += "struct _%s_Data {\n" % self.name + for property in self.common: + header += property.generate_classes_header() + header += "};\n\n" - for property in self.properties: - header += property.generate_classes_header() + # Declare actual data struct for each versions + for cls in self.versions: + header += "#define %s_RESOURCE_URI \\\n" % cls.name.upper() + header += " \"%s\"\n" % cls.uri_info.resourceUri + header += "\n" + header += "struct _%s_Data {\n" % cls.name + for property in cls.properties: + header += property.generate_classes_header() + header += "};\n\n" + header += "SER_DECLARE_TYPE(%s_Data);\n" % cls.name - header += "};\n" - header += "\n" - header += "SER_DECLARE_TYPE(%s_Data);\n" % self.name - header += "\n" + return header + + + def _declare_hypervObject_struct(self): + """Return string for C code declaring hypervObject instance""" + + header = "\n/* must match hypervObject */\n" header += "struct _%s {\n" % self.name - header += " XmlSerializerInfo *serializerInfo;\n" - header += " %s_Data *data;\n" % self.name + header += " union {\n" + + # if there's common use it as "common" else first and only version is + # the "common" member + if self.common is not None: + header += " %s_Data *common;\n" % self.name + else: + header += " %s_Data *common;\n" % self.versions[0].name + + for cls in self.versions: + header += " %s_Data *%s;\n" % (cls.name, cls.version) + + header += " } data;\n" + header += " hypervWmiClassInfoPtr info;\n" header += " %s *next;\n" % self.name header += "};\n" - header += "\n" - header += "\n" - header += "\n" + + header += "\n\n\n" return header - def generate_source(self): - name_upper = self.name.upper() + def _define_WmiInfo_struct(self): + """Return string for C code defining *_WmiInfo struct - source = separator - source += " * %s\n" % self.name - source += " */\n" - source += "\n" - source += "int\n" - source += "hypervGet%sList(hypervPrivate *priv, virBufferPtr query, %s **list)\n" \ - % (self.name.replace("_", ""), self.name) - source += "{\n" - - if self.name.startswith("Win32_") or self.name.startswith("CIM_"): - source += " return hypervEnumAndPull(priv, query, ROOT_CIMV2,\n" - else: - source += " return hypervEnumAndPull(priv, query, ROOT_VIRTUALIZATION,\n" + Those structs hold info with meta-data needed to make wsman requests for + each version of WMI class + """ + + source = "hypervWmiClassInfoListPtr %s_WmiInfo = &(hypervWmiClassInfoList) {\n" % self.name + source += " .count = %d,\n" % len(self.versions) + source += " .objs = (hypervWmiClassInfoPtr []) {\n" + + for cls in self.versions: + source += " &(hypervWmiClassInfo) {\n" + source += " .name = %s_CLASSNAME,\n" % self.name.upper() + source += " .version = \"%s\",\n" % cls.version + source += " .rootUri = %s,\n" % cls.uri_info.rootUri + source += " .resourceUri = %s_RESOURCE_URI,\n" % cls.name.upper() + source += " .serializerInfo = %s_Data_TypeInfo\n" % cls.name + source += " },\n" - source += " %s_Data_TypeInfo,\n" % self.name - source += " %s_RESOURCE_URI,\n" % name_upper - source += " %s_CLASSNAME,\n" % name_upper - source += " (hypervObject **)list);\n" - source += "}\n" - source += "\n" - source += "\n" - source += "\n" + source += " }\n" + source += "};\n" return source - def generate_classes_source(self): - name_upper = self.name.upper() + def _align_property_members(self): + """Identifies common properties in all class versions. - source = separator - source += " * %s\n" % self.name - source += " */\n" - source += "\n" - source += "SER_START_ITEMS(%s_Data)\n" % self.name + Makes sure that properties in all versions are ordered with common + members first and that they are in the same order. This makes the + generated C structs memory aligned and safe to access via the "common" + struct that "shares" members with v1, v2 etc. + """ - for property in self.properties: - source += property.generate_classes_source(self.name) + num_classes = len(self.versions) + common = {} + property_info = {} - source += "SER_END_ITEMS(%s_Data);\n" % self.name - source += "\n" - source += "\n" - source += "\n" + if num_classes < 2: + return + + # count property occurences in all class versions + for cls in self.versions: + for prop in cls.properties: + # consdered same if matches by name AND type + key = "%s_%s" % (prop.name, prop.type) + + if key in property_info: + property_info[key][1] += 1 + else: + property_info[key] = [prop, 1] + + # isolate those that are common for all and keep track of their postions + pos = 0 + for key in property_info: + info = property_info[key] + # exists in all class versions + if info[1] == num_classes: + common[info[0].name] = [info[0], pos] + pos += 1 + + # alter each versions's property list so that common members are first + # and in the same order as in the common dictionary + total = len(common) + for cls in self.versions: + index = 0 + count = len(cls.properties) + + while index < count: + prop = cls.properties[index] + + # it's a "common" proptery + if prop.name in common: + pos = common[prop.name][1] + + # move to the same position as in "common" dictionary + if index != pos: + tmp = cls.properties[pos] + cls.properties[pos] = prop + cls.properties[index] = tmp + else: + index += 1 + else: + index += 1 + + # finally, get common properties as list sorted by position in dictionary + tmp = sorted(common.values(), key=lambda x: x[1]) + self.common = [] + for x in tmp: + self.common.append(x[0]) + + + +class ClassUriInfo: + """Prepares URI information needed for wsman requests.""" + + def __init__(self, wmi_name, version): + self.rootUri = "ROOT_CIMV2" + self.resourceUri = None + baseUri = "http://schemas.microsoft.com/wbem/wsman/1/wmi/root/cimv2" + + if wmi_name.startswith("Msvm_"): + baseUri = "http://schemas.microsoft.com/wbem/wsman/1/wmi/root/virtualization" + self.rootUri = "ROOT_VIRTUALIZATION" + + if version == "v2": + baseUri += "/v2" + self.rootUri = "ROOT_VIRTUALIZATION_V2" + + self.resourceUri = "%s/%s" % (baseUri, wmi_name) + + + +class WmiClassVersion: + """Represents specific verssion of WMI class.""" + + def __init__(self, name, version, properties, uri_info): + self.name = name + self.version = version + self.properties = properties + self.uri_info = uri_info - return source class Property: @@ -155,9 +330,13 @@ class Property: "string" : "STR", "datetime" : "STR", "int8" : "INT8", + "sint8" : "INT8", "int16" : "INT16", + "sint16" : "INT16", "int32" : "INT32", + "sint32" : "INT32", "int64" : "INT64", + "sint64" : "INT64", "uint8" : "UINT8", "uint16" : "UINT16", "uint32" : "UINT32", @@ -189,8 +368,6 @@ class Property: return " SER_NS_%s(%s_RESOURCE_URI, \"%s\", 1),\n" \ % (Property.typemap[self.type], class_name.upper(), self.name) - - def open_and_print(filename): if filename.startswith("./"): print " GEN " + filename[2:] @@ -217,8 +394,15 @@ def parse_class(block): assert header_items[0] == "class" name = header_items[1] - properties = [] + version = "v1" + wmi_name = name + ns_separator = name.find(wmi_version_separator) + + if ns_separator != -1: + version = name[:ns_separator] + wmi_name = name[ns_separator + 1:] + name = "%s_%s" % (wmi_name, version) for line in block[1:]: # expected format: <type> <name> @@ -236,7 +420,13 @@ def parse_class(block): properties.append(Property(type=items[0], name=items[1], is_array=is_array)) - return Class(name=name, properties=properties) + cls = WmiClassVersion(name=name, version=version, properties=properties, + uri_info=ClassUriInfo(wmi_name, version)) + + if wmi_name in wmi_classes_by_name: + wmi_classes_by_name[wmi_name].versions.append(cls) + else: + wmi_classes_by_name[wmi_name] = WmiClass(wmi_name, [cls]) @@ -248,15 +438,13 @@ def main(): input_filename = os.path.join(os.getcwd(), "hyperv_wmi_generator.input") output_dirname = os.getcwd() - header = open_and_print(os.path.join(output_dirname, "hyperv_wmi.generated.h")) - source = open_and_print(os.path.join(output_dirname, "hyperv_wmi.generated.c")) + classes_typedef = open_and_print(os.path.join(output_dirname, "hyperv_wmi_classes.generated.typedef")) classes_header = open_and_print(os.path.join(output_dirname, "hyperv_wmi_classes.generated.h")) classes_source = open_and_print(os.path.join(output_dirname, "hyperv_wmi_classes.generated.c")) - # parse input file + number = 0 - classes_by_name = {} block = None for line in file(input_filename, "rb").readlines(): @@ -268,7 +456,7 @@ def main(): line = line.lstrip().rstrip() if len(line) < 1: - continue + continue if line.startswith("class"): if block is not None: @@ -279,31 +467,22 @@ def main(): if block is not None: if line == "end": if block[0][1].startswith("class"): - cls = parse_class(block) - classes_by_name[cls.name] = cls + parse_class(block) block = None else: block.append((number, line)) - # write output files - notice = "/* Generated by hyperv_wmi_generator.py */\n\n\n\n" - - header.write(notice) - source.write(notice) - classes_typedef.write(notice) - classes_header.write(notice) - classes_source.write(notice) - - names = classes_by_name.keys() + names = wmi_classes_by_name.keys() names.sort() for name in names: - header.write(classes_by_name[name].generate_header()) - source.write(classes_by_name[name].generate_source()) - classes_typedef.write(classes_by_name[name].generate_classes_typedef()) - classes_header.write(classes_by_name[name].generate_classes_header()) - classes_source.write(classes_by_name[name].generate_classes_source()) + cls = wmi_classes_by_name[name] + cls.prepare() + + classes_typedef.write(cls.generate_classes_typedef()) + classes_header.write(cls.generate_classes_header()) + classes_source.write(cls.generate_classes_source()) -- 2.9.3

On Sat, 2017-04-01 at 14:22 +0200, Matthias Bolte wrote:

2017-03-30 22:41 GMT+02:00 Dawid Zamirski <dzamirski@datto.com>:

...

This patch updates the code generator that outputs C headers and code for WMI classes. It has been updated to handle multiple versions (or namespaces) of the same class which were introduced with Hyperv 2012+ ---

changes in v3:  * do not change file mode  * restore accidentally deleted copyright notice

 src/hyperv/hyperv_wmi_generator.py | 387 +++++++++++++++++++++++++++----------  1 file changed, 283 insertions(+), 104 deletions(-)

diff --git a/src/hyperv/hyperv_wmi_generator.py b/src/hyperv/hyperv_wmi_generator.py index 8c62882..ccad04d 100755 --- a/src/hyperv/hyperv_wmi_generator.py +++ b/src/hyperv/hyperv_wmi_generator.py @@ -279,31 +467,22 @@ def main():          if block is not None:              if line == "end":                  if block[0][1].startswith("class"): -                    cls = parse_class(block) -                    classes_by_name[cls.name] = cls +                    parse_class(block)

                 block = None              else:                  block.append((number, line))

-    # write output files -    notice = "/* Generated by hyperv_wmi_generator.py */\n\n\n\n" - -    header.write(notice) -    source.write(notice) -    classes_typedef.write(notice) -    classes_header.write(notice) -    classes_source.write(notice) - -    names = classes_by_name.keys() +    names = wmi_classes_by_name.keys()      names.sort()

     for name in names: -        header.write(classes_by_name[name].generate_header()) -        source.write(classes_by_name[name].generate_source()) -        classes_typedef.write(classes_by_name[name].generate_class es_typedef()) -        classes_header.write(classes_by_name[name].generate_classe s_header()) -        classes_source.write(classes_by_name[name].generate_classe s_source()) +        cls = wmi_classes_by_name[name] +        cls.prepare() + +        classes_typedef.write(cls.generate_classes_typedef()) +        classes_header.write(cls.generate_classes_header()) +        classes_source.write(cls.generate_classes_source())

You're dropping the notice comment from the generated files. Is this intentional?

Nope, not intentional - I'll fix this in v2 for the series. I'll wait till all the remaining patches get reviewed and then send v2 with all the issues addressed. Regards, Dawid