[python-libvirt] RFC:PEP 484 type annotation - how to continue
Hello, currently my Merge Quests <https://gitlab.com/libvirt/libvirt-python/-/merge_requests/9> is stuck because there are some open technical discussions: 1. The Python binding consists of two layers: - libvirtmod* is the low-level C-like API, which mostly is just a thin wrapper around the C-library. It is generated from the API.xml files - The high-level libvirt* modules, providing an object oriented API, which is also generated from the API.xml files. In both cases there are many exceptions defines via libvirt*-override-api.xml or coded in generator.py: - skip_impl: Generate Python code, but skip C implementation - custom_function: Generate neither C nor Python code - hand crafted code - skip_function: Generate neither C nor Python code - not used at all - function_skip_python_impl: Generate C code, but skip python impl Previously the functions listed in "skip_impl" were also duplicated in the "libvirt-*override-api.xml" files, which is somehow hard to maintain. With d19699e48ce9c17481f94b4dc0715e18e05d5adb I removed all those names from "skip_impl", which are also listed in the overrides and added code to add them back when "generator.py" is invoked. For this to work I had to change several "files" attributes to start with "python", which looks okay to me, but still would like to get verified. 2. The low-level library works mostly with bytes, but the high-level Python API uses nested Lists, Tuples and Dicts. To get most out of the Python static type annotations the input and output parameters should be modled in as much detail as possible. This requires to store those types somewhere. My current 46592e28124858ed863bceba65c3dafa2bbb02cd adds some often used types to generator.py and then references them from the overrides.xml:
+ 'PyListSecret': ('', 'List["virSecret"]', '', ''), + 'PyDictAny': ('', 'Dict[str, Any]', '', ''),
+ <return type='PyListSecret' info='the list of secrets or None in case of error'/>
This still is very painful as there are many "used-once" types, which still must be declared and used in two places. So far the "return" type was used for the "C" type and currently I mis-use it for referencing it for my Python types. This doesn't look right, so I would prefer adding a new attribute which can be used to override the Python type. This also would probably nicely fit with functions like "virDomainOpenConsole", when "dev_name" is optional and can be "None", which the current type hint requires a "string". Is extending the libvirt*-override-api.xml files okay or must this be coordinated with the other parts of the project? Thank you for your time Philipp Hahn -- Philipp Hahn Open Source Software Engineer Univention GmbH be open. Mary-Somerville-Str. 1 D-28359 Bremen 📞 +49-421-22232-57 🖶 +49-421-22232-99 ✉️ hahn@univention.de 🌐 https://www.univention.de/ Geschäftsführer: Peter H. Ganten HRB 20755 Amtsgericht Bremen Steuer-Nr.: 71-597-02876
On Mon, Oct 05, 2020 at 09:12:26AM +0200, Philipp Hahn wrote:
Hello,
currently my Merge Quests <https://gitlab.com/libvirt/libvirt-python/-/merge_requests/9> is stuck because there are some open technical discussions:
1. The Python binding consists of two layers: - libvirtmod* is the low-level C-like API, which mostly is just a thin wrapper around the C-library. It is generated from the API.xml files - The high-level libvirt* modules, providing an object oriented API, which is also generated from the API.xml files. In both cases there are many exceptions defines via libvirt*-override-api.xml or coded in generator.py: - skip_impl: Generate Python code, but skip C implementation - custom_function: Generate neither C nor Python code - hand crafted code - skip_function: Generate neither C nor Python code - not used at all - function_skip_python_impl: Generate C code, but skip python impl
Previously the functions listed in "skip_impl" were also duplicated in the "libvirt-*override-api.xml" files, which is somehow hard to maintain. With d19699e48ce9c17481f94b4dc0715e18e05d5adb I removed all those names from "skip_impl", which are also listed in the overrides and added code to add them back when "generator.py" is invoked. For this to work I had to change several "files" attributes to start with "python", which looks okay to me, but still would like to get verified.
The generator.py is such a big hack I don't think it matters too much. The real important thing is that the resulting code is sane.
2. The low-level library works mostly with bytes, but the high-level Python API uses nested Lists, Tuples and Dicts. To get most out of the Python static type annotations the input and output parameters should be modled in as much detail as possible. This requires to store those types somewhere. My current 46592e28124858ed863bceba65c3dafa2bbb02cd adds some often used types to generator.py and then references them from the overrides.xml:
+ 'PyListSecret': ('', 'List["virSecret"]', '', ''), + 'PyDictAny': ('', 'Dict[str, Any]', '', ''),
+ <return type='PyListSecret' info='the list of secrets or None in case of error'/>
This still is very painful as there are many "used-once" types, which still must be declared and used in two places. So far the "return" type was used for the "C" type and currently I mis-use it for referencing it for my Python types. This doesn't look right, so I would prefer adding a new attribute which can be used to override the Python type. This also would probably nicely fit with functions like "virDomainOpenConsole", when "dev_name" is optional and can be "None", which the current type hint requires a "string".
Is extending the libvirt*-override-api.xml files okay or must this be coordinated with the other parts of the project?
These files follow the same schema as the main api.xml files shipped by libvirt itself. That is just a matter of convenience though - the overrides are only used in the libvirt-python generator.py, so there is no reason why we can't record extra information in them. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
participants (2)
-
Daniel P. Berrangé -
Philipp Hahn