[PATCH v4 00/34] Configurable policy for handling deprecated interfaces
This series extends QMP introspection to cover deprecation. Additionally, new option -compat lets you configure what to do when deprecated interfaces get used. This is intended for testing users of the management interfaces. It is experimental. -compat deprecated-input=<in-policy> configures what to do when deprecated input is received. Available policies: * accept: Accept deprecated commands and arguments (default) * reject: Reject them * crash: Crash -compat deprecated-output=<out-policy> configures what to do when deprecated output is sent. Available output policies: * accept: Emit deprecated command results and events (default) * hide: Suppress them For now, -compat covers only deprecated syntactic aspects of QMP. We may want to extend it to cover semantic aspects, CLI, and experimental features. PATCH 01-04: Documentation fixes PATCH 05-10: Test improvements PATCH 11-24: Add feature flags to remaining user-defined types and to struct members PATCH 25-26: New special feature 'deprecated', visible in introspection PATCH 27-34: New -compat to set policy for handling stuff marked with feature 'deprecated' v4: PATCH 05+07: Temporary memory leak plugged [Marc-André] PATCH 23: Rewritten [Marc-André] PATCH 24: Comment typo [Marc-André] PATCH 30: Memory leaks plugged v3: * Rebased, non-trivial conflicts in PATCH 01+26+27+34 due to RST conversion and code motion * PATCH 28-29: Old PATCH 28 split up to ease review * PATCH 30-31: New * PATCH 32-33: Old PATCH 29 split up to ease review Comparison to RFC (24 Oct 2019): * Cover arguments and results in addition to commands and events * Half-baked "[RFC PATCH 18/19] qapi: Include a warning in the response to a deprecated command" dropped See also last item of Subject: Minutes of KVM Forum BoF on deprecating stuff Date: Fri, 26 Oct 2018 16:03:51 +0200 Message-ID: <87mur0ls8o.fsf@dusky.pond.sub.org> https://lists.nongnu.org/archive/html/qemu-devel/2018-10/msg05828.html Cc: Lukáš Doktor <ldoktor@redhat.com> Cc: libguestfs@redhat.com Cc: libvir-list@redhat.com Cc: Daniel P. Berrange <berrange@redhat.com> Cc: Peter Krempa <pkrempa@redhat.com> Cc: Kevin Wolf <kwolf@redhat.com> Markus Armbruster (34): qemu-doc: Belatedly document QMP command arg & result deprecation qapi: Belatedly update doc comment for @wait deprecation docs/devel/qapi-code-gen: Clarify allow-oob introspection docs/devel/qapi-code-gen: Document 'features' introspection tests/test-qmp-cmds: Factor out qmp_dispatch() test helpers tests/test-qmp-cmds: Check responses more thoroughly tests/test-qmp-cmds: Simplify test data setup tests/test-qmp-event: Simplify test data setup tests/test-qmp-event: Use qobject_is_equal() tests/test-qmp-event: Check event is actually emitted qapi/schema: Clean up around QAPISchemaEntity.connect_doc() qapi: Add feature flags to remaining definitions qapi: Consistently put @features parameter right after @ifcond qapi/introspect: Rename *qlit* to reduce confusion qapi/introspect: Factor out _make_tree() qapi/schema: Change _make_features() to a take feature list qapi/schema: Reorder classes so related ones are together qapi/schema: Rename QAPISchemaObjectType{Variant,Variants} qapi/schema: Call QAPIDoc.connect_member() in just one place qapi: Add feature flags to struct members qapi: Inline do_qmp_dispatch() into qmp_dispatch() qapi: Simplify how qmp_dispatch() deals with QCO_NO_SUCCESS_RESP qapi: Simplify how qmp_dispatch() gets the request ID qapi: Replace qmp_dispatch()'s TODO comment by an explanation qapi: New special feature flag "deprecated" qapi: Mark deprecated QMP parts with feature 'deprecated' qemu-options: New -compat to set policy for deprecated interfaces qapi: Implement deprecated-output=hide for QMP command results qapi: Implement deprecated-output=hide for QMP events qapi: Implement deprecated-output=hide for QMP event data qapi: Implement deprecated-output=hide for QMP introspection qapi: Implement deprecated-input=reject for QMP commands qapi: Implement deprecated-input=reject for QMP command arguments qapi: New -compat deprecated-input=crash docs/devel/qapi-code-gen.txt | 79 ++- docs/system/deprecated.rst | 48 +- tests/qapi-schema/doc-good.texi | 32 ++ qapi/block-core.json | 48 +- qapi/block.json | 30 +- qapi/char.json | 1 + qapi/compat.json | 52 ++ qapi/control.json | 11 +- qapi/introspect.json | 28 +- qapi/machine.json | 34 +- qapi/migration.json | 36 +- qapi/misc.json | 13 +- qapi/qapi-schema.json | 1 + include/qapi/compat-policy.h | 20 + include/qapi/qmp/dispatch.h | 1 + include/qapi/qobject-input-visitor.h | 9 + include/qapi/qobject-output-visitor.h | 9 + include/qapi/visitor-impl.h | 3 + include/qapi/visitor.h | 9 + monitor/monitor-internal.h | 3 - monitor/misc.c | 2 - monitor/qmp-cmds-control.c | 102 +++- qapi/qapi-visit-core.c | 9 + qapi/qmp-dispatch.c | 149 +++--- qapi/qobject-input-visitor.c | 29 ++ qapi/qobject-output-visitor.c | 20 + qemu-storage-daemon.c | 2 - softmmu/vl.c | 17 + tests/test-qmp-cmds.c | 249 +++++---- tests/test-qmp-event.c | 203 +++----- qapi/Makefile.objs | 8 +- qapi/trace-events | 1 + qemu-options.hx | 22 + scripts/qapi/commands.py | 20 +- scripts/qapi/doc.py | 16 +- scripts/qapi/events.py | 24 +- scripts/qapi/expr.py | 14 +- scripts/qapi/introspect.py | 104 ++-- scripts/qapi/schema.py | 488 ++++++++++-------- scripts/qapi/types.py | 8 +- scripts/qapi/visit.py | 28 +- tests/Makefile.include | 1 + tests/qapi-schema/alternate-base.err | 2 +- tests/qapi-schema/doc-good.json | 22 +- tests/qapi-schema/doc-good.out | 18 + .../qapi-schema/features-deprecated-type.err | 2 + .../qapi-schema/features-deprecated-type.json | 3 + .../qapi-schema/features-deprecated-type.out | 0 tests/qapi-schema/qapi-schema-test.json | 51 +- tests/qapi-schema/qapi-schema-test.out | 48 +- tests/qapi-schema/test-qapi.py | 26 +- 51 files changed, 1393 insertions(+), 762 deletions(-) create mode 100644 qapi/compat.json create mode 100644 include/qapi/compat-policy.h create mode 100644 tests/qapi-schema/features-deprecated-type.err create mode 100644 tests/qapi-schema/features-deprecated-type.json create mode 100644 tests/qapi-schema/features-deprecated-type.out -- 2.21.1
Hi On Tue, Mar 17, 2020 at 12:55 PM Markus Armbruster <armbru@redhat.com> wrote:
This series extends QMP introspection to cover deprecation. Additionally, new option -compat lets you configure what to do when deprecated interfaces get used. This is intended for testing users of the management interfaces. It is experimental.
-compat deprecated-input=<in-policy> configures what to do when deprecated input is received. Available policies:
* accept: Accept deprecated commands and arguments (default) * reject: Reject them * crash: Crash
-compat deprecated-output=<out-policy> configures what to do when deprecated output is sent. Available output policies:
* accept: Emit deprecated command results and events (default) * hide: Suppress them
For now, -compat covers only deprecated syntactic aspects of QMP. We may want to extend it to cover semantic aspects, CLI, and experimental features.
I suggest to use a qmp- prefix for qmp-related policies.
PATCH 01-04: Documentation fixes PATCH 05-10: Test improvements PATCH 11-24: Add feature flags to remaining user-defined types and to struct members PATCH 25-26: New special feature 'deprecated', visible in introspection PATCH 27-34: New -compat to set policy for handling stuff marked with feature 'deprecated'
v4: PATCH 05+07: Temporary memory leak plugged [Marc-André] PATCH 23: Rewritten [Marc-André] PATCH 24: Comment typo [Marc-André] PATCH 30: Memory leaks plugged
v3: * Rebased, non-trivial conflicts in PATCH 01+26+27+34 due to RST conversion and code motion * PATCH 28-29: Old PATCH 28 split up to ease review * PATCH 30-31: New * PATCH 32-33: Old PATCH 29 split up to ease review
Comparison to RFC (24 Oct 2019): * Cover arguments and results in addition to commands and events * Half-baked "[RFC PATCH 18/19] qapi: Include a warning in the response to a deprecated command" dropped
See also last item of Subject: Minutes of KVM Forum BoF on deprecating stuff Date: Fri, 26 Oct 2018 16:03:51 +0200 Message-ID: <87mur0ls8o.fsf@dusky.pond.sub.org> https://lists.nongnu.org/archive/html/qemu-devel/2018-10/msg05828.html
Cc: Lukáš Doktor <ldoktor@redhat.com> Cc: libguestfs@redhat.com Cc: libvir-list@redhat.com Cc: Daniel P. Berrange <berrange@redhat.com> Cc: Peter Krempa <pkrempa@redhat.com> Cc: Kevin Wolf <kwolf@redhat.com>
Markus Armbruster (34): qemu-doc: Belatedly document QMP command arg & result deprecation qapi: Belatedly update doc comment for @wait deprecation docs/devel/qapi-code-gen: Clarify allow-oob introspection docs/devel/qapi-code-gen: Document 'features' introspection tests/test-qmp-cmds: Factor out qmp_dispatch() test helpers tests/test-qmp-cmds: Check responses more thoroughly tests/test-qmp-cmds: Simplify test data setup tests/test-qmp-event: Simplify test data setup tests/test-qmp-event: Use qobject_is_equal() tests/test-qmp-event: Check event is actually emitted qapi/schema: Clean up around QAPISchemaEntity.connect_doc() qapi: Add feature flags to remaining definitions qapi: Consistently put @features parameter right after @ifcond qapi/introspect: Rename *qlit* to reduce confusion qapi/introspect: Factor out _make_tree() qapi/schema: Change _make_features() to a take feature list qapi/schema: Reorder classes so related ones are together qapi/schema: Rename QAPISchemaObjectType{Variant,Variants} qapi/schema: Call QAPIDoc.connect_member() in just one place qapi: Add feature flags to struct members qapi: Inline do_qmp_dispatch() into qmp_dispatch() qapi: Simplify how qmp_dispatch() deals with QCO_NO_SUCCESS_RESP qapi: Simplify how qmp_dispatch() gets the request ID qapi: Replace qmp_dispatch()'s TODO comment by an explanation qapi: New special feature flag "deprecated" qapi: Mark deprecated QMP parts with feature 'deprecated' qemu-options: New -compat to set policy for deprecated interfaces qapi: Implement deprecated-output=hide for QMP command results qapi: Implement deprecated-output=hide for QMP events qapi: Implement deprecated-output=hide for QMP event data qapi: Implement deprecated-output=hide for QMP introspection qapi: Implement deprecated-input=reject for QMP commands qapi: Implement deprecated-input=reject for QMP command arguments qapi: New -compat deprecated-input=crash
docs/devel/qapi-code-gen.txt | 79 ++- docs/system/deprecated.rst | 48 +- tests/qapi-schema/doc-good.texi | 32 ++ qapi/block-core.json | 48 +- qapi/block.json | 30 +- qapi/char.json | 1 + qapi/compat.json | 52 ++ qapi/control.json | 11 +- qapi/introspect.json | 28 +- qapi/machine.json | 34 +- qapi/migration.json | 36 +- qapi/misc.json | 13 +- qapi/qapi-schema.json | 1 + include/qapi/compat-policy.h | 20 + include/qapi/qmp/dispatch.h | 1 + include/qapi/qobject-input-visitor.h | 9 + include/qapi/qobject-output-visitor.h | 9 + include/qapi/visitor-impl.h | 3 + include/qapi/visitor.h | 9 + monitor/monitor-internal.h | 3 - monitor/misc.c | 2 - monitor/qmp-cmds-control.c | 102 +++- qapi/qapi-visit-core.c | 9 + qapi/qmp-dispatch.c | 149 +++--- qapi/qobject-input-visitor.c | 29 ++ qapi/qobject-output-visitor.c | 20 + qemu-storage-daemon.c | 2 - softmmu/vl.c | 17 + tests/test-qmp-cmds.c | 249 +++++---- tests/test-qmp-event.c | 203 +++----- qapi/Makefile.objs | 8 +- qapi/trace-events | 1 + qemu-options.hx | 22 + scripts/qapi/commands.py | 20 +- scripts/qapi/doc.py | 16 +- scripts/qapi/events.py | 24 +- scripts/qapi/expr.py | 14 +- scripts/qapi/introspect.py | 104 ++-- scripts/qapi/schema.py | 488 ++++++++++-------- scripts/qapi/types.py | 8 +- scripts/qapi/visit.py | 28 +- tests/Makefile.include | 1 + tests/qapi-schema/alternate-base.err | 2 +- tests/qapi-schema/doc-good.json | 22 +- tests/qapi-schema/doc-good.out | 18 + .../qapi-schema/features-deprecated-type.err | 2 + .../qapi-schema/features-deprecated-type.json | 3 + .../qapi-schema/features-deprecated-type.out | 0 tests/qapi-schema/qapi-schema-test.json | 51 +- tests/qapi-schema/qapi-schema-test.out | 48 +- tests/qapi-schema/test-qapi.py | 26 +- 51 files changed, 1393 insertions(+), 762 deletions(-) create mode 100644 qapi/compat.json create mode 100644 include/qapi/compat-policy.h create mode 100644 tests/qapi-schema/features-deprecated-type.err create mode 100644 tests/qapi-schema/features-deprecated-type.json create mode 100644 tests/qapi-schema/features-deprecated-type.out
-- 2.21.1
-- Marc-André Lureau
Marc-André Lureau <marcandre.lureau@gmail.com> writes:
Hi
On Tue, Mar 17, 2020 at 12:55 PM Markus Armbruster <armbru@redhat.com> wrote:
This series extends QMP introspection to cover deprecation. Additionally, new option -compat lets you configure what to do when deprecated interfaces get used. This is intended for testing users of the management interfaces. It is experimental.
-compat deprecated-input=<in-policy> configures what to do when deprecated input is received. Available policies:
* accept: Accept deprecated commands and arguments (default) * reject: Reject them * crash: Crash
-compat deprecated-output=<out-policy> configures what to do when deprecated output is sent. Available output policies:
* accept: Emit deprecated command results and events (default) * hide: Suppress them
For now, -compat covers only deprecated syntactic aspects of QMP. We may want to extend it to cover semantic aspects, CLI, and experimental features.
I suggest to use a qmp- prefix for qmp-related policies.
The interface is general. The implemented infrastructure is QAPI-only. Its application is QMP-only. If our CLI was QAPIfied, I'd certainly apply it there, too. I intend to resume exploring CLI QAPIfication "real soon now". Not covering CLI now is a bit like not covering semantic aspects of QMP. Imagine the thing covered CLI. Would we want to have separate -compat deprecated-qmp-input, deprecated-cli-input? I doubt it. I think we want a single policy for all host interfaces. Imagine it covered deprecated semantic aspects of QMP. Would we want to have a separate flag for that? Again, I doubt it. For what it's worth, the interface is documented as experimental.
Markus Armbruster <armbru@redhat.com> writes:
This series extends QMP introspection to cover deprecation. Additionally, new option -compat lets you configure what to do when deprecated interfaces get used. This is intended for testing users of the management interfaces. It is experimental.
-compat deprecated-input=<in-policy> configures what to do when deprecated input is received. Available policies:
* accept: Accept deprecated commands and arguments (default) * reject: Reject them * crash: Crash
-compat deprecated-output=<out-policy> configures what to do when deprecated output is sent. Available output policies:
* accept: Emit deprecated command results and events (default) * hide: Suppress them
For now, -compat covers only deprecated syntactic aspects of QMP. We may want to extend it to cover semantic aspects, CLI, and experimental features.
PATCH 01-04: Documentation fixes PATCH 05-10: Test improvements PATCH 11-24: Add feature flags to remaining user-defined types and to struct members PATCH 25-26: New special feature 'deprecated', visible in introspection PATCH 27-34: New -compat to set policy for handling stuff marked with feature 'deprecated'
Queued PATCH 01-26. Thanks for the reviews that made this possible! PATCH 27-34 will have to wait for 5.1.
On Tue, Mar 17, 2020 at 12:54:25 +0100, Markus Armbruster wrote:
This series extends QMP introspection to cover deprecation. Additionally, new option -compat lets you configure what to do when deprecated interfaces get used. This is intended for testing users of the management interfaces. It is experimental.
-compat deprecated-input=<in-policy> configures what to do when deprecated input is received. Available policies:
* accept: Accept deprecated commands and arguments (default) * reject: Reject them * crash: Crash
I've noticed that the 'crash' option doesn't manage to log the reason to stderr. Relevant section of libvirt's log file which agregates the stderr/out 2020-04-30 13:24:22.006+0000: 2072883: debug : virCommandHandshakeChild:418 : Handshake with parent is done char device redirected to /dev/pts/0 (label charserial0) 2020-04-30 13:24:31.879+0000: Domain id=4 is tainted: custom-monitor 2020-04-30 13:24:32.330+0000: shutting down, reason=crashed 'handshake' line is last of libvirt's messages pre-start of the qemu process. 'char device redirected' is reported by qemu. 'domain is tainted' is added by libvirt when I've issued the deprecated API via virsh qemu-monitor-command. 'reason=crashed' is added by libvirts VM shutdown hanlder.
-compat deprecated-output=<out-policy> configures what to do when deprecated output is sent. Available output policies:
* accept: Emit deprecated command results and events (default) * hide: Suppress them
For now, -compat covers only deprecated syntactic aspects of QMP. We may want to extend it to cover semantic aspects, CLI, and experimental features.
PATCH 01-04: Documentation fixes PATCH 05-10: Test improvements PATCH 11-24: Add feature flags to remaining user-defined types and to struct members PATCH 25-26: New special feature 'deprecated', visible in introspection
These are cool. I've added support for verifying that any command excercised by the libvirt unit test suite is not deprecated, or we at least know that it is and have a replacement. https://www.redhat.com/archives/libvir-list/2020-April/msg01444.html
PATCH 27-34: New -compat to set policy for handling stuff marked with feature 'deprecated'
While implementing support for this feature I noticed that it's impossible for libvirt to detect that it's available. The idea is to make a developer-centred setting in our config which will enable the compat setting if available and ignore it if not available to prevent us having to fiddle with the settings when testing various qemu versions. Thanks! Peter
Peter Krempa <pkrempa@redhat.com> writes:
On Tue, Mar 17, 2020 at 12:54:25 +0100, Markus Armbruster wrote:
This series extends QMP introspection to cover deprecation. Additionally, new option -compat lets you configure what to do when deprecated interfaces get used. This is intended for testing users of the management interfaces. It is experimental.
-compat deprecated-input=<in-policy> configures what to do when deprecated input is received. Available policies:
* accept: Accept deprecated commands and arguments (default) * reject: Reject them * crash: Crash
I've noticed that the 'crash' option doesn't manage to log the reason to stderr. Relevant section of libvirt's log file which agregates the stderr/out
2020-04-30 13:24:22.006+0000: 2072883: debug : virCommandHandshakeChild:418 : Handshake with parent is done char device redirected to /dev/pts/0 (label charserial0) 2020-04-30 13:24:31.879+0000: Domain id=4 is tainted: custom-monitor 2020-04-30 13:24:32.330+0000: shutting down, reason=crashed
'handshake' line is last of libvirt's messages pre-start of the qemu process. 'char device redirected' is reported by qemu. 'domain is tainted' is added by libvirt when I've issued the deprecated API via virsh qemu-monitor-command. 'reason=crashed' is added by libvirts VM shutdown hanlder.
I'm up to my ears in QOM right now. I'd like to follow up when I pivot back to QAPI/QMP. Please pester me if that takes too long.
-compat deprecated-output=<out-policy> configures what to do when deprecated output is sent. Available output policies:
* accept: Emit deprecated command results and events (default) * hide: Suppress them
For now, -compat covers only deprecated syntactic aspects of QMP. We may want to extend it to cover semantic aspects, CLI, and experimental features.
PATCH 01-04: Documentation fixes PATCH 05-10: Test improvements PATCH 11-24: Add feature flags to remaining user-defined types and to struct members PATCH 25-26: New special feature 'deprecated', visible in introspection
These are cool. I've added support for verifying that any command excercised by the libvirt unit test suite is not deprecated, or we at least know that it is and have a replacement.
https://www.redhat.com/archives/libvir-list/2020-April/msg01444.html
Awesome!
PATCH 27-34: New -compat to set policy for handling stuff marked with feature 'deprecated'
While implementing support for this feature I noticed that it's impossible for libvirt to detect that it's available. The idea is to make a developer-centred setting in our config which will enable the compat setting if available and ignore it if not available to prevent us having to fiddle with the settings when testing various qemu versions.
Again, I'd like to follow up when I pivot back to QAPI/QMP.
participants (3)
-
Marc-André Lureau -
Markus Armbruster -
Peter Krempa