On Wed, Apr 09, 2025 at 09:39:02 +0200, Markus Armbruster via Devel wrote: > Hi Steve, I apologize for the slow response. > > Steve Sistare <steven.sistare(a)oracle.com&gt; writes: > > > Using qom-list and qom-get to get all the nodes and property values in a > > QOM tree can take multiple seconds because it requires 1000's of individual > > QOM requests. Some managers fetch the entire tree or a large subset > > of it when starting a new VM, and this cost is a substantial fraction of > > start up time. > > "Some managers"... could you name one? libvirt is at ~500 qom-get calls during an average startup ... > > To reduce this cost, consider QAPI calls that fetch more information in > > each call: > > * qom-list-get: given a path, return a list of properties and values. > > * qom-list-getv: given a list of paths, return a list of properties and > > values for each path. > > * qom-tree-get: given a path, return all descendant nodes rooted at that > > path, with properties and values for each. > > Libvirt developers, would you be interested in any of these? YES!!!

The getter with value could SO MUCH optimize the startup sequence of a VM where libvirt needs to probe CPU flags: (note the 'id' field in libvirt's monitor is sequential) buf={"execute":"qom-get","arguments":{"path":"/machine/unattached/device[0]","property":"realized"},"id":"libvirt-8"} buf={"execute":"qom-get","arguments":{"path":"/machine/unattached/device[0]","property":"hotplugged"},"id":"libvirt-9"} buf={"execute":"qom-get","arguments":{"path":"/machine/unattached/device[0]","property":"hotpluggable"},"id":"libvirt-10"} [...] buf={"execute":"qom-get","arguments":{"path":"/machine/unattached/device[0]","property":"hv-apicv"},"id":"libvirt-470"} buf={"execute":"qom-get","arguments":{"path":"/machine/unattached/device[0]","property":"xd"},"id":"libvirt-471"} buf={"execute":"qom-get","arguments":{"path":"/machine/unattached/device[0]","property":"sse4_1"},"id":"libvirt-472"} buf={"execute":"qom-get","arguments":{"path":"/machine/unattached/device[0]","property":"unavailable-features"},"id":"libvirt-473"} First and last line's timestamps: 2025-04-08 14:44:28.882+0000: 1481190: info : qemuMonitorIOWrite:340 : QEMU_MONITOR_IO_WRITE: mon=0x7f4678048360 buf={"execute":"qom-get","arguments":{"path":"/machine/unattached/device[0]","property":"realized"},"id":"libvirt-8"} 2025-04-08 14:44:29.149+0000: 1481190: info : qemuMonitorIOWrite:340 : QEMU_MONITOR_IO_WRITE: mon=0x7f4678048360 buf={"execute":"qom-get","arguments":{"path":"/machine/unattached/device[0]","property":"unavailable-features"},"id":"libvirt-473"} Libvirt spent ~170 ms probing cpu flags.

Daniel P. Berrangé <berrange(a)redhat.com&gt; writes: > On Wed, Apr 09, 2025 at 09:58:13AM +0200, Peter Krempa via Devel wrote: >> On Wed, Apr 09, 2025 at 09:39:02 +0200, Markus Armbruster via Devel wrote: >> > Hi Steve, I apologize for the slow response. >> > >> > Steve Sistare <steven.sistare(a)oracle.com&gt; writes: >> > >> > > Using qom-list and qom-get to get all the nodes and property values in a >> > > QOM tree can take multiple seconds because it requires 1000's of individual >> > > QOM requests. Some managers fetch the entire tree or a large subset >> > > of it when starting a new VM, and this cost is a substantial fraction of >> > > start up time. >> > >> > "Some managers"... could you name one? >> >> libvirt is at ~500 qom-get calls during an average startup ... >> >> > > To reduce this cost, consider QAPI calls that fetch more information in >> > > each call: >> > > * qom-list-get: given a path, return a list of properties and values. >> > > * qom-list-getv: given a list of paths, return a list of properties and >> > > values for each path. >> > > * qom-tree-get: given a path, return all descendant nodes rooted at that >> > > path, with properties and values for each. >> > >> > Libvirt developers, would you be interested in any of these? >> >> YES!!! > > Not neccessarily, see below... !!!! > >> >> The getter with value could SO MUCH optimize the startup sequence of a >> VM where libvirt needs to probe CPU flags: >> >> (note the 'id' field in libvirt's monitor is sequential) >> >> buf={"execute":"qom-get","arguments":{"path":"/machine/unattached/device[0]","property":"realized"},"id":"libvirt-8"} >> buf={"execute":"qom-get","arguments":{"path":"/machine/unattached/device[0]","property":"hotplugged"},"id":"libvirt-9"} >> buf={"execute":"qom-get","arguments":{"path":"/machine/unattached/device[0]","property":"hotpluggable"},"id":"libvirt-10"} >> >> [...] >> >> buf={"execute":"qom-get","arguments":{"path":"/machine/unattached/device[0]","property":"hv-apicv"},"id":"libvirt-470"} >> buf={"execute":"qom-get","arguments":{"path":"/machine/unattached/device[0]","property":"xd"},"id":"libvirt-471"} >> buf={"execute":"qom-get","arguments":{"path":"/machine/unattached/device[0]","property":"sse4_1"},"id":"libvirt-472"} >> buf={"execute":"qom-get","arguments":{"path":"/machine/unattached/device[0]","property":"unavailable-features"},"id":"libvirt-473"} >> >> First and last line's timestamps: >> >> 2025-04-08 14:44:28.882+0000: 1481190: info : qemuMonitorIOWrite:340 : QEMU_MONITOR_IO_WRITE: mon=0x7f4678048360 buf={"execute":"qom-get","arguments":{"path":"/machine/unattached/device[0]","property":"realized"},"id":"libvirt-8"} >> >> 2025-04-08 14:44:29.149+0000: 1481190: info : qemuMonitorIOWrite:340 : QEMU_MONITOR_IO_WRITE: mon=0x7f4678048360 buf={"execute":"qom-get","arguments":{"path":"/machine/unattached/device[0]","property":"unavailable-features"},"id":"libvirt-473"} >> >> Libvirt spent ~170 ms probing cpu flags. > > One thing I would point out is that qom-get can be considered an > "escape hatch" to get information when no better QMP command exists. > In this case, libvirt has made the assumption that every CPU feature > is a QOM property. > > Adding qom-list-get doesn't appreciably change that, just makes the > usage more efficient. > > Considering the bigger picture QMP design, when libvirt is trying to > understand QEMU's CPU feature flag expansion, I would ask why we don't > have something like a "query-cpu" command to tell us the current CPU > expansion, avoiding the need for poking at QOM properties directly. How do the existing query-cpu-FOO fall short of what management applications such as libvirt needs?

Daniel P. Berrangé <berrange(a)redhat.com&gt; writes: > On Fri, Apr 11, 2025 at 12:40:46PM +0200, Markus Armbruster wrote: >> Daniel P. Berrangé <berrange(a)redhat.com&gt; writes: >> >>> On Wed, Apr 09, 2025 at 09:58:13AM +0200, Peter Krempa via Devel wrote: >>>> On Wed, Apr 09, 2025 at 09:39:02 +0200, Markus Armbruster via Devel wrote: >>>>> Hi Steve, I apologize for the slow response. >>>>> >>>>> Steve Sistare <steven.sistare(a)oracle.com&gt; writes: >>>>> >>>>>> Using qom-list and qom-get to get all the nodes and property values in a >>>>>> QOM tree can take multiple seconds because it requires 1000's of individual >>>>>> QOM requests. Some managers fetch the entire tree or a large subset >>>>>> of it when starting a new VM, and this cost is a substantial fraction of >>>>>> start up time. >>>>> >>>>> "Some managers"... could you name one? >>>> >>>> libvirt is at ~500 qom-get calls during an average startup ... >>>> >>>>>> To reduce this cost, consider QAPI calls that fetch more information in >>>>>> each call: >>>>>> * qom-list-get: given a path, return a list of properties and values. >>>>>> * qom-list-getv: given a list of paths, return a list of properties and >>>>>> values for each path. >>>>>> * qom-tree-get: given a path, return all descendant nodes rooted at that >>>>>> path, with properties and values for each. >>>>> >>>>> Libvirt developers, would you be interested in any of these? >>>> >>>> YES!!! >>> >>> Not neccessarily, see below... !!!! >>> >>>> >>>> The getter with value could SO MUCH optimize the startup sequence of a >>>> VM where libvirt needs to probe CPU flags: >>>> >>>> (note the 'id' field in libvirt's monitor is sequential) >>>> >>>> buf={"execute":"qom-get","arguments":{"path":"/machine/unattached/device[0]","property":"realized"},"id":"libvirt-8"} >>>> buf={"execute":"qom-get","arguments":{"path":"/machine/unattached/device[0]","property":"hotplugged"},"id":"libvirt-9"} >>>> buf={"execute":"qom-get","arguments":{"path":"/machine/unattached/device[0]","property":"hotpluggable"},"id":"libvirt-10"} >>>> >>>> [...] >>>> >>>> buf={"execute":"qom-get","arguments":{"path":"/machine/unattached/device[0]","property":"hv-apicv"},"id":"libvirt-470"} >>>> buf={"execute":"qom-get","arguments":{"path":"/machine/unattached/device[0]","property":"xd"},"id":"libvirt-471"} >>>> buf={"execute":"qom-get","arguments":{"path":"/machine/unattached/device[0]","property":"sse4_1"},"id":"libvirt-472"} >>>> buf={"execute":"qom-get","arguments":{"path":"/machine/unattached/device[0]","property":"unavailable-features"},"id":"libvirt-473"} >>>> >>>> First and last line's timestamps: >>>> >>>> 2025-04-08 14:44:28.882+0000: 1481190: info : qemuMonitorIOWrite:340 : QEMU_MONITOR_IO_WRITE: mon=0x7f4678048360 buf={"execute":"qom-get","arguments":{"path":"/machine/unattached/device[0]","property":"realized"},"id":"libvirt-8"} >>>> >>>> 2025-04-08 14:44:29.149+0000: 1481190: info : qemuMonitorIOWrite:340 : QEMU_MONITOR_IO_WRITE: mon=0x7f4678048360 buf={"execute":"qom-get","arguments":{"path":"/machine/unattached/device[0]","property":"unavailable-features"},"id":"libvirt-473"} >>>> >>>> Libvirt spent ~170 ms probing cpu flags. >>> >>> One thing I would point out is that qom-get can be considered an >>> "escape hatch" to get information when no better QMP command exists. >>> In this case, libvirt has made the assumption that every CPU feature >>> is a QOM property. >>> >>> Adding qom-list-get doesn't appreciably change that, just makes the >>> usage more efficient. >>> >>> Considering the bigger picture QMP design, when libvirt is trying to >>> understand QEMU's CPU feature flag expansion, I would ask why we don't >>> have something like a "query-cpu" command to tell us the current CPU >>> expansion, avoiding the need for poking at QOM properties directly. >> >> How do the existing query-cpu-FOO fall short of what management >> applications such as libvirt needs? > > It has been along while since I looked at them, but IIRC they were > returning static info about CPU models, whereas libvirt wanted info > on the currently requested '-cpu ARGS'

Libvirt developers, please work with us on design of new commands or improvements to existing ones to better meet libvirt's needs in this area.

On Fri, Apr 11, 2025 at 13:43:39 +0200, Markus Armbruster wrote: > Daniel P. Berrangé <berrange(a)redhat.com&gt; writes: > > On Fri, Apr 11, 2025 at 12:40:46PM +0200, Markus Armbruster wrote: > >> Daniel P. Berrangé <berrange(a)redhat.com&gt; writes: > >> > Considering the bigger picture QMP design, when libvirt is trying to > >> > understand QEMU's CPU feature flag expansion, I would ask why we don't > >> > have something like a "query-cpu" command to tell us the current CPU > >> > expansion, avoiding the need for poking at QOM properties directly. > >> > >> How do the existing query-cpu-FOO fall short of what management > >> applications such as libvirt needs? > > > > It has been along while since I looked at them, but IIRC they were > > returning static info about CPU models, whereas libvirt wanted info > > on the currently requested '-cpu ARGS' > > Libvirt developers, please work with us on design of new commands or > improvements to existing ones to better meet libvirt's needs in this > area. The existing commands (query-cpu-definitions, query-cpu-model-expansion) are useful for probing before starting a domain. But what we use qom-get for is to get a view of the currently instantiated virtual CPU created by QEMU according to -cpu when we're starting a domain. In other words, we start QEMU with -S and before starting vCPUs we need to know exactly what features were enabled and if any feature we requested was disabled by QEMU. Currently we query QOM for CPU properties as that's what we were advised to use ages ago. The reason behind querying such info is ensuring stable guest ABI during migration. Asking QEMU for a specific CPU model and features does not mean we'll get exactly what we asked for (this is not a bug) so we need to record the differences so that we can start QEMU for incoming migration with a CPU matching exactly the one provided on the source. As Peter said, the current way is terribly inefficient as it requires several hundreds of QMP commands so the goal is to have a single QMP command that would tell us all we need to know about the virtual CPU. That is all enabled features and all features that could not be enabled even though we asked for them.

Hi Steve, I apologize for the slow response. Steve Sistare <steven.sistare(a)oracle.com&gt; writes: > Using qom-list and qom-get to get all the nodes and property values in a > QOM tree can take multiple seconds because it requires 1000's of individual > QOM requests. Some managers fetch the entire tree or a large subset > of it when starting a new VM, and this cost is a substantial fraction of > start up time. "Some managers"... could you name one?

> To reduce this cost, consider QAPI calls that fetch more information in > each call: > * qom-list-get: given a path, return a list of properties and values. > * qom-list-getv: given a list of paths, return a list of properties and > values for each path. > * qom-tree-get: given a path, return all descendant nodes rooted at that > path, with properties and values for each. Libvirt developers, would you be interested in any of these? > In all cases, a returned property is represented by ObjectPropertyValue, > with fields name, type, value, and error. If an error occurs when reading > a value, the value field is omitted, and the error message is returned in the > the error field. Thus an error for one property will not cause a bulk fetch > operation to fail. Returning errors this way is highly unusual. Observation; I'm not rejecting this out of hand. Can you elaborate a bit on why it's useful?

> To evaluate each method, I modified scripts/qmp/qom-tree to use the method, > verified all methods produce the same output, and timed each using: > > qemu-system-x86_64 -display none \ > -chardev socket,id=monitor0,path=/tmp/vm1.sock,server=on,wait=off \ > -mon monitor0,mode=control & > > time qom-tree -s /tmp/vm1.sock > /dev/null Cool! > I only measured once per method, but the variation is low after a warm up run. > The 'real - user - sys' column is a proxy for QEMU CPU time. > > method real(s) user(s) sys(s) (real - user - sys)(s) > qom-list / qom-get 2.048 0.932 0.057 1.059 > qom-list-get 0.402 0.230 0.029 0.143 > qom-list-getv 0.200 0.132 0.015 0.053 > qom-tree-get 0.143 0.123 0.012 0.008 > > qom-tree-get is the clear winner, reducing elapsed time by a factor of 14X, > and reducing QEMU CPU time by 132X. > > qom-list-getv is slower when fetching the entire tree, but can beat > qom-tree-get when only a subset of the tree needs to be fetched (not shown). > > qom-list-get is shown for comparison only, and is not included in this series. If we have qom-list-getv, then qom-list-get is not worth having.

Steven Sistare <steven.sistare(a)oracle.com&gt; writes: > On 4/9/2025 3:39 AM, Markus Armbruster wrote: >> Hi Steve, I apologize for the slow response. >> >> Steve Sistare <steven.sistare(a)oracle.com&gt; writes: >> >>> Using qom-list and qom-get to get all the nodes and property values in a >>> QOM tree can take multiple seconds because it requires 1000's of individual >>> QOM requests. Some managers fetch the entire tree or a large subset >>> of it when starting a new VM, and this cost is a substantial fraction of >>> start up time. >> >> "Some managers"... could you name one? > > My personal experience is with Oracle's OCI, but likely others could benefit. Peter Krempa tells us libvirt would benefit. >>> To reduce this cost, consider QAPI calls that fetch more information in >>> each call: >>> * qom-list-get: given a path, return a list of properties and values. >>> * qom-list-getv: given a list of paths, return a list of properties and >>> values for each path. >>> * qom-tree-get: given a path, return all descendant nodes rooted at that >>> path, with properties and values for each. >> >> Libvirt developers, would you be interested in any of these? >> >>> In all cases, a returned property is represented by ObjectPropertyValue, >>> with fields name, type, value, and error. If an error occurs when reading >>> a value, the value field is omitted, and the error message is returned in the >>> the error field. Thus an error for one property will not cause a bulk fetch >>> operation to fail. >> >> Returning errors this way is highly unusual. Observation; I'm not >> rejecting this out of hand. Can you elaborate a bit on why it's useful? > > It is considered an error to read some properties if they are not valid for > the configuration. And some properties are write-only and return an error > if they are read. Examples: > > legacy-i8042: <EXCEPTION: Property 'vmmouse.legacy-i8042' is not readable> (str) > legacy-memory: <EXCEPTION: Property 'qemu64-x86_64-cpu.legacy-memory' is not readable> (str) > crash-information: <EXCEPTION: No crash occurred> (GuestPanicInformation) > > With conventional error handling, if any of these poison pills falls in the > scope of a bulk get operation, the entire operation fails. I suspect many of these poison pills are design mistakes. If a property is not valid for the configuration, why does it exist? QOM is by design dynamic. I wish it wasn't, but as long as it is dynamic, I can't see why we should create properties we know to be unusable. Why is reading crash-information an error when no crash occured? This is the *normal* case. Errors are for the abnormal. Anyway, asking you to fix design mistakes all over the place wouldn't be fair. So I'm asking you something else instead: do you actually need the error information?

Steven Sistare <steven.sistare(a)oracle.com&gt; writes: > On 4/9/2025 9:34 AM, Markus Armbruster wrote: >> Steven Sistare <steven.sistare(a)oracle.com&gt; writes: >>> On 4/9/2025 3:39 AM, Markus Armbruster wrote: >>>> Hi Steve, I apologize for the slow response. >>>> >>>> Steve Sistare <steven.sistare(a)oracle.com&gt; writes: >>>> >>>>> Using qom-list and qom-get to get all the nodes and property values in a >>>>> QOM tree can take multiple seconds because it requires 1000's of individual >>>>> QOM requests. Some managers fetch the entire tree or a large subset >>>>> of it when starting a new VM, and this cost is a substantial fraction of >>>>> start up time. >>>> >>>> "Some managers"... could you name one? >>> >>> My personal experience is with Oracle's OCI, but likely others could benefit. >> >> Peter Krempa tells us libvirt would benefit. >> >>>>> To reduce this cost, consider QAPI calls that fetch more information in >>>>> each call: >>>>> * qom-list-get: given a path, return a list of properties and values. >>>>> * qom-list-getv: given a list of paths, return a list of properties and >>>>> values for each path. >>>>> * qom-tree-get: given a path, return all descendant nodes rooted at that >>>>> path, with properties and values for each. >>>> >>>> Libvirt developers, would you be interested in any of these? >>>> >>>>> In all cases, a returned property is represented by ObjectPropertyValue, >>>>> with fields name, type, value, and error. If an error occurs when reading >>>>> a value, the value field is omitted, and the error message is returned in the >>>>> the error field. Thus an error for one property will not cause a bulk fetch >>>>> operation to fail. >>>> >>>> Returning errors this way is highly unusual. Observation; I'm not >>>> rejecting this out of hand. Can you elaborate a bit on why it's useful? >>> >>> It is considered an error to read some properties if they are not valid for >>> the configuration. And some properties are write-only and return an error >>> if they are read. Examples: >>> >>> legacy-i8042: <EXCEPTION: Property 'vmmouse.legacy-i8042' is not readable> (str) >>> legacy-memory: <EXCEPTION: Property 'qemu64-x86_64-cpu.legacy-memory' is not readable> (str) >>> crash-information: <EXCEPTION: No crash occurred> (GuestPanicInformation) >>> >>> With conventional error handling, if any of these poison pills falls in the >>> scope of a bulk get operation, the entire operation fails. >> >> I suspect many of these poison pills are design mistakes. >> >> If a property is not valid for the configuration, why does it exist? >> QOM is by design dynamic. I wish it wasn't, but as long as it is >> dynamic, I can't see why we should create properties we know to be >> unusable. >> >> Why is reading crash-information an error when no crash occured? This >> is the *normal* case. Errors are for the abnormal. >> >> Anyway, asking you to fix design mistakes all over the place wouldn't be >> fair. So I'm asking you something else instead: do you actually need >> the error information? > > I don't need the specific error message. > > I could return a boolean meaning "property not available" instead of returning > the exact error message, as long as folks are OK with the output of the qom-tree > script changing for these properties. Let's put aside the qom-tree script for a moment. In your patches, the queries return an object's properties as a list of ObjectPropertyValue, defined as { 'struct': 'ObjectPropertyValue', 'data': { 'name': 'str', 'type': 'str', '*value': 'any', '*error': 'str' } } As far as I understand, exactly one of @value and @error are present. The list has no duplicates, i.e. no two elements have the same value of "name". Say we're interested in property "foo". Three cases: * The list has an element with "name": "foo", and the element has member "value": the property exists and "value" has its value. * The list has an element with "name": "foo", and the element does not have member "value": the property exists, but its value cannot be gotten; member "error" has the error message. * The list has no element with "name": "foo": the property does not exist. If we simply drop ObjectPropertyValue member @error, we lose 'member "error" has the error message'. That's all. If a need for more error information should arise later, we could add member @error. Or something else entirely. Or tell people to qom-get any properties qom-tree-get couldn't get for error information. My point is: dropping @error now does not tie our hands as far as I can tell.

Back to qom-tree. I believe this script is a development aid that exists because qom-get is painful to use for humans. Your qom-tree command would completely obsolete it. I wouldn't worry about it. If you think I'm wrong there, please speak up!

On 4/9/2025 3:39 AM, Markus Armbruster wrote: > Hi Steve, I apologize for the slow response. > > Steve Sistare <steven.sistare(a)oracle.com&gt; writes: > >> Using qom-list and qom-get to get all the nodes and property values in a >> QOM tree can take multiple seconds because it requires 1000's of individual >> QOM requests. Some managers fetch the entire tree or a large subset >> of it when starting a new VM, and this cost is a substantial fraction of >> start up time. > > "Some managers"... could you name one? My personal experience is with Oracle's OCI, but likely others could benefit.

On 4/28/2025 4:04 AM, Markus Armbruster wrote: > Steven Sistare <steven.sistare(a)oracle.com&gt; writes: > >> On 4/9/2025 3:39 AM, Markus Armbruster wrote: >>> Hi Steve, I apologize for the slow response. >>> >>> Steve Sistare <steven.sistare(a)oracle.com&gt; writes: >>> >>>> Using qom-list and qom-get to get all the nodes and property values in a >>>> QOM tree can take multiple seconds because it requires 1000's of individual >>>> QOM requests. Some managers fetch the entire tree or a large subset >>>> of it when starting a new VM, and this cost is a substantial fraction of >>>> start up time. >>> >>> "Some managers"... could you name one? >> >> My personal experience is with Oracle's OCI, but likely others could benefit. > > Elsewhere in this thread, we examined libvirt's use qom-get. Its use of > qom-get is also noticably slow, and your work could speed it up. > However, most of its use is for working around QMP interface > shortcomings around probing CPU flags. Addressing these would help it > even more. > > This makes me wonder what questions Oracle's OCI answers with the help > of qom-get. Can you briefly describe them? > > Even if OCI would likewise be helped more by better QMP queries, your > fast qom tree get work might still be useful. We already optimized our queries as a first step, but what remains is still significant, which is why I submitted this RFE.