Hi All, I’m using qmp command via “virsh qemu-monitor-command” to perform block related jobs. And in my case, I execute block-commit to commit the changes in the top image to the base image which is encrypted. So in the base parameter, I need a json string which is single quoted to support the encryption parameters. However, the virsh is failed to parse the whole json qmp command.

Attempt 1: Single quote without backflash, failed root@host:~# virsh qemu-monitor-command i-9wdfw2x8  "{\"execute\": \"block-commit\", \"arguments\": { \"device\": \"drive-virtio-disk2\", \"job-id\": \"job100\", \"base\":'json:{\"encrypt.key-secret\":\"vol-38973xjl.secret\",\"driver\":\"qcow2\",\"file\":{\"driver\":\"file\",\"filename\":\"/pitrix/data/container/vol-38973xjl.img\"}}', \"top\": \"/pitrix/data/container/vol-38973xjl_ss-2tw7v0mm.img\"}}" error: internal error: cannot parse json {"execute": "block-commit", "arguments": { "device": "drive-virtio-disk2", "job-id": "job100", "base":'json:{"encrypt.key-secret":"vol-38973xjl.secret","driver":"qcow2","file":{"driver":"file","filename":"/pitrix/data/container/vol-38973xjl.img"}}', "top": "/pitrix/data/container/vol-38973xjl_ss-2tw7v0mm.img"}}: lexical error: invalid char in json text.           , "job-id": "job100", "base":'json:{"encrypt.key-secret":"vo

On 1/31/20 4:38 AM, Peter Luo wrote: > error: internal error: cannot parse json {"execute": "block-commit", "arguments": { "device": "drive-virtio-disk2", "job-id": "job100", "base":'json:{"encrypt.key-secret":"vol-38973xjl.secret","driver":"qcow2","file":{"driver":"file","filename":"/pitrix/data/container/vol-38973xjl.img"}}', "top": "/pitrix/data/container/vol-38973xjl_ss-2tw7v0mm.img"}}: lexical error: invalid char in json text. > >           , "job-id": "job100", "base":'json:{"encrypt.key-secret":"vo > >                      (right here) ------^ > qemu's QMP language has an extension where it accepts 'string' in addition to the JSON "string". But it uses a hand-rolled parser, so it can do whatever it wants.

[adding Markus] On 2/3/20 4:13 AM, Daniel P. Berrangé wrote: > On Fri, Jan 31, 2020 at 06:44:42AM -0600, Eric Blake wrote: >> On 1/31/20 4:38 AM, Peter Luo wrote: >> >>> error: internal error: cannot parse json {"execute": "block-commit", "arguments": { "device": "drive-virtio-disk2", "job-id": "job100", "base":'json:{"encrypt.key-secret":"vol-38973xjl.secret","driver":"qcow2","file":{"driver":"file","filename":"/pitrix/data/container/vol-38973xjl.img"}}', "top": "/pitrix/data/container/vol-38973xjl_ss-2tw7v0mm.img"}}: lexical error: invalid char in json text. >>> >>>           , "job-id": "job100", "base":'json:{"encrypt.key-secret":"vo >>> >>>                      (right here) ------^ >>> >> >> qemu's QMP language has an extension where it accepts 'string' in addition >> to the JSON "string". But it uses a hand-rolled parser, so it can do >> whatever it wants. > > Can we deprecate & remove this extension in QEMU ?

We could start a deprecation clock, if desired, but I don't know how many external users would be impacted (at least qemu's testsuite heavily relies on the extension of single quotes).

Are there any third-party libraries that parse JSON5? https://json5.org/ documents that one of the nice features of JSON5 is that single quotes work out of the box. Right now, even though qemu does NOT parse the same thing as pure JSON (which is https://datatracker.ietf.org/doc/rfc8259/), it does appear to parse a subset of JSON5, where the additional features of JSON5 (such as allowing a trailing comma, escaped newlines within strings, allowing bareword key:"value" syntax, a larger set of accepted numeric values, allowing comments) don't seem that onerous.

> If we're going to call QMP protocol JSON, then IMHO QEMU should follow > the JSON spec as closely as possible, without extensions. As it is, qemu explicitly does NOT parse all valid JSON - it rejects non-ASCII bytes (whether as UTF-8 or as \u escape sequences) in strings. So the fact that it already extensions in some places and limitations in others is a burden for clients to be aware of.

> On the flip side, if we're going to support extensions like single quoting, > then we should make it clear to applications that this is not really JSON > and that they need to provide an impl that is 100% matching QEMU's dialect. > This effectively means they need just import a copy of QEMU's code.

On Tue, Feb 04, 2020 at 09:11:27AM +0100, Markus Armbruster wrote: > Eric Blake <eblake(a)redhat.com&gt; writes: > > > [adding Markus] > > > > On 2/3/20 4:13 AM, Daniel P. Berrangé wrote: > >> On Fri, Jan 31, 2020 at 06:44:42AM -0600, Eric Blake wrote: > >>> On 1/31/20 4:38 AM, Peter Luo wrote: > >>> > >>>> error: internal error: cannot parse json {"execute": "block-commit", "arguments": { "device": "drive-virtio-disk2", "job-id": "job100", "base":'json:{"encrypt.key-secret":"vol-38973xjl.secret","driver":"qcow2","file":{"driver":"file","filename":"/pitrix/data/container/vol-38973xjl.img"}}', "top": "/pitrix/data/container/vol-38973xjl_ss-2tw7v0mm.img"}}: lexical error: invalid char in json text. > >>>> > >>>>           , "job-id": "job100", "base":'json:{"encrypt.key-secret":"vo > >>>> > >>>>                      (right here) ------^ > >>>> > >>> > >>> qemu's QMP language has an extension where it accepts 'string' in addition > >>> to the JSON "string". But it uses a hand-rolled parser, so it can do > >>> whatever it wants. > >> > >> Can we deprecate & remove this extension in QEMU ? [snip] > >> On the flip side, if we're going to support extensions like single quoting, > >> then we should make it clear to applications that this is not really JSON > >> and that they need to provide an impl that is 100% matching QEMU's dialect. > >> This effectively means they need just import a copy of QEMU's code. > > To the best of my knowledge, the JSON parser interprets any valid strict > JSON input in accordance to RFC 8259. In other words, you don't notice > the extensions unless you use them, or rely on invalid strict JSON to be > rejected. > > Peter Luo's input uses one of QEMU's JSON parser's extensions like this: > > "base":'json:{...}' > > This is not valid strict JSON. Libvirt's JSON parser doesn't accept it. > > The problem is not presence of extensions in QEMU, it's the use of these > extensions in input for libvirt. Removing the extensions from QEMU will > not affect the error. Removing their use from the input will. The issue that I see is that QEMU accepts this input string when it parsers JSON provided by the user. This in turn means the user has an expectation that other tools based on QEMU will accept this exact same document. This is not the case because the other tools are using a stricter impl of JSON. IOW QEMU's extensions have mislead the users into believing their JSON input is valid for any tool based on QEMU. Thus overall I think it would be beneficial for any places where QEMU accepts JSON from external users or apps, to be restricted to common JSON syntax only, without any QEMU specific extensions.

[adding Markus] On 2/3/20 4:13 AM, Daniel P. Berrangé wrote: > On Fri, Jan 31, 2020 at 06:44:42AM -0600, Eric Blake wrote: >> On 1/31/20 4:38 AM, Peter Luo wrote: >> >>> error: internal error: cannot parse json {"execute": "block-commit", >>> "arguments": { "device": "drive-virtio-disk2", "job-id": "job100", >>> "base":'json:{"encrypt.key-secret":"vol-38973xjl.secret","driver":"qcow2","file":{"driver":"file","filename":"/pitrix/data/container/vol-38973xjl.img"}}', >>> "top": "/pitrix/data/container/vol-38973xjl_ss-2tw7v0mm.img"}}: >>> lexical error: invalid char in json text. >>> >>>             , "job-id": "job100", >>> "base":'json:{"encrypt.key-secret":"vo >>> >>>                        (right here) ------^ >>> >> >> qemu's QMP language has an extension where it accepts 'string' in >> addition >> to the JSON "string".  But it uses a hand-rolled parser, so it can do >> whatever it wants. > > Can we deprecate & remove this extension in QEMU ? We could start a deprecation clock, if desired, but I don't know how many external users would be impacted (at least qemu's testsuite heavily relies on the extension of single quotes). Are there any third-party libraries that parse JSON5? https://json5.org/ documents that one of the nice features of JSON5 is that single quotes work out of the box.  Right now, even though qemu does NOT parse the same thing as pure JSON (which is https://datatracker.ietf.org/doc/rfc8259/), it does appear to parse a subset of JSON5, where the additional features of JSON5 (such as allowing a trailing comma, escaped newlines within strings, allowing bareword key:"value" syntax, a larger set of accepted numeric values, allowing comments) don't seem that onerous. > > If we're going to call QMP protocol JSON, then IMHO QEMU should follow > the JSON spec as closely as possible, without extensions. As it is, qemu explicitly does NOT parse all valid JSON - it rejects non-ASCII bytes (whether as UTF-8 or as \u escape sequences) in strings.  So the fact that it already extensions in some places and limitations in others is a burden for clients to be aware of. > > On the flip side, if we're going to support extensions like single > quoting, > then we should make it clear to applications that this is not really JSON > and that they need to provide an impl that is 100% matching QEMU's > dialect. > This effectively means they need just import a copy of QEMU's code. > > Regards, > Daniel >