Re: [libvirt] [Qemu-devel] QMP; unsigned 64-bit ints; JSON standards compliance

Daniel P. Berrangé <berrange(a)redhat.com&gt; writes: As we'll see below, speaking of "strict standard compliance" is meaningless, because the standard is too weak. What you mean is "lowest common denominator of relevant implementations". This is the ECMAScript standard, not the JSON standard. For JSON, the situation is more confused. See below. Correct. JavaScript has only one type of number, and they're all represented as double precision floating-point. Perfectly sensible for a prototype done in ten days. Move fast and leave things broken forever. JSON is derived from JavaScript. RFC 4672 is notoriously useless on precision and range. RFC 7159 attempts to contain the damage by spelling out how useless it: This specification allows implementations to set limits on the range and precision of numbers accepted. Since software that implements IEEE 754-2008 binary64 (double precision) numbers [IEEE754] is generally available and widely used, good interoperability can be achieved by implementations that expect no more precision or range than these provide, in the sense that implementations will approximate JSON numbers within the expected precision. A JSON number such as 1E400 or 3.141592653589793238462643383279 may indicate potential interoperability problems, since it suggests that the software that created it expects receiving software to have greater capabilities for numeric magnitude and precision than is widely available. Note that when such software is used, numbers that are integers and are in the range [-(2**53)+1, (2**53)-1] are interoperable in the sense that implementations will agree exactly on their numeric values. RFC 8259 keeps this unchanged. QEMU does conform to RFC 8259. It sets limits on range and precision of numbers accepted: double precision \union the integers between INT64_MIN and UINT64_MAX inclusive. The trouble is that conformance doesn't imply interoperability. Many other implementations follow JavaScript and limit numbers to double. Correct. Support for integers 2^63..2^64-1 is relatively recent: commit 2bc7cfea095 (v2.10, 2017). Since we really needed these, the QObject input visitor silently casts negative integers to uint64_t. It still does for backward compatibility. Commit 5923f85fb82 (right after 2bc7cfea095) explains The input visitor will cast i64 input to u64 for compatibility reasons (existing json QMP client already use negative i64 for large u64, and expect an implicit cast in qemu). Note: before the patch, uint64_t values above INT64_MAX are sent over json QMP as negative values, e.g. UINT64_MAX is sent as -1. After the patch, they are sent unmodified. Clearly a bug fix, but we have to consider compatibility issues anyway. libvirt should cope fine, because its parsing of unsigned integers accepts negative values modulo 2^64. There's hope that other clients will, too. JSON is terrible at interoperability, so good luck with that. If you reduce your order to "the commonly used JSON libraries we know", we can talk. Ugly. It's also what we did until v2.10, August 2017. QMP's input direction still does it, for backward compatibility. Differently ugly. Yet another ugly. We might be able to do option 1 without capability negotiation. v2.10's change from option 1 to what we have now produced zero complaints. On the other hand, we made that change for a reason, so we may want a "send large integers as negative integers" capability regardless. There's no such thing. You mean "we maximize interoperability with common implementations of JSON". Let's talk implementation for a bit. Encoding and decoding integers in funny ways should be fairly easy in the QObject visitors. The generated QMP marshallers all use them. Trouble is a few commands still bypass the generated marshallers, and mess with the QObject themselves: * query-qmp-schema: minor hack explained in qmp_query_qmp_schema()'s comment. Should be harmless. * netdev_add: not QAPIfied. Eric's patches to QAPIfy it got stuck because they reject some abuses like passing numbers and bools as strings. * device_add: not QAPIfied. We're not sure QAPIfication is feasible. netdev_add and device_add both use qemu_opts_from_qdict(). Perhaps we could hack that to mirror what the QObject visitor do. Else, we might have to do it in the JSON parser. Should be possible, but I'd rather not. QMP sending all integers in decimal is inconvenient for some values, such as addresses. QMP sending all (large) integers in hexadecimal would be inconvenient for other values. Let's keep it simple & stupid. If you want sophistication, JSON is the wrong choice. Option 1 feels simplest. Option 2 feels ugliest. Less simple, more interoperable than option 1. Option 3 is like option 2, just not quite as ugly. Can we agree to eliminate option 2 from the race?