Now that all calls to parse_error have a token, add the line and column
to the message. As far as I can see the two important TODOs (better
errors and better EOI handling) are done, and the others (token range
information and "parsed size"?) do not really matter or are handled
better by json-streamer.c. So remove the list, which had sat unchanged
since 2009.
This needs some adjustments to provide a good x and y for error messages.
First of all, they switch from zero-based to one-based, which is safe
because they were both sitting unused. Second, right now the x and y
are those of the *last* character in the token. Modify json-lexer.c to
freeze tok->x and tok->y at the first character added to the GString.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-ID: <20260626101727.1727389-7-pbonzini@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
This is not needed with a push parser. Since it processes tokens
immediately, the JSONToken can be created directly on the stack
and does not need to copy the lexer's string data.
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-ID: <20260626101727.1727389-6-pbonzini@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Now fully exploit the push parser, feeding it one token at a time
without having to wait until braces and brackets are balanced.
While the nesting counts are retained for error recovery purposes,
the system can now report the first parsing error without waiting
for parentheses to be balanced. This also means that JSON_ERROR
can be handled in json-parser.c, not json-streamer.c.
After reporting the error, json-streamer.c then enters an error recovery
mode where subsequent errors are suppressed. This mimics the previous
error reporting behavior, but it provides prompt feedback on parsing
errors. As an example, here is an example interaction with qemu-ga.
BEFORE (error reported only once braces are balanced):
>> {"execute":foo
>> }
<< {"error": {"class": "GenericError", "desc": "JSON parse error, invalid keyword 'foo'"}}
>> {"execute":"somecommand"}
<< {"error": {"class": "CommandNotFound", "desc": "The command somecommand has not been found"}}
AFTER (error reported immediately, but similar error recovery as before):
>> {"execute":foo
<< {"error": {"class": "GenericError", "desc": "JSON parse error, invalid keyword 'foo'"}}
>> }
>> {"execute":"somecommand"}
<< {"error": {"class": "CommandNotFound", "desc": "The command somecommand has not been found"}}
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-ID: <20260626101727.1727389-5-pbonzini@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
[Token size limit check off-by-one fixed]
Signed-off-by: Markus Armbruster <armbru@redhat.com>
It makes no sense to let brace_count and bracket_count go negative,
also because it immediately ends error recovery and sets them both
back to zero. Instead set them to zero *before* choosing
whether to process the token queue; this makes it possible to
have the fields as unsigned.
Note that JSON_END_OF_INPUT now forces the parentheses to appear
balanced, so that the queue is emptied and an error is reported;
hence, the "type != JSON_END_OF_INPUT" condition can be removed.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-ID: <20260626101727.1727389-4-pbonzini@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
[Comment tweaked]
Signed-off-by: Markus Armbruster <armbru@redhat.com>
In order to avoid stashing all the tokens corresponding to a JSON value,
embed the parsing stack and state machine in JSONParser. This is more
efficient and allows for more prompt error recovery; it also does not
make the code substantially larger than the current recursive descent
parser, though the state machine is probably a bit harder to follow.
The stack consists of QLists and QDicts corresponding to open
brackets and braces, plus optionally a QString with the current
key on top of each QDict.
After each value is parsed, it is added to the top array or dictionary
or, if the stack is empty, json_parser_feed returns the complete
QObject.
For now, json-streamer.c keeps tracking the tokens up until braces
and brackets are balanced, and then shoves the whole queue of tokens
into the push parser. The only logic change is that JSON_END_OF_INPUT
always triggers the emptying of the queue; the parser takes notice and
checks that there is nothing on the stack. Not using brace_count
and bracket_count for this is the first step towards improved separation
of concerns between json-parser.c and json-streamer.c.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-ID: <20260626101727.1727389-2-pbonzini@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
[Minor comment improvements]
Signed-off-by: Markus Armbruster <armbru@redhat.com>
json_writer_new() creates the output GString with g_string_new(NULL),
which starts at the GLib default of 64 bytes. Serializing typical
QMP responses then requires multiple reallocations as the buffer
grows -- for query-qmp-schema the GString is reallocated 12+ times.
Preallocate JSON_WRITER_INITIAL_SIZE (4096) bytes. This covers
most QMP responses without any reallocation. The JSONWriter is a
short-lived object so the preallocation does not accumulate.
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Bin Guo <guobin@linux.alibaba.com>
Message-ID: <20260603022538.92780-1-guobin@linux.alibaba.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
The general expectation is that header files should follow the same
file/path naming scheme as the corresponding source file. There are
various historical exceptions to this practice in QEMU, with one of
the most notable being the include/qapi/qmp/ directory. Most of the
headers there correspond to source files in qobject/.
This patch corrects most of that inconsistency by creating
include/qobject/ and moving the headers for qobject/ there.
This also fixes MAINTAINERS for include/qapi/qmp/dispatch.h:
scripts/get_maintainer.pl now reports "QAPI" instead of "No
maintainers found".
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Zhao Liu <zhao1.liu@intel.com>
Acked-by: Halil Pasic <pasic@linux.ibm.com> #s390x
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-ID: <20241118151235.2665921-2-armbru@redhat.com>
[Rebased]
Convert the qmp-spec.txt document to restructuredText.
Notable points about the conversion:
* numbers at the start of section headings are removed, to match
the style of the rest of the manual
* cross-references to other sections or documents are hyperlinked
* various formatting tweaks (notably the examples, which need the
-> and <- prefixed so the QMP code-block lexer will accept them)
* English prose fixed in a few places
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
Message-Id: <20230515162245.3964307-2-peter.maydell@linaro.org>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
[.. code-block:: dumbed down to :: to work around CI failure]
One less qemu-specific macro. It also helps to make some headers/units
only depend on glib, and thus moved in standalone projects eventually.
Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Reviewed-by: Richard W.M. Jones <rjones@redhat.com>
Building QEMU on Fedora 37 (Rawhide Prerelease) ppc64le failed with the
following error:
$ ../configure --prefix=/usr/local/qemu-disabletcg --target-list=ppc-softmmu,ppc64-softmmu --disable-tcg --disable-linux-user
...
$ make -j$(nproc)
...
In file included from /root/qemu/include/qapi/qmp/qdict.h:16,
from /root/qemu/include/block/qdict.h:13,
from ../qobject/block-qdict.c:11:
/root/qemu/include/qapi/qmp/qobject.h: In function ‘qdict_array_split’:
/root/qemu/include/qapi/qmp/qobject.h:49:17: error: ‘subqdict’ may be used uninitialized [-Werror=maybe-uninitialized]
49 | typeof(obj) _obj = (obj); \
| ^~~~
../qobject/block-qdict.c:227:16: note: ‘subqdict’ declared here
227 | QDict *subqdict;
| ^~~~~~~~
cc1: all warnings being treated as errors
Fix build failure by expanding the ternary operation.
Tested with `make check-unit` (the check-block-qdict test passed).
Signed-off-by: Murilo Opsfelder Araujo <muriloo@linux.ibm.com>
Cc: Kevin Wolf <kwolf@redhat.com>
Cc: Hanna Reitz <hreitz@redhat.com>
Cc: Markus Armbruster <armbru@redhat.com>
Message-Id: <20220311221634.58288-1-muriloo@linux.ibm.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
The functions to modify a QString's string are all unused now. Drop
them, and make the string immutable. Saves 16 bytes per QString on my
system.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20201211171152.146877-21-armbru@redhat.com>
QString supports modifying its string, but it's quite limited: you can
only append. The remaining callers use it for building an initial
string, never for modifying it later.
Change parse_string() to do build the initial string with GString.
This is another step towards making QString immutable.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20201211171152.146877-18-armbru@redhat.com>
We have two JSON writers written in C: qobject/qjson.c provides
qobject_to_json(), and migration/qjson.c provides a more low level
imperative interface. They don't share code. The latter tacitly
limits numbers to int64_t, and strings contents to characters that
don't need escaping.
Factor out qobject_to_json()'s JSON writer as qobject/json-writer.c.
Straightforward, except for numbers: since the writer is to be
independent of QObject, it can't use qnum_to_string(). Open-code it
instead. This is actually an improvement of sorts, because it
liberates qnum_to_string() from JSON's needs: its JSON-related FIXMEs
move to the JSON writer, where they belong.
The next commit will replace migration/qjson.c.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20201211171152.146877-16-armbru@redhat.com>
qobject_to_json() and qobject_to_json_pretty() build a GString, then
covert it to QString. Just one of the callers actually needs a
QString: qemu_rbd_parse_filename(). A few others need a string they
can modify: qmp_send_response(), qga's send_response(), to_json_str(),
and qmp_fd_vsend_fds(). The remainder just need a string.
Change qobject_to_json() and qobject_to_json_pretty() to return the
GString.
qemu_rbd_parse_filename() now has to convert to QString. All others
save a QString temporary. to_json_str() actually becomes a bit
simpler, because GString provides more convenient modification
functions.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20201211171152.146877-6-armbru@redhat.com>
QString supports modifying its string, but it's quite limited: you can
only append. The remaining callers use it for building an initial
string, never for modifying it later.
Use of GString for building the initial string is actually more
convenient here. Change qobject_to_json() & friends to do that.
Once all such uses are replaced this way, QString can become immutable.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20201211171152.146877-5-armbru@redhat.com>
We should serialize numbers to JSON so that they deserialize back to
the same number. We fail to do so.
The culprit is qnum_to_string(): it uses format %f with trailing '0'
trimmed. Results in pretty output for "nice" numbers, but is prone to
nasty rounding errors. For instance, numbers between 0 and 0.0000005
get flushed to zero.
Where exactly the incorrect rounding can bite is tiresome to gauge.
Here's my take.
* In QMP output, type 'number':
- query-blockstats value avg_rd_queue_depth
- QMP query-migrate values mbps, cache-miss-rate, encoding-rate,
busy-rate, compression-rate.
Relatively harmless, I guess.
* In tracing QMP input. Harmless.
* In qemu-ga output, type 'number': guest-get-users value login-time.
Harmless.
* In output of HMP qom-get. Harmless.
Not affected, because double values don't actually occur there (I
think):
* QMP output, type 'any':
* qom-get value
* qom-list, qom-list-properties value default-value
* query-cpu-model-comparison, query-cpu-model-baseline,
query-cpu-model-expansion value props.
* qemu-img --output json output.
* "json:" pseudo-filenames generated by bdrv_refresh_filename().
* The rbd block driver's "=keyvalue-pairs" hack.
* In -object help on property default values. Aside: use of JSON
feels inappropriate here.
* Output of HMP qom-get.
* Argument conversion to QemuOpts for qdev_device_add() and HMP with
qemu_opts_from_qdict()
QMP and HMP device_add, virtio-net failover primary creation,
xen-usb "usb-host" creation, HMP netdev_add, object_add.
* The uses of qobject_input_visitor_new_flat_confused()
As far as I can tell, none of the visited types contain double
values.
* Dumping ImageInfoSpecific with dump_qobject()
Fix by formatting with %.17g. 17 decimal digits always suffice for
IEEE double.
The change to expected test output illustrates the effect: the
rounding errors are gone, but some seemingly "nice" numbers now get
converted to not so nice strings, e.g. 0.42 to "0.41999999999999998".
This is because 0.42 is not representable exactly in double. It's
more accurate in this example than strictly necessary, though.
If ugly accuracy bothers us, we can we can try using the least number
of digits that still converts back to the same double. In this
example, "0.42" would do.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20201210161452.2813491-7-armbru@redhat.com>
In qobject_type(), NULL is returned when the 'QObject' returned from parse_value() is not of QString type,
and this 'QObject' memory will leaked.
So we need to first cache the 'QObject' returned from parse_value(), and finally
free 'QObject' memory at the end of the function.
Also, we add a testcast about invalid dict key.
The memleak stack is as follows:
Direct leak of 32 byte(s) in 1 object(s) allocated from:
#0 0xfffe4b3c34fb in __interceptor_malloc (/lib64/libasan.so.4+0xd34fb)
#1 0xfffe4ae48aa3 in g_malloc (/lib64/libglib-2.0.so.0+0x58aa3)
#2 0xaaab3557d9f7 in qnum_from_int qemu/qobject/qnum.c:25
#3 0xaaab35584d23 in parse_literal qemu/qobject/json-parser.c:511
#4 0xaaab35584d23 in parse_value qemu/qobject/json-parser.c:554
#5 0xaaab35583d77 in parse_pair qemu/qobject/json-parser.c:270
#6 0xaaab355845db in parse_object qemu/qobject/json-parser.c:327
#7 0xaaab355845db in parse_value qemu/qobject/json-parser.c:546
#8 0xaaab35585b1b in json_parser_parse qemu/qobject/json-parser.c:580
#9 0xaaab35583703 in json_message_process_token qemu/qobject/json-streamer.c:92
#10 0xaaab355ddccf in json_lexer_feed_char qemu/qobject/json-lexer.c:313
#11 0xaaab355de0eb in json_lexer_feed qemu/qobject/json-lexer.c:350
#12 0xaaab354aff67 in tcp_chr_read qemu/chardev/char-socket.c:525
#13 0xfffe4ae429db in g_main_context_dispatch (/lib64/libglib-2.0.so.0+0x529db)
#14 0xfffe4ae42d8f (/lib64/libglib-2.0.so.0+0x52d8f)
#15 0xfffe4ae430df in g_main_loop_run (/lib64/libglib-2.0.so.0+0x530df)
#16 0xaaab34d70bff in iothread_run qemu/iothread.c:82
#17 0xaaab3559d71b in qemu_thread_start qemu/util/qemu-thread-posix.c:519
Fixes: 532fb53284 ("qapi: Make more of qobject_to()")
Reported-by: Euler Robot <euler.robot@huawei.com>
Signed-off-by: Alex Chen <alex.chen@huawei.com>
Signed-off-by: Chen Qun <kuhn.chenqun@huawei.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20201113145525.85151-1-alex.chen@huawei.com>
[Commit message tweaked]
This shows how to do some "computations" in meson.build using its array
and dictionary data structures, and also a basic usage of the sourceset
module for conditional compilation.
Notice the new "if have_system" part of util/meson.build, which fixes
a bug in the old build system was buggy: util/dbus.c was built even for
non-softmmu builds, but the dependency on -lgio was lost when the linking
was done through libqemuutil.a. Because all of its users required gio
otherwise, the bug was hidden. Meson instead propagates libqemuutil's
dependencies down to its users, and shows the problem.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Reported by GCC9 when building with CFLAG -Wimplicit-fallthrough=2:
qobject/json-parser.c: In function ‘parse_literal’:
qobject/json-parser.c:492:24: error: this statement may fall through [-Werror=implicit-fallthrough=]
492 | case JSON_INTEGER: {
| ^
qobject/json-parser.c:524:5: note: here
524 | case JSON_FLOAT:
| ^~~~
Correctly place the 'fall through' comment.
Reported-by: Stefan Weil <sw@weilnetz.de>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20190719131425.10835-2-philmd@redhat.com>
Signed-off-by: Laurent Vivier <laurent@vivier.eu>
The assert checking if the value of lexer->state in next_state(),
which is used as an index to the 'json_lexer' array, incorrectly
checks for an index value less than or equal to ARRAY_SIZE(json_lexer).
Fix assert so that it just checks for an index less than the array size.
Signed-off-by: Liam Merwick <liam.merwick@oracle.com>
Message-Id: <1553169472-25325-1-git-send-email-liam.merwick@oracle.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Li Qiang <liq3ea@gmail.com>
Reviewed-by: Stefano Garzarella <sgarzare@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Commit 8bca4613 added support for %% in json strings when interpolating,
but in doing so broke handling of % when not interpolating.
When parse_string() is fed a string token containing '%', it skips the
'%' regardless of ctxt->ap, i.e. even it's not interpolating. If the
'%' is the string's last character, it fails an assertion. Else, it
"merely" swallows the '%'.
Fix parse_string() to handle '%' specially only when interpolating.
To gauge the bug's impact, let's review non-interpolating users of this
parser, i.e. code passing NULL context to json_message_parser_init():
* tests/check-qjson.c, tests/test-qobject-input-visitor.c,
tests/test-visitor-serialization.c
Plenty of tests, but we still failed to cover the buggy case.
* monitor.c: QMP input
* qga/main.c: QGA input
* qobject_from_json():
- qobject-input-visitor.c: JSON command line option arguments of
-display and -blockdev
Reproducer: -blockdev '{"%"}'
- block.c: JSON pseudo-filenames starting with "json:"
Reproducer: https://bugzilla.redhat.com/show_bug.cgi?id=1668244#c3
- block/rbd.c: JSON key pairs
Pseudo-filenames starting with "rbd:".
Command line, QMP and QGA input are trusted.
Filenames are trusted when they come from command line, QMP or HMP.
They are untrusted when they come from from image file headers.
Example: QCOW2 backing file name. Note that this is *not* the security
boundary between host and guest. It's the boundary between host and an
image file from an untrusted source.
Neither failing an assertion nor skipping a character in a filename of
your choice looks exploitable. Note that we don't support compiling
with NDEBUG.
Fixes: 8bca4613e6
Cc: qemu-stable@nongnu.org
Signed-off-by: Christophe Fergeau <cfergeau@redhat.com>
Message-Id: <20190102140535.11512-1-cfergeau@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Tested-by: Richard W.M. Jones <rjones@redhat.com>
[Commit message extended to discuss impact]
Signed-off-by: Markus Armbruster <armbru@redhat.com>
The lexer ignores whitespace like this:
on whitespace on non-ws spontaneously
IN_START --> IN_WHITESPACE --> JSON_SKIP --> IN_START
^ |
\__/ on whitespace
This accumulates a whitespace token in state IN_WHITESPACE, only to
throw it away on the transition via JSON_SKIP to the start state.
Wasteful. Go from IN_START to IN_START on whitespace directly,
dropping the whitespace character.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20180831075841.13363-7-armbru@redhat.com>
When the lexer chokes on an input character, it consumes the
character, emits a JSON error token, and enters its start state. This
can lead to suboptimal error recovery. For instance, input
0123 ,
produces the tokens
JSON_ERROR 01
JSON_INTEGER 23
JSON_COMMA ,
Make the lexer skip characters after a lexical error until a
structural character ('[', ']', '{', '}', ':', ','), an ASCII control
character, or '\xFE', or '\xFF'.
Note that we must not skip ASCII control characters, '\xFE', '\xFF',
because those are documented to force the JSON parser into known-good
state, by docs/interop/qmp-spec.txt.
The lexer now produces
JSON_ERROR 01
JSON_COMMA ,
Update qmp-test for the nicer error recovery: QMP now reports just one
error for input %p instead of two. Also drop the newline after %p; it
was needed to tease out the second error.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20180831075841.13363-5-armbru@redhat.com>
[Conflict with commit ebb4d82d88 resolved]
The lexer uses macro TERMINAL_NEEDED_LOOKAHEAD() to decide whether a
state transition consumes the input character. It returns true when
the state transition is defined with the TERMINAL() macro. To detect
that, it checks whether input '\0' would have resulted in the same
state transition, and the new state is not IN_ERROR.
Why does that even work? For all states, the new state on input '\0'
is either IN_ERROR or defined with TERMINAL(). If the state
transition equals the one we'd get for input '\0', it goes to IN_ERROR
or to the argument of TERMINAL(). We never use TERMINAL(IN_ERROR),
because it makes no sense. Thus, if it doesn't go to IN_ERROR, it
must be defined with TERMINAL().
Since this isn't quite confusing enough, we negate the result to get
@char_consumed, and ignore it when @flush is true.
Instead of deriving the lookahead bit from the state transition, make
it explicit. This is easier to understand, and a bit more flexible,
too.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20180831075841.13363-4-armbru@redhat.com>
