The company 'Arm' went through a rebranding many years back
involving a recapitalization from 'ARM' to 'Arm'. As a result
our documentation is a bit inconsistent between the two forms.
It's not worth trying to update everywhere in QEMU, but it's
easy enough to make docs/ consistent.
We last did this in commit 6fe6d6c9a in 2020, but a few new
uses of the wrong capitalization have crept back in since.
As before, "ARMv8" and similar architecture names, and
older CPU names like "ARM926" still retain all-caps.
In a few places we make minor grammar fixups as we touch
the sentences we're fixing.
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
Reviewed-by: Cédric Le Goater <clg@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Message-id: 20260115150545.669444-1-peter.maydell@linaro.org
The functional tests currently don't allow a single test to be
selected for execution by dotted name, e.g:
./build/run tests/functional/ppc64/test_pseries.py PseriesMachine.test_ppc64_linux_boot
^
The issue is that the testcase.py main function passes the test
module's name as the second argument to unittest.main(), which makes
it ignore all other positional arguments (presumably because the
module is already the superset of all tests).
After commit cac08383f0 ("tests/functional: expose sys.argv to
unittest.main"), the situation improves by passing the rest of the
argv from the command line invocation into unittest.main(), but it
still doesn't fix the issue. The short form options are now accepted,
so the -k option could be used to filter for a pattern, which is
useful, but not the same as listing the test names.
Fix this by passing the test module name via the "module" argument to
unittest.main() and stop touching argv. The ways of invoking tests are
now as per unittests documentation (-k still works):
Examples:
test_pseries.py - run default set of tests
test_pseries.py MyTestSuite - run suite 'MyTestSuite'
test_pseries.py MyTestCase.testSomething - run MyTestCase.testSomething
test_pseries.py MyTestCase - run all 'test*' test methods in MyTestCase
Note that ever since we've been programatically passing the module
name to unittest.main(), the usage 'test_pseries.py test_pseries' was
never valid. It used to "work" just the same as 'test_pseries.py
foobar' would. After this patch, that usage results in an error.
Also note that testcase.py:main() pertains to running the test module
that invoked it via QemuSystemTest.main(), i.e. module == __main__. So
the 'discover' usage of unittest doesn't apply here, the module is
already discovered because that's where this code was called from to
begin with. This patch could just as well call unittest.main() instead
of unittest.main(test_module), but the latter provides nicer error
messages prefixed with the module name.
Tested-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Fabiano Rosas <farosas@suse.de>
Message-ID: <20260102181700.11886-1-farosas@suse.de>
Signed-off-by: Thomas Huth <thuth@redhat.com>
Various aspects of the development workflow are complicated by the need
to set env variables ahead of time, or use specific paths. Meson
provides a 'devenv' command that can be used to launch a command with a
number of appropriate project specific environment variables preset.
By default it will modify $PATH to point to any build directory that
contains a binary built by the project.
This further augments that to replicate the venv 'activate' script:
* Add $BUILD_DIR/pyvenv/bin to $PATH
* Set VIRTUAL_ENV to $BUILD_DIR/pyvenv
And then makes functional tests more easily executable
* Add $SRC_DIR/tests/functional and $SRC_DIR/python to $PYTHONPATH
To see the benefits of this consider this command:
$ source ./build/pyvenv/bin/activate
$ ./scripts/qmp/qmp-shell-wrap ./build/qemu-system-x86_64
which is now simplified to
$ ./build/run ./scripts/qmp/qmp-shell-wrap qemu-system-x86_64 [args..]
This avoids the need repeat './build' several times and avoids polluting
the current terminal's environment and/or avoids errors from forgetting
to source the venv settings.
As another example running functional tests
$ export PYTHONPATH=./python:./tests/functional
$ export QEMU_TEST_QEMU_BINARY=./build/qemu-system-x86_64
$ build/pyvenv/bin/python3 ./tests/functional/x86_64/test_virtio_version.py
which is now simplified to
$ export QEMU_TEST_QEMU_BINARY=qemu-system-x86_64
$ ./build/run ./tests/functional/x86_64/test_virtio_version.py
This usefulness of this will be further enhanced with the pending
removal of the QEMU python APIs from git, as that will require the use
of the python venv in even more scenarios that today.
The 'run' script does not let 'meson devenv' directly launch the command
to be run because it always requires $BUILD_DIR as the current working
directory. It is desired that 'run' script always honour the current
working directory of the terminal that invokes is. Thus the '--dump'
flag is used to export the devenv variables into the 'run' script's
shell.
This takes the liberty to assign 'run.in' to the "Build system" section
in the MAINTAINERS file, given that it leverages meson's 'devenv'
feature.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Link: https://lore.kernel.org/r/20251222113859.182395-1-berrange@redhat.com
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
It is possible to get qtest to read fuzzer reproducers from a file
rather than directly from stdio; this is useful when you want to run
QEMU under gdb to debug the failure. Document how to do this, which
was previously only written down in the commit message for
5b18a6bf44 ("chardev: Allow setting file chardev input file on the
command line").
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
Reviewed-by: Fabiano Rosas <farosas@suse.de>
Reviewed-by: Manos Pitsidianakis <manos.pitsidianakis@linaro.org>
Message-id: 20251028165236.3327658-1-peter.maydell@linaro.org
The QAPI_TEST_UPDATE env var can be set when running the QAPI
schema tests to regenerate the reference output. For consistent
naming with the tracetool test, change the env var name to
QEMU_TEST_REGENERATE.
The test is modified to provide a hint about use of the new
env var and it is also added to the developer documentation.document its usage.
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Message-id: 20250916081638.764020-8-berrange@redhat.com
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
When reviewing tracetool patches it is often very unclear what the
expected output will be for the generated backends. Compounding
this is that a default build will only enable the 'log' trace
backend, so developers won't see generated code for other backends
without making a special effort. Some backends are also platform
specific, so can't be enabled in QEMU builds, even though tracetool
could generate the code.
To address this, introduce a test suite for tracetool which is
conceptually similar to the qapi-schema test. It is a simple
python program that runs tracetool and compares the actual output
to historical reference output kept in git. The test directly
emits TAP format logs for ease of integration with meson.
This can be run with
make check-tracetool
to make it easier for developers changing generated output, the
sample expected content can be auto-recreated
QEMU_TEST_REGENERATE=1 make check-tracetool
and the changes reviewed and added to the commit. This will also
assist reviewers interpreting the change.
Developers are reminded of this in the test output on failure:
$ make check-tracetool
1/6 qemu:tracetool / dtrace OK 0.14s
2/6 qemu:tracetool / ftrace FAIL 0.06s exit status 1
――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
1..2
ok 1 - ftrace.c
#
not ok 1 - ftrace.h (set QEMU_TEST_REGENERATE=1 to recreate reference output if tracetool generator was intentionally changed)
――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
3/6 qemu:tracetool / log OK 0.06s
4/6 qemu:tracetool / simple OK 0.06s
5/6 qemu:tracetool / syslog OK 0.06s
6/6 qemu:tracetool / ust OK 0.11s
Summary of Failures:
2/6 qemu:tracetool / ftrace FAIL 0.06s exit status 1
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-id: 20250916081638.764020-6-berrange@redhat.com
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Now that all Avocado tests have been converted to or been replaced by
other functional tests, we can delete the remainders of the Avocado
tests from the QEMU source tree.
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Message-ID: <20250414113031.151105-16-thuth@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
This file was meant for defining the vocabulary for our testing
efforts, but it did not age well: First, the definitions are not
only about the CI part, but also about testing in general, so most
of the information should rather reside in main.rst instead.
Second, some vocabulary has never been really adopted by the QEMU
project, for example we never really use the word "system testing"
since "system" rather means the system emulator binaries in the
QEMU project (and we also don't do any testing with other components
like libvirt and virt-managers here). It also defines that the qtests
are the "functional" tests in QEMU, which is not really up to date
anymore after the "tests/functional" framework has been introduced
a couple of months ago (FWIW, the qtests could rather be seen as a
mix between unit testing and functional testing).
To solve this problem, move the useful parts of this file into
main.rst and directly into ci.rst, and drop the ones (like "system
testing") that we don't really need anymore.
Message-ID: <20250314085959.1585568-1-thuth@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
Since we don't run the Avocado jobs in the CI anymore, rename
these variables to QEMU_JOB_FUNCTIONAL and QEMU_CI_FUNCTIONAL.
Also, there was a mismatch between the documentation and the
implementation of QEMU_CI_AVOCADO_TESTING: While the documentation
said that you had to "Set this variable to have the tests using the
Avocado framework run automatically", you indeed needed to set it
to make the pipelines appear in your dashboard - but they were never
run automatically in forks and had to be triggered manually. Let's
improve this now: No need to hide these pipelines from the users
by default anymore (the functional tests should be stable enough
nowadays), and rather allow the users to run the pipelines auto-
matically with this switch now instead, as was documented.
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Message-ID: <20250414113031.151105-15-thuth@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
The 'qemu_bin' field is currently set on the class, despite being
accessed as if it were an object instance field with 'self.qemu_bin'.
This is no obvious need to have it as a class field, so move it into
the object instance.
Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Message-ID: <20250228102738.3064045-3-berrange@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
Some tests have a very long runtime and might run into timeout issues
e.g. when QEMU has been compiled with --enable-debug. Add a decorator
for marking them more easily. Rename the corresponding environment
variable to be more in sync with the other QEMU_TEST_ALLOW_* switches
that we already have, and add a paragraph about it in the documentation.
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Message-ID: <20250128152839.184599-2-thuth@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
Make it clear that the commands have to be run from the folder with the
build, and use the python3 from our pyvenv to make sure that the
pycotap module is available.
Message-ID: <20241112115302.470527-1-thuth@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
Support the QEMU_TEST_QMP_BACKDOOR=backdoor.sock env variable as a
way to get a QMP backdoor for debugging a stalled QEMU test. Most
typically this would be used if running the tests directly:
$ QEMU_TEST_QMP_BACKDOOR=backdoor.sock \
QEMU_TEST_QEMU_BINARY=./build/qemu-system-arm \
PYTHONPATH=./python \
./tests/functional/test_arm_tuxrun.py
And then, when the test stalls, in a second shell run:
$ ./scripts/qmp/qmp-shell backdoor.sock
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20241121154218.1423005-18-berrange@redhat.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20241121165806.476008-18-alex.bennee@linaro.org>
The build/tests/functional subdirectories are consuming huge amounts
of disk space.
Split the location for scratch files into a 'scratch' sub-directory,
separate from log files, and delete it upon completion of each test.
The new env variable QEMU_TEST_KEEP_SCRATCH can be set to preserve
this scratch dir for debugging access if required.
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20241121154218.1423005-3-berrange@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20241121165806.476008-3-alex.bennee@linaro.org>
Replace the footnotes with inline links whenever the footnote text
consists of nothing but the URL. While at it, make the link texts
consistent in the surrounding areas, for example avoiding usage of
"here" for the link's text.
In the case of acpi-bits.rst this fixes a build failure with Sphinx
8.1.0, because the FOSDEM link was duplicated in the paragraph and the
new version is a lot stricter about unreferenced footnotes.
Reviewed-by: Peter Maydell <peter.maydell@linaro.org>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
We do not always want both address and undefined behavior
sanitizers running at the same time.
For the gitlab custom-runners, drop to only --enable-ubsan.
These jobs are not run by default, but as will be obvious in the
next patch, we don't run ASan on x86 either, and it seems wrong
to hold aarch64 and s390x to a different standard.
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Message-ID: <20240813095216.306555-2-richard.henderson@linaro.org>
Signed-off-by: Thomas Huth <thuth@redhat.com>
Document the new functional testing framework. The text is originally
based on the Avocado documentation, but heavily modified to match the
new framework.
Message-ID: <20240830133841.142644-45-thuth@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
The avocado_qemu.Test class has been renamed a while back in commit
2283b627bc ("tests/avocado: Rename avocado_qemu.Test -> QemuSystemTest"),
so we should reflect this now in the documentation, too.
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Message-ID: <20240830133841.142644-44-thuth@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
Building and testing are two separate topics, so let's split the testing
into a separate category and move the related files into a separate folder.
Message-ID: <20240830133841.142644-42-thuth@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
