1 :Authors: Jason Lowe-Power
4 This file explains how to use gem5's updated testing infrastructure. Running
5 tests before submitting a patch is *incredibly important* so unexpected bugs
8 gem5's testing infrastructure has the following goals:
9 * Simple for *all* users to run
10 * Fast execution in the simple case
11 * High coverage of gem5 code
15 gem5 comes with unit tests, created using the Google Test framework. These can
16 be built through SCons.
18 To build and run all the unit tests:
21 scons build/NULL/unittests.opt
24 All unit tests should be run prior to posting a patch to
25 https://gem5-review.googlesource.com
27 To compile and run just one set of tests (e.g. those declared within
28 `src/base/bitunion.test.cc`):
31 scons build/NULL/base/bitunion.test.opt
32 ./build/NULL/base/bitunion.test.opt
35 To list the available test functions from a test file:
38 ./build/NULL/base/bitunion.test.opt --gtest_list_tests
41 To run a specific test function (e.g., BitUnionData.NormalBitfield):
44 ./build/NULL/base/bitunion.test.opt --gtest_filter=BitUnionData.NormalBitfield
47 # Running system-level tests
49 Within the `tests` directory we have system-level tests. These tests run
50 the gem5 framework against various hardware configurations, with different
51 ISAs, then verify the simulations execute correctly. These should be seen as
52 high-level, coarse-grained tests to compliment the unit-tests.
54 Below is the most common way the tests are run. This will run all of the
55 "quick" tests for X86, ARM, and RISC-V. These tests make up our best-supported
56 platforms and use cases. When running these tests, you will likely want to us
57 the option `-j <CPUs>` where `CPUs` is as large as you can make it.
58 Additionally, it is often a good idea to run longer tests (e.g., linux boot)
59 before submitting your patch.
66 The above is the *minumum* you should run before posting a patch to
67 https://gem5-review.googlesource.com
69 ## Specifying a subset of tests to run
71 You can use the tag query interface to specify the exact tests you want to run.
72 For instance, if you want to run only with `gem5.opt`, you can use
75 ./main.py run --variant opt
78 Or, if you want to just run X86 tests with the `gem5.opt` binary:
81 ./main.py run --length quick --variant opt --isa X86
85 To view all of the available tags, use
88 ./main.py list --all-tags
91 The output is split into tag *types* (e.g., isa, variant, length) and the
92 tags for each type are listed after the type name.
94 You can specify "or" between tags within the same type by using the tag flag
95 multiple times. For instance, to run everything that is tagged "opt" or "fast"
99 ./main.py run --variant opt --variant fast
102 You can also specify "and" between different types of tags by specifying more
103 than one type on the command line. For instance, this will only run tests with
104 both the "X86" and "opt" tags.
107 ./main.py run --isa X86 --variant opt
110 ## Running tests in batch
112 The testing infrastructure provides the two needed methods to run tests in
113 batch. First, you can list all of the tests based on the same tags as above in
114 a machine-readable format by passing the `-q` flag. This will list all of the
115 *suites* that match the given tag(s).
118 ./main.py list -q --suites
119 SuiteUID:tests/gem5/hello_se/test_hello_se.py:testhello64-static-X86-opt
120 SuiteUID:tests/gem5/hello_se/test_hello_se.py:testhello64-dynamic-X86-opt
121 SuiteUID:tests/gem5/hello_se/test_hello_se.py:testhello32-static-X86-opt
122 SuiteUID:tests/gem5/hello_se/test_hello_se.py:testhello64-static-ARM-opt
123 SuiteUID:tests/gem5/hello_se/test_hello_se.py:testhello32-static-ARM-opt
124 SuiteUID:tests/gem5/m5_util/test_exit.py:m5_exit_test-X86-opt
125 SuiteUID:tests/gem5/test_build/test_build.py:build-X86-opt
126 SuiteUID:tests/gem5/test_build/test_build.py:build-RISCV-opt
127 SuiteUID:tests/gem5/test_build/test_build.py:build-ARM-opt
130 Next, you can run a single *suite* from the command line by passing the option
131 `--uid`. For instance,
134 ./main.py run --skip-build \
135 --uid SuiteUID:tests/gem5/m5_util/test_exit.py:m5_exit_test-X86-opt
138 With this method, you can only run a *single* suite at a time. If you want to
139 run more than one uid, you must call `./main.py` multiple times.
141 Currently, you must specify `--skip-build` if you want to run a single suite or
142 run in batch mode. Otherwise, you will build gem5 for all architectures.
144 ## Rerunning failed tests
146 While developing software a common practice is to run tests, make a change, and
147 assert that the tests still pass. If tests fail you'll likely want to
148 rerun and fix those specific tests without running redundant ones. The testing
149 infrastructure allows you to rerun tests which failed in the last execution by
150 using the `rerun` command.
158 # Rerun only the failed test suites (not the ones which passed).
162 ## If something goes wrong
164 The first step is to turn up the verbosity of the output using `-v`. This will
165 allow you to see what tests are running and why a test is failing.
167 If a test fails, the temporary directory where the gem5 output was saved is kept
168 and the path to the directory is printed in the terminal.
170 ## Debugging the testing infrastructure
172 Every command takes an option for the verbosity. `-v`, `-vv`, `-vvv` will
173 increase the verbosity level. If something isn't working correctly, you can
176 Most of the code for the testing infrastructure is in ext/testlib. This code
177 contains the base code for tests, suites, fixtures, etc. The code in tests/gem5
178 is *gem5-specific* code. For the most part, the code in tests/gem5 extends the
179 structures in ext/testlib.
183 You may see a number of lines of output during test discovery that look like
187 Tried to load tests from ... but failed with an exception.
188 Tried to load tests from ... but failed with an exception.
192 The testing library searches all python files in the `tests/` directory. The
193 test library executes each python file it finds searching for tests. It's okay
194 if the file causes an exception. This means there are no tests in that file
195 (e.g., it's not a new-style test).
198 ## Binary test applications
200 The code for test binaries that are run in the gem5 guest during testing are
201 found in `tests/test-progs`.
202 There's one directory per test application.
203 The source code is under the `source` directory.
205 You may have a `bin` directory as well.
206 The `bin` directory is automatically created when running the test case that
207 uses the test binary. The binary is downloaded from the gem5 servers the first
208 time it is referenced by a test.
210 ## Updating the test binaries
212 The test infrastructure should check with the gem5 servers to ensure you have
213 the latest binaries. However, if you believe your binaries are out of date,
214 simply delete the `bin` directory and they will be re-downloaded to your local
217 ## Building (new-style) test binaries
219 In each `src/` directory under `tests/test-progs`, there is a Makefile.
220 This Makefile downloads a docker image and builds the test binary for some ISA
221 (e.g., Makefile.x86 builds the binary for x86). Additionally, if you run `make
222 upload` it will upload the binaries to the gem5 server, if you have access to
223 modify the binaries. *If you need to modify the binaries for updating a test or
224 adding a new test and you don't have access to the gem5 server, contact a
225 maintainer (see MAINTAINERS).*
228 ## Running Tests in Parallel
230 Whimsy has support for parallel testing baked in. This system supports
231 running multiple suites at the same time on the same computer. To run
232 suites in parallel, supply the `-t <number-tests>` flag to the run command.
234 For example, to run up to three test suites at the same time::
236 ./main.py run --skip-build -t 3