tests: arch-power: Add 64-bit hello binaries
[gem5.git] / TESTING.md
1 This file explains how to use gem5's updated testing infrastructure. Running
2 tests before submitting a patch is *incredibly important* so unexpected bugs
3 don't creep into gem5.
5 gem5's testing infrastructure has the following goals:
6 * Simple for *all* users to run
7 * Fast execution in the simple case
8 * High coverage of gem5 code
10 # Running unit tests
12 gem5 comes with unit tests, created using the Google Test framework. These can
13 be built through SCons.
15 To build and run all the unit tests:
17 ```shell
18 scons build/NULL/unittests.opt
19 ```
21 All unit tests should be run prior to posting a patch to
22 https://gem5-review.googlesource.com
24 To compile and run just one set of tests (e.g. those declared within
25 `src/base/bitunion.test.cc`):
27 ```shell
28 scons build/NULL/base/bitunion.test.opt
29 ./build/NULL/base/bitunion.test.opt
30 ```
32 To list the available test functions from a test file:
34 ```shell
35 ./build/NULL/base/bitunion.test.opt --gtest_list_tests
36 ```
38 To run a specific test function (e.g., BitUnionData.NormalBitfield):
40 ```shell
41 ./build/NULL/base/bitunion.test.opt --gtest_filter=BitUnionData.NormalBitfield
42 ```
44 # Running system-level tests
46 Within the `tests` directory we have system-level tests. These tests run
47 the gem5 framework against various hardware configurations, with different
48 ISAs, then verify the simulations execute correctly. These should be seen as
49 high-level, coarse-grained tests to compliment the unit-tests.
51 Below is the most common way the tests are run. This will run all of the
52 "quick" tests for X86, ARM, and RISC-V. These tests make up our best-supported
53 platforms and use cases. When running these tests, you will likely want to us
54 the option `-j <CPUs>` where `CPUs` is as large as you can make it.
55 Additionally, it is often a good idea to run longer tests (e.g., linux boot)
56 before submitting your patch.
58 ```shell
59 cd tests
60 ./main.py run
61 ```
63 The above is the *minumum* you should run before posting a patch to
64 https://gem5-review.googlesource.com
66 ## Running tests from multiple directories
68 The command line above will walk the directory tree starting from the cwd
69 (tests), and it will run every test it encounters in its path. It is possible
70 to specify multiple root directories by providing several positional
71 arguments:
73 ```shell
74 ./main.py run <directory1> <directory2> [...]
75 ```
77 This will load every test in directory1 and directory2 (and their
78 subdirectories).
80 ## Specifying a subset of tests to run
82 You can use the tag query interface to specify the exact tests you want to run.
83 For instance, if you want to run only with `gem5.opt`, you can use
85 ```shell
86 ./main.py run --variant opt
87 ```
89 Or, if you want to just run X86 tests with the `gem5.opt` binary:
91 ```shell
92 ./main.py run --length quick --variant opt --isa X86
93 ```
96 To view all of the available tags, use
98 ```shell
99 ./main.py list --all-tags
100 ```
102 The output is split into tag *types* (e.g., isa, variant, length) and the
103 tags for each type are listed after the type name.
105 You can specify "or" between tags within the same type by using the tag flag
106 multiple times. For instance, to run everything that is tagged "opt" or "fast"
107 use
109 ```shell
110 ./main.py run --variant opt --variant fast
111 ```
113 You can also specify "and" between different types of tags by specifying more
114 than one type on the command line. For instance, this will only run tests with
115 both the "X86" and "opt" tags.
117 ```shell
118 ./main.py run --isa X86 --variant opt
119 ```
121 ## Running tests in batch
123 The testing infrastructure provides the two needed methods to run tests in
124 batch. First, you can list all of the tests based on the same tags as above in
125 a machine-readable format by passing the `-q` flag. This will list all of the
126 *suites* that match the given tag(s).
128 ```shell
129 ./main.py list -q --suites
130 SuiteUID:tests/gem5/hello_se/test_hello_se.py:testhello64-static-X86-opt
131 SuiteUID:tests/gem5/hello_se/test_hello_se.py:testhello64-dynamic-X86-opt
132 SuiteUID:tests/gem5/hello_se/test_hello_se.py:testhello32-static-X86-opt
133 SuiteUID:tests/gem5/hello_se/test_hello_se.py:testhello64-static-ARM-opt
134 SuiteUID:tests/gem5/hello_se/test_hello_se.py:testhello32-static-ARM-opt
135 SuiteUID:tests/gem5/m5_util/test_exit.py:m5_exit_test-X86-opt
136 SuiteUID:tests/gem5/test_build/test_build.py:build-X86-opt
137 SuiteUID:tests/gem5/test_build/test_build.py:build-RISCV-opt
138 SuiteUID:tests/gem5/test_build/test_build.py:build-ARM-opt
139 ```
141 Next, you can run a single *suite* from the command line by passing the option
142 `--uid`. For instance,
144 ```shell
145 ./main.py run --skip-build \
146 --uid SuiteUID:tests/gem5/m5_util/test_exit.py:m5_exit_test-X86-opt
147 ```
149 With this method, you can only run a *single* suite at a time. If you want to
150 run more than one uid, you must call `./main.py` multiple times.
152 Currently, you must specify `--skip-build` if you want to run a single suite or
153 run in batch mode. Otherwise, you will build gem5 for all architectures.
155 ## Rerunning failed tests
157 While developing software a common practice is to run tests, make a change, and
158 assert that the tests still pass. If tests fail you'll likely want to
159 rerun and fix those specific tests without running redundant ones. The testing
160 infrastructure allows you to rerun tests which failed in the last execution by
161 using the `rerun` command.
163 ```shell
164 ./main.py run
165 #
166 # Some tests fail...
167 #
169 # Rerun only the failed test suites (not the ones which passed).
170 ./main.py rerun
171 ```
173 ## If something goes wrong
175 The first step is to turn up the verbosity of the output using `-v`. This will
176 allow you to see what tests are running and why a test is failing.
178 If a test fails, the temporary directory where the gem5 output was saved is kept
179 and the path to the directory is printed in the terminal.
181 ## Debugging the testing infrastructure
183 Every command takes an option for the verbosity. `-v`, `-vv`, `-vvv` will
184 increase the verbosity level. If something isn't working correctly, you can
185 start here.
187 Most of the code for the testing infrastructure is in ext/testlib. This code
188 contains the base code for tests, suites, fixtures, etc. The code in tests/gem5
189 is *gem5-specific* code. For the most part, the code in tests/gem5 extends the
190 structures in ext/testlib.
192 ## Common errors
194 You may see a number of lines of output during test discovery that look like
195 the following:
197 ```shell
198 Tried to load tests from ... but failed with an exception.
199 Tried to load tests from ... but failed with an exception.
200 ...
201 ```
203 The testing library searches all python files in the `tests/` directory. The
204 test library executes each python file it finds searching for tests. It's okay
205 if the file causes an exception. This means there are no tests in that file
206 (e.g., it's not a new-style test).
209 ## Binary test applications
211 The code for some test binaries that are run in the gem5 guest during
212 testing can be found in `tests/test-progs`.
213 There's one directory per test application.
214 The source code is under the `source` directory.
216 You may have a `bin` directory as well.
217 The `bin` directory is automatically created when running the test case that
218 uses the test binary.
219 This is not the case when a test is run via the --bin-path option.
220 In that scenario a bin directory will be created in the selected path
221 rather than in `tests/test-progs`.
222 The binary is downloaded from the gem5 servers the first
223 time it is referenced by a test.
225 Some other tests (like Linux-boot) don't have sources inside gem5 and
226 are simply downloaded from gem5 servers.
228 ## Updating the test binaries
230 The test infrastructure should check with the gem5 servers to ensure you have
231 the latest binaries. However, if you believe your binaries are out of date,
232 simply delete the `bin` directory and they will be re-downloaded to your local
233 machine.
235 ## Building (new-style) test binaries
237 In each `src/` directory under `tests/test-progs`, there is a Makefile.
238 This Makefile downloads a docker image and builds the test binary for some ISA
239 (e.g., Makefile.x86 builds the binary for x86). Additionally, if you run `make
240 upload` it will upload the binaries to the gem5 server, if you have access to
241 modify the binaries. *If you need to modify the binaries for updating a test or
242 adding a new test and you don't have access to the gem5 server, contact a
243 maintainer (see MAINTAINERS).*
246 ## Running Tests in Parallel
248 Whimsy has support for parallel testing baked in. This system supports
249 running multiple suites at the same time on the same computer. To run
250 suites in parallel, supply the `-t <number-tests>` flag to the run command.
252 For example, to run up to three test suites at the same time::
254 ./main.py run --skip-build -t 3