TESTING.md

   1 This file explains how to use gem5's updated testing infrastructure. Running
   2 tests before submitting a patch is *incredibly important* so unexpected bugs
   3 don't creep into gem5.
   4
   5 gem5's testing infrastructure has the following goals:
   6  * Simple for *all* users to run
   7  * Fast execution in the simple case
   8  * High coverage of gem5 code
   9
  10 # Running unit tests
  11
  12 gem5 comes with unit tests, created using the Google Test framework. These can
  13 be built through SCons.
  14
  15 To build and run all the unit tests:
  16
  17 ```shell
  18 scons build/NULL/unittests.opt
  19 ```
  20
  21 All unit tests should be run prior to posting a patch to
  22 https://gem5-review.googlesource.com
  23
  24 To compile and run just one set of tests (e.g. those declared within
  25 `src/base/bitunion.test.cc`):
  26
  27 ```shell
  28 scons build/NULL/base/bitunion.test.opt
  29 ./build/NULL/base/bitunion.test.opt
  30 ```
  31
  32 To list the available test functions from a test file:
  33
  34 ```shell
  35 ./build/NULL/base/bitunion.test.opt --gtest_list_tests
  36 ```
  37
  38 To run a specific test function (e.g., BitUnionData.NormalBitfield):
  39
  40 ```shell
  41 ./build/NULL/base/bitunion.test.opt --gtest_filter=BitUnionData.NormalBitfield
  42 ```
  43
  44 # Running system-level tests
  45
  46 Within the `tests` directory we have system-level tests. These tests run
  47 the gem5 framework against various hardware configurations, with different
  48 ISAs, then verify the simulations execute correctly. These should be seen as
  49 high-level, coarse-grained tests to compliment the unit-tests.
  50
  51 Below is the most common way the tests are run. This will run all of the
  52 "quick" tests for X86, ARM, and RISC-V. These tests make up our best-supported
  53 platforms and use cases. When running these tests, you will likely want to us
  54 the option `-j <CPUs>` where `CPUs` is as large as you can make it.
  55 Additionally, it is often a good idea to run longer tests (e.g., linux boot)
  56 before submitting your patch.
  57
  58 ```shell
  59 cd tests
  60 ./main.py run
  61 ```
  62
  63 The above is the *minumum* you should run before posting a patch to
  64 https://gem5-review.googlesource.com
  65
  66 ## Running tests from multiple directories
  67
  68 The command line above will walk the directory tree starting from the cwd
  69 (tests), and it will run every test it encounters in its path. It is possible
  70 to specify multiple root directories by providing several positional
  71 arguments:
  72
  73 ```shell
  74 ./main.py run <directory1> <directory2> [...]
  75 ```
  76
  77 This will load every test in directory1 and directory2 (and their
  78 subdirectories).
  79
  80 ## Specifying a subset of tests to run
  81
  82 You can use the tag query interface to specify the exact tests you want to run.
  83 For instance, if you want to run only with `gem5.opt`, you can use
  84
  85 ```shell
  86 ./main.py run --variant opt
  87 ```
  88
  89 Or, if you want to just run X86 tests with the `gem5.opt` binary:
  90
  91 ```shell
  92 ./main.py run --length quick --variant opt --isa X86
  93 ```
  94
  95
  96 To view all of the available tags, use
  97
  98 ```shell
  99 ./main.py list --all-tags
 100 ```
 101
 102 The output is split into tag *types* (e.g., isa, variant, length) and the
 103 tags for each type are listed after the type name.
 104
 105 You can specify "or" between tags within the same type by using the tag flag
 106 multiple times. For instance, to run everything that is tagged "opt" or "fast"
 107 use
 108
 109 ```shell
 110 ./main.py run --variant opt --variant fast
 111 ```
 112
 113 You can also specify "and" between different types of tags by specifying more
 114 than one type on the command line. For instance, this will only run tests with
 115 both the "X86" and "opt" tags.
 116
 117 ```shell
 118 ./main.py run --isa X86 --variant opt
 119 ```
 120
 121 ## Running tests in batch
 122
 123 The testing infrastructure provides the two needed methods to run tests in
 124 batch. First, you can list all of the tests based on the same tags as above in
 125 a machine-readable format by passing the `-q` flag. This will list all of the
 126 *suites* that match the given tag(s).
 127
 128 ```shell
 129 ./main.py list -q --suites
 130 SuiteUID:tests/gem5/hello_se/test_hello_se.py:testhello64-static-X86-opt
 131 SuiteUID:tests/gem5/hello_se/test_hello_se.py:testhello64-dynamic-X86-opt
 132 SuiteUID:tests/gem5/hello_se/test_hello_se.py:testhello32-static-X86-opt
 133 SuiteUID:tests/gem5/hello_se/test_hello_se.py:testhello64-static-ARM-opt
 134 SuiteUID:tests/gem5/hello_se/test_hello_se.py:testhello32-static-ARM-opt
 135 SuiteUID:tests/gem5/m5_util/test_exit.py:m5_exit_test-X86-opt
 136 SuiteUID:tests/gem5/test_build/test_build.py:build-X86-opt
 137 SuiteUID:tests/gem5/test_build/test_build.py:build-RISCV-opt
 138 SuiteUID:tests/gem5/test_build/test_build.py:build-ARM-opt
 139 ```
 140
 141 Next, you can run a single *suite* from the command line by passing the option
 142 `--uid`. For instance,
 143
 144 ```shell
 145 ./main.py run --skip-build \
 146     --uid SuiteUID:tests/gem5/m5_util/test_exit.py:m5_exit_test-X86-opt
 147 ```
 148
 149 With this method, you can only run a *single* suite at a time. If you want to
 150 run more than one uid, you must call `./main.py` multiple times.
 151
 152 Currently, you must specify `--skip-build` if you want to run a single suite or
 153 run in batch mode. Otherwise, you will build gem5 for all architectures.
 154
 155 ## Rerunning failed tests
 156
 157 While developing software a common practice is to run tests, make a change, and
 158 assert that the tests still pass. If tests fail you'll likely want to
 159 rerun and fix those specific tests without running redundant ones. The testing
 160 infrastructure allows you to rerun tests which failed in the last execution by
 161 using the `rerun` command.
 162
 163 ```shell
 164 ./main.py run
 165 #
 166 #  Some tests fail...
 167 #
 168
 169 # Rerun only the failed test suites (not the ones which passed).
 170 ./main.py rerun
 171 ```
 172
 173 ## If something goes wrong
 174
 175 The first step is to turn up the verbosity of the output using `-v`. This will
 176 allow you to see what tests are running and why a test is failing.
 177
 178 If a test fails, the temporary directory where the gem5 output was saved is kept
 179 and the path to the directory is printed in the terminal.
 180
 181 ## Debugging the testing infrastructure
 182
 183 Every command takes an option for the verbosity. `-v`, `-vv`, `-vvv` will
 184 increase the verbosity level. If something isn't working correctly, you can
 185 start here.
 186
 187 Most of the code for the testing infrastructure is in ext/testlib. This code
 188 contains the base code for tests, suites, fixtures, etc. The code in tests/gem5
 189 is *gem5-specific* code. For the most part, the code in tests/gem5 extends the
 190 structures in ext/testlib.
 191
 192 ## Common errors
 193
 194 You may see a number of lines of output during test discovery that look like
 195 the following:
 196
 197 ```shell
 198     Tried to load tests from ... but failed with an exception.
 199     Tried to load tests from ... but failed with an exception.
 200     ...
 201 ```
 202
 203 The testing library searches all python files in the `tests/` directory. The
 204 test library executes each python file it finds searching for tests. It's okay
 205 if the file causes an exception. This means there are no tests in that file
 206 (e.g., it's not a new-style test).
 207
 208
 209 ## Binary test applications
 210
 211 The code for some test binaries that are run in the gem5 guest during
 212 testing can be found in `tests/test-progs`.
 213 There's one directory per test application.
 214 The source code is under the `source` directory.
 215
 216 You may have a `bin` directory as well.
 217 The `bin` directory is automatically created when running the test case that
 218 uses the test binary.
 219 This is not the case when a test is run via the --bin-path option.
 220 In that scenario a bin directory will be created in the selected path
 221 rather than in `tests/test-progs`.
 222 The binary is downloaded from the gem5 servers the first
 223 time it is referenced by a test.
 224
 225 Some other tests (like Linux-boot) don't have sources inside gem5 and
 226 are simply downloaded from gem5 servers.
 227
 228 ## Updating the test binaries
 229
 230 The test infrastructure should check with the gem5 servers to ensure you have
 231 the latest binaries. However, if you believe your binaries are out of date,
 232 simply delete the `bin` directory and they will be re-downloaded to your local
 233 machine.
 234
 235 ## Building (new-style) test binaries
 236
 237 In each `src/` directory under `tests/test-progs`, there is a Makefile.
 238 This Makefile downloads a docker image and builds the test binary for some ISA
 239 (e.g., Makefile.x86 builds the binary for x86). Additionally, if you run `make
 240 upload` it will upload the binaries to the gem5 server, if you have access to
 241 modify the binaries. *If you need to modify the binaries for updating a test or
 242 adding a new test and you don't have access to the gem5 server, contact a
 243 maintainer (see MAINTAINERS).*
 244
 245
 246 ## Running Tests in Parallel
 247
 248 Whimsy has support for parallel testing baked in. This system supports
 249 running multiple suites at the same time on the same computer. To run
 250 suites in parallel, supply the `-t <number-tests>` flag to the run command.
 251
 252 For example, to run up to three test suites at the same time::
 253
 254     ./main.py run --skip-build -t 3
 255