TESTING.md

   1 This file explains how to use gem5's updated testing infrastructure. Running
   2 tests before submitting a patch is *incredibly important* so unexpected bugs
   3 don't creep into gem5.
   4
   5 gem5's testing infrastructure has the following goals:
   6  * Simple for *all* users to run
   7  * Fast execution in the simple case
   8  * High coverage of gem5 code
   9
  10 # Running unit tests
  11
  12 gem5 comes with unit tests, created using the Google Test framework. These can
  13 be built through SCons.
  14
  15 To build and run all the unit tests:
  16
  17 ```shell
  18 scons build/NULL/unittests.opt
  19 ```
  20
  21 All unit tests should be run prior to posting a patch to
  22 https://gem5-review.googlesource.com
  23
  24 To compile and run just one set of tests (e.g. those declared within
  25 `src/base/bitunion.test.cc`):
  26
  27 ```shell
  28 scons build/NULL/base/bitunion.test.opt
  29 ./build/NULL/base/bitunion.test.opt
  30 ```
  31
  32 To list the available test functions from a test file:
  33
  34 ```shell
  35 ./build/NULL/base/bitunion.test.opt --gtest_list_tests
  36 ```
  37
  38 To run a specific test function (e.g., BitUnionData.NormalBitfield):
  39
  40 ```shell
  41 ./build/NULL/base/bitunion.test.opt --gtest_filter=BitUnionData.NormalBitfield
  42 ```
  43
  44 # Running system-level tests
  45
  46 Within the `tests` directory we have system-level tests. These tests run
  47 the gem5 framework against various hardware configurations, with different
  48 ISAs, then verify the simulations execute correctly. These should be seen as
  49 high-level, coarse-grained tests to compliment the unit-tests.
  50
  51 Below is the most common way the tests are run. This will run all of the
  52 "quick" tests for X86, ARM, and RISC-V. These tests make up our best-supported
  53 platforms and use cases. When running these tests, you will likely want to us
  54 the option `-j <CPUs>` where `CPUs` is as large as you can make it.
  55 Additionally, it is often a good idea to run longer tests (e.g., linux boot)
  56 before submitting your patch.
  57
  58 ```shell
  59 cd tests
  60 ./main.py run
  61 ```
  62
  63 The above is the *minumum* you should run before posting a patch to
  64 https://gem5-review.googlesource.com
  65
  66 ## Specifying a subset of tests to run
  67
  68 You can use the tag query interface to specify the exact tests you want to run.
  69 For instance, if you want to run only with `gem5.opt`, you can use
  70
  71 ```shell
  72 ./main.py run --variant opt
  73 ```
  74
  75 Or, if you want to just run X86 tests with the `gem5.opt` binary:
  76
  77 ```shell
  78 ./main.py run --length quick --variant opt --isa X86
  79 ```
  80
  81
  82 To view all of the available tags, use
  83
  84 ```shell
  85 ./main.py list --all-tags
  86 ```
  87
  88 The output is split into tag *types* (e.g., isa, variant, length) and the
  89 tags for each type are listed after the type name.
  90
  91 You can specify "or" between tags within the same type by using the tag flag
  92 multiple times. For instance, to run everything that is tagged "opt" or "fast"
  93 use
  94
  95 ```shell
  96 ./main.py run --variant opt --variant fast
  97 ```
  98
  99 You can also specify "and" between different types of tags by specifying more
 100 than one type on the command line. For instance, this will only run tests with
 101 both the "X86" and "opt" tags.
 102
 103 ```shell
 104 ./main.py run --isa X86 --variant opt
 105 ```
 106
 107 ## Running tests in batch
 108
 109 The testing infrastructure provides the two needed methods to run tests in
 110 batch. First, you can list all of the tests based on the same tags as above in
 111 a machine-readable format by passing the `-q` flag. This will list all of the
 112 *suites* that match the given tag(s).
 113
 114 ```shell
 115 ./main.py list -q --suites
 116 SuiteUID:tests/gem5/hello_se/test_hello_se.py:testhello64-static-X86-opt
 117 SuiteUID:tests/gem5/hello_se/test_hello_se.py:testhello64-dynamic-X86-opt
 118 SuiteUID:tests/gem5/hello_se/test_hello_se.py:testhello32-static-X86-opt
 119 SuiteUID:tests/gem5/hello_se/test_hello_se.py:testhello64-static-ARM-opt
 120 SuiteUID:tests/gem5/hello_se/test_hello_se.py:testhello32-static-ARM-opt
 121 SuiteUID:tests/gem5/m5_util/test_exit.py:m5_exit_test-X86-opt
 122 SuiteUID:tests/gem5/test_build/test_build.py:build-X86-opt
 123 SuiteUID:tests/gem5/test_build/test_build.py:build-RISCV-opt
 124 SuiteUID:tests/gem5/test_build/test_build.py:build-ARM-opt
 125 ```
 126
 127 Next, you can run a single *suite* from the command line by passing the option
 128 `--uid`. For instance,
 129
 130 ```shell
 131 ./main.py run --skip-build \
 132     --uid SuiteUID:tests/gem5/m5_util/test_exit.py:m5_exit_test-X86-opt
 133 ```
 134
 135 With this method, you can only run a *single* suite at a time. If you want to
 136 run more than one uid, you must call `./main.py` multiple times.
 137
 138 Currently, you must specify `--skip-build` if you want to run a single suite or
 139 run in batch mode. Otherwise, you will build gem5 for all architectures.
 140
 141 ## Rerunning failed tests
 142
 143 While developing software a common practice is to run tests, make a change, and
 144 assert that the tests still pass. If tests fail you'll likely want to
 145 rerun and fix those specific tests without running redundant ones. The testing
 146 infrastructure allows you to rerun tests which failed in the last execution by
 147 using the `rerun` command.
 148
 149 ```shell
 150 ./main.py run
 151 #
 152 #  Some tests fail...
 153 #
 154
 155 # Rerun only the failed test suites (not the ones which passed).
 156 ./main.py rerun
 157 ```
 158
 159 ## If something goes wrong
 160
 161 The first step is to turn up the verbosity of the output using `-v`. This will
 162 allow you to see what tests are running and why a test is failing.
 163
 164 If a test fails, the temporary directory where the gem5 output was saved is kept
 165 and the path to the directory is printed in the terminal.
 166
 167 ## Debugging the testing infrastructure
 168
 169 Every command takes an option for the verbosity. `-v`, `-vv`, `-vvv` will
 170 increase the verbosity level. If something isn't working correctly, you can
 171 start here.
 172
 173 Most of the code for the testing infrastructure is in ext/testlib. This code
 174 contains the base code for tests, suites, fixtures, etc. The code in tests/gem5
 175 is *gem5-specific* code. For the most part, the code in tests/gem5 extends the
 176 structures in ext/testlib.
 177
 178 ## Common errors
 179
 180 You may see a number of lines of output during test discovery that look like
 181 the following:
 182
 183 ```shell
 184     Tried to load tests from ... but failed with an exception.
 185     Tried to load tests from ... but failed with an exception.
 186     ...
 187 ```
 188
 189 The testing library searches all python files in the `tests/` directory. The
 190 test library executes each python file it finds searching for tests. It's okay
 191 if the file causes an exception. This means there are no tests in that file
 192 (e.g., it's not a new-style test).
 193
 194
 195 ## Binary test applications
 196
 197 The code for some test binaries that are run in the gem5 guest during
 198 testing can be found in `tests/test-progs`.
 199 There's one directory per test application.
 200 The source code is under the `source` directory.
 201
 202 You may have a `bin` directory as well.
 203 The `bin` directory is automatically created when running the test case that
 204 uses the test binary.
 205 This is not the case when a test is run via the --bin-path option.
 206 In that scenario a bin directory will be created in the selected path
 207 rather than in `tests/test-progs`.
 208 The binary is downloaded from the gem5 servers the first
 209 time it is referenced by a test.
 210
 211 Some other tests (like Linux-boot) don't have sources inside gem5 and
 212 are simply downloaded from gem5 servers.
 213
 214 ## Updating the test binaries
 215
 216 The test infrastructure should check with the gem5 servers to ensure you have
 217 the latest binaries. However, if you believe your binaries are out of date,
 218 simply delete the `bin` directory and they will be re-downloaded to your local
 219 machine.
 220
 221 ## Building (new-style) test binaries
 222
 223 In each `src/` directory under `tests/test-progs`, there is a Makefile.
 224 This Makefile downloads a docker image and builds the test binary for some ISA
 225 (e.g., Makefile.x86 builds the binary for x86). Additionally, if you run `make
 226 upload` it will upload the binaries to the gem5 server, if you have access to
 227 modify the binaries. *If you need to modify the binaries for updating a test or
 228 adding a new test and you don't have access to the gem5 server, contact a
 229 maintainer (see MAINTAINERS).*
 230
 231
 232 ## Running Tests in Parallel
 233
 234 Whimsy has support for parallel testing baked in. This system supports
 235 running multiple suites at the same time on the same computer. To run
 236 suites in parallel, supply the `-t <number-tests>` flag to the run command.
 237
 238 For example, to run up to three test suites at the same time::
 239
 240     ./main.py run --skip-build -t 3
 241