TESTING.md

   1 :Authors: Jason Lowe-Power
   2           Sean Wilson
   3
   4 This file explains how to use gem5's updated testing infrastructure. Running
   5 tests before submitting a patch is *incredibly important* so unexpected bugs
   6 don't creep into gem5.
   7
   8 gem5's testing infrastructure has the following goals:
   9  * Simple for *all* users to run
  10  * Fast execution in the simple case
  11  * High coverage of gem5 code
  12
  13 # Running unit tests
  14
  15 gem5 comes with unit tests, created using the Google Test framework. These can
  16 be built through SCons.
  17
  18 To build and run all the unit tests:
  19
  20 ```shell
  21 scons build/NULL/unittests.opt
  22 ```
  23
  24 All unit tests should be run prior to posting a patch to
  25 https://gem5-review.googlesource.com
  26
  27 To compile and run just one set of tests (e.g. those declared within
  28 `src/base/bitunion.test.cc`):
  29
  30 ```shell
  31 scons build/NULL/base/bitunion.test.opt
  32 ./build/NULL/base/bitunion.test.opt
  33 ```
  34
  35 To list the available test functions from a test file:
  36
  37 ```shell
  38 ./build/NULL/base/bitunion.test.opt --gtest_list_tests
  39 ```
  40
  41 To run a specific test function (e.g., BitUnionData.NormalBitfield):
  42
  43 ```shell
  44 ./build/NULL/base/bitunion.test.opt --gtest_filter=BitUnionData.NormalBitfield
  45 ```
  46
  47 # Running system-level tests
  48
  49 Within the `tests` directory we have system-level tests. These tests run
  50 the gem5 framework against various hardware configurations, with different
  51 ISAs, then verify the simulations execute correctly. These should be seen as
  52 high-level, coarse-grained tests to compliment the unit-tests.
  53
  54 Below is the most common way the tests are run. This will run all of the
  55 "quick" tests for X86, ARM, and RISC-V. These tests make up our best-supported
  56 platforms and use cases. When running these tests, you will likely want to us
  57 the option `-j <CPUs>` where `CPUs` is as large as you can make it.
  58 Additionally, it is often a good idea to run longer tests (e.g., linux boot)
  59 before submitting your patch.
  60
  61 ```shell
  62 cd tests
  63 ./main.py run
  64 ```
  65
  66 The above is the *minumum* you should run before posting a patch to
  67 https://gem5-review.googlesource.com
  68
  69 ## Specifying a subset of tests to run
  70
  71 You can use the tag query interface to specify the exact tests you want to run.
  72 For instance, if you want to run only with `gem5.opt`, you can use
  73
  74 ```shell
  75 ./main.py run --variant opt
  76 ```
  77
  78 Or, if you want to just run X86 tests with the `gem5.opt` binary:
  79
  80 ```shell
  81 ./main.py run --length quick --variant opt --isa X86
  82 ```
  83
  84
  85 To view all of the available tags, use
  86
  87 ```shell
  88 ./main.py list --all-tags
  89 ```
  90
  91 The output is split into tag *types* (e.g., isa, variant, length) and the
  92 tags for each type are listed after the type name.
  93
  94 You can specify "or" between tags within the same type by using the tag flag
  95 multiple times. For instance, to run everything that is tagged "opt" or "fast"
  96 use
  97
  98 ```shell
  99 ./main.py run --variant opt --variant fast
 100 ```
 101
 102 You can also specify "and" between different types of tags by specifying more
 103 than one type on the command line. For instance, this will only run tests with
 104 both the "X86" and "opt" tags.
 105
 106 ```shell
 107 ./main.py run --isa X86 --variant opt
 108 ```
 109
 110 ## Running tests in batch
 111
 112 The testing infrastructure provides the two needed methods to run tests in
 113 batch. First, you can list all of the tests based on the same tags as above in
 114 a machine-readable format by passing the `-q` flag. This will list all of the
 115 *suites* that match the given tag(s).
 116
 117 ```shell
 118 ./main.py list -q --suites
 119 SuiteUID:tests/gem5/hello_se/test_hello_se.py:testhello64-static-X86-opt
 120 SuiteUID:tests/gem5/hello_se/test_hello_se.py:testhello64-dynamic-X86-opt
 121 SuiteUID:tests/gem5/hello_se/test_hello_se.py:testhello32-static-X86-opt
 122 SuiteUID:tests/gem5/hello_se/test_hello_se.py:testhello64-static-ARM-opt
 123 SuiteUID:tests/gem5/hello_se/test_hello_se.py:testhello32-static-ARM-opt
 124 SuiteUID:tests/gem5/m5_util/test_exit.py:m5_exit_test-X86-opt
 125 SuiteUID:tests/gem5/test_build/test_build.py:build-X86-opt
 126 SuiteUID:tests/gem5/test_build/test_build.py:build-RISCV-opt
 127 SuiteUID:tests/gem5/test_build/test_build.py:build-ARM-opt
 128 ```
 129
 130 Next, you can run a single *suite* from the command line by passing the option
 131 `--uid`. For instance,
 132
 133 ```shell
 134 ./main.py run --skip-build \
 135     --uid SuiteUID:tests/gem5/m5_util/test_exit.py:m5_exit_test-X86-opt
 136 ```
 137
 138 With this method, you can only run a *single* suite at a time. If you want to
 139 run more than one uid, you must call `./main.py` multiple times.
 140
 141 Currently, you must specify `--skip-build` if you want to run a single suite or
 142 run in batch mode. Otherwise, you will build gem5 for all architectures.
 143
 144 ## Rerunning failed tests
 145
 146 While developing software a common practice is to run tests, make a change, and
 147 assert that the tests still pass. If tests fail you'll likely want to
 148 rerun and fix those specific tests without running redundant ones. The testing
 149 infrastructure allows you to rerun tests which failed in the last execution by
 150 using the `rerun` command.
 151
 152 ```shell
 153 ./main.py run
 154 #
 155 #  Some tests fail...
 156 #
 157
 158 # Rerun only the failed test suites (not the ones which passed).
 159 ./main.py rerun
 160 ```
 161
 162 ## If something goes wrong
 163
 164 The first step is to turn up the verbosity of the output using `-v`. This will
 165 allow you to see what tests are running and why a test is failing.
 166
 167 If a test fails, the temporary directory where the gem5 output was saved is kept
 168 and the path to the directory is printed in the terminal.
 169
 170 ## Debugging the testing infrastructure
 171
 172 Every command takes an option for the verbosity. `-v`, `-vv`, `-vvv` will
 173 increase the verbosity level. If something isn't working correctly, you can
 174 start here.
 175
 176 Most of the code for the testing infrastructure is in ext/testlib. This code
 177 contains the base code for tests, suites, fixtures, etc. The code in tests/gem5
 178 is *gem5-specific* code. For the most part, the code in tests/gem5 extends the
 179 structures in ext/testlib.
 180
 181 ## Common errors
 182
 183 You may see a number of lines of output during test discovery that look like
 184 the following:
 185
 186 ```shell
 187     Tried to load tests from ... but failed with an exception.
 188     Tried to load tests from ... but failed with an exception.
 189     ...
 190 ```
 191
 192 The testing library searches all python files in the `tests/` directory. The
 193 test library executes each python file it finds searching for tests. It's okay
 194 if the file causes an exception. This means there are no tests in that file
 195 (e.g., it's not a new-style test).
 196
 197
 198 ## Binary test applications
 199
 200 The code for some test binaries that are run in the gem5 guest during
 201 testing can be found in `tests/test-progs`.
 202 There's one directory per test application.
 203 The source code is under the `source` directory.
 204
 205 You may have a `bin` directory as well.
 206 The `bin` directory is automatically created when running the test case that
 207 uses the test binary.
 208 This is not the case when a test is run via the --bin-path option.
 209 In that scenario a bin directory will be created in the selected path
 210 rather than in `tests/test-progs`.
 211 The binary is downloaded from the gem5 servers the first
 212 time it is referenced by a test.
 213
 214 Some other tests (like Linux-boot) don't have sources inside gem5 and
 215 are simply downloaded from gem5 servers.
 216
 217 ## Updating the test binaries
 218
 219 The test infrastructure should check with the gem5 servers to ensure you have
 220 the latest binaries. However, if you believe your binaries are out of date,
 221 simply delete the `bin` directory and they will be re-downloaded to your local
 222 machine.
 223
 224 ## Building (new-style) test binaries
 225
 226 In each `src/` directory under `tests/test-progs`, there is a Makefile.
 227 This Makefile downloads a docker image and builds the test binary for some ISA
 228 (e.g., Makefile.x86 builds the binary for x86). Additionally, if you run `make
 229 upload` it will upload the binaries to the gem5 server, if you have access to
 230 modify the binaries. *If you need to modify the binaries for updating a test or
 231 adding a new test and you don't have access to the gem5 server, contact a
 232 maintainer (see MAINTAINERS).*
 233
 234
 235 ## Running Tests in Parallel
 236
 237 Whimsy has support for parallel testing baked in. This system supports
 238 running multiple suites at the same time on the same computer. To run
 239 suites in parallel, supply the `-t <number-tests>` flag to the run command.
 240
 241 For example, to run up to three test suites at the same time::
 242
 243     ./main.py run --skip-build -t 3
 244