TESTING.md

   1 :Authors: Jason Lowe-Power
   2           Sean Wilson
   3
   4 This file explains how to use gem5's updated testing infrastructure. Running
   5 tests before submitting a patch is *incredibly important* so unexpected bugs
   6 don't creep into gem5.
   7
   8 gem5's testing infrastructure has the following goals:
   9  * Simple for *all* users to run
  10  * Fast execution in the simple case
  11  * High coverage of gem5 code
  12
  13 # Running tests
  14
  15 Below is the most common way the tests are run. This will run all of the
  16 "quick" tests for X86, ARM, and RISC-V. These tests make up our best-supported
  17 platforms and use cases. When running these tests, you will likely want to us
  18 the option `-j <CPUs>` where `CPUs` is as large as you can make it.
  19 Additionally, it is often a good idea to run longer tests (e.g., linux boot)
  20 before submitting your patch.
  21
  22 ```shell
  23 cd tests
  24 ./main.py run
  25 ```
  26
  27 The above is the *minumum* you should run before posting a patch to
  28 https://gem5-review.googlesource.com
  29
  30 ## Specifying a subset of tests to run
  31
  32 You can use the tag query interface to specify the exact tests you want to run.
  33 For instance, if you want to run only with `gem5.opt`, you can use
  34
  35 ```shell
  36 ./main.py run --variant opt
  37 ```
  38
  39 Or, if you want to just run X86 tests with the `gem5.opt` binary:
  40
  41 ```shell
  42 ./main.py run --length quick --variant opt --isa X86
  43 ```
  44
  45
  46 To view all of the available tags, use
  47
  48 ```shell
  49 ./main.py list --all-tags
  50 ```
  51
  52 The output is split into tag *types* (e.g., isa, variant, length) and the
  53 tags for each type are listed after the type name.
  54
  55 You can specify "or" between tags within the same type by using the tag flag
  56 multiple times. For instance, to run everything that is tagged "opt" or "fast"
  57 use
  58
  59 ```shell
  60 ./main.py run --variant opt --variant fast
  61 ```
  62
  63 You can also specify "and" between different types of tags by specifying more
  64 than one type on the command line. For instance, this will only run tests with
  65 both the "X86" and "opt" tags.
  66
  67 ```shell
  68 ./main.py run --isa X86 --variant opt
  69 ```
  70
  71 ## Running tests in batch
  72
  73 The testing infrastructure provides the two needed methods to run tests in
  74 batch. First, you can list all of the tests based on the same tags as above in
  75 a machine-readable format by passing the `-q` flag. This will list all of the
  76 *suites* that match the given tag(s).
  77
  78 ```shell
  79 ./main.py list -q --suites
  80 SuiteUID:tests/gem5/hello_se/test_hello_se.py:testhello64-static-X86-opt
  81 SuiteUID:tests/gem5/hello_se/test_hello_se.py:testhello64-dynamic-X86-opt
  82 SuiteUID:tests/gem5/hello_se/test_hello_se.py:testhello32-static-X86-opt
  83 SuiteUID:tests/gem5/hello_se/test_hello_se.py:testhello64-static-ARM-opt
  84 SuiteUID:tests/gem5/hello_se/test_hello_se.py:testhello32-static-ARM-opt
  85 SuiteUID:tests/gem5/m5_util/test_exit.py:m5_exit_test-X86-opt
  86 SuiteUID:tests/gem5/test_build/test_build.py:build-X86-opt
  87 SuiteUID:tests/gem5/test_build/test_build.py:build-RISCV-opt
  88 SuiteUID:tests/gem5/test_build/test_build.py:build-ARM-opt
  89 ```
  90
  91 Next, you can run a single *suite* from the command line by passing the option
  92 `--uid`. For instance,
  93
  94 ```shell
  95 ./main.py run --skip-build \
  96     --uid SuiteUID:tests/gem5/m5_util/test_exit.py:m5_exit_test-X86-opt
  97 ```
  98
  99 With this method, you can only run a *single* suite at a time. If you want to
 100 run more than one uid, you must call `./main.py` multiple times.
 101
 102 Currently, you must specify `--skip-build` if you want to run a single suite or
 103 run in batch mode. Otherwise, you will build gem5 for all architectures.
 104
 105 ## Rerunning failed tests
 106
 107 While developing software a common practice is to run tests, make a change, and
 108 assert that the tests still pass. If tests fail you'll likely want to
 109 rerun and fix those specific tests without running redundant ones. The testing
 110 infrastructure allows you to rerun tests which failed in the last execution by
 111 using the `rerun` command.
 112
 113 ```shell
 114 ./main.py run
 115 #
 116 #  Some tests fail...
 117 #
 118
 119 # Rerun only the failed test suites (not the ones which passed).
 120 ./main.py rerun
 121 ```
 122
 123 # If something goes wrong
 124
 125 The first step is to turn up the verbosity of the output using `-v`. This will
 126 allow you to see what tests are running and why a test is failing.
 127
 128 If a test fails, the temporary directory where the gem5 output was saved is kept
 129 and the path to the directory is printed in the terminal.
 130
 131 ## Debugging the testing infrastructure
 132
 133 Every command takes an option for the verbosity. `-v`, `-vv`, `-vvv` will
 134 increase the verbosity level. If something isn't working correctly, you can
 135 start here.
 136
 137 Most of the code for the testing infrastructure is in ext/testlib. This code
 138 contains the base code for tests, suites, fixtures, etc. The code in tests/gem5
 139 is *gem5-specific* code. For the most part, the code in tests/gem5 extends the
 140 structures in ext/testlib.
 141
 142 ## Common errors
 143
 144 You may see a number of lines of output during test discovery that look like
 145 the following:
 146
 147 ```shell
 148     Tried to load tests from ... but failed with an exception.
 149     Tried to load tests from ... but failed with an exception.
 150     ...
 151 ```
 152
 153 The testing library searches all python files in the `tests/` directory. The
 154 test library executes each python file it finds searching for tests. It's okay
 155 if the file causes an exception. This means there are no tests in that file
 156 (e.g., it's not a new-style test).
 157
 158
 159 # Binary test applications
 160
 161 The code for test binaries that are run in the gem5 guest during testing are
 162 found in `tests/test-progs`.
 163 There's one directory per test application.
 164 The source code is under the `source` directory.
 165
 166 You may have a `bin` directory as well.
 167 The `bin` directory is automatically created when running the test case that
 168 uses the test binary. The binary is downloaded from the gem5 servers the first
 169 time it is referenced by a test.
 170
 171 ## Updating the test binaries
 172
 173 The test infrastructure should check with the gem5 servers to ensure you have
 174 the latest binaries. However, if you believe your binaries are out of date,
 175 simply delete the `bin` directory and they will be re-downloaded to your local
 176 machine.
 177
 178 ## Building (new-style) test binaries
 179
 180 In each `src/` directory under `tests/test-progs`, there is a Makefile.
 181 This Makefile downloads a docker image and builds the test binary for some ISA
 182 (e.g., Makefile.x86 builds the binary for x86). Additionally, if you run `make
 183 upload` it will upload the binaries to the gem5 server, if you have access to
 184 modify the binaries. *If you need to modify the binaries for updating a test or
 185 adding a new test and you don't have access to the gem5 server, contact a
 186 maintainer (see MAINTAINERS).*
 187
 188
 189 # Running Tests in Parallel
 190
 191 Whimsy has support for parallel testing baked in. This system supports
 192 running multiple suites at the same time on the same computer. To run
 193 suites in parallel, supply the `-t <number-tests>` flag to the run command.
 194
 195 For example, to run up to three test suites at the same time::
 196
 197     ./main.py run --skip-build -t 3
 198