docs/testapi.mdwn

   1 #Power ISA Test API
   2
   3 Links:
   4
   5 * <https://bugs.libre-soc.org/show_bug.cgi?id=717>
   6
   7
   8 ##Overview
   9
  10 It has been stated many times but bears repeating, testing is important.
  11 At this level, mistakes can be quite costly.  One usually does not have
  12 the luxury of patching silicon.
  13
  14 This API helps to define a standard way to collect and compare the results
  15 from different implementations by abstracting away a certain level of
  16 complexity.  It will evolve to include more features and refinements in
  17 the future.
  18
  19 Basic workflow for using the Test API:
  20
  21 ###Test Cases
  22
  23 These are what you write to verify the functionality and correctness of
  24 a Power ISA implementation.
  25
  26 ###Test Issuer
  27
  28 Once written, the Test Issuer submits the tests to the Test Runner with
  29 optional parameters.
  30
  31 ###Test Runner
  32
  33 The Test Runner runs the tests using testing parameters supplied by
  34 Test Issuer.
  35
  36 ###Results Comparison
  37
  38 The results between the implementation objects are compared and failures
  39 are displayed.
  40
  41 By increasing the number of tests and the number of different
  42 implemntations to compare, the probability for correctness increases.
  43 The Test API helps to accomplish this by reducing the burden of the
  44 testing process.
  45
  46 ##Test API components
  47
  48 ###Objects and States
  49
  50 The objects are what simulate the test.  In this document, we simulate
  51 tests using ISACaller and the nMigen HDL simulator and then capture their
  52 states for comparison.  An additional object, the ExpectedState class can
  53 also be used in testing.  In the future, additional objects to test with
  54 (such as qemu) can be added.  This API provides a means to compare them
  55 in a uniform fashion.
  56
  57 States are a snapshot of registers and memory after the run of a
  58 simulation.  As of now, these include GPRs, control registers, program
  59 counter, XERs, and memory.
  60
  61 The [base state
  62 class](https://git.libre-soc.org/?p=openpower-isa.git;a=blob;f=src/openpower/test/state.py;h=7219919670cd1d3b9737d9c3cc91f1e2bf1181b5;hb=HEAD#l60)
  63 gives the methods that are required under the get_state() method to
  64 obtain the values from an implementation object.
  65
  66 * get_intregs() Retrieves the integer GPRs (0-31).  Stored as a list.
  67 * get_crregs() Retrieves the control registers (0-7).  Stored as a list.
  68 * get_xregs() Retrieves the XPERs.  Stored as individual members.
  69 * get_pc() Retrieves the program counter.  Stored as an individual member.
  70 * get_mem() Retrieves the memory.  Stored as a dictionary {location: data}
  71
  72 The [compare and compare_mem
  73 methods](https://git.libre-soc.org/?p=openpower-isa.git;a=blob;f=src/openpower/test/state.py;h=7219919670cd1d3b9737d9c3cc91f1e2bf1181b5;hb=HEAD#l78)
  74 are provided within the class.  They do simple asserts against another
  75 state to verify they are equal.  If one of them fails, the test will
  76 fail.  Compare_mem will also pad memory when needed before the compare.
  77 For example, one object may store only the few scattered memory locations
  78 used instead of another object storing memory as a continuous range.
  79
  80 [SimState](https://git.libre-soc.org/?p=openpower-isa.git;a=blob;f=src/openpower/test/state.py;h=7219919670cd1d3b9737d9c3cc91f1e2bf1181b5;hb=HEAD#l123)
  81 implements the methods to retrieve registers and memory from a passed
  82 in ISACaller object.
  83
  84 [HDLState](https://git.libre-soc.org/?p=soc.git;a=blob;f=src/soc/simple/test/teststate.py;h=d2f4b51ff74b865c0e758c34e49db1f92f094634;hb=HEAD)
  85 implements the methods to retrieve registers and memory from a passed
  86 in nmigen simulator object.
  87
  88 You will notice that between the two different types of states, the
  89 methods to gather the underlying registers and memory are quite different.
  90 However, they are stored in their respective states in the same format
  91 allowing comparisons to be straight forward.  Also, if implementing your
  92 own state class, the use of yields in the methods are required.
  93
  94 [ExpectedState](https://git.libre-soc.org/?p=openpower-isa.git;a=blob;f=src/openpower/test/state.py;h=7219919670cd1d3b9737d9c3cc91f1e2bf1181b5;hb=HEAD#l183),
  95 while somewhat sparse, serves a very useful function.  It allows one
  96 to manually define what a state should be after a test is run.  This is
  97 useful for both educational purposes, catching regressions, and ensuring
  98 correct behavior.  By default, ExpectedState will initialize everything
  99 to 0.  Therefore, any possible register changes that happens during a
 100 test must be set before comparing to another state object.  An example
 101 of using ExpectedState is provided in the Test Cases section below.
 102
 103 ###Test Cases
 104
 105 For this section, we
 106 will look at a set of test cases for [shift and rotate
 107 instructions](https://git.libre-soc.org/?p=openpower-isa.git;a=blob;f=src/openpower/test/shift_rot/shift_rot_cases2.py;h=2ab6a2ef52a9d04ca1fec63ea4609fbdc1b84c64;hb=HEAD)
 108 and focus on
 109 [case_srw_1](https://git.libre-soc.org/?p=openpower-isa.git;a=blob;f=src/openpower/test/shift_rot/shift_rot_cases2.py;h=2ab6a2ef52a9d04ca1fec63ea4609fbdc1b84c64;hb=HEAD#l23)
 110 since it is a fairly basic "shift right word" instruction.
 111
 112 First off is the case name itself.  It should be somewhat short and
 113 descriptive, but also needs to have "case_" as the prefix otherwise Test
 114 Issuer will ignore it completely.
 115
 116 Next comes the instruction(s) we are testing which in this
 117 case is "sraw 3, 1, 2". It will shift the contents of register
 118 1 the number of bits specified in register 2 and store the result
 119 in register 3.  You can have a test run multiple instructions, such as
 120 [case_shift_once](https://git.libre-soc.org/?p=openpower-isa.git;a=blob;f=src/openpower/test/shift_rot/shift_rot_cases2.py;h=2ab6a2ef52a9d04ca1fec63ea4609fbdc1b84c64;hb=HEAD#l59).
 121 Just know what is being tested is the final result of all the
 122 instructions, and not each one individually inside the test case.
 123
 124 The following lines setup the initial registers for the test:
 125
 126     25  initial_regs = [0] * 32      # Set all the GPRs to 0
 127     26  initial_regs[1] = 0x12345678 # Set gpr1 to the value to shift
 128     27  initial_regs[2] = 8          # Set gpr2 to number of bits to shift right
 129
 130 With the testing items in place, we can move to what we expect the
 131 outcome of the test to be.  This is done by using the ExpectedState class:
 132
 133     28  e = ExpectedState(initial_regs, 4)  # Create an object 'e' from the above values
 134     29  e.intregs[3] = 0x123456             # This is the expected result we are testing
 135
 136 In the above lines, we are setting a blank expected state to what we
 137 think the results will be after the test.  On line 28 we are loading
 138 this expected state with the set of initial registers and setting the
 139 program counter (PC) to 4 because that is the location where we expect
 140 the next instruction (if there was one) to be located.
 141
 142 Line 29 is setting register 3 to our expected result from the instruction
 143 provided.  Remember, we are shifting 0x12345678 8 bytes to the right and
 144 storing the result in register 3.  The result after simulation should
 145 be 0x123456 in register 3.
 146
 147 The final step is adding the test case with:
 148
 149     30  self.add_case(Program(lst, bigendian), initial_regs, expected=e)
 150
 151 Using an expected state is optional, although recommended.  You may see
 152 tests that use random values and loops.  These tests cannot use expected
 153 values because we obviously can't predict a random outcome!
 154
 155
 156 ###Test Issuer
 157
 158 [This is a basic
 159 TestIssuer](https://git.libre-soc.org/?p=soc.git;a=blob;f=src/soc/simple/test/test_issuer.py;h=c01e8c68c5c68e4c107c18337f66fcdacf041194;hb=HEAD)
 160 that can either run all the tests listed or specific ones and calling
 161 TestRunner with the specified options.  You'll notice our shift_rot_cases2
 162 we looked at previously appear here in the list of imported test cases.
 163
 164 In this Test Issuer, we see the line:
 165
 166     suite.addTest(TestRunner(data, svp64=svp64))
 167
 168 which sets up the TestRunner with the test cases (including expected
 169 states if specified within the test case) and whether to use svp64.
 170 By default, TestRunner will run both the ISACaller simulator and the
 171 nMigen HDL simulator.  The parameters run_sim and run_hdl can either be
 172 set to False to instruct TestRunner not to run those components.
 173
 174 ###Test Runner and Verification
 175
 176     TODO
 177
 178 ## Additional Notes on Testing and the API
 179
 180 * ###Start small
 181
 182 Writing tests can be daunting at first.  It is helpful to start with
 183 easier instructions at first with simple values and work up from there.
 184
 185 For example, in the Test Cases section we looked at case_srw_1 which is
 186 a simple shift.  The test case_srw_2 which follows it doesn't look much
 187 different however produces quite a different outcome since it causes carry
 188 and sign extension because the value of the register exceeds 0x7fffffff.
 189
 190 While the Power ISA manual provides the pseudo code for how instructions
 191 work, the manual will sometimes give alternate instructions that give
 192 the equivalent result.  This can be helpful in    understanding certain
 193 instructions as well.
 194
 195 * ###Don't comment out tests
 196
 197 If a test needs to be skipped for some reason, use the @unittest.skip("the
 198 reason") decorator before the test.
 199
 200 * ###You can test for expected failure
 201
 202 Sometimes a test can be useful when we expect it to fail
 203 and want to verify that it does.  This can be done using the
 204 @unittest.expectedFailure decorator.  Examples can be found in
 205 [test_state_class](https://git.libre-soc.org/?p=openpower-isa.git;a=blob;f=src/openpower/test/test_state_class.py;h=918958cfe4c43af4c0b22d08f22933b273805885;hb=HEAD).
 206
 207    (TODO: add example)
 208
 209 * ###API Modificatons and Additions
 210
 211 Great care must be taken when making potential changes to the API itself.
 212 You will notice nearly every area contains logging events.  This helps
 213 verify that those parts are indeed being executed.  A misplaced and not
 214 utilized yield for example can cause sections not to execute.  Therefore
 215 it is prudent to redirect test_issuer output with > /tmp/something and
 216 look for messages where changed or added areas are in fact being executed.
 217
 218 **Make small and incremental changes and test often.**
 219
 220 This is clearly documented and stated in the [[HDL_workflow]]
 221
 222