docs/testapi.mdwn

   1 #Power ISA Test API
   2
   3 ##Overview
   4 It has been stated many times but bears repeating, testing is important.  At this level, mistakes can be quite costly.  One usually does not have the luxury of patching silicon.
   5
   6 This API helps to define a standard way to collect and compare the results from different implementations by abstracting away a certain level of complexity.  It will evolve to include more features and refinements in the future.
   7
   8 Basic workflow for using the Test API:
   9
  10 ###Test Cases
  11 These are what you write to verify the functionality and correctness of a Power ISA implementation.
  12
  13 ###Test Issuer
  14 Once written, the Test Issuer submits the tests to the Test Runner with optional parameters.
  15
  16 ###Test Runner
  17 The Test Runner runs the tests using testing parameters supplied by Test Issuer.
  18
  19 ###Results Comparison
  20 The results between the implementation objects are compared and failures are displayed.
  21
  22 By increasing the number of tests and the number of different implemntations to compare, the probability for correctness increases.  The Test API helps to accomplish this by reducing the burden of the testing process.
  23
  24 ##Test API components
  25 ###Objects and States
  26 The objects are what simulate the test.  In this document, we simulate tests using ISACaller and the nMigen HDL simulator and then capture their states for comparison.  An additional object, the ExpectedState class can also be used in testing.  In the future, additional objects to test with (such as qemu) can be added.  This API provides a means to compare them in a uniform fashion.
  27
  28 States are a snapshot of registers and memory after the run of a simulation.  As of now, these include GPRs, control registers, program counter, XERs, and memory.
  29
  30 The [base state class](https://git.libre-soc.org/?p=openpower-isa.git;a=blob;f=src/openpower/test/state.py;h=7219919670cd1d3b9737d9c3cc91f1e2bf1181b5;hb=HEAD#l60) gives the methods that are required under the get_state() method to obtain the values from an implementation object.
  31 * get_intregs()
  32
  33 Retrieves the integer GPRs (0-31).  Stored as a list.
  34
  35 * get_crregs()
  36
  37 Retrieves the control registers (0-7).  Stored as a list.
  38
  39 * get_xregs()
  40
  41 Retrieves the XPERs.  Stored as individual members.
  42
  43 * get_pc()
  44
  45 Retrieves the program counter.  Stored as an individual member.
  46
  47 * get_mem()
  48
  49 Retrieves the memory.  Stored as a dictionary {location: data}
  50
  51 The [compare and compare_mem methods](https://git.libre-soc.org/?p=openpower-isa.git;a=blob;f=src/openpower/test/state.py;h=7219919670cd1d3b9737d9c3cc91f1e2bf1181b5;hb=HEAD#l78) are provided within the class.  They do simple asserts against another state to verify they are equal.  If one of them fails, the test will fail.  Compare_mem will also pad memory when needed before the compare.  For example, one object may store only the few scattered memory locations used instead of another object storing memory as a continuous range.
  52
  53 [SimState](https://git.libre-soc.org/?p=openpower-isa.git;a=blob;f=src/openpower/test/state.py;h=7219919670cd1d3b9737d9c3cc91f1e2bf1181b5;hb=HEAD#l123) implements the methods to retrieve registers and memory from a passed in ISACaller object.
  54
  55 [HDLState](https://git.libre-soc.org/?p=soc.git;a=blob;f=src/soc/simple/test/teststate.py;h=d2f4b51ff74b865c0e758c34e49db1f92f094634;hb=HEAD) implements the methods to retrieve registers and memory from a passed in nmigen simulator object.
  56
  57 You will notice that between the two different types of states, the methods to gather the underlying registers and memory are quite different.  However, they are stored in their respective states in the same format allowing comparisons to be straight forward.  Also, if implementing your own state class, the use of yields in the methods are required.
  58
  59 [ExpectedState](https://git.libre-soc.org/?p=openpower-isa.git;a=blob;f=src/openpower/test/state.py;h=7219919670cd1d3b9737d9c3cc91f1e2bf1181b5;hb=HEAD#l183), while somewhat sparse, serves a very useful function.  It allows one to manually define what a state should be after a test is run.  This is useful for both educational purposes, catching regressions, and ensuring correct behavior.  By default, ExpectedState will initialize everything to 0.  Therefore, any possible register changes that happens during a test must be set before comparing to another state object.  An example of using ExpectedState is provided in the Test Cases section below.
  60
  61 ###Test Cases
  62
  63 For this section, we will look at a set of test cases for [shift and rotate instructions](https://git.libre-soc.org/?p=openpower-isa.git;a=blob;f=src/openpower/test/shift_rot/shift_rot_cases2.py;h=2ab6a2ef52a9d04ca1fec63ea4609fbdc1b84c64;hb=HEAD) and focus on [case_srw_1](https://git.libre-soc.org/?p=openpower-isa.git;a=blob;f=src/openpower/test/shift_rot/shift_rot_cases2.py;h=2ab6a2ef52a9d04ca1fec63ea4609fbdc1b84c64;hb=HEAD#l23) since it is a fairly basic "shift right word" instruction.
  64
  65 First off is the case name itself.  It should be somewhat short and descriptive, but also needs to have "case_" as the prefix otherwise Test Issuer will ignore it completely.
  66
  67 Next comes the instruction(s) we are testing which in this case is "sraw 3, 1, 2". It will shift the contents of register 1 the number of bits specified in register 2 and store the result in register 3.  You can have a test run multiple instructions, such as [case_shift_once](https://git.libre-soc.org/?p=openpower-isa.git;a=blob;f=src/openpower/test/shift_rot/shift_rot_cases2.py;h=2ab6a2ef52a9d04ca1fec63ea4609fbdc1b84c64;hb=HEAD#l59).  Just know what is being tested is the final result of all the instructions, and not each one individually inside the test case.
  68
  69 The following lines setup the initial registers for the test:
  70
  71         25      initial_regs = [0] * 32         # Set all the GPRs to 0
  72         26      initial_regs[1] = 0x12345678    # Set gpr1 to the value to shift
  73         27      initial_regs[2] = 8             # Set gpr2 to number of bits to shift right
  74
  75 With the testing items in place, we can move to what we expect the outcome of the test to be.  This is done by using the ExpectedState class:
  76
  77         28      e = ExpectedState(initial_regs, 4)  # Create an object 'e' from the above values
  78         29      e.intregs[3] = 0x123456             # This is the expected result we are testing
  79
  80 In the above lines, we are setting a blank expected state to what we think the results will be after the test.  On line 28 we are loading this expected state with the set of initial registers and setting the program counter (PC) to 4 because that is the location where we expect the next instruction (if there was one) to be located.
  81
  82 Line 29 is setting register 3 to our expected result from the instruction provided.  Remember, we are shifting 0x12345678 8 bytes to the right and storing the result in register 3.  The result after simulation should be 0x123456 in register 3.
  83
  84 The final step is adding the test case with:
  85
  86         30      self.add_case(Program(lst, bigendian), initial_regs, expected=e)
  87
  88 Using an expected state is optional, although recommended.  You may see tests that use random values and loops.  These tests cannot use expected values because we obviously can't predict a random outcome!
  89
  90
  91 ###Test Issuer
  92
  93 [This is a basic TestIssuer](https://git.libre-soc.org/?p=soc.git;a=blob;f=src/soc/simple/test/test_issuer.py;h=c01e8c68c5c68e4c107c18337f66fcdacf041194;hb=HEAD) that can either run all the tests listed or specific ones and calling TestRunner with the specified options.  You'll notice our shift_rot_cases2 we looked at previously appear here in the list of imported test cases.
  94
  95 In this Test Issuer, we see the line:
  96
  97 suite.addTest(TestRunner(data, svp64=svp64))
  98
  99 which sets up the TestRunner with the test cases (including expected states if specified within the test case) and whether to use svp64.  By default, TestRunner will run both the ISACaller simulator and the nMigen HDL simulator.  The parameters run_sim and run_hdl can either be set to False to instruct TestRunner not to run those components.
 100
 101 ###Test Runner and Verification
 102
 103     TODO
 104
 105
 106 ## Additional Notes on Testing and the API
 107
 108 * ###Start small
 109
 110 Writing tests can be daunting at first.  It is helpful to start with easier instructions at first with simple values and work up from there.
 111
 112 For example, in the Test Cases section we looked at case_srw_1 which is a simple shift.  The test case_srw_2 which follows it doesn't look much different however produces quite a different outcome since it causes carry and sign extension because the value of the register exceeds 0x7fffffff.
 113
 114 While the Power ISA manual provides the pseudo code for how instructions work, the manual will sometimes give alternate instructions that give the equivalent result.  This can be helpful in    understanding certain instructions as well.
 115
 116 * ###Don't comment out tests
 117
 118 If a test needs to be skipped for some reason, use the @unittest.skip("the reason") decorator before the test.
 119
 120 * ###You can test for expected failure
 121
 122 Sometimes a test can be useful when we expect it to fail and want to verify that it does.  This can be done using the @unittest.expectedFailure decorator.  Examples can be found in [test_state_class](https://git.libre-soc.org/?p=openpower-isa.git;a=blob;f=src/openpower/test/test_state_class.py;h=918958cfe4c43af4c0b22d08f22933b273805885;hb=HEAD).
 123
 124 * ###API Modificatons and Additions
 125
 126 Great care must be taken when making potential changes to the API itself.  You will notice nearly every area contains logging events.  This helps verify that those parts are indeed being executed.  A misplaced and not utilized yield for example can cause sections not to execute.  Therefore it is prudent to redirect test_issuer output with > /tmp/something and look for messages where changed or added areas are in fact being executed.
 127
 128 **Make small and incremental changes and test often.**
 129
 130
 131 Links:
 132 https://bugs.libre-soc.org/show_bug.cgi?id=717
 133
 134
 135
 136
 137
 138
 139
 140