5 * <https://bugs.libre-soc.org/show_bug.cgi?id=717>
10 It has been stated many times but bears repeating, testing is important.
11 At this level, mistakes can be quite costly. One usually does not have
12 the luxury of patching silicon.
14 This API helps to define a standard way to collect and compare the results
15 from different implementations by abstracting away a certain level of
16 complexity. It will evolve to include more features and refinements in
19 Basic workflow for using the Test API:
23 These are what you write to verify the functionality and correctness of
24 a Power ISA implementation.
28 Once written, the Test Issuer submits the tests to the Test Runner with
33 The Test Runner runs the tests using testing parameters supplied by
38 The results between the implementation objects are compared and failures
41 By increasing the number of tests and the number of different
42 implemntations to compare, the probability for correctness increases.
43 The Test API helps to accomplish this by reducing the burden of the
50 The objects are what simulate the test. In this document, we simulate
51 tests using ISACaller and the nMigen HDL simulator and then capture their
52 states for comparison. An additional object, the ExpectedState class can
53 also be used in testing. In the future, additional objects to test with
54 (such as qemu) can be added. This API provides a means to compare them
57 States are a snapshot of registers and memory after the run of a
58 simulation. As of now, these include GPRs, control registers, program
59 counter, XERs, and memory.
62 class](https://git.libre-soc.org/?p=openpower-isa.git;a=blob;f=src/openpower/test/state.py;h=7219919670cd1d3b9737d9c3cc91f1e2bf1181b5;hb=HEAD#l60)
63 gives the methods that are required under the get_state() method to
64 obtain the values from an implementation object.
66 * get_intregs() Retrieves the integer GPRs (0-31). Stored as a list.
67 * get_crregs() Retrieves the control registers (0-7). Stored as a list.
68 * get_xregs() Retrieves the XPERs. Stored as individual members.
69 * get_pc() Retrieves the program counter. Stored as an individual member.
70 * get_mem() Retrieves the memory. Stored as a dictionary {location: data}
72 The [compare and compare_mem
73 methods](https://git.libre-soc.org/?p=openpower-isa.git;a=blob;f=src/openpower/test/state.py;h=7219919670cd1d3b9737d9c3cc91f1e2bf1181b5;hb=HEAD#l78)
74 are provided within the class. They do simple asserts against another
75 state to verify they are equal. If one of them fails, the test will
76 fail. Compare_mem will also pad memory when needed before the compare.
77 For example, one object may store only the few scattered memory locations
78 used instead of another object storing memory as a continuous range.
80 [SimState](https://git.libre-soc.org/?p=openpower-isa.git;a=blob;f=src/openpower/test/state.py;h=7219919670cd1d3b9737d9c3cc91f1e2bf1181b5;hb=HEAD#l123)
81 implements the methods to retrieve registers and memory from a passed
84 [HDLState](https://git.libre-soc.org/?p=soc.git;a=blob;f=src/soc/simple/test/teststate.py;h=d2f4b51ff74b865c0e758c34e49db1f92f094634;hb=HEAD)
85 implements the methods to retrieve registers and memory from a passed
86 in nmigen simulator object.
88 You will notice that between the two different types of states, the
89 methods to gather the underlying registers and memory are quite different.
90 However, they are stored in their respective states in the same format
91 allowing comparisons to be straight forward. Also, if implementing your
92 own state class, the use of yields in the methods are required.
94 [ExpectedState](https://git.libre-soc.org/?p=openpower-isa.git;a=blob;f=src/openpower/test/state.py;h=7219919670cd1d3b9737d9c3cc91f1e2bf1181b5;hb=HEAD#l183),
95 while somewhat sparse, serves a very useful function. It allows one
96 to manually define what a state should be after a test is run. This is
97 useful for both educational purposes, catching regressions, and ensuring
98 correct behavior. By default, ExpectedState will initialize everything
99 to 0. Therefore, any possible register changes that happens during a
100 test must be set before comparing to another state object. An example
101 of using ExpectedState is provided in the Test Cases section below.
106 will look at a set of test cases for [shift and rotate
107 instructions](https://git.libre-soc.org/?p=openpower-isa.git;a=blob;f=src/openpower/test/shift_rot/shift_rot_cases2.py;h=2ab6a2ef52a9d04ca1fec63ea4609fbdc1b84c64;hb=HEAD)
109 [case_srw_1](https://git.libre-soc.org/?p=openpower-isa.git;a=blob;f=src/openpower/test/shift_rot/shift_rot_cases2.py;h=2ab6a2ef52a9d04ca1fec63ea4609fbdc1b84c64;hb=HEAD#l23)
110 since it is a fairly basic "shift right word" instruction.
112 First off is the case name itself. It should be somewhat short and
113 descriptive, but also needs to have "case_" as the prefix otherwise Test
114 Issuer will ignore it completely.
116 Next comes the instruction(s) we are testing which in this
117 case is "sraw 3, 1, 2". It will shift the contents of register
118 1 the number of bits specified in register 2 and store the result
119 in register 3. You can have a test run multiple instructions, such as
120 [case_shift_once](https://git.libre-soc.org/?p=openpower-isa.git;a=blob;f=src/openpower/test/shift_rot/shift_rot_cases2.py;h=2ab6a2ef52a9d04ca1fec63ea4609fbdc1b84c64;hb=HEAD#l59).
121 Just know what is being tested is the final result of all the
122 instructions, and not each one individually inside the test case.
124 The following lines setup the initial registers for the test:
126 25 initial_regs = [0] * 32 # Set all the GPRs to 0
127 26 initial_regs[1] = 0x12345678 # Set gpr1 to the value to shift
128 27 initial_regs[2] = 8 # Set gpr2 to number of bits to shift right
130 With the testing items in place, we can move to what we expect the
131 outcome of the test to be. This is done by using the ExpectedState class:
133 28 e = ExpectedState(initial_regs, 4) # Create an object 'e' from the above values
134 29 e.intregs[3] = 0x123456 # This is the expected result we are testing
136 In the above lines, we are setting a blank expected state to what we
137 think the results will be after the test. On line 28 we are loading
138 this expected state with the set of initial registers and setting the
139 program counter (PC) to 4 because that is the location where we expect
140 the next instruction (if there was one) to be located.
142 Line 29 is setting register 3 to our expected result from the instruction
143 provided. Remember, we are shifting 0x12345678 8 bytes to the right and
144 storing the result in register 3. The result after simulation should
145 be 0x123456 in register 3.
147 The final step is adding the test case with:
149 30 self.add_case(Program(lst, bigendian), initial_regs, expected=e)
151 Using an expected state is optional, although recommended. You may see
152 tests that use random values and loops. These tests cannot use expected
153 values because we obviously can't predict a random outcome!
159 TestIssuer](https://git.libre-soc.org/?p=soc.git;a=blob;f=src/soc/simple/test/test_issuer.py;h=c01e8c68c5c68e4c107c18337f66fcdacf041194;hb=HEAD)
160 that can either run all the tests listed or specific ones and calling
161 TestRunner with the specified options. You'll notice our shift_rot_cases2
162 we looked at previously appear here in the list of imported test cases.
164 In this Test Issuer, we see the line:
166 suite.addTest(TestRunner(data, svp64=svp64))
168 which sets up the TestRunner with the test cases (including expected
169 states if specified within the test case) and whether to use svp64.
170 By default, TestRunner will run both the ISACaller simulator and the
171 nMigen HDL simulator. The parameters run_sim and run_hdl can either be
172 set to False to instruct TestRunner not to run those components.
174 ###Test Runner and Verification
178 ## Additional Notes on Testing and the API
182 Writing tests can be daunting at first. It is helpful to start with
183 easier instructions at first with simple values and work up from there.
185 For example, in the Test Cases section we looked at case_srw_1 which is
186 a simple shift. The test case_srw_2 which follows it doesn't look much
187 different however produces quite a different outcome since it causes carry
188 and sign extension because the value of the register exceeds 0x7fffffff.
190 While the Power ISA manual provides the pseudo code for how instructions
191 work, the manual will sometimes give alternate instructions that give
192 the equivalent result. This can be helpful in understanding certain
193 instructions as well.
195 * ###Don't comment out tests
197 If a test needs to be skipped for some reason, use the @unittest.skip("the
198 reason") decorator before the test.
200 * ###You can test for expected failure
202 Sometimes a test can be useful when we expect it to fail
203 and want to verify that it does. This can be done using the
204 @unittest.expectedFailure decorator. Examples can be found in
205 [test_state_class](https://git.libre-soc.org/?p=openpower-isa.git;a=blob;f=src/openpower/test/test_state_class.py;h=918958cfe4c43af4c0b22d08f22933b273805885;hb=HEAD).
209 * ###API Modificatons and Additions
211 Great care must be taken when making potential changes to the API itself.
212 You will notice nearly every area contains logging events. This helps
213 verify that those parts are indeed being executed. A misplaced and not
214 utilized yield for example can cause sections not to execute. Therefore
215 it is prudent to redirect test_issuer output with > /tmp/something and
216 look for messages where changed or added areas are in fact being executed.
218 **Make small and incremental changes and test often.**
220 This is clearly documented and stated in the [[HDL_workflow]]