-# Test API
+#Power ISA Test API
Links:
* <https://bugs.libre-soc.org/show_bug.cgi?id=717>
+
+
+##Overview
+
+It has been stated many times but bears repeating, testing is important.
+At this level, mistakes can be quite costly. One usually does not have
+the luxury of patching silicon.
+
+This API helps to define a standard way to collect and compare the results
+from different implementations by abstracting away a certain level of
+complexity. It will evolve to include more features and refinements in
+the future.
+
+Basic workflow for using the Test API:
+
+###Test Cases
+
+These are what you write to verify the functionality and correctness of
+a Power ISA implementation.
+
+###Test Issuer
+
+Once written, the Test Issuer submits the tests to the Test Runner with
+optional parameters.
+
+###Test Runner
+
+The Test Runner runs the tests using testing parameters supplied by
+Test Issuer.
+
+###Results Comparison
+
+The results between the implementation objects are compared and failures
+are displayed.
+
+By increasing the number of tests and the number of different
+implemntations to compare, the probability for correctness increases.
+The Test API helps to accomplish this by reducing the burden of the
+testing process.
+
+##Test API components
+
+###Objects and States
+
+The objects are what simulate the test. In this document, we simulate
+tests using ISACaller and the nMigen HDL simulator and then capture their
+states for comparison. An additional object, the ExpectedState class can
+also be used in testing. In the future, additional objects to test with
+(such as qemu) can be added. This API provides a means to compare them
+in a uniform fashion.
+
+States are a snapshot of registers and memory after the run of a
+simulation. As of now, these include GPRs, control registers, program
+counter, XERs, and memory.
+
+The [base state
+class](https://git.libre-soc.org/?p=openpower-isa.git;a=blob;f=src/openpower/test/state.py;h=7219919670cd1d3b9737d9c3cc91f1e2bf1181b5;hb=HEAD#l60)
+gives the methods that are required under the get_state() method to
+obtain the values from an implementation object.
+
+* get_intregs() Retrieves the integer GPRs (0-31). Stored as a list.
+* get_crregs() Retrieves the control registers (0-7). Stored as a list.
+* get_xregs() Retrieves the XPERs. Stored as individual members.
+* get_pc() Retrieves the program counter. Stored as an individual member.
+* get_mem() Retrieves the memory. Stored as a dictionary {location: data}
+
+The [compare and compare_mem
+methods](https://git.libre-soc.org/?p=openpower-isa.git;a=blob;f=src/openpower/test/state.py;h=7219919670cd1d3b9737d9c3cc91f1e2bf1181b5;hb=HEAD#l78)
+are provided within the class. They do simple asserts against another
+state to verify they are equal. If one of them fails, the test will
+fail. Compare_mem will also pad memory when needed before the compare.
+For example, one object may store only the few scattered memory locations
+used instead of another object storing memory as a continuous range.
+
+[SimState](https://git.libre-soc.org/?p=openpower-isa.git;a=blob;f=src/openpower/test/state.py;h=7219919670cd1d3b9737d9c3cc91f1e2bf1181b5;hb=HEAD#l123)
+implements the methods to retrieve registers and memory from a passed
+in ISACaller object.
+
+[HDLState](https://git.libre-soc.org/?p=soc.git;a=blob;f=src/soc/simple/test/teststate.py;h=d2f4b51ff74b865c0e758c34e49db1f92f094634;hb=HEAD)
+implements the methods to retrieve registers and memory from a passed
+in nmigen simulator object.
+
+You will notice that between the two different types of states, the
+methods to gather the underlying registers and memory are quite different.
+However, they are stored in their respective states in the same format
+allowing comparisons to be straight forward. Also, if implementing your
+own state class, the use of yields in the methods are required.
+
+[ExpectedState](https://git.libre-soc.org/?p=openpower-isa.git;a=blob;f=src/openpower/test/state.py;h=7219919670cd1d3b9737d9c3cc91f1e2bf1181b5;hb=HEAD#l183),
+while somewhat sparse, serves a very useful function. It allows one
+to manually define what a state should be after a test is run. This is
+useful for both educational purposes, catching regressions, and ensuring
+correct behavior. By default, ExpectedState will initialize everything
+to 0. Therefore, any possible register changes that happens during a
+test must be set before comparing to another state object. An example
+of using ExpectedState is provided in the Test Cases section below.
+
+###Test Cases
+
+For this section, we
+will look at a set of test cases for [shift and rotate
+instructions](https://git.libre-soc.org/?p=openpower-isa.git;a=blob;f=src/openpower/test/shift_rot/shift_rot_cases2.py;h=2ab6a2ef52a9d04ca1fec63ea4609fbdc1b84c64;hb=HEAD)
+and focus on
+[case_srw_1](https://git.libre-soc.org/?p=openpower-isa.git;a=blob;f=src/openpower/test/shift_rot/shift_rot_cases2.py;h=2ab6a2ef52a9d04ca1fec63ea4609fbdc1b84c64;hb=HEAD#l23)
+since it is a fairly basic "shift right word" instruction.
+
+First off is the case name itself. It should be somewhat short and
+descriptive, but also needs to have "case_" as the prefix otherwise Test
+Issuer will ignore it completely.
+
+Next comes the instruction(s) we are testing which in this
+case is "sraw 3, 1, 2". It will shift the contents of register
+1 the number of bits specified in register 2 and store the result
+in register 3. You can have a test run multiple instructions, such as
+[case_shift_once](https://git.libre-soc.org/?p=openpower-isa.git;a=blob;f=src/openpower/test/shift_rot/shift_rot_cases2.py;h=2ab6a2ef52a9d04ca1fec63ea4609fbdc1b84c64;hb=HEAD#l59).
+Just know what is being tested is the final result of all the
+instructions, and not each one individually inside the test case.
+
+The following lines setup the initial registers for the test:
+
+ 25 initial_regs = [0] * 32 # Set all the GPRs to 0
+ 26 initial_regs[1] = 0x12345678 # Set gpr1 to the value to shift
+ 27 initial_regs[2] = 8 # Set gpr2 to number of bits to shift right
+
+With the testing items in place, we can move to what we expect the
+outcome of the test to be. This is done by using the ExpectedState class:
+
+ 28 e = ExpectedState(initial_regs, 4) # Create an object 'e' from the above values
+ 29 e.intregs[3] = 0x123456 # This is the expected result we are testing
+
+In the above lines, we are setting a blank expected state to what we
+think the results will be after the test. On line 28 we are loading
+this expected state with the set of initial registers and setting the
+program counter (PC) to 4 because that is the location where we expect
+the next instruction (if there was one) to be located.
+
+Line 29 is setting register 3 to our expected result from the instruction
+provided. Remember, we are shifting 0x12345678 8 bytes to the right and
+storing the result in register 3. The result after simulation should
+be 0x123456 in register 3.
+
+The final step is adding the test case with:
+
+ 30 self.add_case(Program(lst, bigendian), initial_regs, expected=e)
+
+Using an expected state is optional, although recommended. You may see
+tests that use random values and loops. These tests cannot use expected
+values because we obviously can't predict a random outcome!
+
+
+###Test Issuer
+
+[This is a basic
+TestIssuer](https://git.libre-soc.org/?p=soc.git;a=blob;f=src/soc/simple/test/test_issuer.py;h=c01e8c68c5c68e4c107c18337f66fcdacf041194;hb=HEAD)
+that can either run all the tests listed or specific ones and calling
+TestRunner with the specified options. You'll notice our shift_rot_cases2
+we looked at previously appear here in the list of imported test cases.
+
+In this Test Issuer, we see the line:
+
+ suite.addTest(TestRunner(data, svp64=svp64))
+
+which sets up the TestRunner with the test cases (including expected
+states if specified within the test case) and whether to use svp64.
+By default, TestRunner will run both the ISACaller simulator and the
+nMigen HDL simulator. The parameters run_sim and run_hdl can either be
+set to False to instruct TestRunner not to run those components.
+
+###Test Runner and Verification
+
+ TODO
+
+## Additional Notes on Testing and the API
+
+* ###Start small
+
+Writing tests can be daunting at first. It is helpful to start with
+easier instructions at first with simple values and work up from there.
+
+For example, in the Test Cases section we looked at case_srw_1 which is
+a simple shift. The test case_srw_2 which follows it doesn't look much
+different however produces quite a different outcome since it causes carry
+and sign extension because the value of the register exceeds 0x7fffffff.
+
+While the Power ISA manual provides the pseudo code for how instructions
+work, the manual will sometimes give alternate instructions that give
+the equivalent result. This can be helpful in understanding certain
+instructions as well.
+
+* ###Don't comment out tests
+
+If a test needs to be skipped for some reason, use the @unittest.skip("the
+reason") decorator before the test.
+
+* ###You can test for expected failure
+
+Sometimes a test can be useful when we expect it to fail
+and want to verify that it does. This can be done using the
+@unittest.expectedFailure decorator. Examples can be found in
+[test_state_class](https://git.libre-soc.org/?p=openpower-isa.git;a=blob;f=src/openpower/test/test_state_class.py;h=918958cfe4c43af4c0b22d08f22933b273805885;hb=HEAD).
+
+ (TODO: add example)
+
+* ###API Modificatons and Additions
+
+Great care must be taken when making potential changes to the API itself.
+You will notice nearly every area contains logging events. This helps
+verify that those parts are indeed being executed. A misplaced and not
+utilized yield for example can cause sections not to execute. Therefore
+it is prudent to redirect test_issuer output with > /tmp/something and
+look for messages where changed or added areas are in fact being executed.
+
+**Make small and incremental changes and test often.**
+
+This is clearly documented and stated in the [[HDL_workflow]]
+
+
projects / libreriscv.git / blobdiff