(no commit message)

[libreriscv.git] / simple_v_extension / specification.mdwn

# Simple-V (Parallelism Extension Proposal) Specification

* Copyright (C) 2017, 2018, 3029 Luke Kenneth Casson Leighton
* Status: DRAFTv0.6
* Last edited: 21 jun 2019
* Ancillary resource: [[opcodes]] [[sv_prefix_proposal]]

With thanks to:

* Allen Baum
* Bruce Hoult
* comp.arch
* Jacob Bachmeyer
* Guy Lemurieux
* Jacob Lifshay
* Terje Mathisen
* The RISC-V Founders, without whom this all would not be possible.

[[!toc ]]

# Summary and Background: Rationale

Simple-V is a uniform parallelism API for RISC-V hardware that has several
unplanned side-effects including code-size reduction, expansion of
HINT space and more.  The reason for
creating it is to provide a manageable way to turn a pre-existing design
into a parallel one, in a step-by-step incremental fashion, allowing
the implementor to focus on adding hardware where it is needed and necessary.
The primary target is for mobile-class 3D GPUs and VPUs, with secondary
goals being to reduce executable size and reduce context-switch latency.

Critically: **No new instructions are added**.  The parallelism (if any
is implemented) is implicitly added by tagging *standard* scalar registers
for redirection.  When such a tagged register is used in any instruction,
it indicates that the PC shall **not** be incremented; instead a loop
is activated where *multiple* instructions are issued to the pipeline
(as determined by a length CSR), with contiguously incrementing register
numbers starting from the tagged register.  When the last "element"
has been reached, only then is the PC permitted to move on.  Thus
Simple-V effectively sits (slots) *in between* the instruction decode phase
and the ALU(s).

The barrier to entry with SV is therefore very low.  The minimum
compliant implementation is software-emulation (traps), requiring
only the CSRs and CSR tables, and that an exception be thrown if an
instruction's registers are detected to have been tagged.  The looping
that would otherwise be done in hardware is thus carried out in software,
instead.  Whilst much slower, it is "compliant" with the SV specification,
and may be suited for implementation in RV32E and also in situations
where the implementor wishes to focus on certain aspects of SV, without
unnecessary time and resources into the silicon, whilst also conforming
strictly with the API.  A good area to punt to software would be the
polymorphic element width capability for example.

Hardware Parallelism, if any, is therefore added at the implementor's
discretion to turn what would otherwise be a sequential loop into a
parallel one.

To emphasise that clearly: Simple-V (SV) is *not*:

* A SIMD system
* A SIMT system
* A Vectorisation Microarchitecture
* A microarchitecture of any specific kind
* A mandary parallel processor microarchitecture of any kind
* A supercomputer extension

SV does **not** tell implementors how or even if they should implement
parallelism: it is a hardware "API" (Application Programming Interface)
that, if implemented, presents a uniform and consistent way to *express*
parallelism, at the same time leaving the choice of if, how, how much,
when and whether to parallelise operations **entirely to the implementor**.

# Basic Operation

The principle of SV is as follows:

* Standard RV instructions are "prefixed" either to a 48 format (single instruction option) or a variable
 length VLIW-like prefix (multi or "grouped" option) that indicates
 which registers are "tagged" as "vectorised". Predicates can also be added.
* A "Vector Length" CSR is set, indicating the span of any future
  "parallel" operations.
* If any operation (a **scalar** standard RV opcode)
 uses a register that has been so "marked",
 a hardware "macro-unrolling loop" is activated, of length
  VL, that effectively issues **multiple** identical instructions
  using contiguous sequentially-incrementing register numbers.
* **Whether they be executed sequentially or in parallel or a
  mixture of both or punted to software-emulation in a trap handler
  is entirely up to the implementor**.

In this way an entire scalar algorithm may be vectorised with
the minimum of modification to the hardware and to compiler toolchains.

To reiterate: **There are *no* new opcodes**. The scheme works *entirely* on hidden context that augments *scalar* RISCV instructions.

# CSRs <a name="csrs"></a>

* An optional "reshaping" CSR key-value table which remaps from a 1D
  linear shape to 2D or 3D, including full transposition.

There are also five additional User mode CSRs :

* uMVL (the Maximum Vector Length)
* uVL (which has different characteristics from standard CSRs)
* uSUBVL (effectively a kind of SIMD)
* uEPCVLIW (a copy of the sub-execution Program Counter, that is relative to the start of the current VLIW Group, set on a trap).
* uSTATE (useful for saving and restoring during context switch,
  and for providing fast transitions)

There are also five additional CSRs for Supervisor-Mode:

* SMVL
* SVL
* SSUBVL
* SEPCVLIW
* SSTATE

And likewise for M-Mode:

* MMVL
* MVL
* MSUBVL
* MEPCVLIW
* MSTATE

Both Supervisor and M-Mode have their own CSR registers, independent of the other privilege levels, in order to make it easier to use Vectorisation in each level without affecting other privilege levels.

The access pattern for these groups of CSRs in each mode follows the
same pattern for other CSRs that have M-Mode and S-Mode "mirrors":

* In M-Mode, the S-Mode and U-Mode CSRs are separate and distinct.
* In S-Mode, accessing and changing of the M-Mode CSRs is identical
  to changing the S-Mode CSRs.  Accessing and changing the U-Mode
  CSRs is permitted.
* In U-Mode, accessing and changing of the S-Mode and U-Mode CSRs
  is prohibited.

In M-Mode, only the M-Mode CSRs are in effect, i.e. it is only the
M-Mode MVL, the M-Mode STATE and so on that influences the processor
behaviour.  Likewise for S-Mode, and likewise for U-Mode.

This has the interesting benefit of allowing M-Mode (or S-Mode)
to be set up, for context-switching to take place, and, on return
back to the higher privileged mode, the CSRs of that mode will be
exactly as they were.  Thus, it becomes possible for example to
set up CSRs suited best to aiding and assisting low-latency fast
context-switching *once and only once*, without the need for
re-initialising the CSRs needed to do so.

The xEPCVLIW CSRs must be treated exactly like their corresponding xepc equivalents. See VLIW section for details.

## MAXVECTORLENGTH (MVL) <a name="mvl" />

MAXVECTORLENGTH is the same concept as MVL in RVV, except that it
is variable length and may be dynamically set.  MVL is
however limited to the regfile bitwidth XLEN (1-32 for RV32,
1-64 for RV64 and so on).

The reason for setting this limit is so that predication registers, when
marked as such, may fit into a single register as opposed to fanning out
over several registers.  This keeps the implementation a little simpler.

The other important factor to note is that the actual MVL is **offset
by one**, so that it can fit into only 6 bits (for RV64) and still cover
a range up to XLEN bits.  So, when setting the MVL CSR to 0, this actually
means that MVL==1.  When setting the MVL CSR to 3, this actually means
that MVL==4, and so on.  This is expressed more clearly in the "pseudocode"
section, where there are subtle differences between CSRRW and CSRRWI.

## Vector Length (VL) <a name="vl" />

VSETVL is slightly different from RVV.  Like RVV, VL is set to be within
the range 1 <= VL <= MVL (where MVL in turn is limited to 1 <= MVL <= XLEN)

    VL = rd = MIN(vlen, MVL)

where 1 <= MVL <= XLEN

However just like MVL it is important to note that the range for VL has
subtle design implications, covered in the "CSR pseudocode" section

The fixed (specific) setting of VL allows vector LOAD/STORE to be used
to switch the entire bank of registers using a single instruction (see
Appendix, "Context Switch Example").  The reason for limiting VL to XLEN
is down to the fact that predication bits fit into a single register of
length XLEN bits.

The second change is that when VSETVL is requested to be stored
into x0, it is *ignored* silently (VSETVL x0, x5)

The third and most important change is that, within the limits set by
MVL, the value passed in **must** be set in VL (and in the
destination register).

This has implication for the microarchitecture, as VL is required to be
set (limits from MVL notwithstanding) to the actual value
requested.  RVV has the option to set VL to an arbitrary value that suits
the conditions and the micro-architecture: SV does *not* permit this.

The reason is so that if SV is to be used for a context-switch or as a
substitute for LOAD/STORE-Multiple, the operation can be done with only
2-3 instructions (setup of the CSRs, VSETVL x0, x0, #{regfilelen-1},
single LD/ST operation).  If VL does *not* get set to the register file
length when VSETVL is called, then a software-loop would be needed.
To avoid this need, VL *must* be set to exactly what is requested
(limits notwithstanding).

Therefore, in turn, unlike RVV, implementors *must* provide
pseudo-parallelism (using sequential loops in hardware) if actual
hardware-parallelism in the ALUs is not deployed.  A hybrid is also
permitted (as used in Broadcom's VideoCore-IV) however this must be
*entirely* transparent to the ISA.

The fourth change is that VSETVL is implemented as a CSR, where the
behaviour of CSRRW (and CSRRWI) must be changed to specifically store
the *new* value in the destination register, **not** the old value.
Where context-load/save is to be implemented in the usual fashion
by using a single CSRRW instruction to obtain the old value, the
*secondary* CSR must be used (SVSTATE).  This CSR behaves
exactly as standard CSRs, and contains more than just VL.

One interesting side-effect of using CSRRWI to set VL is that this
may be done with a single instruction, useful particularly for a
context-load/save.  There are however limitations: CSRWI's immediate
is limited to 0-31 (representing VL=1-32).

Note that when VL is set to 1, all parallel operations cease: the
hardware loop is reduced to a single element: scalar operations.

## SUBVL - Sub Vector Length

This is a "group by quantity" that effectively divides VL into groups of elements of length SUBVL. VL itself must therefore be set in advance to a multiple of SUBVL.

Legal values are 1, 2, 3 and 4, and the STATE CSR must hold the 2 bit values 0b00 thru 0b11.

Setting this CSR to 0 must raise an exception.  Setting it to a value greater than 4 likewise.

The main effect of SUBVL is that predication bits are applied per **group**, 
rather than by individual element.

This saves a not insignificant number of instructions when handling 3D vectors, as otherwise a much longer predicate mask would have to be set up with regularly-repeated bit patterns.

## STATE

This is a standard CSR that contains sufficient information for a
full context save/restore.  It contains (and permits setting of)
MVL, VL, SUBVL,
the destination element offset of the current parallel
instruction being executed, and, for twin-predication, the source
element offset as well.  Interestingly it may hypothetically
also be used to make the immediately-following instruction to skip a
certain number of elements, however the recommended method to do
this is predication or using the offset mode of the REMAP CSRs.

Setting destoffs and srcoffs is realistically intended for saving state
so that exceptions (page faults in particular) may be serviced and the
hardware-loop that was being executed at the time of the trap, from
user-mode (or Supervisor-mode), may be returned to and continued from
where it left off.  The reason why this works is because setting
User-Mode STATE will not change (not be used) in M-Mode or S-Mode
(and is entirely why M-Mode and S-Mode have their own STATE CSRs).

The format of the STATE CSR is as follows:

| (28..27) | (26..24) | (23..18) | (17..12) | (11..6) | (5...0) |
| -------- | -------- | -------- | -------- | ------- | ------- |
| rsvd    | subvl   | destoffs | srcoffs  | vl      | maxvl   |

When setting this CSR, the following characteristics will be enforced:

* **MAXVL** will be truncated (after offset) to be within the range 1 to XLEN
* **VL** will be truncated (after offset) to be within the range 1 to MAXVL
* **SUBVL** which sets a SIMD-like quantity, has only 4 values however if VL is not a multiple of SUBVL an exception will be raised.
* **srcoffs** will be truncated to be within the range 0 to VL-1
* **destoffs** will be truncated to be within the range 0 to VL-1

## MVL and VL  Pseudocode

The pseudo-code for get and set of VL and MVL are as follows:

    set_mvl_csr(value, rd):
        regs[rd] = MVL
        MVL = MIN(value, MVL)

    get_mvl_csr(rd):
        regs[rd] = VL

    set_vl_csr(value, rd):
        VL = MIN(value, MVL)
        regs[rd] = VL # yes returning the new value NOT the old CSR
        return VL

    get_vl_csr(rd):
        regs[rd] = VL
        return VL

Note that where setting MVL behaves as a normal CSR, unlike standard CSR
behaviour, setting VL will return the **new** value of VL **not** the old
one.

For CSRRWI, the range of the immediate is restricted to 5 bits.  In order to
maximise the effectiveness, an immediate of 0 is used to set VL=1,
an immediate of 1 is used to set VL=2 and so on:

    CSRRWI_Set_MVL(value):
        set_mvl_csr(value+1, x0)

    CSRRWI_Set_VL(value):
        set_vl_csr(value+1, x0)

However for CSRRW the following pseudocode is used for MVL and VL,
where setting the value to zero will cause an exception to be raised.
The reason is that if VL or MVL are set to zero, the STATE CSR is
not capable of returning that value.

    CSRRW_Set_MVL(rs1, rd):
        value = regs[rs1]
        if value == 0:
            raise Exception
        set_mvl_csr(value, rd)

    CSRRW_Set_VL(rs1, rd):
        value = regs[rs1]
        if value == 0:
            raise Exception
        set_vl_csr(value, rd)

In this way, when CSRRW is utilised with a loop variable, the value
that goes into VL (and into the destination register) may be used
in an instruction-minimal fashion:

     CSRvect1 = {type: F, key: a3, val: a3, elwidth: dflt}
     CSRvect2 = {type: F, key: a7, val: a7, elwidth: dflt}
     CSRRWI MVL, 3          # sets MVL == **4** (not 3)
     j zerotest             # in case loop counter a0 already 0
    loop:
     CSRRW VL, t0, a0       # vl = t0 = min(mvl, a0)
     ld     a3, a1          # load 4 registers a3-6 from x
     slli   t1, t0, 3       # t1 = vl * 8 (in bytes)
     ld     a7, a2          # load 4 registers a7-10 from y
     add    a1, a1, t1      # increment pointer to x by vl*8
     fmadd a7, a3, fa0, a7 # v1 += v0 * fa0 (y = a * x + y)
     sub    a0, a0, t0      # n -= vl (t0)
     st     a7, a2          # store 4 registers a7-10 to y
     add    a2, a2, t1      # increment pointer to y by vl*8
    zerotest:
     bnez   a0, loop        # repeat if n != 0

With the STATE CSR, just like with CSRRWI, in order to maximise the
utilisation of the limited bitspace, "000000" in binary represents
VL==1, "00001" represents VL==2 and so on (likewise for MVL):

    CSRRW_Set_SV_STATE(rs1, rd):
        value = regs[rs1]
        get_state_csr(rd)
        MVL = set_mvl_csr(value[11:6]+1)
        VL = set_vl_csr(value[5:0]+1)
        destoffs = value[23:18]>>18
        srcoffs = value[23:18]>>12

    get_state_csr(rd):
        regs[rd] = (MVL-1) | (VL-1)<<6 | (srcoffs)<<12 |
                   (destoffs)<<18
        return regs[rd]

In both cases, whilst CSR read of VL and MVL return the exact values
of VL and MVL respectively, reading and writing the STATE CSR returns
those values **minus one**.  This is absolutely critical to implement
if the STATE CSR is to be used for fast context-switching.

## Register CSR key-value (CAM) table <a name="regcsrtable" />

The purpose of the Register CSR table is four-fold:

* To mark integer and floating-point registers as requiring "redirection"
  if it is ever used as a source or destination in any given operation.
  This involves a level of indirection through a 5-to-7-bit lookup table,
  such that **unmodified** operands with 5 bit (3 for Compressed) may
  access up to **128** registers.
* To indicate whether, after redirection through the lookup table, the
  register is a vector (or remains a scalar).
* To over-ride the implicit or explicit bitwidth that the operation would
  normally give the register.

16 bit format:

| RegCAM | | 15       | (14..8)  | 7   | (6..5) | (4..0)  |
| ------ | | -        | -        | -   | ------ | ------- |
| 0      | | isvec0   | regidx0  | i/f | vew0   | regkey  |
| 1      | | isvec1   | regidx1  | i/f | vew1   | regkey  |
| ..     | | isvec..  | regidx.. | i/f | vew..  | regkey  |
| 15     | | isvec15  | regidx15 | i/f | vew15  | regkey  |

8 bit format:

| RegCAM | | 7   | (6..5) | (4..0)  |
| ------ | | -   | ------ | ------- |
| 0      | | i/f | vew0   | regnum  |

i/f is set to "1" to indicate that the redirection/tag entry is to be applied
to integer registers; 0 indicates that it is relevant to floating-point
registers.

The 8 bit format is used for a much more compact expression. "isvec" is implicit and, as in [[sv-prefix-proposal]], the target vector is "regnum<<2", implicitly. Contrast this with the 16-bit format where the target vector is *explicitly* named in bits 8 to 14, and bit 15 may optionally set "scalar" mode.

vew has the following meanings, indicating that the instruction's
operand size is "over-ridden" in a polymorphic fashion:

| vew | bitwidth            |
| --- | ------------------- |
| 00  | default (XLEN/FLEN) |
| 01  | 8 bit               |
| 10  | 16 bit              |
| 11  | 32 bit              |

As the above table is a CAM (key-value store) it may be appropriate
(faster, implementation-wise) to expand it as follows:

    struct vectorised fp_vec[32], int_vec[32];

    for (i = 0; i < 16; i++) // 16 CSRs?
       tb = int_vec if CSRvec[i].type == 0 else fp_vec
       idx = CSRvec[i].regkey // INT/FP src/dst reg in opcode
       tb[idx].elwidth  = CSRvec[i].elwidth
       tb[idx].regidx   = CSRvec[i].regidx  // indirection
       tb[idx].isvector = CSRvec[i].isvector // 0=scalar
       tb[idx].packed   = CSRvec[i].packed  // SIMD or not

The actual size of the CSR Register table depends on the platform
and on whether other Extensions are present (RV64G, RV32E, etc.).
For details see "Subsets" section.






## Predication CSR <a name="predication_csr_table"></a>

TODO: update CSR tables, now 7-bit for regidx

The Predication CSR is a key-value store indicating whether, if a given
destination register (integer or floating-point) is referred to in an
instruction, it is to be predicated.  Tt is particularly important to note
that the *actual* register used can be *different* from the one that is
in the instruction, due to the redirection through the lookup table.

* regidx is the actual register that in combination with the
  i/f flag, if that integer or floating-point register is referred to,
  results in the lookup table being referenced to find the predication
  mask to use on the operation in which that (regidx) register has
  been used
* predidx (in combination with the bank bit in the future) is the
  *actual* register to be used for the predication mask.  Note:
  in effect predidx is actually a 6-bit register address, as the bank
  bit is the MSB (and is nominally set to zero for now).
* inv indicates that the predication mask bits are to be inverted
  prior to use *without* actually modifying the contents of the
  register itself.
* zeroing is either 1 or 0, and if set to 1, the operation must
  place zeros in any element position where the predication mask is
  set to zero.  If zeroing is set to 0, unpredicated elements *must*
  be left alone.  Some microarchitectures may choose to interpret
  this as skipping the operation entirely.  Others which wish to
  stick more closely to a SIMD architecture may choose instead to
  interpret unpredicated elements as an internal "copy element"
  operation (which would be necessary in SIMD microarchitectures
  that perform register-renaming)

16 bit format:

| PrCSR | (15..11) | 10     | 9     | 8   | (7..1)  | 0       |
| ----- | -        | -      | -     | -   | ------- | ------- |
| 0     | predkey  | zero0  | inv0  | i/f | regidx  | rsrvd |
| 1     | predkey  | zero1  | inv1  | i/f | regidx  | rsvd |
| ...   | predkey  | .....  | ....  | i/f | ....... | ....... |
| 15    | predkey  | zero15 | inv15 | i/f | regidx  | rsvd |


8 bit format:

| PrCSR | 7     | 6     | 5   | (4..0)  |
| ----- | -     | -     | -   | ------- |
| 0     | zero0 | inv0  | i/f | regnum  |

The 8 bit format is a compact and less expressive variant of the full 16 bit format.  Using the 8 bit formatis very different: the predicate register to use is implicit, and numbering begins inplicitly from x9. The regnum is still used to "activate" predication.

The 16 bit Predication CSR Table is a key-value store, so implementation-wise
it will be faster to turn the table around (maintain topologically
equivalent state):

    struct pred {
        bool zero;
        bool inv;
        bool enabled;
        int predidx; // redirection: actual int register to use
    }

    struct pred fp_pred_reg[32];   // 64 in future (bank=1)
    struct pred int_pred_reg[32];  // 64 in future (bank=1)

    for (i = 0; i < 16; i++)
      tb = int_pred_reg if CSRpred[i].type == 0 else fp_pred_reg;
      idx = CSRpred[i].regidx
      tb[idx].zero = CSRpred[i].zero
      tb[idx].inv  = CSRpred[i].inv
      tb[idx].predidx  = CSRpred[i].predidx
      tb[idx].enabled  = true

So when an operation is to be predicated, it is the internal state that
is used.  In Section 6.4.2 of Hwacha's Manual (EECS-2015-262) the following
pseudo-code for operations is given, where p is the explicit (direct)
reference to the predication register to be used:

    for (int i=0; i<vl; ++i)
        if ([!]preg[p][i])
           (d ? vreg[rd][i] : sreg[rd]) =
            iop(s1 ? vreg[rs1][i] : sreg[rs1],
                s2 ? vreg[rs2][i] : sreg[rs2]); // for insts with 2 inputs

This instead becomes an *indirect* reference using the *internal* state
table generated from the Predication CSR key-value store, which is used
as follows.

    if type(iop) == INT:
        preg = int_pred_reg[rd]
    else:
        preg = fp_pred_reg[rd]

    for (int i=0; i<vl; ++i)
        predicate, zeroing = get_pred_val(type(iop) == INT, rd):
        if (predicate && (1<<i))
           (d ? regfile[rd+i] : regfile[rd]) =
            iop(s1 ? regfile[rs1+i] : regfile[rs1],
                s2 ? regfile[rs2+i] : regfile[rs2]); // for insts with 2 inputs
        else if (zeroing)
           (d ? regfile[rd+i] : regfile[rd]) = 0

Note:

* d, s1 and s2 are booleans indicating whether destination,
  source1 and source2 are vector or scalar
* key-value CSR-redirection of rd, rs1 and rs2 have NOT been included
  above, for clarity.  rd, rs1 and rs2 all also must ALSO go through
  register-level redirection (from the Register CSR table) if they are
  vectors.

If written as a function, obtaining the predication mask (and whether
zeroing takes place) may be done as follows:

    def get_pred_val(bool is_fp_op, int reg):
       tb = int_reg if is_fp_op else fp_reg
       if (!tb[reg].enabled):
          return ~0x0, False       // all enabled; no zeroing
       tb = int_pred if is_fp_op else fp_pred
       if (!tb[reg].enabled):
          return ~0x0, False       // all enabled; no zeroing
       predidx = tb[reg].predidx   // redirection occurs HERE
       predicate = intreg[predidx] // actual predicate HERE
       if (tb[reg].inv):
          predicate = ~predicate   // invert ALL bits
       return predicate, tb[reg].zero

Note here, critically, that **only** if the register is marked
in its CSR **register** table entry as being "active" does the testing
proceed further to check if the CSR **predicate** table entry is
also active.

Note also that this is in direct contrast to branch operations
for the storage of comparisions: in these specific circumstances
the requirement for there to be an active CSR *register* entry
is removed.

## REMAP CSR <a name="remap" />

(Note: both the REMAP and SHAPE sections are best read after the
 rest of the document has been read)

There is one 32-bit CSR which may be used to indicate which registers,
if used in any operation, must be "reshaped" (re-mapped) from a linear
form to a 2D or 3D transposed form, or "offset" to permit arbitrary
access to elements within a register.

The 32-bit REMAP CSR may reshape up to 3 registers:

| 29..28 | 27..26 | 25..24 | 23 | 22..16  | 15 | 14..8   | 7  | 6..0    |
| ------ | ------ | ------ | -- | ------- | -- | ------- | -- | ------- |
| shape2 | shape1 | shape0 | 0  | regidx2 | 0  | regidx1 | 0  | regidx0 |

regidx0-2 refer not to the Register CSR CAM entry but to the underlying
*real* register (see regidx, the value) and consequently is 7-bits wide.
When set to zero (referring to x0), clearly reshaping x0 is pointless,
so is used to indicate "disabled".
shape0-2 refers to one of three SHAPE CSRs.  A value of 0x3 is reserved.
Bits 7, 15, 23, 30 and 31 are also reserved, and must be set to zero.

It is anticipated that these specialist CSRs not be very often used.
Unlike the CSR Register and Predication tables, the REMAP CSRs use
the full 7-bit regidx so that they can be set once and left alone,
whilst the CSR Register entries pointing to them are disabled, instead.

## SHAPE 1D/2D/3D vector-matrix remapping CSRs

(Note: both the REMAP and SHAPE sections are best read after the
 rest of the document has been read)

There are three "shape" CSRs, SHAPE0, SHAPE1, SHAPE2, 32-bits in each,
which have the same format.  When each SHAPE CSR is set entirely to zeros,
remapping is disabled: the register's elements are a linear (1D) vector.

| 26..24  | 23      | 22..16  | 15      | 14..8   | 7       | 6..0    |
| ------- | --      | ------- | --      | ------- | --      | ------- |
| permute | offs[2] | zdimsz  | offs[1] | ydimsz  | offs[0] | xdimsz  |

offs is a 3-bit field, spread out across bits 7, 15 and 23, which
is added to the element index during the loop calculation.

xdimsz, ydimsz and zdimsz are offset by 1, such that a value of 0 indicates
that the array dimensionality for that dimension is 1.  A value of xdimsz=2
would indicate that in the first dimension there are 3 elements in the
array.  The format of the array is therefore as follows:

    array[xdim+1][ydim+1][zdim+1]

However whilst illustrative of the dimensionality, that does not take the
"permute" setting into account.  "permute" may be any one of six values
(0-5, with values of 6 and 7 being reserved, and not legal).  The table
below shows how the permutation dimensionality order works:

| permute | order | array format             |
| ------- | ----- | ------------------------ |
| 000     | 0,1,2 | (xdim+1)(ydim+1)(zdim+1) |
| 001     | 0,2,1 | (xdim+1)(zdim+1)(ydim+1) |
| 010     | 1,0,2 | (ydim+1)(xdim+1)(zdim+1) |
| 011     | 1,2,0 | (ydim+1)(zdim+1)(xdim+1) |
| 100     | 2,0,1 | (zdim+1)(xdim+1)(ydim+1) |
| 101     | 2,1,0 | (zdim+1)(ydim+1)(xdim+1) |

In other words, the "permute" option changes the order in which
nested for-loops over the array would be done.  The algorithm below
shows this more clearly, and may be executed as a python program:

    # mapidx = REMAP.shape2
    xdim = 3 # SHAPE[mapidx].xdim_sz+1
    ydim = 4 # SHAPE[mapidx].ydim_sz+1
    zdim = 5 # SHAPE[mapidx].zdim_sz+1

    lims = [xdim, ydim, zdim]
    idxs = [0,0,0] # starting indices
    order = [1,0,2] # experiment with different permutations, here
    offs = 0        # experiment with different offsets, here

    for idx in range(xdim * ydim * zdim):
        new_idx = offs + idxs[0] + idxs[1] * xdim + idxs[2] * xdim * ydim
        print new_idx,
        for i in range(3):
            idxs[order[i]] = idxs[order[i]] + 1
            if (idxs[order[i]] != lims[order[i]]):
                break
            print
            idxs[order[i]] = 0

Here, it is assumed that this algorithm be run within all pseudo-code
throughout this document where a (parallelism) for-loop would normally
run from 0 to VL-1 to refer to contiguous register
elements; instead, where REMAP indicates to do so, the element index
is run through the above algorithm to work out the **actual** element
index, instead.  Given that there are three possible SHAPE entries, up to
three separate registers in any given operation may be simultaneously
remapped:

    function op_add(rd, rs1, rs2) # add not VADD!
      ...
      ...
      for (i = 0; i < VL; i++)
        if (predval & 1<<i) # predication uses intregs
           ireg[rd+remap(id)] <= ireg[rs1+remap(irs1)] +
                                 ireg[rs2+remap(irs2)];
           if (!int_vec[rd ].isvector) break;
        if (int_vec[rd ].isvector)  { id += 1; }
        if (int_vec[rs1].isvector)  { irs1 += 1; }
        if (int_vec[rs2].isvector)  { irs2 += 1; }

By changing remappings, 2D matrices may be transposed "in-place" for one
operation, followed by setting a different permutation order without
having to move the values in the registers to or from memory.  Also,
the reason for having REMAP separate from the three SHAPE CSRs is so
that in a chain of matrix multiplications and additions, for example,
the SHAPE CSRs need only be set up once; only the REMAP CSR need be
changed to target different registers.

Note that:

* Over-running the register file clearly has to be detected and
  an illegal instruction exception thrown
* When non-default elwidths are set, the exact same algorithm still
  applies (i.e. it offsets elements *within* registers rather than
  entire registers).
* If permute option 000 is utilised, the actual order of the
  reindexing does not change!
* If two or more dimensions are set to zero, the actual order does not change!
* The above algorithm is pseudo-code **only**.  Actual implementations
  will need to take into account the fact that the element for-looping
  must be **re-entrant**, due to the possibility of exceptions occurring.
  See MSTATE CSR, which records the current element index.
* Twin-predicated operations require **two** separate and distinct
  element offsets.  The above pseudo-code algorithm will be applied
  separately and independently to each, should each of the two
  operands be remapped.  *This even includes C.LDSP* and other operations
  in that category, where in that case it will be the **offset** that is
  remapped (see Compressed Stack LOAD/STORE section).
* Offset is especially useful, on its own, for accessing elements
  within the middle of a register.  Without offsets, it is necessary
  to either use a predicated MV, skipping the first elements, or
  performing a LOAD/STORE cycle to memory.
  With offsets, the data does not have to be moved.
* Setting the total elements (xdim+1) times (ydim+1) times (zdim+1) to
  less than MVL is **perfectly legal**, albeit very obscure.  It permits
  entries to be regularly presented to operands **more than once**, thus
  allowing the same underlying registers to act as an accumulator of
  multiple vector or matrix operations, for example.

Clearly here some considerable care needs to be taken as the remapping
could hypothetically create arithmetic operations that target the
exact same underlying registers, resulting in data corruption due to
pipeline overlaps.  Out-of-order / Superscalar micro-architectures with
register-renaming will have an easier time dealing with this than
DSP-style SIMD micro-architectures.

# Instruction Execution Order

Simple-V behaves as if it is a hardware-level "macro expansion system",
substituting and expanding a single instruction into multiple sequential
instructions with contiguous and sequentially-incrementing registers.
As such, it does **not** modify - or specify - the behaviour and semantics of
the execution order: that may be deduced from the **existing** RV
specification in each and every case.

So for example if a particular micro-architecture permits out-of-order
execution, and it is augmented with Simple-V, then wherever instructions
may be out-of-order then so may the "post-expansion" SV ones.

If on the other hand there are memory guarantees which specifically
prevent and prohibit certain instructions from being re-ordered
(such as the Atomicity Axiom, or FENCE constraints), then clearly
those constraints **MUST** also be obeyed "post-expansion".

It should be absolutely clear that SV is **not** about providing new
functionality or changing the existing behaviour of a micro-architetural
design, or about changing the RISC-V Specification.
It is **purely** about compacting what would otherwise be contiguous
instructions that use sequentially-increasing register numbers down
to the **one** instruction.

# Instructions <a name="instructions" />

Despite being a 98% complete and accurate topological remap of RVV
concepts and functionality, no new instructions are needed.
Compared to RVV: *All* RVV instructions can be re-mapped, however xBitManip
becomes a critical dependency for efficient manipulation of predication
masks (as a bit-field).  Despite the removal of all operations,
with the exception of CLIP and VSELECT.X
*all instructions from RVV Base are topologically re-mapped and retain their
complete functionality, intact*.  Note that if RV64G ever had
a MV.X added as well as FCLIP, the full functionality of RVV-Base would
be obtained in SV.

Three instructions, VSELECT, VCLIP and VCLIPI, do not have RV Standard
equivalents, so are left out of Simple-V.  VSELECT could be included if
there existed a MV.X instruction in RV (MV.X is a hypothetical
non-immediate variant of MV that would allow another register to
specify which register was to be copied).  Note that if any of these three
instructions are added to any given RV extension, their functionality
will be inherently parallelised.

With some exceptions, where it does not make sense or is simply too
challenging, all RV-Base instructions are parallelised:

* CSR instructions, whilst a case could be made for fast-polling of
  a CSR into multiple registers, or for being able to copy multiple
  contiguously addressed CSRs into contiguous registers, and so on,
  are the fundamental core basis of SV.  If parallelised, extreme
  care would need to be taken.  Additionally, CSR reads are done
  using x0, and it is *really* inadviseable to tag x0.
* LUI, C.J, C.JR, WFI, AUIPC are not suitable for parallelising so are
  left as scalar.
* LR/SC could hypothetically be parallelised however their purpose is
  single (complex) atomic memory operations where the LR must be followed
  up by a matching SC.  A sequence of parallel LR instructions followed
  by a sequence of parallel SC instructions therefore is guaranteed to
  not be useful. Not least: the guarantees of a Multi-LR/SC
  would be impossible to provide if emulated in a trap.
* EBREAK, NOP, FENCE and others do not use registers so are not inherently
  paralleliseable anyway.

All other operations using registers are automatically parallelised.
This includes AMOMAX, AMOSWAP and so on, where particular care and
attention must be paid.

Example pseudo-code for an integer ADD operation (including scalar operations).
Floating-point uses fp csrs.

    function op_add(rd, rs1, rs2) # add not VADD!
      int i, id=0, irs1=0, irs2=0;
      predval = get_pred_val(FALSE, rd);
      rd  = int_vec[rd ].isvector ? int_vec[rd ].regidx : rd;
      rs1 = int_vec[rs1].isvector ? int_vec[rs1].regidx : rs1;
      rs2 = int_vec[rs2].isvector ? int_vec[rs2].regidx : rs2;
      for (i = 0; i < VL; i++)
        if (predval & 1<<i) # predication uses intregs
           ireg[rd+id] <= ireg[rs1+irs1] + ireg[rs2+irs2];
           if (!int_vec[rd ].isvector) break;
        if (int_vec[rd ].isvector)  { id += 1; }
        if (int_vec[rs1].isvector)  { irs1 += 1; }
        if (int_vec[rs2].isvector)  { irs2 += 1; }

Note that for simplicity there is quite a lot missing from the above
pseudo-code: element widths, zeroing on predication, dimensional
reshaping and offsets and so on.  However it demonstrates the basic
principle.  Augmentations that produce the full pseudo-code are covered in
other sections.

## Instruction Format

It is critical to appreciate that there are
**no operations added to SV, at all**.

Instead, by using CSRs to tag registers as an indication of "changed behaviour",
SV *overloads* pre-existing branch operations into predicated
variants, and implicitly overloads arithmetic operations, MV,
FCVT, and LOAD/STORE depending on CSR configurations for bitwidth
and predication.  **Everything** becomes parallelised.  *This includes
Compressed instructions* as well as any future instructions and Custom
Extensions.

Note: CSR tags to change behaviour of instructions is nothing new, including
in RISC-V.  UXL, SXL and MXL change the behaviour so that XLEN=32/64/128.
FRM changes the behaviour of the floating-point unit, to alter the rounding
mode.  Other architectures change the LOAD/STORE byte-order from big-endian
to little-endian on a per-instruction basis.  SV is just a little more...
comprehensive in its effect on instructions.

## Branch Instructions

### Standard Branch <a name="standard_branch"></a>

Branch operations use standard RV opcodes that are reinterpreted to
be "predicate variants" in the instance where either of the two src
registers are marked as vectors (active=1, vector=1).

Note that the predication register to use (if one is enabled) is taken from
the *first* src register, and that this is used, just as with predicated
arithmetic operations, to mask whether the comparison operations take
place or not.  The target (destination) predication register
to use (if one is enabled) is taken from the *second* src register.

If either of src1 or src2 are scalars (whether by there being no
CSR register entry or whether by the CSR entry specifically marking
the register as "scalar") the comparison goes ahead as vector-scalar
or scalar-vector.

In instances where no vectorisation is detected on either src registers
the operation is treated as an absolutely standard scalar branch operation.
Where vectorisation is present on either or both src registers, the
branch may stil go ahead if any only if *all* tests succeed (i.e. excluding
those tests that are predicated out).

Note that when zero-predication is enabled (from source rs1),
a cleared bit in the predicate indicates that the result
of the compare is set to "false", i.e. that the corresponding
destination bit (or result)) be set to zero.  Contrast this with
when zeroing is not set: bits in the destination predicate are
only *set*; they are **not** cleared.  This is important to appreciate,
as there may be an expectation that, going into the hardware-loop,
the destination predicate is always expected to be set to zero:
this is **not** the case.  The destination predicate is only set
to zero if **zeroing** is enabled.

Note that just as with the standard (scalar, non-predicated) branch
operations, BLE, BGT, BLEU and BTGU may be synthesised by inverting
src1 and src2.

In Hwacha EECS-2015-262 Section 6.7.2 the following pseudocode is given
for predicated compare operations of function "cmp":

    for (int i=0; i<vl; ++i)
      if ([!]preg[p][i])
         preg[pd][i] = cmp(s1 ? vreg[rs1][i] : sreg[rs1],
                           s2 ? vreg[rs2][i] : sreg[rs2]);

With associated predication, vector-length adjustments and so on,
and temporarily ignoring bitwidth (which makes the comparisons more
complex), this becomes:

    s1 = reg_is_vectorised(src1);
    s2 = reg_is_vectorised(src2);

    if not s1 && not s2
        if cmp(rs1, rs2) # scalar compare
            goto branch
        return

    preg = int_pred_reg[rd]
    reg = int_regfile

    ps = get_pred_val(I/F==INT, rs1);
    rd = get_pred_val(I/F==INT, rs2); # this may not exist

    if not exists(rd) or zeroing:
        result = 0
    else
        result = preg[rd]

    for (int i = 0; i < VL; ++i)
      if (zeroing)
        if not (ps & (1<<i))
           result &= ~(1<<i);
      else if (ps & (1<<i))
          if (cmp(s1 ? reg[src1+i]:reg[src1],
                               s2 ? reg[src2+i]:reg[src2])
              result |= 1<<i;
          else
              result &= ~(1<<i);

     if not exists(rd)
        if result == ps
            goto branch
     else
        preg[rd] = result # store in destination
        if preg[rd] == ps
            goto branch

Notes:

* Predicated SIMD comparisons would break src1 and src2 further down
  into bitwidth-sized chunks (see Appendix "Bitwidth Virtual Register
  Reordering") setting Vector-Length times (number of SIMD elements) bits
  in Predicate Register rd, as opposed to just Vector-Length bits.
* The execution of "parallelised" instructions **must** be implemented
  as "re-entrant" (to use a term from software).  If an exception (trap)
  occurs during the middle of a vectorised
  Branch (now a SV predicated compare) operation, the partial results
  of any comparisons must be written out to the destination
  register before the trap is permitted to begin.  If however there
  is no predicate, the **entire** set of comparisons must be **restarted**,
  with the offset loop indices set back to zero.  This is because
  there is no place to store the temporary result during the handling
  of traps.

TODO: predication now taken from src2.  also branch goes ahead
if all compares are successful.

Note also that where normally, predication requires that there must
also be a CSR register entry for the register being used in order
for the **predication** CSR register entry to also be active,
for branches this is **not** the case.  src2 does **not** have
to have its CSR register entry marked as active in order for
predication on src2 to be active.

Also note: SV Branch operations are **not** twin-predicated
(see Twin Predication section).  This would require three
element offsets: one to track src1, one to track src2 and a third
to track where to store the accumulation of the results.  Given
that the element offsets need to be exposed via CSRs so that
the parallel hardware looping may be made re-entrant on traps
and exceptions, the decision was made not to make SV Branches
twin-predicated.

### Floating-point Comparisons

There does not exist floating-point branch operations, only compare.
Interestingly no change is needed to the instruction format because
FP Compare already stores a 1 or a zero in its "rd" integer register
target, i.e. it's not actually a Branch at all: it's a compare.

In RV (scalar) Base, a branch on a floating-point compare is
done via the sequence "FEQ x1, f0, f5; BEQ x1, x0, #jumploc".
This does extend to SV, as long as x1 (in the example sequence given)
is vectorised.  When that is the case, x1..x(1+VL-1) will also be
set to 0 or 1 depending on whether f0==f5, f1==f6, f2==f7 and so on.
The BEQ that follows will *also* compare x1==x0, x2==x0, x3==x0 and
so on.  Consequently, unlike integer-branch, FP Compare needs no
modification in its behaviour.

In addition, it is noted that an entry "FNE" (the opposite of FEQ) is missing,
and whilst in ordinary branch code this is fine because the standard
RVF compare can always be followed up with an integer BEQ or a BNE (or
a compressed comparison to zero or non-zero), in predication terms that
becomes more of an impact.  To deal with this, SV's predication has
had "invert" added to it.

Also: note that FP Compare may be predicated, using the destination
integer register (rd) to determine the predicate.  FP Compare is **not**
a twin-predication operation, as, again, just as with SV Branches,
there are three registers involved: FP src1, FP src2 and INT rd.

### Compressed Branch Instruction

Compressed Branch instructions are, just like standard Branch instructions,
reinterpreted to be vectorised and predicated based on the source register
(rs1s) CSR entries.  As however there is only the one source register,
given that c.beqz a10 is equivalent to beqz a10,x0, the optional target
to store the results of the comparisions is taken from CSR predication
table entries for **x0**.

The specific required use of x0 is, with a little thought, quite obvious,
but is counterintuitive.  Clearly it is **not** recommended to redirect
x0 with a CSR register entry, however as a means to opaquely obtain
a predication target it is the only sensible option that does not involve
additional special CSRs (or, worse, additional special opcodes).

Note also that, just as with standard branches, the 2nd source
(in this case x0 rather than src2) does **not** have to have its CSR
register table marked as "active" in order for predication to work.

## Vectorised Dual-operand instructions

There is a series of 2-operand instructions involving copying (and
sometimes alteration):

* C.MV
* FMV, FNEG, FABS, FCVT, FSGNJ, FSGNJN and FSGNJX
* C.LWSP, C.SWSP, C.LDSP, C.FLWSP etc.
* LOAD(-FP) and STORE(-FP)

All of these operations follow the same two-operand pattern, so it is
*both* the source *and* destination predication masks that are taken into
account.  This is different from
the three-operand arithmetic instructions, where the predication mask
is taken from the *destination* register, and applied uniformly to the
elements of the source register(s), element-for-element.

The pseudo-code pattern for twin-predicated operations is as
follows:

    function op(rd, rs):
      rd = int_csr[rd].active ? int_csr[rd].regidx : rd;
      rs = int_csr[rs].active ? int_csr[rs].regidx : rs;
      ps = get_pred_val(FALSE, rs); # predication on src
      pd = get_pred_val(FALSE, rd); # ... AND on dest
      for (int i = 0, int j = 0; i < VL && j < VL;):
        if (int_csr[rs].isvec) while (!(ps & 1<<i)) i++;
        if (int_csr[rd].isvec) while (!(pd & 1<<j)) j++;
        reg[rd+j] = SCALAR_OPERATION_ON(reg[rs+i])
        if (int_csr[rs].isvec) i++;
        if (int_csr[rd].isvec) j++; else break

This pattern covers scalar-scalar, scalar-vector, vector-scalar
and vector-vector, and predicated variants of all of those.
Zeroing is not presently included (TODO).  As such, when compared
to RVV, the twin-predicated variants of C.MV and FMV cover
**all** standard vector operations: VINSERT, VSPLAT, VREDUCE,
VEXTRACT, VSCATTER, VGATHER, VCOPY, and more.

Note that:

* elwidth (SIMD) is not covered in the pseudo-code above
* ending the loop early in scalar cases (VINSERT, VEXTRACT) is also
  not covered
* zero predication is also not shown (TODO).

### C.MV Instruction <a name="c_mv"></a>

There is no MV instruction in RV however there is a C.MV instruction.
It is used for copying integer-to-integer registers (vectorised FMV
is used for copying floating-point).

If either the source or the destination register are marked as vectors
C.MV is reinterpreted to be a vectorised (multi-register) predicated
move operation.  The actual instruction's format does not change:

[[!table  data="""
15  12 | 11   7 | 6  2 | 1  0 |
funct4 | rd     | rs   | op   |
4      | 5      | 5    | 2    |
C.MV   | dest   | src  | C0   |
"""]]

A simplified version of the pseudocode for this operation is as follows:

    function op_mv(rd, rs) # MV not VMV!
      rd = int_csr[rd].active ? int_csr[rd].regidx : rd;
      rs = int_csr[rs].active ? int_csr[rs].regidx : rs;
      ps = get_pred_val(FALSE, rs); # predication on src
      pd = get_pred_val(FALSE, rd); # ... AND on dest
      for (int i = 0, int j = 0; i < VL && j < VL;):
        if (int_csr[rs].isvec) while (!(ps & 1<<i)) i++;
        if (int_csr[rd].isvec) while (!(pd & 1<<j)) j++;
        ireg[rd+j] <= ireg[rs+i];
        if (int_csr[rs].isvec) i++;
        if (int_csr[rd].isvec) j++; else break

There are several different instructions from RVV that are covered by
this one opcode:

[[!table  data="""
src    | dest    | predication   | op             |
scalar | vector  | none          | VSPLAT         |
scalar | vector  | destination   | sparse VSPLAT  |
scalar | vector  | 1-bit dest    | VINSERT        |
vector | scalar  | 1-bit? src    | VEXTRACT       |
vector | vector  | none          | VCOPY          |
vector | vector  | src           | Vector Gather  |
vector | vector  | dest          | Vector Scatter |
vector | vector  | src & dest    | Gather/Scatter |
vector | vector  | src == dest   | sparse VCOPY   |
"""]]

Also, VMERGE may be implemented as back-to-back (macro-op fused) C.MV
operations with inversion on the src and dest predication for one of the
two C.MV operations.

Note that in the instance where the Compressed Extension is not implemented,
MV may be used, but that is a pseudo-operation mapping to addi rd, x0, rs.
Note that the behaviour is **different** from C.MV because with addi the
predication mask to use is taken **only** from rd and is applied against
all elements: rs[i] = rd[i].

### FMV, FNEG and FABS Instructions

These are identical in form to C.MV, except covering floating-point
register copying.  The same double-predication rules also apply.
However when elwidth is not set to default the instruction is implicitly
and automatic converted to a (vectorised) floating-point type conversion
operation of the appropriate size covering the source and destination
register bitwidths.

(Note that FMV, FNEG and FABS are all actually pseudo-instructions)

### FVCT Instructions

These are again identical in form to C.MV, except that they cover
floating-point to integer and integer to floating-point.  When element
width in each vector is set to default, the instructions behave exactly
as they are defined for standard RV (scalar) operations, except vectorised
in exactly the same fashion as outlined in C.MV.

However when the source or destination element width is not set to default,
the opcode's explicit element widths are *over-ridden* to new definitions,
and the opcode's element width is taken as indicative of the SIMD width
(if applicable i.e. if packed SIMD is requested) instead.

For example FCVT.S.L would normally be used to convert a 64-bit
integer in register rs1 to a 64-bit floating-point number in rd.
If however the source rs1 is set to be a vector, where elwidth is set to
default/2 and "packed SIMD" is enabled, then the first 32 bits of
rs1 are converted to a floating-point number to be stored in rd's
first element and the higher 32-bits *also* converted to floating-point
and stored in the second.  The 32 bit size comes from the fact that
FCVT.S.L's integer width is 64 bit, and with elwidth on rs1 set to
divide that by two it means that rs1 element width is to be taken as 32.

Similar rules apply to the destination register.

## LOAD / STORE Instructions and LOAD-FP/STORE-FP <a name="load_store"></a>

An earlier draft of SV modified the behaviour of LOAD/STORE (modified
the interpretation of the instruction fields).  This
actually undermined the fundamental principle of SV, namely that there
be no modifications to the scalar behaviour (except where absolutely
necessary), in order to simplify an implementor's task if considering
converting a pre-existing scalar design to support parallelism.

So the original RISC-V scalar LOAD/STORE and LOAD-FP/STORE-FP functionality
do not change in SV, however just as with C.MV it is important to note
that dual-predication is possible.

In vectorised architectures there are usually at least two different modes
for LOAD/STORE:

* Read (or write for STORE) from sequential locations, where one
  register specifies the address, and the one address is incremented
  by a fixed amount.  This is usually known as "Unit Stride" mode.
* Read (or write) from multiple indirected addresses, where the
  vector elements each specify separate and distinct addresses.

To support these different addressing modes, the CSR Register "isvector"
bit is used.  So, for a LOAD, when the src register is set to
scalar, the LOADs are sequentially incremented by the src register
element width, and when the src register is set to "vector", the
elements are treated as indirection addresses.  Simplified
pseudo-code would look like this:

    function op_ld(rd, rs) # LD not VLD!
      rdv = int_csr[rd].active ? int_csr[rd].regidx : rd;
      rsv = int_csr[rs].active ? int_csr[rs].regidx : rs;
      ps = get_pred_val(FALSE, rs); # predication on src
      pd = get_pred_val(FALSE, rd); # ... AND on dest
      for (int i = 0, int j = 0; i < VL && j < VL;):
        if (int_csr[rs].isvec) while (!(ps & 1<<i)) i++;
        if (int_csr[rd].isvec) while (!(pd & 1<<j)) j++;
        if (int_csr[rd].isvec)
          # indirect mode (multi mode)
          srcbase = ireg[rsv+i];
        else
          # unit stride mode
          srcbase = ireg[rsv] + i * XLEN/8; # offset in bytes
        ireg[rdv+j] <= mem[srcbase + imm_offs];
        if (!int_csr[rs].isvec &&
            !int_csr[rd].isvec) break # scalar-scalar LD
        if (int_csr[rs].isvec) i++;
        if (int_csr[rd].isvec) j++;

Notes:

* For simplicity, zeroing and elwidth is not included in the above:
  the key focus here is the decision-making for srcbase; vectorised
  rs means use sequentially-numbered registers as the indirection
  address, and scalar rs is "offset" mode.
* The test towards the end for whether both source and destination are
  scalar is what makes the above pseudo-code provide the "standard" RV
  Base behaviour for LD operations.
* The offset in bytes (XLEN/8) changes depending on whether the
  operation is a LB (1 byte), LH (2 byes), LW (4 bytes) or LD
  (8 bytes), and also whether the element width is over-ridden
  (see special element width section).

## Compressed Stack LOAD / STORE Instructions <a name="c_ld_st"></a>

C.LWSP / C.SWSP and floating-point etc. are also source-dest twin-predicated,
where it is implicit in C.LWSP/FLWSP etc. that x2 is the source register.
It is therefore possible to use predicated C.LWSP to efficiently
pop registers off the stack (by predicating x2 as the source), cherry-picking
which registers to store to (by predicating the destination).  Likewise
for C.SWSP.  In this way, LOAD/STORE-Multiple is efficiently achieved.

The two modes ("unit stride" and multi-indirection) are still supported,
as with standard LD/ST.  Essentially, the only difference is that the
use of x2 is hard-coded into the instruction.

**Note**: it is still possible to redirect x2 to an alternative target
register.  With care, this allows C.LWSP / C.SWSP (and C.FLWSP) to be used as
general-purpose LOAD/STORE operations.

## Compressed LOAD / STORE Instructions

Compressed LOAD and STORE are again exactly the same as scalar LOAD/STORE,
where the same rules apply and the same pseudo-code apply as for
non-compressed LOAD/STORE.  Again: setting scalar or vector mode
on the src for LOAD and dest for STORE switches mode from "Unit Stride"
to "Multi-indirection", respectively.

# Element bitwidth polymorphism <a name="elwidth"></a>

Element bitwidth is best covered as its own special section, as it
is quite involved and applies uniformly across-the-board.  SV restricts
bitwidth polymorphism to default, 8-bit, 16-bit and 32-bit.

The effect of setting an element bitwidth is to re-cast each entry
in the register table, and for all memory operations involving
load/stores of certain specific sizes, to a completely different width.
Thus In c-style terms, on an RV64 architecture, effectively each register
now looks like this:

    typedef union {
        uint8_t  b[8];
        uint16_t s[4];
        uint32_t i[2];
        uint64_t l[1];
    } reg_t;

    // integer table: assume maximum SV 7-bit regfile size
    reg_t int_regfile[128];

where the CSR Register table entry (not the instruction alone) determines
which of those union entries is to be used on each operation, and the
VL element offset in the hardware-loop specifies the index into each array.

However a naive interpretation of the data structure above masks the
fact that setting VL greater than 8, for example, when the bitwidth is 8,
accessing one specific register "spills over" to the following parts of
the register file in a sequential fashion.  So a much more accurate way
to reflect this would be:

    typedef union {
        uint8_t  actual_bytes[8]; // 8 for RV64, 4 for RV32, 16 for RV128
        uint8_t  b[0]; // array of type uint8_t
        uint16_t s[0];
        uint32_t i[0];
        uint64_t l[0];
        uint128_t d[0];
    } reg_t;

    reg_t int_regfile[128];

where when accessing any individual regfile[n].b entry it is permitted
(in c) to arbitrarily over-run the *declared* length of the array (zero),
and thus "overspill" to consecutive register file entries in a fashion
that is completely transparent to a greatly-simplified software / pseudo-code
representation.
It is however critical to note that it is clearly the responsibility of
the implementor to ensure that, towards the end of the register file,
an exception is thrown if attempts to access beyond the "real" register
bytes is ever attempted.

Now we may modify pseudo-code an operation where all element bitwidths have
been set to the same size, where this pseudo-code is otherwise identical
to its "non" polymorphic versions (above):

    function op_add(rd, rs1, rs2) # add not VADD!
      ...
      ...
      for (i = 0; i < VL; i++)
           ...
           ...
           // TODO, calculate if over-run occurs, for each elwidth
           if (elwidth == 8) {
               int_regfile[rd].b[id] <= int_regfile[rs1].i[irs1] +
                                        int_regfile[rs2].i[irs2];
            } else if elwidth == 16 {
               int_regfile[rd].s[id] <= int_regfile[rs1].s[irs1] +
                                        int_regfile[rs2].s[irs2];
            } else if elwidth == 32 {
               int_regfile[rd].i[id] <= int_regfile[rs1].i[irs1] +
                                        int_regfile[rs2].i[irs2];
            } else { // elwidth == 64
               int_regfile[rd].l[id] <= int_regfile[rs1].l[irs1] +
                                        int_regfile[rs2].l[irs2];
            }
           ...
           ...

So here we can see clearly: for 8-bit entries rd, rs1 and rs2 (and registers
following sequentially on respectively from the same) are "type-cast"
to 8-bit; for 16-bit entries likewise and so on.

However that only covers the case where the element widths are the same.
Where the element widths are different, the following algorithm applies:

* Analyse the bitwidth of all source operands and work out the
  maximum.  Record this as "maxsrcbitwidth"
* If any given source operand requires sign-extension or zero-extension
  (ldb, div, rem, mul, sll, srl, sra etc.), instead of mandatory 32-bit
  sign-extension / zero-extension or whatever is specified in the standard
  RV specification, **change** that to sign-extending from the respective
  individual source operand's bitwidth from the CSR table out to
  "maxsrcbitwidth" (previously calculated), instead.
* Following separate and distinct (optional) sign/zero-extension of all
  source operands as specifically required for that operation, carry out the
  operation at "maxsrcbitwidth".  (Note that in the case of LOAD/STORE or MV
  this may be a "null" (copy) operation, and that with FCVT, the changes
  to the source and destination bitwidths may also turn FVCT effectively
  into a copy).
* If the destination operand requires sign-extension or zero-extension,
  instead of a mandatory fixed size (typically 32-bit for arithmetic,
  for subw for example, and otherwise various: 8-bit for sb, 16-bit for sw
  etc.), overload the RV specification with the bitwidth from the
  destination register's elwidth entry.
* Finally, store the (optionally) sign/zero-extended value into its
  destination: memory for sb/sw etc., or an offset section of the register
  file for an arithmetic operation.

In this way, polymorphic bitwidths are achieved without requiring a
massive 64-way permutation of calculations **per opcode**, for example
(4 possible rs1 bitwidths times 4 possible rs2 bitwidths times 4 possible
rd bitwidths).  The pseudo-code is therefore as follows:

    typedef union {
        uint8_t  b;
        uint16_t s;
        uint32_t i;
        uint64_t l;
    } el_reg_t;

    bw(elwidth):
        if elwidth == 0:
            return xlen
        if elwidth == 1:
            return xlen / 2
        if elwidth == 2:
            return xlen * 2
        // elwidth == 3:
        return 8

    get_max_elwidth(rs1, rs2):
        return max(bw(int_csr[rs1].elwidth), # default (XLEN) if not set
                   bw(int_csr[rs2].elwidth)) # again XLEN if no entry

    get_polymorphed_reg(reg, bitwidth, offset):
        el_reg_t res;
        res.l = 0; // TODO: going to need sign-extending / zero-extending
        if bitwidth == 8:
            reg.b = int_regfile[reg].b[offset]
        elif bitwidth == 16:
            reg.s = int_regfile[reg].s[offset]
        elif bitwidth == 32:
            reg.i = int_regfile[reg].i[offset]
        elif bitwidth == 64:
            reg.l = int_regfile[reg].l[offset]
        return res

    set_polymorphed_reg(reg, bitwidth, offset, val):
        if (!int_csr[reg].isvec):
            # sign/zero-extend depending on opcode requirements, from
            # the reg's bitwidth out to the full bitwidth of the regfile
            val = sign_or_zero_extend(val, bitwidth, xlen)
            int_regfile[reg].l[0] = val
        elif bitwidth == 8:
            int_regfile[reg].b[offset] = val
        elif bitwidth == 16:
            int_regfile[reg].s[offset] = val
        elif bitwidth == 32:
            int_regfile[reg].i[offset] = val
        elif bitwidth == 64:
            int_regfile[reg].l[offset] = val

      maxsrcwid =  get_max_elwidth(rs1, rs2) # source element width(s)
      destwid = int_csr[rs1].elwidth         # destination element width
      for (i = 0; i < VL; i++)
        if (predval & 1<<i) # predication uses intregs
           // TODO, calculate if over-run occurs, for each elwidth
           src1 = get_polymorphed_reg(rs1, maxsrcwid, irs1)
           // TODO, sign/zero-extend src1 and src2 as operation requires
           if (op_requires_sign_extend_src1)
              src1 = sign_extend(src1, maxsrcwid)
           src2 = get_polymorphed_reg(rs2, maxsrcwid, irs2)
           result = src1 + src2 # actual add here
           // TODO, sign/zero-extend result, as operation requires
           if (op_requires_sign_extend_dest)
              result = sign_extend(result, maxsrcwid)
           set_polymorphed_reg(rd, destwid, ird, result)
           if (!int_vec[rd].isvector) break
        if (int_vec[rd ].isvector)  { id += 1; }
        if (int_vec[rs1].isvector)  { irs1 += 1; }
        if (int_vec[rs2].isvector)  { irs2 += 1; }

Whilst specific sign-extension and zero-extension pseudocode call
details are left out, due to each operation being different, the above
should be clear that;

* the source operands are extended out to the maximum bitwidth of all
  source operands
* the operation takes place at that maximum source bitwidth (the
  destination bitwidth is not involved at this point, at all)
* the result is extended (or potentially even, truncated) before being
  stored in the destination.  i.e. truncation (if required) to the
  destination width occurs **after** the operation **not** before.
* when the destination is not marked as "vectorised", the **full**
  (standard, scalar) register file entry is taken up, i.e. the
  element is either sign-extended or zero-extended to cover the
  full register bitwidth (XLEN) if it is not already XLEN bits long.

Implementors are entirely free to optimise the above, particularly
if it is specifically known that any given operation will complete
accurately in less bits, as long as the results produced are
directly equivalent and equal, for all inputs and all outputs,
to those produced by the above algorithm.

## Polymorphic floating-point operation exceptions and error-handling

For floating-point operations, conversion takes place without
raising any kind of exception.  Exactly as specified in the standard
RV specification, NAN (or appropriate) is stored if the result
is beyond the range of the destination, and, again, exactly as
with the standard RV specification just as with scalar
operations, the floating-point flag is raised (FCSR).  And, again, just as
with scalar operations, it is software's responsibility to check this flag.
Given that the FCSR flags are "accrued", the fact that multiple element
operations could have occurred is not a problem.

Note that it is perfectly legitimate for floating-point bitwidths of
only 8 to be specified.  However whilst it is possible to apply IEEE 754
principles, no actual standard yet exists.  Implementors wishing to
provide hardware-level 8-bit support rather than throw a trap to emulate
in software should contact the author of this specification before
proceeding.

## Polymorphic shift operators

A special note is needed for changing the element width of left and right
shift operators, particularly right-shift.  Even for standard RV base,
in order for correct results to be returned, the second operand RS2 must
be truncated to be within the range of RS1's bitwidth.  spike's implementation
of sll for example is as follows:

    WRITE_RD(sext_xlen(zext_xlen(RS1) << (RS2 & (xlen-1))));

which means: where XLEN is 32 (for RV32), restrict RS2 to cover the
range 0..31 so that RS1 will only be left-shifted by the amount that
is possible to fit into a 32-bit register.  Whilst this appears not
to matter for hardware, it matters greatly in software implementations,
and it also matters where an RV64 system is set to "RV32" mode, such
that the underlying registers RS1 and RS2 comprise 64 hardware bits
each.

For SV, where each operand's element bitwidth may be over-ridden, the
rule about determining the operation's bitwidth *still applies*, being
defined as the maximum bitwidth of RS1 and RS2.  *However*, this rule
**also applies to the truncation of RS2**.  In other words, *after*
determining the maximum bitwidth, RS2's range must **also be truncated**
to ensure a correct answer.  Example:

* RS1 is over-ridden to a 16-bit width
* RS2 is over-ridden to an 8-bit width
* RD is over-ridden to a 64-bit width
* the maximum bitwidth is thus determined to be 16-bit - max(8,16)
* RS2 is **truncated to a range of values from 0 to 15**: RS2 & (16-1)

Pseudocode (in spike) for this example would therefore be:

    WRITE_RD(sext_xlen(zext_16bit(RS1) << (RS2 & (16-1))));

This example illustrates that considerable care therefore needs to be
taken to ensure that left and right shift operations are implemented
correctly.  The key is that

* The operation bitwidth is determined by the maximum bitwidth
  of the *source registers*, **not** the destination register bitwidth
* The result is then sign-extend (or truncated) as appropriate.

## Polymorphic MULH/MULHU/MULHSU

MULH is designed to take the top half MSBs of a multiply that
does not fit within the range of the source operands, such that
smaller width operations may produce a full double-width multiply
in two cycles.  The issue is: SV allows the source operands to
have variable bitwidth.

Here again special attention has to be paid to the rules regarding
bitwidth, which, again, are that the operation is performed at
the maximum bitwidth of the **source** registers.  Therefore:

* An 8-bit x 8-bit multiply will create a 16-bit result that must
  be shifted down by 8 bits
* A 16-bit x 8-bit multiply will create a 24-bit result that must
  be shifted down by 16 bits (top 8 bits being zero)
* A 16-bit x 16-bit multiply will create a 32-bit result that must
  be shifted down by 16 bits
* A 32-bit x 16-bit multiply will create a 48-bit result that must
  be shifted down by 32 bits
* A 32-bit x 8-bit multiply will create a 40-bit result that must
  be shifted down by 32 bits

So again, just as with shift-left and shift-right, the result
is shifted down by the maximum of the two source register bitwidths.
And, exactly again, truncation or sign-extension is performed on the
result.  If sign-extension is to be carried out, it is performed
from the same maximum of the two source register bitwidths out
to the result element's bitwidth.

If truncation occurs, i.e. the top MSBs of the result are lost,
this is "Officially Not Our Problem", i.e. it is assumed that the
programmer actually desires the result to be truncated.  i.e. if the
programmer wanted all of the bits, they would have set the destination
elwidth to accommodate them.

## Polymorphic elwidth on LOAD/STORE <a name="elwidth_loadstore"></a>

Polymorphic element widths in vectorised form means that the data
being loaded (or stored) across multiple registers needs to be treated
(reinterpreted) as a contiguous stream of elwidth-wide items, where
the source register's element width is **independent** from the destination's.

This makes for a slightly more complex algorithm when using indirection
on the "addressed" register (source for LOAD and destination for STORE),
particularly given that the LOAD/STORE instruction provides important
information about the width of the data to be reinterpreted.

Let's illustrate the "load" part, where the pseudo-code for elwidth=default
was as follows, and i is the loop from 0 to VL-1:

    srcbase = ireg[rs+i];
    return mem[srcbase + imm]; // returns XLEN bits

Instead, when elwidth != default, for a LW (32-bit LOAD), elwidth-wide
chunks are taken from the source memory location addressed by the current
indexed source address register, and only when a full 32-bits-worth
are taken will the index be moved on to the next contiguous source
address register:

    bitwidth = bw(elwidth);             // source elwidth from CSR reg entry
    elsperblock = 32 / bitwidth         // 1 if bw=32, 2 if bw=16, 4 if bw=8
    srcbase = ireg[rs+i/(elsperblock)]; // integer divide
    offs = i % elsperblock;             // modulo
    return &mem[srcbase + imm + offs];  // re-cast to uint8_t*, uint16_t* etc.

Note that the constant "32" above is replaced by 8 for LB, 16 for LH, 64 for LD
and 128 for LQ.

The principle is basically exactly the same as if the srcbase were pointing
at the memory of the *register* file: memory is re-interpreted as containing
groups of elwidth-wide discrete elements.

When storing the result from a load, it's important to respect the fact
that the destination register has its *own separate element width*.  Thus,
when each element is loaded (at the source element width), any sign-extension
or zero-extension (or truncation) needs to be done to the *destination*
bitwidth.  Also, the storing has the exact same analogous algorithm as
above, where in fact it is just the set\_polymorphed\_reg pseudocode
(completely unchanged) used above.

One issue remains: when the source element width is **greater** than
the width of the operation, it is obvious that a single LB for example
cannot possibly obtain 16-bit-wide data.  This condition may be detected
where, when using integer divide, elsperblock (the width of the LOAD
divided by the bitwidth of the element) is zero.

The issue is "fixed" by ensuring that elsperblock is a minimum of 1:

    elsperblock = min(1, LD_OP_BITWIDTH / element_bitwidth)

The elements, if the element bitwidth is larger than the LD operation's
size, will then be sign/zero-extended to the full LD operation size, as
specified by the LOAD (LDU instead of LD, LBU instead of LB), before
being passed on to the second phase.

As LOAD/STORE may be twin-predicated, it is important to note that
the rules on twin predication still apply, except where in previous
pseudo-code (elwidth=default for both source and target) it was
the *registers* that the predication was applied to, it is now the
**elements** that the predication is applied to.

Thus the full pseudocode for all LD operations may be written out
as follows:

    function LBU(rd, rs):
        load_elwidthed(rd, rs, 8, true)
    function LB(rd, rs):
        load_elwidthed(rd, rs, 8, false)
    function LH(rd, rs):
        load_elwidthed(rd, rs, 16, false)
    ...
    ...
    function LQ(rd, rs):
        load_elwidthed(rd, rs, 128, false)

    # returns 1 byte of data when opwidth=8, 2 bytes when opwidth=16..
    function load_memory(rs, imm, i, opwidth):
        elwidth = int_csr[rs].elwidth
        bitwidth = bw(elwidth);
        elsperblock = min(1, opwidth / bitwidth)
        srcbase = ireg[rs+i/(elsperblock)];
        offs = i % elsperblock;
        return mem[srcbase + imm + offs]; # 1/2/4/8/16 bytes

    function load_elwidthed(rd, rs, opwidth, unsigned):
      destwid = int_csr[rd].elwidth # destination element width
      rd = int_csr[rd].active ? int_csr[rd].regidx : rd;
      rs = int_csr[rs].active ? int_csr[rs].regidx : rs;
      ps = get_pred_val(FALSE, rs); # predication on src
      pd = get_pred_val(FALSE, rd); # ... AND on dest
      for (int i = 0, int j = 0; i < VL && j < VL;):
        if (int_csr[rs].isvec) while (!(ps & 1<<i)) i++;
        if (int_csr[rd].isvec) while (!(pd & 1<<j)) j++;
        val = load_memory(rs, imm, i, opwidth)
        if unsigned:
            val = zero_extend(val, min(opwidth, bitwidth))
        else:
            val = sign_extend(val, min(opwidth, bitwidth))
        set_polymorphed_reg(rd, bitwidth, j, val)
        if (int_csr[rs].isvec) i++;
        if (int_csr[rd].isvec) j++; else break;

Note:

* when comparing against for example the twin-predicated c.mv
  pseudo-code, the pattern of independent incrementing of rd and rs
  is preserved unchanged.
* just as with the c.mv pseudocode, zeroing is not included and must be
  taken into account (TODO).
* that due to the use of a twin-predication algorithm, LOAD/STORE also
  take on the same VSPLAT, VINSERT, VREDUCE, VEXTRACT, VGATHER and
  VSCATTER characteristics.
* that due to the use of the same set\_polymorphed\_reg pseudocode,
  a destination that is not vectorised (marked as scalar) will
  result in the element being fully sign-extended or zero-extended
  out to the full register file bitwidth (XLEN).  When the source
  is also marked as scalar, this is how the compatibility with
  standard RV LOAD/STORE is preserved by this algorithm.

### Example Tables showing LOAD elements

This section contains examples of vectorised LOAD operations, showing
how the two stage process works (three if zero/sign-extension is included).


#### Example: LD x8, x5(0), x8 CSR-elwidth=32, x5 CSR-elwidth=16, VL=7

This is:

* a 64-bit load, with an offset of zero
* with a source-address elwidth of 16-bit
* into a destination-register with an elwidth of 32-bit
* where VL=7
* from register x5 (actually x5-x6) to x8 (actually x8 to half of x11)
* RV64, where XLEN=64 is assumed.

First, the memory table, which, due to the
element width being 16 and the operation being LD (64), the 64-bits
loaded from memory are subdivided into groups of **four** elements.
And, with VL being 7 (deliberately to illustrate that this is reasonable
and possible), the first four are sourced from the offset addresses pointed
to by x5, and the next three from the ofset addresses pointed to by
the next contiguous register, x6:

[[!table  data="""
addr | byte 0 | byte 1 | byte 2 | byte 3 | byte 4 | byte 5 | byte 6 | byte 7 |
@x5  | elem 0         || elem 1         || elem 2         || elem 3         ||
@x6  | elem 4         || elem 5         || elem 6         || not loaded     ||
"""]]

Next, the elements are zero-extended from 16-bit to 32-bit, as whilst
the elwidth CSR entry for x5 is 16-bit, the destination elwidth on x8 is 32.

[[!table  data="""
byte 3 | byte 2 | byte 1 | byte 0 |
0x0    | 0x0    | elem0          ||
0x0    | 0x0    | elem1          ||
0x0    | 0x0    | elem2          ||
0x0    | 0x0    | elem3          ||
0x0    | 0x0    | elem4          ||
0x0    | 0x0    | elem5          ||
0x0    | 0x0    | elem6          ||
0x0    | 0x0    | elem7          ||
"""]]

Lastly, the elements are stored in contiguous blocks, as if x8 was also
byte-addressable "memory".  That "memory" happens to cover registers
x8, x9, x10 and x11, with the last 32 "bits" of x11 being **UNMODIFIED**:

[[!table  data="""
reg# | byte 7 | byte 6 | byte 5 | byte 4 | byte 3 | byte 2 | byte 1 | byte 0 |
x8   | 0x0    | 0x0    | elem 1         || 0x0    | 0x0    | elem 0         ||
x9   | 0x0    | 0x0    | elem 3         || 0x0    | 0x0    | elem 2         ||
x10  | 0x0    | 0x0    | elem 5         || 0x0    | 0x0    | elem 4         ||
x11  | **UNMODIFIED**                 |||| 0x0    | 0x0    | elem 6         ||
"""]]

Thus we have data that is loaded from the **addresses** pointed to by
x5 and x6, zero-extended from 16-bit to 32-bit, stored in the **registers**
x8 through to half of x11.
The end result is that elements 0 and 1 end up in x8, with element 8 being
shifted up 32 bits, and so on, until finally element 6 is in the
LSBs of x11.

Note that whilst the memory addressing table is shown left-to-right byte order,
the registers are shown in right-to-left (MSB) order.  This does **not**
imply that bit or byte-reversal is carried out: it's just easier to visualise
memory as being contiguous bytes, and emphasises that registers are not
really actually "memory" as such.

## Why SV bitwidth specification is restricted to 4 entries

The four entries for SV element bitwidths only allows three over-rides:

* default bitwidth for a given operation *divided* by two
* default bitwidth for a given operation *multiplied* by two
* 8-bit

At first glance this seems completely inadequate: for example, RV64
cannot possibly operate on 16-bit operations, because 64 divided by
2 is 32.  However, the reader may have forgotten that it is possible,
at run-time, to switch a 64-bit application into 32-bit mode, by
setting UXL.  Once switched, opcodes that formerly had 64-bit
meanings now have 32-bit meanings, and in this way, "default/2"
now reaches **16-bit** where previously it meant "32-bit".

There is however an absolutely crucial aspect oF SV here that explicitly
needs spelling out, and it's whether the "vectorised" bit is set in
the Register's CSR entry.

If "vectorised" is clear (not set), this indicates that the operation
is "scalar".  Under these circumstances, when set on a destination (RD),
then sign-extension and zero-extension, whilst changed to match the
override bitwidth (if set), will erase the **full** register entry
(64-bit if RV64).

When vectorised is *set*, this indicates that the operation now treats
**elements** as if they were independent registers, so regardless of
the length, any parts of a given actual register that are not involved
in the operation are **NOT** modified, but are **PRESERVED**.

SIMD micro-architectures may implement this by using predication on
any elements in a given actual register that are beyond the end of
multi-element operation.

Example:

* rs1, rs2 and rd are all set to 8-bit
* VL is set to 3
* RV64 architecture is set (UXL=64)
* add operation is carried out
* bits 0-23 of RD are modified to be rs1[23..16] + rs2[23..16]
  concatenated with similar add operations on bits 15..8 and 7..0
* bits 24 through 63 **remain as they originally were**.

Example SIMD micro-architectural implementation:

* SIMD architecture works out the nearest round number of elements
  that would fit into a full RV64 register (in this case: 8)
* SIMD architecture creates a hidden predicate, binary 0b00000111
  i.e. the bottom 3 bits set (VL=3) and the top 5 bits clear
* SIMD architecture goes ahead with the add operation as if it
  was a full 8-wide batch of 8 adds
* SIMD architecture passes top 5 elements through the adders
  (which are "disabled" due to zero-bit predication)
* SIMD architecture gets the 5 unmodified top 8-bits back unmodified
  and stores them in rd.

This requires a read on rd, however this is required anyway in order
to support non-zeroing mode.

## Polymorphic floating-point

Standard scalar RV integer operations base the register width on XLEN,
which may be changed (UXL in USTATUS, and the corresponding MXL and
SXL in MSTATUS and SSTATUS respectively).  Integer LOAD, STORE and
arithmetic operations are therefore restricted to an active XLEN bits,
with sign or zero extension to pad out the upper bits when XLEN has
been dynamically set to less than the actual register size.

For scalar floating-point, the active (used / changed) bits are
specified exclusively by the operation: ADD.S specifies an active
32-bits, with the upper bits of the source registers needing to
be all 1s ("NaN-boxed"), and the destination upper bits being
*set* to all 1s (including on LOAD/STOREs).

Where elwidth is set to default (on any source or the destination)
it is obvious that this NaN-boxing behaviour can and should be
preserved.  When elwidth is non-default things are less obvious,
so need to be thought through.  Here is a normal (scalar) sequence,
assuming an RV64 which supports Quad (128-bit) FLEN:

* FLD loads 64-bit wide from memory.  Top 64 MSBs are set to all 1s
* ADD.D performs a 64-bit-wide add.  Top 64 MSBs of destination set to 1s.
* FSD stores lowest 64-bits from the 128-bit-wide register to memory:
  top 64 MSBs ignored.

Therefore it makes sense to mirror this behaviour when, for example,
elwidth is set to 32.  Assume elwidth set to 32 on all source and
destination registers:

* FLD loads 64-bit wide from memory as **two** 32-bit single-precision
  floating-point numbers.
* ADD.D performs **two** 32-bit-wide adds, storing one of the adds
  in bits 0-31 and the second in bits 32-63.
* FSD stores lowest 64-bits from the 128-bit-wide register to memory

Here's the thing: it does not make sense to overwrite the top 64 MSBs
of the registers either during the FLD **or** the ADD.D.  The reason
is that, effectively, the top 64 MSBs actually represent a completely
independent 64-bit register, so overwriting it is not only gratuitous
but may actually be harmful for a future extension to SV which may
have a way to directly access those top 64 bits.

The decision is therefore **not** to touch the upper parts of floating-point
registers whereever elwidth is set to non-default values, including
when "isvec" is false in a given register's CSR entry.  Only when the
elwidth is set to default **and** isvec is false will the standard
RV behaviour be followed, namely that the upper bits be modified.

Ultimately if elwidth is default and isvec false on *all* source
and destination registers, a SimpleV instruction defaults completely
to standard RV scalar behaviour (this holds true for **all** operations,
right across the board).

The nice thing here is that ADD.S, ADD.D and ADD.Q when elwidth are
non-default values are effectively all the same: they all still perform
multiple ADD operations, just at different widths.  A future extension
to SimpleV may actually allow ADD.S to access the upper bits of the
register, effectively breaking down a 128-bit register into a bank
of 4 independently-accesible 32-bit registers.

In the meantime, although when e.g. setting VL to 8 it would technically
make no difference to the ALU whether ADD.S, ADD.D or ADD.Q is used,
using ADD.Q may be an easy way to signal to the microarchitecture that
it is to receive a higher VL value.  On a superscalar OoO architecture
there may be absolutely no difference, however on simpler SIMD-style
microarchitectures they may not necessarily have the infrastructure in
place to know the difference, such that when VL=8 and an ADD.D instruction
is issued, it completes in 2 cycles (or more) rather than one, where
if an ADD.Q had been issued instead on such simpler microarchitectures
it would complete in one.

## Specific instruction walk-throughs

This section covers walk-throughs of the above-outlined procedure
for converting standard RISC-V scalar arithmetic operations to
polymorphic widths, to ensure that it is correct.

### add

Standard Scalar RV32/RV64 (xlen):

* RS1 @ xlen bits
* RS2 @ xlen bits
* add @ xlen bits
* RD @ xlen bits

Polymorphic variant:

* RS1 @ rs1 bits, zero-extended to max(rs1, rs2) bits
* RS2 @ rs2 bits, zero-extended to max(rs1, rs2) bits
* add @ max(rs1, rs2) bits
* RD @ rd bits.  zero-extend to rd if rd > max(rs1, rs2) otherwise truncate

Note here that polymorphic add zero-extends its source operands,
where addw sign-extends.

### addw

The RV Specification specifically states that "W" variants of arithmetic
operations always produce 32-bit signed values.  In a polymorphic
environment it is reasonable to assume that the signed aspect is
preserved, where it is the length of the operands and the result
that may be changed.

Standard Scalar RV64 (xlen):

* RS1 @ xlen bits
* RS2 @ xlen bits
* add @ xlen bits
* RD @ xlen bits, truncate add to 32-bit and sign-extend to xlen.

Polymorphic variant:

* RS1 @ rs1 bits, sign-extended to max(rs1, rs2) bits
* RS2 @ rs2 bits, sign-extended to max(rs1, rs2) bits
* add @ max(rs1, rs2) bits
* RD @ rd bits. sign-extend to rd if rd > max(rs1, rs2) otherwise truncate

Note here that polymorphic addw sign-extends its source operands,
where add zero-extends.

This requires a little more in-depth analysis.  Where the bitwidth of
rs1 equals the bitwidth of rs2, no sign-extending will occur.  It is
only where the bitwidth of either rs1 or rs2 are different, will the
lesser-width operand be sign-extended.

Effectively however, both rs1 and rs2 are being sign-extended (or truncated),
where for add they are both zero-extended.  This holds true for all arithmetic
operations ending with "W".

### addiw

Standard Scalar RV64I:

* RS1 @ xlen bits, truncated to 32-bit
* immed @ 12 bits, sign-extended to 32-bit
* add @ 32 bits
* RD @ rd bits.  sign-extend to rd if rd > 32, otherwise truncate.

Polymorphic variant:

* RS1 @ rs1 bits
* immed @ 12 bits, sign-extend to max(rs1, 12) bits
* add @ max(rs1, 12) bits
* RD @ rd bits. sign-extend to rd if rd > max(rs1, 12) otherwise truncate

# Predication Element Zeroing

The introduction of zeroing on traditional vector predication is usually
intended as an optimisation for lane-based microarchitectures with register
renaming to be able to save power by avoiding a register read on elements
that are passed through en-masse through the ALU.  Simpler microarchitectures
do not have this issue: they simply do not pass the element through to
the ALU at all, and therefore do not store it back in the destination.
More complex non-lane-based micro-architectures can, when zeroing is
not set, use the predication bits to simply avoid sending element-based
operations to the ALUs, entirely: thus, over the long term, potentially
keeping all ALUs 100% occupied even when elements are predicated out.

SimpleV's design principle is not based on or influenced by
microarchitectural design factors: it is a hardware-level API.
Therefore, looking purely at whether zeroing is *useful* or not,
(whether less instructions are needed for certain scenarios),
given that a case can be made for zeroing *and* non-zeroing, the
decision was taken to add support for both.

## Single-predication (based on destination register)

Zeroing on predication for arithmetic operations is taken from
the destination register's predicate.  i.e. the predication *and*
zeroing settings to be applied to the whole operation come from the
CSR Predication table entry for the destination register.
Thus when zeroing is set on predication of a destination element,
if the predication bit is clear, then the destination element is *set*
to zero (twin-predication is slightly different, and will be covered
next).

Thus the pseudo-code loop for a predicated arithmetic operation
is modified to as follows:

      for (i = 0; i < VL; i++)
        if not zeroing: # an optimisation
           while (!(predval & 1<<i) && i < VL)
             if (int_vec[rd ].isvector)  { id += 1; }
             if (int_vec[rs1].isvector)  { irs1 += 1; }
             if (int_vec[rs2].isvector)  { irs2 += 1; }
           if i == VL:
             break
        if (predval & 1<<i)
           src1 = ....
           src2 = ...
           else:
               result = src1 + src2 # actual add (or other op) here
           set_polymorphed_reg(rd, destwid, ird, result)
           if (!int_vec[rd].isvector) break
        else if zeroing:
           result = 0
           set_polymorphed_reg(rd, destwid, ird, result)
        if (int_vec[rd ].isvector)  { id += 1; }
        else if (predval & 1<<i) break;
        if (int_vec[rs1].isvector)  { irs1 += 1; }
        if (int_vec[rs2].isvector)  { irs2 += 1; }

The optimisation to skip elements entirely is only possible for certain
micro-architectures when zeroing is not set.  However for lane-based
micro-architectures this optimisation may not be practical, as it
implies that elements end up in different "lanes".  Under these
circumstances it is perfectly fine to simply have the lanes
"inactive" for predicated elements, even though it results in
less than 100% ALU utilisation.

## Twin-predication (based on source and destination register)

Twin-predication is not that much different, except that that
the source is independently zero-predicated from the destination.
This means that the source may be zero-predicated *or* the
destination zero-predicated *or both*, or neither.

When with twin-predication, zeroing is set on the source and not
the destination, if a predicate bit is set it indicates that a zero
data element is passed through the operation (the exception being:
if the source data element is to be treated as an address - a LOAD -
then the data returned *from* the LOAD is zero, rather than looking up an
*address* of zero.

When zeroing is set on the destination and not the source, then just
as with single-predicated operations, a zero is stored into the destination
element (or target memory address for a STORE).

Zeroing on both source and destination effectively result in a bitwise
NOR operation of the source and destination predicate: the result is that
where either source predicate OR destination predicate is set to 0,
a zero element will ultimately end up in the destination register.

However: this may not necessarily be the case for all operations;
implementors, particularly of custom instructions, clearly need to
think through the implications in each and every case.

Here is pseudo-code for a twin zero-predicated operation:

    function op_mv(rd, rs) # MV not VMV!
      rd = int_csr[rd].active ? int_csr[rd].regidx : rd;
      rs = int_csr[rs].active ? int_csr[rs].regidx : rs;
      ps, zerosrc = get_pred_val(FALSE, rs); # predication on src
      pd, zerodst = get_pred_val(FALSE, rd); # ... AND on dest
      for (int i = 0, int j = 0; i < VL && j < VL):
        if (int_csr[rs].isvec && !zerosrc) while (!(ps & 1<<i)) i++;
        if (int_csr[rd].isvec && !zerodst) while (!(pd & 1<<j)) j++;
        if ((pd & 1<<j))
            if ((pd & 1<<j))
                sourcedata = ireg[rs+i];
            else
                sourcedata = 0
            ireg[rd+j] <= sourcedata
        else if (zerodst)
            ireg[rd+j] <= 0
        if (int_csr[rs].isvec)
            i++;
        if (int_csr[rd].isvec)
            j++;
        else
            if ((pd & 1<<j))
                break;

Note that in the instance where the destination is a scalar, the hardware
loop is ended the moment a value *or a zero* is placed into the destination
register/element.  Also note that, for clarity, variable element widths
have been left out of the above.

# Exceptions

TODO: expand.  Exceptions may occur at any time, in any given underlying
scalar operation.  This implies that context-switching (traps) may
occur, and operation must be returned to where it left off.  That in
turn implies that the full state - including the current parallel
element being processed - has to be saved and restored.  This is
what the **STATE** CSR is for.

The implications are that all underlying individual scalar operations
"issued" by the parallelisation have to appear to be executed sequentially.
The further implications are that if two or more individual element
operations are underway, and one with an earlier index causes an exception,
it may be necessary for the microarchitecture to **discard** or terminate
operations with higher indices.

This being somewhat dissatisfactory, an "opaque predication" variant
of the STATE CSR is being considered.

# Hints

A "HINT" is an operation that has no effect on architectural state,
where its use may, by agreed convention, give advance notification
to the microarchitecture: branch prediction notification would be
a good example.  Usually HINTs are where rd=x0.

With Simple-V being capable of issuing *parallel* instructions where
rd=x0, the space for possible HINTs is expanded considerably.  VL
could be used to indicate different hints.  In addition, if predication
is set, the predication register itself could hypothetically be passed
in as a *parameter* to the HINT operation.

No specific hints are yet defined in Simple-V

# VLIW Format <a name="vliw-format"></a>

One issue with SV is the setup and teardown time of the CSRs.  The cost
of the use of a full CSRRW (requiring LI) is quite high.  A VLIW format
therefore makes sense.

A suitable prefix, which fits the Expanded Instruction-Length encoding
for "(80 + 16 times instruction_length)", as defined in Section 1.5
of the RISC-V ISA, is as follows:

| 15    | 14:12 | 11:10 | 9:8   | 7    | 6:0     |
| -     | ----- | ----- | ----- | ---  | ------- |
| vlset | 16xil | pplen | rplen | mode | 1111111 |

An optional VL Block, optional predicate entries, optional register entries and finally some 16/32/48 bit standard RV or SVPrefix opcodes follow.

The variable-length format from Section 1.5 of the RISC-V ISA:

| base+4 ... base+2         | base             | number of bits             |
| ------ ------------------- | ----------------  -------------------------- |
| ..xxxx  xxxxxxxxxxxxxxxx | xnnnxxxxx1111111 | (80+16\*nnn)-bit, nnn!=111 |
| {ops}{Pred}{Reg}{VL Block}    | SV Prefix        |                            |

VL/MAXVL/SubVL Block:

| 31-30 | 29:28 | 27:22  | 21:17  - 16  |
| -     | ----- | ------ | ------ - -   |
| 0     | SubVL | VLdest | VLEN     vlt |
| 1     | SubVL | VLdest | VLEN         |

If vlt is 0, VLEN is a 5 bit immediate value. If vlt is 1, it specifies
the scalar register from which VL is set by this VLIW instruction
group. VL, whether set from the register or the immediate, is then
modified (truncated) to be MIN(VL, MAXVL), and the result stored in the
scalar register specified in VLdest. If VLdest is zero, no store in the
regfile occurs (however VL is still set).

This option will typically be used to start vectorised loops, where
the VLIW instruction effectively embeds an optional "SETSUBVL, SETVL"
sequence (in compact form).

When bit 15 is set to 1, MAXVL and VL are both set to the immediate,
VLEN, which is 6 bits in length, and the same value stored in scalar
register VLdest (if that register is nonzero).

This option will typically not be used so much for loops as it will be
for one-off instructions such as saving the entire register file to the
stack with a single one-off Vectorised and predicated LD/ST.

CSRs needed:

* mepcvliw
* sepcvliw
* uepcvliw
* hepcvliw

Notes:

* Bit 7 specifies if the prefix block format is the full 16 bit format
  (1) or the compact less expressive format (0). In the 8 bit format,
  pplen is multiplied by 2.
* 8 bit format predicate numbering is implicit and begins from x9. Thus it is critical to put blocks in the correct order as required.
* Bit 7 also specifies if the register block format is 16 bit (1) or 8 bit
  (0). In the 8 bit format, rplen is multiplied by 2. If only an odd number
  of entries are needed the last may be set to 0x00, indicating "unused".
* Bit 15 specifies if the VL Block is present. If set to 1, the VL Block immediately follows the VLIW instruction Prefix
* Bits 8 and 9 define how many RegCam entries (0 to 3 if bit 15 is 1, otherwise 0 to 6) follow the (optional) VL Block.
* Bits 10 and 11 define how many PredCam entries (0 to 3 if bit 7 is 1, otherwise 0 to 6) follow the (optional) RegCam entries
* Bits 14 to 12 (IL) define the actual length of the instruction: total
  number of bits is 80 + 16 times IL.  Standard RV32, RVC and also
  SVPrefix (P48-\*-Type) instructions fit into this space, after the
  (optional) VL / RegCam / PredCam entries
* Anything - any registers - within the VLIW-prefixed format *MUST* have the
  RegCam and PredCam entries applied to it.
* At the end of the VLIW Group, the RegCam and PredCam entries
  *no longer apply*.  VL, MAXVL and SUBVL on the other hand remain at
  the values set by the last instruction (whether a CSRRW or the VL
  Block header).
* Although an inefficient use of resources, it is fine to set the MAXVL,
  VL and SUBVL CSRs with standard CSRRW instructions, within a VLIW block.

All this would greatly reduce the amount of space utilised by Vectorised
instructions, given that 64-bit CSRRW requires 3, even 4 32-bit opcodes: the
CSR itself, a LI, and the setting up of the value into the RS register
of the CSR, which, again, requires a LI / LUI to get the 32 bit
data into the CSR.  To get 64-bit data into the register in order to put
it into the CSR(s), LOAD operations from memory are needed!

Given that each 64-bit CSR can hold only 4x PredCAM entries (or 4 RegCAM
entries), that's potentially 6 to eight 32-bit instructions, just to
establish the Vector State!

Not only that: even CSRRW on VL and MAXVL requires 64-bits (even more bits if
VL needs to be set to greater than 32).  Bear in mind that in SV, both MAXVL
and VL need to be set.

By contrast, the VLIW prefix is only 16 bits, the VL/MAX/SubVL block is
only 16 bits, and as long as not too many predicates and register vector
qualifiers are specified, several 32-bit and 16-bit opcodes can fit into
the format. If the full flexibility of the 16 bit block formats are not
needed, more space is saved by using the 8 bit formats.

In this light, embedding the VL/MAXVL, PredCam and RegCam CSR entries into
a VLIW format makes a lot of sense.

Open Questions:

* Is it necessary to stick to the RISC-V 1.5 format?  Why not go with
  using the 15th bit to allow 80 + 16\*0bnnnn bits?  Perhaps to be sane,
  limit to 256 bits (16 times 0-11).
* Could a "hint" be used to set which operations are parallel and which
  are sequential?
* Could a new sub-instruction opcode format be used, one that does not
  conform precisely to RISC-V rules, but *unpacks* to RISC-V opcodes?
  no need for byte or bit-alignment
* Could a hardware compression algorithm be deployed?  Quite likely,
  because of the sub-execution context (sub-VLIW PC)

## Limitations on instructions.

To greatly simplify implementations, it is required to treat the VLIW
group as a separate sub-program with its own separate PC. The sub-pc
advances separately whilst the main PC remains pointing at the beginning
of the VLIW instruction (not to be confused with how VL works, which
is exactly the same principle, except it is VStart in the STATE CSR
that increments).

This has implications, namely that a new set of CSRs identical to xepc
(mepc, srpc, hepc and uepc) must be created and managed and respected
as being a sub extension of the xepc set of CSRs.  Thus, xepcvliw CSRs
must be context switched and saved / restored in traps.

The VStart indices in the STATE CSR may be similarly regarded as another
sub-execution context, giving in effect two sets of nested sub-levels
of the RISCV Program Counter.

In addition, as xepcvliw CSRs are relative to the beginning of the VLIW
block, branches MUST be restricted to within the block, i.e. addressing
is now restricted to the start (and very short) length of the block.

Also: calling subroutines is inadviseable, unless they can be entirely
accomplished within a block.

A normal jump and a normal function call may only be taken by letting
the VLIW end, returning to "normal" standard RV mode, using RVC, 32 bit
or P48-*-type opcodes.

## Links

* <https://groups.google.com/d/msg/comp.arch/yIFmee-Cx-c/jRcf0evSAAAJ>

# Subsets of RV functionality

This section describes the differences when SV is implemented on top of
different subsets of RV.

## Common options

It is permitted to limit the size of either (or both) the register files
down to the original size of the standard RV architecture.  However, below
the mandatory limits set in the RV standard will result in non-compliance
with the SV Specification.

## RV32 / RV32F

When RV32 or RV32F is implemented, XLEN is set to 32, and thus the
maximum limit for predication is also restricted to 32 bits.  Whilst not
actually specifically an "option" it is worth noting.

## RV32G

Normally in standard RV32 it does not make much sense to have
RV32G, The critical instructions that are missing in standard RV32
are those for moving data to and from the double-width floating-point
registers into the integer ones, as well as the FCVT routines.

In an earlier draft of SV, it was possible to specify an elwidth
of double the standard register size: this had to be dropped,
and may be reintroduced in future revisions.

## RV32 (not RV32F / RV32G) and RV64 (not RV64F / RV64G)

When floating-point is not implemented, the size of the User Register and
Predication CSR tables may be halved, to only 4 2x16-bit CSRs (8 entries
per table).

## RV32E

In embedded scenarios the User Register and Predication CSRs may be
dropped entirely, or optionally limited to 1 CSR, such that the combined
number of entries from the M-Mode CSR Register table plus U-Mode
CSR Register table is either 4 16-bit entries or (if the U-Mode is
zero) only 2 16-bit entries (M-Mode CSR table only).  Likewise for
the Predication CSR tables.

RV32E is the most likely candidate for simply detecting that registers
are marked as "vectorised", and generating an appropriate exception
for the VL loop to be implemented in software.

## RV128

RV128 has not been especially considered, here, however it has some
extremely large possibilities: double the element width implies
256-bit operands, spanning 2 128-bit registers each, and predication
of total length 128 bit given that XLEN is now 128.

# Under consideration <a name="issues"></a>

for element-grouping, if there is unused space within a register
(3 16-bit elements in a 64-bit register for example), recommend:

* For the unused elements in an integer register, the used element
  closest to the MSB is sign-extended on write and the unused elements
  are ignored on read.
* The unused elements in a floating-point register are treated as-if
  they are set to all ones on write and are ignored on read, matching the
  existing standard for storing smaller FP values in larger registers.

---

info register,

> One solution is to just not support LR/SC wider than a fixed
> implementation-dependent size, which must be at least 
>1 XLEN word, which can be read from a read-only CSR
> that can also be used for info like the kind and width of 
> hw parallelism supported (128-bit SIMD, minimal virtual 
> parallelism, etc.) and other things (like maybe the number 
> of registers supported). 

> That CSR would have to have a flag to make a read trap so
> a hypervisor can simulate different values.

----

> And what about instructions like JALR? 

answer: they're not vectorised, so not a problem

----

* if opcode is in the RV32 group, rd, rs1 and rs2 bitwidth are
  XLEN if elwidth==default
* if opcode is in the RV32I group, rd, rs1 and rs2 bitwidth are
  *32* if elwidth == default

---

TODO: update elwidth to be default / 8 / 16 / 32

---

TODO: document different lengths for INT / FP regfiles, and provide
as part of info register. 00=32, 01=64, 10=128, 11=reserved.

---

TODO, update to remove RegCam and PredCam CSRs, just use SVprefix and VLIW format