(no commit message)

[libreriscv.git] / openpower / sv / SimpleV_rationale.mdwn

[[!tag whitepapers]]

**Revision History**

* v0.00 05may2021 first created
* v0.01 06may2021 initial first draft
* v0.02 08may2021 add scenarios / use-cases
* v0.03 09may2021 add draft image for scenario

**Table of Contents**

[[!toc]]

# Why in the 2020s would you invent a new Vector ISA

*(The short answer: you don't. Extend existing technology: on the shoulders of giants)*

Inventing a new Scalar ISA from scratch is over a decade-long task
including simulators and compilers: OpenRISC 1200 took 12 years to
mature.  Stable ISAs require Standards and Compliance Suites that
take more. A Vector or Packed SIMD ISA to reach stable *general-purpose*
auto-vectorisation compiler support has never been achieved in the
history of computing, not with the combined resources of ARM, Intel,
AMD, MIPS, Sun Microsystems, SGI, Cray, and many more. (*Hand-crafted
assembler and direct use of intrinsics is the Industry-standard norm
to achieve high-performance optimisation where it matters*).
GPUs fill this void both in hardware and software terms by having
ultra-specialist compilers (CUDA) that are designed from the ground up
to support Vector/SIMD parallelism, and associated standards
(SPIR-V, Vulkan, OpenCL) managed by
the Khronos Group, with multi-man-century development committment from
multiple billion-dollar-revenue companies, to sustain them.

Therefore it begs the question, why on earth would anyone consider
this task, and what, in Computer Science, actually needs solving?

First hints are that whilst memory bitcells have not increased in speed
since the 90s (around 150 mhz), increasing the bank width, striping, and
datapath widths and speeds to the same has, with significant relative
latency penalties, allowed
apparent speed increases: 3200 mhz DDR4 and even faster DDR5,
and other advanced Memory interfaces such as HBM, Gen-Z, and OpenCAPI's OMI,
all make an effort (all simply increasing the parallel deployment of
the underlying 150 mhz bitcells), but these efforts are dwarfed by the
two nearly three orders of magnitude increase in CPU horsepower
over the same timeframe. Seymour
Cray, from his amazing in-depth knowledge, predicted that the mismatch
would become a serious limitation, over two decades ago.

The latency gap between that bitcell speed and the CPU speed can do nothing to help Random Access (unpredictable reads/writes). Cacheing helps only so
much, but not with some types of workloads (FFTs are one of the worst)
even though
they are fully deterministic.
Some systems
at the time of writing are now approaching a *Gigabyte* of L4 Cache,
by way of compensation, and as we know from experience even that will
be considered inadequate in future.

Efforts to solve this problem by moving the processing closer to or
directly integrated into the memory have traditionally not gone well:
Aspex Microelectronics, Elixent, these are parallel processing companies
that very few have heard of, because their software stack was so
specialist that it required heavy investment by customers to utilise.
D-Matrix, a Systolic Array Processor, is a modern incarnation of the exact same
"specialist parallel processing" mistake, betting heavily on AI with
Matrix and Convolution Engines that can do no other task.  Aspex only
survived by being bought by Ericsson, where its specialised suitability
for massive wide Baseband FFTs saved it from going under.
The huge risk is that any "better
AI mousetrap" created by an innovative competitor
that comes along quickly renders a too-specialist design obsolete.

NVIDIA and other GPUs have taken a different approach again: massive
parallelism with more Turing-complete ISAs in each, and dedicated
slower parallel memory paths (GDDR5) suited to the specific tasks of
3D, Parallel Compute and AI. The complexity of this approach is only dwarfed
by the amount of money poured into the software ecosystem in order
to make it accessible, and even then, GPU Programmers are a specialist
and rare (expensive) breed.
 
Second hints as to the answer emerge from an article
"[SIMD considered harmful](https://www.sigarch.org/simd-instructions-considered-harmful/)"
which illustrates a catastrophic rabbit-hole taken by Industry Giants
ARM, Intel, AMD, since the 90s (over 3 decades) whereby SIMD, an
Order(N^6) opcode proliferation nightmare, with its mantra "make it
easy for hardware engineers, let software sort out the mess" literally
overwhelming programmers with thousands of instructions. Specialists charging
clients for assembly-code Optimisation Services are finding that AVX-512,
to take an
example, is anything but optimal: overall performance of AVX-512 actually
*decreases* even as power consumption goes up.

Cray-style Vectors solved, over thirty years ago, the opcode proliferation
nightmare.  Only the NEC SX Aurora however truly kept the Cray Vector
flame alive, until RISC-V RVV and now SVP64 and recently MRISC32 joined
it.  ARM's SVE/SVE2 is critically flawed (lacking the Cray `setvl`
instruction that makes a truly ubiquitous Vector ISA) in ways that
will become apparent over time as adoption increases. In the meantime
programmers are, in direct violation of ARM's advice on how to use SVE2,
trying desperately to understand it by applying their experience
of Packed SIMD NEON.  The advice from ARM
not to create SVE2 assembler that is hardcoded to fixed widths is being
disregarded, in favour of writing *multiple identical implementations*
of a function, each with a different hardware width, and compelling
software to choose one at runtime after probing the hardware.

Even RISC-V, for all that we can be grateful to the RISC-V Founders
for reviving Cray Vectors, has severe performance and implementation
limitations that are only really apparent to exceptionally experienced
assembly-level developers with a wide, diverse depth in multiple ISAs:
one of the best and clearest is a
[ycombinator post](https://news.ycombinator.com/item?id=24459041)
by adrian_b.

Adrian logically and concisely points out that the fundamental design
assumptions and simplifications that went into the RISC-V ISA have an
irrevocably damaging effect on its viability for high performance use.
That is not to say that its use in low-performance embedded scenarios is
not ideal: in private custom secretive commercial usage it is perfect.
Trinamic, an early adopter, created their TMC2660 Stepper IC replacing
ARM with RISC-V and saving themselves USD 1 in licensing royalties
per product are a classic case study.  Ubiquitous and common everyday
usage in scenarios currently occupied by ARM, Intel, AMD and IBM? not
so much. Even though RISC-V has Cray-style Vectors, the whole ISA is,
unfortunately, fundamentally flawed as far as power efficient high
performance is concerned.

Slowly, at this point, a realisation should be sinking in that, actually,
there aren't as many really truly viable Vector ISAs out there, as the
ones that are evolving in the general direction of Vectorisation are,
in various completely different ways, flawed.

**Successfully identifying a limitation marks the beginning of an
opportunity**

We are nowhere near done, however, because a Vector ISA is a superset of a
Scalar ISA, and even a Scalar ISA takes over a decade to develop compiler
support, and even longer to get the software ecosystem up and running.

Which ISAs, therefore, have or have had, at one point in time, a decent
Software Ecosystem? Debian supports most of these including s390:

* SPARC, created by Sun Microsystems and all but abandoned by Oracle.
  Gaisler Research maintains the LEON Open Source Cores but with Oracle's
  reputation nobody wants to go near SPARC.
* MIPS, created by SGI and only really commonly used in Network switches.
  Exceptions: Ingenic with embedded CPUs,
  and China ICT with the Loongson supercomputers.
* x86, the most well-known ISA and also one of the most heavily
  litigously-protected.
* ARM, well known in embedded and smartphone scenarios, very slowly
  making its way into data centres.
* OpenRISC, an entirely Open ISA suitable for embedded systems.
* s390, a Mainframe ISA very similar to Power.
* Power ISA, a Supercomputing-class ISA, as demonstrated by
  two out of three of the top500.org supercomputers using
  around 2 million IBM POWER9 Cores each.
* ARC, a competitor at the time to ARM, best known for use in
  Broadcom VideoCore IV.
* RISC-V, with a software ecosystem heavily in development
  and with rapid expansion
  in an uncontrolled fashion, is set on an unstoppable
  and inevitable trainwreck path to replicate the
  opcode conflict nightmare that plagued the Power ISA,
  two decades ago.
* Tensilica, Andes STAR and Western Digital for successful
  commercial proprietary ISAs: Tensilica in Baseband Modems,
  Andes in Audio DSPs, WD in HDDs and SSDs. These are all
  astoundingly commercially successful
  multi-billion-unit mass volume markets that almost nobody
  knows anything about. Included for completeness.

In order of least controlled to most controlled, the viable
candidates for further advancement are:

* OpenRISC 1200, not controlled or restricted by anyone. no patent
  protection.
* RISC-V, touted as "Open" but actually strictly controlled under
  Trademark License: too new to have adequate patent pool protection,
  as evidenced by multiple adopters having been hit by patent lawsuits.
  (Agreements between RISC-V *Members* to not engage in patent litigation
   does nothing to stop third party patents that *legitimately pre-date*
   the newly-created RISC-V ISA)
* MIPS, SPARC, ARC, and others, simply have no viable publicly
  managed ecosystem. They work well within their niche markets.
* Power ISA: protected by IBM's extensive patent portfolio for Members
  of the OpenPOWER Foundation, covered by Trademarks, permitting
  and encouraging contributions, and having software support for over
  20 years.
* ARM, not permitting Open Licensing, they survived in the early 90s
  only by doing a deal with Samsung for an in-perpetuity
  Royalty-free License, in exchange
  for GBP 3 million and legal protection through Samsung Research.
  Several large Corporations (Apple most notably) have licensed the ISA
  but not ARM designs: the barrier to entry is high and the ISA itself
  protected from interference as a result.
* x86, famous for an unprecedented
  Court Ruling in 2004 where a Judge "banged heads
  together" and ordered AMD and Intel to stop wasting his time,
  make peace, and cross-license each other's patents, anyone wishing
  to use the x86 ISA need only look at Transmeta, SiS, the Vortex x86,
  and VIA EDEN processors, and see how they fared.
* s390, IBM's mainframe ISA. Nowhere near as well-known as x86 lawsuits,
  but the 800lb "Corporate Gorilla Syndrome" seems not to have deterred one
  particularly disingenuous group from performing illegal
  Reverse-Engineering.

By asking the question, "which ISA would be the best and most stable to
base a Vector Supercomputing-class Extension on?" where patent protection,
software ecosystem, open-ness and pedigree all combine to reduce risk
and increase the chances of success, there is really only one candidate.

**Of all of these, the only one with the most going for it is the Power ISA.**

The summary of advantages, then, of the Power ISA is that:

* It has a 25-year software ecosystem, with RHEL, Fedora, Debian
  and more.
* Amongst many other features
  it has Condition Registers which can be used by Branches, greatly
  reducing pressure on the main register files.
* IBM's extensive 20+ years of patents is available, royalty-free,
  to protect implementors as long as they are also members of the
  OpenPOWER Foundation
* IBM designed and maintained the Power ISA as a Supercomputing
  class ISA from its inception over 25 years ago.
* Coherent distributed memory access is possible through OpenCAPI
* Extensions to the Power ISA may be submitted through an External
  RFC Process that does not require membership of OPF.

From this strong base, the next step is: how to leverage this
foundation to take a leap forward in performance and performance/watt,
*without* losing all the advantages of an ubiquitous software ecosystem,
the lack of which has historically plagued other systems and relegated
them to a risky niche market?

# How do you turn a Scalar ISA into a Vector one?

The most obvious question before that is: why on earth would you want to?
As explained in the "SIMD Considered Harmful" article, Cray-style
Vector ISAs break the link between data element batches and the
underlying architectural back-end parallel processing capability.
Packed SIMD explicitly smashes that width right in the face of the
programmer and expects them to like it.  As the article immediately
demonstrates, an arbitrary-sized data set has to contend with
an insane power-of-two Packed SIMD cascade at both setup and teardown
that routinely adds literally an order
of magnitude increase in the number of hand-written lines of assembler
compared to a well-designed Cray-style Vector ISA with a `setvl`
instruction.

*<blockquote>
Packed SIMD looped algorithms actually have to
contain multiple implementations processing fragments of data at
different SIMD widths: Cray-style Vectors have just the one, covering not
just current architectural implementations but future ones with
wider back-end ALUs as well.
</blockquote>*

Assuming then that variable-length Vectors are obviously desirable,
it becomes a matter of how, not if.  Both Cray and NEC SX Aurora
went the way of adding explicit Vector opcodes, a style which RVV
copied and modernised. In the case of RVV this introduced 192 new
instructions on top of an existing 95+ for base RV64GC.  Adding
200% more instructions than the base ISA seems unwise: at least,
it feels like there should be a better way, particularly on
close inspection of RVV as an example, the basic arithmetic
operations are massively duplicated: scalar-scalar from the base
is joined by both scalar-vector and vector-vector *and* predicate
mask management, and transfer instructions between all the same,
which goes a long way towards explaining why there are twice as many
Vector instructions in RISC-V as there are in the RV64GC Scalar base.

The question then becomes: with all the duplication of arithmetic
operations just to make the registers scalar or vector, why not
leverage the *existing* Scalar ISA with some sort of "context"
or prefix that augments its behaviour? Make "Scalar instruction"
synonymous with "Vector Element instruction" and through nothing
more than contextual
augmentation the Scalar ISA *becomes* the Vector ISA.
Then, by not having to have any Vector instructions at all,
the Instruction Decode
phase is greatly simplified, reducing design complexity and leaving
plenty of headroom for further expansion.

Remarkably this is not a new idea.  Intel's x86 `REP` instruction
gives the base concept, but in 1994 it was Peter Hsu, the designer
of the MIPS R8000, who first came up with the idea of Vector-augmented
prefixing of an existing Scalar ISA.  Relying on a multi-issue Out-of-Order Execution Engine,
the prefix would mark which of the registers were to be treated as
Scalar and which as Vector, then, treating the Scalar "suffix" instruction
as a guide and making "scalar instruction" synonymous with "Vector element",
perform a `REP`-like loop that
jammed multiple scalar operations into the Multi-Issue Execution
Engine.  The only reason that the team did not take this forward
into a commercial product
was because they could not work out how to cleanly do OoO
multi-issue at the time.

In its simplest form, then, this "prefixing" idea is a matter
of:

* Defining the format of the prefix
* Adding a `setvl` instruction
* Adding Vector-context SPRs and working out how to do
  context-switches with them
* Writing an awful lot of Specification Documentation
  (4 years and counting)

Once the basics of this concept have sunk in, early
advancements quickly follow naturally from analysis
of the problem-space:

* Expanding the size of GPR, FPR and CR register files to
  provide 128 entries in each. This is a bare minimum for GPUs
  in order to keep processing workloads as close to a LOAD-COMPUTE-STORE
  batching as possible.
* Predication (an absolutely critical component for a Vector ISA),
  then the next logical advancement is to allow separate predication masks
  to be applied to *both* the source *and* the destination, independently.
  (*Readers familiar with Vector ISAs will recognise this as a back-to-back
  `VGATHER-VSCATTER`*)
* Element-width overrides: most Scalar ISAs today are 64-bit only,
  with primarily Load and Store being able to handle 8/16/32/64
  and sometimes 128-bit (quad-word), where Vector ISAs need to
  go as low as 8-bit arithmetic, even 8-bit Floating-Point for
  high-performance AI. Rather than waste opcode space adding all
  such operations at different bitwidths, let the prefix
  *redefine* (override) the element width, without actually altering
  the Scalar ISA at all.
* "Reordering" of the assumption of linear sequential element
  access, for Matrices, rotations, transposition, Convolutions,
  DCT, FFT, Parallel Prefix-Sum and other common transformations
  that require significant programming effort in other ISAs.

All of these things come entirely from "Augmentation" of the Scalar operation
being prefixed: at no time is the Scalar operation significantly
altered.
From there, several more "Modes" can be added, including

* saturation,
which is needed for Audio and Video applications
*  "Reverse Gear"
which runs the Element Loop in reverse order (needed for Prefix
Sum)
* Data-dependent Fail-First, which emerged from asking the simple
  question, "If modern Vector ISAs have Load/Store Fail-First,
  and the Power ISA has Condition Codes, why not make Conditional
  early-exit from Arithmetic operation looping?"
* over 500 Branch-Conditional Modes emerge from application of
  Boolean Logic in a Vector context, on top of an already-powerful
  Scalar Branch-Conditional/Counter instruction

**What is missing from Power Scalar ISA that a Vector ISA needs?**

Remarkably, very little: the devil is in the details though.

* The traditional `iota` instruction may be
  synthesised with an overlapping add, that stacks up incrementally
  and sequentially.  Although it requires two instructions (one to
  start the sum-chain) the technique has the advantage of allowing
  increments by arbitrary amounts, and is not limited to addition,
  either.
* Big-integer addition (arbitrary-precision arithmetic) is an
  emergent characteristic from the carry-in, carry-out capability of
  Power ISA `adde` instruction. `sv.adde` as a BigNum add
  naturally emerges from the
  sequential carry-flag chaining of these scalar instructions.
* The Condition Register Fields of the Power ISA make a great candidate
  for use as Predicate Masks, particularly when combined with
  Vectorised `cmp` and Vectorised `crand`, `crxor` etc.

It is only when looking slightly deeper into the Power ISA that
certain things turn out to be missing, and this is down in part to IBM's
primary focus on the 750 Packed SIMD opcodes at the expense of the 250 or
so Scalar ones.  Examples include that transfer operations between the
Integer and Floating-point Scalar register files were dropped approximately
a decade ago after the Packed SIMD variants were considered to be
duplicates.  With it being completely inappropriate to attempt to Vectorise
a Packed SIMD ISA designed 20 years ago with no Predication of any kind,
the Scalar ISA, a much better all-round candidate for Vectorisation is
left anaemic.

A particular key instruction that is missing is `MV.X` which is
illustrated as `GPR(dest) = GPR(GPR(src))`. This horrendously
expensive instruction causing a huge swathe of Register Hazards
in one single hit is almost never added to a Scalar ISA but
is almost always added to a Vector one. When `MV.X` is
Vectorised it allows for arbitrary
remapping of elements within a Vector to positions specified
by another Vector. A typical Scalar ISA will use Memory to
achieve this task, but with Vector ISAs the Vector Register Files are
usually so enormous, and so far away from Memory, that it is easier and
more efficient, architecturally, to provide these Indexing instructions.

Fortunately, with the ISA Working Group being willing
to consider RFCs (Requests For Change) these omissions have the potential
to be corrected.

One deliberate decision in SVP64 involves Predication. Typical Vector
ISAs have quite comprehensive arithmetic and logical operations on
Predicate Masks, and it turns out, unsurprisingly, that the Scalar Integer
side of Power ISA already has most of them.
If CR Fields were the only predicates in SVP64
it would put pressure on to start adding the exact same arithmetic and logical
operations that already exist in the Integer opcodes, which is less
than desirable.
Instead of taking that route the decision was made to allow *both*
Integer *and* CR Fields to be Predicate Masks, and to create Draft
instructions that provide better transfer capability between CR Fields
and Integer Register files. 

Beyond that, further extensions to the Power ISA become much more
domain-specific, such as adding bitmanipulation for Audio, Video
and Cryptographic use-cases, or adding Transcendentals (`LOG1P`,
`ATAN2` etc) for 3D and other GPU workloads. The huge advantage here
of the SVP64 "Prefix" approach is that anything added to the Scalar ISA
*automatically* is inherently added to the Vector one as well, and
because these GPU and Video opcodes have been added to the CPU ISA,
Software Driver development and debugging is dramatically simplified.

Which brings us to the next important question: how is any of these
CPU-centric Vector-centric improvements relevant to power efficiency
and making more effective use of resources?

# Simpler more compact programs saves power

The first and most obvious saving is that, just as with any Vector
ISA, the amount of data processing requested
and controlled by each instruction is enormous, and leaves the
Decode and Issue Engines idle, as well as the L1 I-Cache. With
programs being smaller, chances are higher that they fit into
L1 Cache, or that the L1 Cache may be made smaller: either way
is a considerable O(N^2) power-saving.

Even a Packed SIMD ISA could take limited advantage of a higher
bang-per-buck for limited specific workloads, as long as the
stripmining setup and teardown is not required.  However a
2-wide Packed SIMD instruction is nowhere near as high a bang-per-buck
ratio as a 64-wide Vector Length.

Realistically, for general use cases however it is extremely common
to have the Packed SIMD setup and teardown. `strncpy` for VSX is an
astounding 240 hand-coded assembler instructions where it is around
12 to 14 for both RVV and SVP64. Worst case (full algorithm unrolling
for Massive FFTs) the L1 I-Cache becomes completely ineffective, and in
the case of the IBM POWER9 with a little-known design flaw not
normally otherwise encountered this results in
contention between the L1 D and I Caches at the L2 Bus, slowing down
execution even further.  Power ISA 3.1 MMA (Matrix-Multiply-Assist)
requires loop-unrolling to contend with non-power-of-two Matrix
sizes: SVP64 does not (as hinted at below).
[Figures 8 and 9](https://arxiv.org/abs/2104.03142)
illustrate the process of concatenating copies of data in order
to match RADIX2 limitations of MMA.

Additional savings come in the form of `SVREMAP`. Like the
hardware-assist of Google's TPU mentioned on p9 of the above MMA paper,
`SVREMAP` is a hardware
index transformation system where the normally sequentially-linear
Vector element access may be "Re-Mapped" to limited but algorithmic-tailored
commonly-used deterministic schedules, for example Matrix Multiply,
DCT, or FFT.  A full in-register-file 5x7 Matrix Multiply or a 3x4 or
2x6 with optional *in-place* transpose, mirroring or rotation
on any source or destination Matrix
may be performed in as little as 4 instructions, one of which
is to zero-initialise the accumulator Vector used to store the result.
If addition to another Matrix is also required then it is only three
instructions.

Not only that, but because the "Schedule" is an abstract
concept separated from the mathematical operation, there is no reason
why Matrix Multiplication Schedules may not be applied to Integer
Mul-and-Accumulate, Galois Field Mul-and-Accumulate, Logical
AND-and-OR, or any other future instruction such as Complex-Number
Multiply-and-Accumulate that a future version of the Power ISA might
support.  The flexibility is not only enormous, but the compactness
unprecedented.  RADIX2 in-place DCT Triple-loop Schedules may be created in
around 11 instructions. The only other processors well-known to have
this type of compact capability are both VLIW DSPs: TI's TMS320 Series
and Qualcom's Hexagon, and both are targetted at FFTs only.

There is no reason at all why future algorithmic schedules should not
be proposed as extensions to SVP64 (sorting algorithms,
compression algorithms, Sparse Data Sets, Graph Node walking
for example). (*Bear in mind that
the submission process will be
entirely at the discretion of the OpenPOWER Foundation ISA WG,
something that is both encouraged and welcomed by the OPF.*)

One of SVP64's current limitations is that it was initially designed
for 3D and Video workloads as a hybrid GPU-VPU-CPU. This resulted in
a heavy focus on adding hardware-for-loops onto the *Registers*.
After more than three years of development the realisation hit that
the SVP64 concept could be expanded to Coherent Distributed Memory.
This astoundingly powerful concept is explored in the next section.

# Coherent Deterministic Hybrid Distributed In-Memory Processing

It is not often that a heading in an article can legitimately
contain quite so many comically-chained buzzwords, but in this section
they are justified.  As hinted at in the first section, the last time
that memory was the same speed as processors was the Pentium III
and Motorola 88100 era: 133 and 166 mhz SDRAM was available, and
CPUs were about the same rate.  DRAM bitcells *simply cannot exceed
these rates*, yet the pressure from Software Engineers is to
make *sequential* algorithm processing faster and faster because
parallelising of algorithms is simply too difficult to master, and always
has been.  Thus whilst DRAM has to go parallel (like RAID Striping) to
keep up, CPUs are now at 8-way Multi-Issue 5 ghz clock rates and
are at an astonishing four levels of cache (L1 to L4).

It should therefore come as no surprise that attempts are being made
to move (distribute) processing closer to the DRAM Memory, firmly
on the *opposite* side of the main CPU's L1/2/3/4 Caches,
where a simple `LOAD-COMPUTE-STORE-LOOP` workload easily illustrates
why this approach is compelling.  However
the alarm bells ring here at the keyword "distributed", because by
moving the processing down next to the Memory, even onto
the same die as the DRAM, the speed of any
of the parallel Processing Elements (PEs) would likely drop
by almost two orders of magnitude (5 ghz down to 150 mhz),
the simplicity of each PE has, for pure pragmatic reasons,
to drop by several
orders of magnitude as well.
Things that the average "sequential algorithm"
programmer
takes for granted such as SMP, Cache Coherency, Virtual Memory,
spinlocks (atomic locking, mutexes), all of these are either outright gone
or expected that the programmer shall explicitly contend with
(even if that programmer is the Compiler Developer). There's definitely
not going to be a standard OS: the PEs will be too basic, too
resource-constrained, and definitely too busy.

To give an extreme example: Aspex's Array-String Processor, which
was 4096 2-bit SIMD PEs each with 256 bytes of Content Addressable
Memory, was capable of literally a hundred-fold improvement in
performance over Scalar CPUs such as the Pentium III of its era,
all on a 3 watt budget at only 250 mhz in 130 nm.  Yet to take
proper advantage of its capability required an astounding 5-10
*days* per line of assembly code because multiple versions of
an algorithm had to be hand-crafted then compared, and only
the best one selected: all others discarded. 20 lines of optimised
Assembler taking three to six months to write can in no way be termed
"productive", yet this extreme level of unproductivity is an inherent
side-effect of going down the parallel-processing rabbithole where
the cost of providing "Traditional" programmabilility (Virtual Memory,
SMP) is worse than counter-productive, it's often outright impossible.

**In short, we are in "Programmer's nightmare" territory**

Having dug a proverbial hole that rivals the Grand Canyon, and
jumped in it feet-first, the next
task is to piece together a strategy to climb back out and show
how falling back in can be avoided. This takes some explaining,
and first requires some background on various research efforts and
commercial designs.  Once the context is clear, their synthesis
can be proposed.  These are:

* [ZOLC: Zero-Overhead Loop Control](https://ieeexplore.ieee.org/abstract/document/1692906/)
* [OpenCAPI and Extra-V](https://dl.acm.org/doi/abs/10.14778/3137765.3137776)
* [Snitch](https://arxiv.org/abs/2002.10143)

**ZOLC: Zero-Overhead Loop Control**

Zero-Overhead Looping is the concept of automatically running a set sequence
of instructions a predetermined number of times, without requiring
a branch. This is conceptually similar but
slightly different from using Power ISA `bc` in `CTR`
(Counter) Mode to create loops, because in ZOLC the branch-back is automatic.

The simplest longest commercially successful deployment of Zero-overhead looping
has been in Texas Instruments TMS320 DSPs. Up to fourteen sub-instructions
within the VLIW word may be repeatedly deployed on successive clock
cycles until a countdown reaches zero. This extraordinarily simple
concept needs no branches, and has no complex Register Hazard
Management in the hardware
because it is down to the programmer (or, the compiler),
to ensure data overlaps do not occur.  Careful crafting of those
14 instructions can keep the ALUs 100% occupied for sustained periods,
and the iconic example for which the TI DSPs are renowned
is that an entire inner loop for large FFTs
can be done with that one VLIW word: no stalls, no stopping, no fuss,
an entire 1024 or 4096 wide FFT Layer in one instruction.

<blockquote>
The key aspect of these
very simplistic countdown loops as far as we are concerned:
is: *they are deterministic*.
</blockquote>

Zero-Overhead Loop Control takes this basic "single loop" concept
way further: both nested loops and conditional exit are included,
but also arbitrary control-jumping from the current inner loop
out to an entirely different loop, all based on conditions determined
dynamically at runtime.

Even when deployed on as basic a CPU as a single-issue in-order RISC
core, the performance and power-savings were astonishing: between 20
and **80%** reduction in algorithm completion times were achieved compared
to a more traditional branch-speculative in-order RISC CPU.  MPEG
Decode, the target algorithm specifically picked by the researcher
due to its high complexity with 6-deep nested loops and conditional
execution that frequently jumped in and out of at least 2 loops,
came out with an astonishing 43% improvement in completion time. 43%
less instructions executed is an almost unheard-of level of optimisation:
most ISA designers are elated if they can achieve 5 to 10%. The reduction
was so compelling that ST Microelectronics put it into commercial
production in one of their embedded CPUs, the ST120 DSP-MCU.

The kicker: when implementing SVP64's Matrix REMAP Schedule, the VLSI
design of its triple-nested for-loop system
turned out to be remarkably similar to the
core nested for-loop engine of ZOLC. In hindsight this should not
have come as a surprise, because both are basically nested for-loops
that do not need branches to issue instructions.

The important insight is, however, that if ZOLC can be general-purpose
and apply deterministic nested looped instruction
schedules to more than just registers
(unlike SVP64 in its current incarnation) then so can SVP64.

**OpenCAPI and Extra-V**

OpenCAPI is a deterministic high-performance, high-bandwidth, low-latency
cache-coherent Memory-access Protocol that is integrated into IBM's Supercomputing-class POWER9 and POWER10 processors.

<blockquote>(Side note:
POWER10 *only*
has OpenCAPI Memory interfaces: an astounding number of them,
with overall bandwidth so high it's actually difficult to conceptualise.
An OMI-to-DDR4/5 Bridge PHY is therefore required
to connect to standard Memory DIMMs.)
</blockquote>

Extra-V appears to be a remarkable research project based on OpenCAPI that,
by assuming that the map of edges (excluding the actual data)
in any given arbitrary data graph
could be kept by the main CPU in-memory, could distribute and delegate
a limited-capability deterministic but most importantly *data-dependent*
node-walking schedule actually right down into the memory itself (on the other side of that L1-4 cache barrier).  A miniature processor 
(non-Turing-complete) analysed
the data it had read (at the Memory), and determined if it should
notify the main processor that this "Node" is worth investigating,
or if the Graph node-walk should split in a different direction.
Thanks to the OpenCAPI Standard, which takes care of Virtual Memory
abstraction, locking, and cache-coherency, many of the nightmare problems
of other more explicit parallel processing paradigms disappear.

The similarity to ZOLC should not have gone unnoticed: where ZOLC
has nested conditional for-loops Extra-V appears to have just the
one conditional for-loop, but the key strategically-crucial
part of this multi-faceted puzzle is that due to the deterministic and
coherent nature of Extra-V, the processing of the loops, which
requires a tiny non-Turing-Complete processor, is not
done close to or by the main CPU at all: it is
*embedded right next to the memory*.

The similarity to the D-Matrix Systolic Array Processing, Aspex Microelectronics
Array-String Processing, and Elixent 2D Array Processing, should
also not have gone unnoticed.  All of these solutions utilised
or utilise
a more comprehensive Turing-complete von-Neumann "Management Core"
to coordinate data passed in and out of PEs: none of them have or
had something
as powerful as OpenCAPI as part of that picture.

The fact that Neural Networks may be expressed as arbitrary Graphs,
and comprise Sparse Matrices, should also have been noted by the reader
interested in AI.

**Snitch**

Snitch is an elegant Memory-Coherent Barrel-Processor where registers
become "tagged" with a Memory-access Mode that went out of fashion
over forty years ago: Load-then-Auto-Increment. Expressed in c as
`src = *x++`, and requiring special Address Registers (PDP-11, 68000),
thanks to the RISC paradigm having gone too far,
the efficiency and effectiveness
of these Load-Store-with-Increment instructions has been
forgotten until Snitch.

What the designers did however was not to add any new Load-Store
or Arithmetic instructions to the underlying RISC-V at all, but instead to "mark"
registers with a tag which *augmented* (altered) the behaviour
of *existing* instructions.  These tags tell the CPU: when you are asked to
carry out
an add instruction on r6 and r7, do not take r6 or r7 from the register
file, instead please perform a Cache-coherent Load-with-Increment
on each, using special (hidden, implicit)
Address Registers for each.  Each new use
of r6 therefore brings in an entirely new value *directly from
memory*. Likewise on the second operand, r7, and likewise on
the destination result which can be an automatic Coherent
Store-and-increment
directly into Memory. 

<blockquote>
*The act of "reading" or "writing" a register has been decoupled
and intercepted, then connected transparently to a completely
separate Coherent Memory Subsystem*
</blockquote>

On top of a barrel-architecture the slowness of Memory access
was not a problem because the Deterministic nature of classic
Load-Store-Increment can be compensated for by having 8 Memory
accesses scheduled underway and interleaved in a time-sliced
fashion with an FPU that is correspondingly 8 times faster than
the Coherent Memory accesses.

This design is reminiscent of the early Vector Processors
of the late 1950s and early 1960s, which also critically relied
on implicit auto-increment addressing.
The [CDC STAR-100](https://en.m.wikipedia.org/wiki/CDC_STAR-100)
for example was specifically designed as a Memory-to-Memory Vector
Processor. The barrel-architecture of Snitch neatly
solves one of the inherent problems of those early designs (a mismatch
with memory
speed) and the presence of a full register file (non-tagged,
normal, standard scalar registers) caters for a
second limitation of pure Memory-based Vector Processors: temporary
variables needed in the computation of intermediate results, which
also had to go through memory, put
an awfully high artificial load on Memory bandwidth.

The similarity to SVP64 should be clear: SVP64 Prefixing and the
associated REMAP system is just another form of register "tagging"
that augments what was formerly designated by its original authors
as "just a Scalar ISA", tagging allows for dramatic implicit alteration
with advanced behaviour not previously envisaged.

What Snitch brings to the table therefore is a further illustration of
the concept introduced by Extra-V: where Extra-V brought information
about Sparse-Distributed Data to the attention of the main CPU in
a coherent fashion *without the CPU having to ask for it*, Snitch
demonstrates a classic LOAD-COMPUTE-STORE cycle in the same
distributed coherent manner, and does so with dramatically-reduced
power consumption.

**Bringing it all together**

At this point we are well into a future revision of SVP64, one that
clearly has some startlingly powerful potential: Supercomputing-class
Multi-Issue Vector Engines kept 100% occupied in a 100% long-term
sustained fashion with reduced complexity, reduced power consumption
and reduced completion time, thanks to Deterministic Coherent Scheduling
of the data fed in and out, or even moved down next to Memory.

This last part is where it normally gets hair-raising, but as ZOLC shows
there is no reason at all why even complex algorithms such as MPEG cannot
be run in a partially-deterministic manner, and anything that is 
deterministic can be Scheduled, coherently.  Combine that with OpenCAPI
which solves the many issues associated with SMP Virtual Memory and so on
yet still allows Cache-Coherent Distributed Memory Access, and what was
previously an intractable Computer Science problem for decades begins to
look like there is a potential solution.

The Deterministic Schedules created by ZOLC should even be possible to identify their
suitability for full off-CPU distributed processing, as long as OpenCAPI
is integrated into the mix.  What a compiler - or even the hardware -
will be looking out for is a Basic Block of instructions that:

* begins with a LOAD (to be handled by OpenCAPI)
* contains some instructions that a given PE is capable of executing
* ends with a STORE (again: OpenCAPI)

For best results that would be wrapped with a Zero-Overhead Loop
(which is offloaded - in full - down to the PE), where
the Compiler (or hardware at runtime) could easily identify, in advance,
the full range of Memory Addresses that the Loop is to encounter.  Copies
of loop-invariant data would need to be passed down to the remote PE:
again, for simple-enough Basic Blocks, with assistance from the Compiler,
loop-invariant inputs are easily identified. Parallel Processing
opportunities should also be easy enough to create, simply by farming out
different parts of a given Deterministic Zero-Overhead Loop to
different PEs based on their proximity, bandwidth or ease of access to
given Memory.

The importance of OpenCAPI in this mix cannot be underestimated, because
it will be the means by which the main CPU coordinates its activities
with the remote PEs, ensuring that LOAD/STORE Memory Hazards are not
violated. It should also be straightforward to ensure that the offloading
is entirely transparent to the developer, in fact this is a hard requirement
because at any given moment there is the possibility that the PEs may be
busy and it is the main CPU that has to complete the Processing Task itself.

It is also important to note that we are not necessarily talking about
the Remote PEs executing the Power ISA, but if they do so it becomes
much easier for the main CPU to take over in the event that PEs are
currently occupied.  Plus, the twin lessons that inventing ISAs, even
a small one, is hard (mostly in compiler writing) and how complex
GPU Task Scheduling is, are being heard loud and clear.

Put another way: if the PEs run a foriegn ISA, then the Basic Blocks embedded inside the ZOLC Loops must be in that ISA and therefore:

* In order that the main CPU can execute the same sequence if necessary,
  the CPU must support dual ISAs: Power and PE **OR**
* There must be a JIT binary-translator which either turns PE code
 into Power ISA code or vice-versa **OR**
* The compiler dual-compiles the original source code, and embeds
  both a Power binary and a PE binary into the ZOLC Basic Block **OR**
* All binaries are stored in an Intermediate Representation
  (LLVM-IR, SPIR-V) and JIT-compiled on-demand.

All of these would work, but it is simpler and a lot less work
just to have the PEs
execute the exact same ISA (or a subset of it). If however the
concept of Hybrid PE-Memory Processing were to become a JEDEC Standard,
which would increase adoption and reduce cost, a bit more thought
is required here because ARM or Intel or MIPS might not necessarily
be happy that a Processing Element has to execute Power ISA binaries.
At least the Power ISA is much richer, more powerful, still RISC,
and is an Open Standard, as discussed in a earlier sections.

# Transparently-Distributed Vector Processing

It is very strange to the author to be describing what amounts to a
"Holy Grail" solution to a decades-long intractable problem that
mitigates the anticipated end of Moore's Law: how to make it easy for
well-defined workloads, expressed as a perfectly normal
sequential program, compiled to a standard well-known ISA, to have
the potential of being offloaded transparently to Parallel Compute Engines,
all without the Software Developer being excessively burdened with
a Parallel-Processing Paradigm that is alien to all their experience
and training, as well as Industry-wide common knowledge.

Will it be that easy? ZOLC is, honestly, in its current incarnation,
not that straightforward: programs
have to be "massaged" by tools that insert intrinsics into the
source code, in order to identify the Basic Blocks that the Zero-Overhead
Loops can run. Can this be merged into standard gcc and llvm
compilers? As intrinsics: of course. Can it become part of auto-vectorisation? Probably,
if an infinite supply of money and engineering time is thrown at it.
Is a half-way-house solution of compiler intrinsics good enough?
Intel, ARM, MIPS, Power ISA and RISC-V have all already said "yes" on that,
for several decades, and advanced programmers are comfortable with the
practice.

Additional questions remain as to whether OpenCAPI or its use for this
particular scenario requires that the PEs, even quite basic ones,
implement a full RADIX MMU, and associated TLB lookup? In order to ensure
that programs may be cleanly and seamlessly transferred between PEs
and CPU the answer is quite likely to be "yes", which is interesting
in and of itself.  Fortunately, the associated L1 Cache with TLB
Translation does not have to be large, and the actual RADIX Tree Walk
need not explicitly be done by the PEs, it can be handled by the main
CPU as a software-extension: PEs generate a TLB Miss notification
to the main CPU over OpenCAPI, and the main CPU feeds back the new
TLB entries to the PE in response.

Also in practical terms, with the PEs anticipated to be so small as to
make running a full SMP-aware OS impractical it will not just be their TLB
pages that need remote management but their entire register file including
the Program Counter will need to be set up, and the ZOLC Context as
well. With OpenCAPI packet formats being quite large a concern is that
the context management increases latency to the point where the premise
of this paper is invalidated. Research is needed here as to whether a
bare-bones microkernel
would be viable, or a Management Core closer to the PEs (on the same
die or Multi-Chip-Module as the PEs) would allow better bandwidth and
reduce Management Overhead on the main CPUs.

**Use-case: Matrix and Convolutions**

* **Horizontal-First**: (aka standard Cray Vectors) walk
  through **elements** first before moving to next **instruction**
* **Vertical-First**: walk through **instructions** before
  moving to next **element**.  Currently managed by `svstep`,
  ZOLC may be deployed to manage the stepping, in a Deterministic manner.

Imagine a large Matrix scenario, with several values close to zero that
could be skipped: no need to include zero-multiplications, but a
traditional CPU in no way can help: only by loading the data through
the L1-L4 Cache and Virtual Memory Barriers is it possible to
ascertain, retrospectively, that time and power had just been wasted.

SVP64 is able to do what is termed "Vertical-First" Vectorisation,
combined with SVREMAP Matrix Schedules.  Imagine that SVREMAP has been
extended, Snitch-style, to perform a deterministic memory-array walk of
a large Matrix.

Let us also imagine that the Matrices are stored in Memory with PEs
attached, and that the PEs are fully functioning Power ISA with Draft
SVP64, but their Multiply capability is not as good as the main CPU.
Therefore:
we want the PEs to conditionally
feed sparse data to the main CPU, a la "Extra-V".

* The ZOLC SVREMAP System running on the main CPU generates a Matrix
  Memory-Load Schedule.
* The Schedule is sent to the PEs, next to the Memory, via OpenCAPI
* The PEs are also sent the Basic Block to be executed on each
  Memory Load (each element of the Matrices to be multiplied)
* The PEs execute the Basic Block and **exclude**, in a deterministic
  fashion, any elements containing Zero values
* Non-zero elements are sent, via OpenCAPI, to the main CPU, which
  queues sequences of Multiply-and-Accumulate, and feeds the results
  back to Memory, again via OpenCAPI, to the PEs.
* The PEs, which are tracking the Sparse Conditions, know where
  to store the results received

In essence this is near-identical to the original Snitch concept
except that there are, like Extra-V, PEs able to perform
conditional testing of the data as it goes both to and from the
main CPU.  In this way a large Sparse Matrix Multiply or Convolution
may be achieved without having to pass unnecessary data through
L1/L2/L3 Caches only to find, at the CPU, that it is zero.

The reason in this case for the use of Vertical-First Mode is the
conditional execution of the Multiply-and-Accumulate.
Horizontal-First Mode is the standard Cray-Style Vectorisation:
loop on all *elements* with the same instruction before moving
on to the next instruction. Predication needs to be pre-calculated
for the entire Vector in order to exclude certain elements from
the computation. In this case, that's an expensive inconvenience 
(similar to the problems associated with Memory-to-Memory
Vector Machines such as the CDC Star-100).

Vertical-First allows *scalar* instructions and
*scalar* temporary registers to be utilised
in the assessment as to whether a particular Vector element should
be skipped, utilising a straight Branch instruction *(or ZOLC
Conditions)*.  The Vertical Vector technique
is pioneered by Mitch Alsup and is a key feature of his VVM Extension
to MyISA 66000.  Careful analysis of the registers within the
Vertical-First Loop allows a Multi-Issue Out-of-Order Engine to
*amortise in-flight scalar looped operations into SIMD batches*
as long as the loop is kept small enough to entirely fit into
in-flight Reservation Stations in the first place.

*<blockquote>
(With thanks and gratitude to Mitch Alsup on comp.arch for
spending considerable time explaining VVM, how its Loop
Construct explicitly identifies loop-invariant registers,
and how that helps Register Hazards and SIMD amortisation
on a GB-OoO Micro-architecture)
</blockquote>*

Draft Image (placeholder):

<img src="/openpower/sv/zolc_svp64_extrav.jpg" width=800 />

The program being executed is a simple loop with a conditional
test that ignores the multiply if the input is zero.

* In the CPU-only case (top) the data goes through L1/L2
  Cache before reaching the CPU.
* However the PE version does not send zero-data to the CPU,
  and even when it does it goes into a Coherent FIFO: no real
  compelling need to enter L1/L2 Cache or even the CPU Register
  File (one of the key reasons why Snitch saves so much power).
* The PE-only version (see next use-case) the CPU is mostly
  idle, serving RADIX MMU TLB requests for PEs, and OpenCAPI
  requests.

**Use-case variant: More powerful in-memory PEs**

An obvious variant of the above is that, if there is inherently
more parallelism in the data set, then the PEs get their own
Multiply-and-Accumulate instruction, and rather than send the
data to the CPU over OpenCAPI, perform the Matrix-Multiply
directly themselves.

However the source code and binary would be near-identical if
not identical in every respect, and the PEs implementing the full
ZOLC capability in order to compact binary size to the bare minimum.
The main CPU's role would be to coordinate and manage the PEs
over OpenCAPI.

One key strategic question does remain: do the PEs need to have
a RADIX MMU and associated TLB-aware minimal L1 Cache, in order
to support OpenCAPI properly? The answer is very likely to be yes.
The saving grace here is that with
the expectation of running only hot-loops with ZOLC-driven
binaries, the size of each PE's TLB-aware
L1 Cache needed would be miniscule compared
to the average high-end CPU.

**Roadmap summary of Advanced SVP64**

The future direction for SVP64, then, is:

* To overcome its current limitation of REMAP Schedules being
  restricted to Register Files, leveraging the Snitch-style
  register interception "tagging" technique.
* To adopt ZOLC and merge REMAP Schedules into ZOLC
* To bring OpenCAPI Memory Access into ZOLC as a first-level
  concept that mirrors Snitch's Coherent Memory interception
* To add the Graph-Node Walking Capability of Extra-V
  to ZOLC / SVREMAP
* To make it possible, in a combination of hardware and software,
  to easily identify ZOLC / SVREMAP Blocks
  that may be transparently pushed down closer to Memory, for
  localised distributed parallel execution, by OpenCAPI-aware PEs,
  exploiting both the Deterministic nature of ZOLC / SVREMAP
  combined with the Cache-Coherent nature of OpenCAPI,
  to the maximum extent possible.
* To make the exploitation of this powerful solution as simple
  and straightforward as possible for Software Engineers to use,
  in standard common-usage compilers, gcc and llvm.
* To propose extensions to Public Standards that allow all of
  the above to become part of everyday ubiquitous mass-volume
  computing.

Even the first of these - merging Snitch-style register tagging
into SVP64 - would
expand SVP64's capability for Matrices, currently limited to
around 5x7 to 6x6 Matrices and constrained by the size of
the register files (128 64-bit entries), to arbitrary (massive) sizes.

**Summary**

Bottom line is that there is a clear roadmap towards solving a long
standing problem facing Computer Science and doing so in a way that
reduces power consumption reduces algorithm completion time and reduces
the need for complex hardware microarchitectures in favour of much
smaller distributed coherent Processing Elements.