dcache: Reduce latencies and improve timing

dcache: Reduce latencies and improve timing

This implements various improvements to the dcache with the aim of
making it go faster.

- We can now execute operations that don't need to access main memory
  (cacheable loads that hit in the cache and TLB operations) as soon
  as any previous operation has completed, without waiting for the
  state machine to become idle.

- Cache line refills start with the doubleword that is needed to
  satisfy the load that initiated them.

- Cacheable loads that miss return their data and complete as soon as
  the requested doubleword comes back from memory; they don't wait for
  the refill to finish.

- We now have per-doubleword valid bits for the cache line being
  refilled, meaning that if a load comes in for a line that is in the
  process of being refilled, we can return the data and complete it
  within a couple of cycles of the doubleword coming in from memory.

- There is now a bypass path for data being written to the cache RAM
  so that we can do a store hit followed immediately by a load hit to
  the same doubleword.  This also makes the data from a refill
  available to load hits one cycle earlier than it would be otherwise.

- Stores complete in the cycle where their wishbone operation is
  initiated, without waiting for the wishbone cycle to complete.

- During the wishbone cycle for a store, if another store comes in
  that is to the same page, and we don't have a stall from the
  wishbone, we can send out the write for the second store in the same
  wishbone cycle and without going through the IDLE state first.  We
  limit it to 7 outstanding writes that have not yet been
  acknowledged.

- The cache tag RAM is now read on a clock edge rather than being
  combinatorial for reading.  Its width is rounded up to a multiple of
  8 bits per way so that byte enables can be used for writing
  individual tags.

- The cache tag RAM is now written a cycle later than previously, in
  order to ease timing.

- Data for a store hit is now written one cycle later than
  previously.  This eases timing since we don't have to get through
  the tag matching and on to the write enable within a single cycle.
  The 2-stage bypass path means we can still handle a load hit on
  either of the two cycles after the store and return the correct
  data.  (A load hit 3 or more cycles later will get the correct data
  from the BRAM.)

- Operations can sit in r0 while there is an uncompleted operation in
  r1.  Once the operation in r1 is completed, the operation in r0
  spends one cycle in r0 for TLB/cache tag lookup and then gets put
  into r1.req.  This can happen before r1 gets to the IDLE state.
  Some operations can then be completed before r1 gets to the IDLE
  state - a load miss to the cache line being refilled, or a store to
  the same page as a previous store.

Signed-off-by: Paul Mackerras <paulus@ozlabs.org>