2 * Copyright (c) 1999-2008 Mark D. Hill and David A. Wood
5 * Redistribution and use in source and binary forms, with or without
6 * modification, are permitted provided that the following conditions are
7 * met: redistributions of source code must retain the above copyright
8 * notice, this list of conditions and the following disclaimer;
9 * redistributions in binary form must reproduce the above copyright
10 * notice, this list of conditions and the following disclaimer in the
11 * documentation and/or other materials provided with the distribution;
12 * neither the name of the copyright holders nor the names of its
13 * contributors may be used to endorse or promote products derived from
14 * this software without specific prior written permission.
16 * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
17 * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
18 * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
19 * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
20 * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
21 * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
22 * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
23 * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
24 * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
25 * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
26 * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
30 * Description: This module simulates a basic DDR-style memory controller
31 * (and can easily be extended to do FB-DIMM as well).
33 * This module models a single channel, connected to any number of
34 * DIMMs with any number of ranks of DRAMs each. If you want multiple
35 * address/data channels, you need to instantiate multiple copies of
38 * Each memory request is placed in a queue associated with a specific
39 * memory bank. This queue is of finite size; if the queue is full
40 * the request will back up in an (infinite) common queue and will
41 * effectively throttle the whole system. This sort of behavior is
42 * intended to be closer to real system behavior than if we had an
43 * infinite queue on each bank. If you want the latter, just make
44 * the bank queues unreasonably large.
46 * The head item on a bank queue is issued when all of the
48 * the bank is available
49 * the address path to the DIMM is available
50 * the data path to or from the DIMM is available
52 * Note that we are not concerned about fixed offsets in time. The bank
53 * will not be used at the same moment as the address path, but since
54 * there is no queue in the DIMM or the DRAM it will be used at a constant
55 * number of cycles later, so it is treated as if it is used at the same
58 * We are assuming closed bank policy; that is, we automatically close
59 * each bank after a single read or write. Adding an option for open
60 * bank policy is for future work.
62 * We are assuming "posted CAS"; that is, we send the READ or WRITE
63 * immediately after the ACTIVATE. This makes scheduling the address
64 * bus trivial; we always schedule a fixed set of cycles. For DDR-400,
65 * this is a set of two cycles; for some configurations such as
66 * DDR-800 the parameter tRRD forces this to be set to three cycles.
68 * We assume a four-bit-time transfer on the data wires. This is
69 * the minimum burst length for DDR-2. This would correspond
70 * to (for example) a memory where each DIMM is 72 bits wide
71 * and DIMMs are ganged in pairs to deliver 64 bytes at a shot.
72 * This gives us the same occupancy on the data wires as on the
73 * address wires (for the two-address-cycle case).
75 * The only non-trivial scheduling problem is the data wires.
76 * A write will use the wires earlier in the operation than a read
77 * will; typically one cycle earlier as seen at the DRAM, but earlier
78 * by a worst-case round-trip wire delay when seen at the memory controller.
79 * So, while reads from one rank can be scheduled back-to-back
80 * every two cycles, and writes (to any rank) scheduled every two cycles,
81 * when a read is followed by a write we need to insert a bubble.
82 * Furthermore, consecutive reads from two different ranks may need
83 * to insert a bubble due to skew between when one DRAM stops driving the
84 * wires and when the other one starts. (These bubbles are parameters.)
86 * This means that when some number of reads and writes are at the
87 * heads of their queues, reads could starve writes, and/or reads
88 * to the same rank could starve out other requests, since the others
89 * would never see the data bus ready.
90 * For this reason, we have implemented an anti-starvation feature.
91 * A group of requests is marked "old", and a counter is incremented
92 * each cycle as long as any request from that batch has not issued.
93 * if the counter reaches twice the bank busy time, we hold off any
94 * newer requests until all of the "old" requests have issued.
96 * We also model tFAW. This is an obscure DRAM parameter that says
97 * that no more than four activate requests can happen within a window
98 * of a certain size. For most configurations this does not come into play,
99 * or has very little effect, but it could be used to throttle the power
100 * consumption of the DRAM. In this implementation (unlike in a DRAM
101 * data sheet) TFAW is measured in memory bus cycles; i.e. if TFAW = 16
102 * then no more than four activates may happen within any 16 cycle window.
103 * Refreshes are included in the activates.
107 #include "base/cprintf.hh"
108 #include "mem/gems_common/Map.hh"
109 #include "mem/ruby/common/Address.hh"
110 #include "mem/ruby/common/Consumer.hh"
111 #include "mem/ruby/common/Global.hh"
112 #include "mem/ruby/network/Network.hh"
113 #include "mem/ruby/profiler/Profiler.hh"
114 #include "mem/ruby/slicc_interface/NetworkMessage.hh"
115 #include "mem/ruby/slicc_interface/RubySlicc_ComponentMapping.hh"
116 #include "mem/ruby/system/MemoryControl.hh"
117 #include "mem/ruby/system/System.hh"
123 // Value to reset watchdog timer to.
124 // If we're idle for this many memory control cycles,
125 // shut down our clock (our rescheduling of ourselves).
126 // Refresh shuts down as well.
127 // When we restart, we'll be in a different phase
128 // with respect to ruby cycles, so this introduces
129 // a slight inaccuracy. But it is necessary or the
130 // ruby tester never terminates because the event
131 // queue is never empty.
132 #define IDLECOUNT_MAX_VALUE 1000
134 // Output operator definition
137 operator<<(ostream
& out
, const MemoryControl
& obj
)
145 // ****************************************************************
148 MemoryControl::MemoryControl(const Params
*p
)
151 m_mem_bus_cycle_multiplier
= p
->mem_bus_cycle_multiplier
;
152 m_banks_per_rank
= p
->banks_per_rank
;
153 m_ranks_per_dimm
= p
->ranks_per_dimm
;
154 m_dimms_per_channel
= p
->dimms_per_channel
;
155 m_bank_bit_0
= p
->bank_bit_0
;
156 m_rank_bit_0
= p
->rank_bit_0
;
157 m_dimm_bit_0
= p
->dimm_bit_0
;
158 m_bank_queue_size
= p
->bank_queue_size
;
159 m_bank_busy_time
= p
->bank_busy_time
;
160 m_rank_rank_delay
= p
->rank_rank_delay
;
161 m_read_write_delay
= p
->read_write_delay
;
162 m_basic_bus_busy_time
= p
->basic_bus_busy_time
;
163 m_mem_ctl_latency
= p
->mem_ctl_latency
;
164 m_refresh_period
= p
->refresh_period
;
166 m_mem_random_arbitrate
= p
->mem_random_arbitrate
;
167 m_mem_fixed_delay
= p
->mem_fixed_delay
;
169 m_profiler_ptr
= new MemCntrlProfiler(name(),
172 m_dimms_per_channel
);
176 MemoryControl::init()
182 assert(m_tFaw
<= 62); // must fit in a uint64 shift register
184 m_total_banks
= m_banks_per_rank
* m_ranks_per_dimm
* m_dimms_per_channel
;
185 m_total_ranks
= m_ranks_per_dimm
* m_dimms_per_channel
;
186 m_refresh_period_system
= m_refresh_period
/ m_total_banks
;
188 m_bankQueues
= new list
<MemoryNode
> [m_total_banks
];
189 assert(m_bankQueues
);
191 m_bankBusyCounter
= new int [m_total_banks
];
192 assert(m_bankBusyCounter
);
194 m_oldRequest
= new int [m_total_banks
];
195 assert(m_oldRequest
);
197 for (int i
= 0; i
< m_total_banks
; i
++) {
198 m_bankBusyCounter
[i
] = 0;
202 m_busBusyCounter_Basic
= 0;
203 m_busBusyCounter_Write
= 0;
204 m_busBusyCounter_ReadNewRank
= 0;
205 m_busBusy_WhichRank
= 0;
215 // Each tfaw shift register keeps a moving bit pattern
216 // which shows when recent activates have occurred.
217 // m_tfaw_count keeps track of how many 1 bits are set
218 // in each shift register. When m_tfaw_count is >= 4,
219 // new activates are not allowed.
220 m_tfaw_shift
= new uint64
[m_total_ranks
];
221 m_tfaw_count
= new int[m_total_ranks
];
222 for (int i
= 0; i
< m_total_ranks
; i
++) {
228 MemoryControl::~MemoryControl()
230 delete [] m_bankQueues
;
231 delete [] m_bankBusyCounter
;
232 delete [] m_oldRequest
;
233 delete m_profiler_ptr
;
236 // enqueue new request from directory
238 MemoryControl::enqueue(const MsgPtr
& message
, int latency
)
240 Time current_time
= g_eventQueue_ptr
->getTime();
241 Time arrival_time
= current_time
+ latency
;
242 const MemoryMsg
* memMess
= safe_cast
<const MemoryMsg
*>(message
.get());
243 physical_address_t addr
= memMess
->getAddress().getAddress();
244 MemoryRequestType type
= memMess
->getType();
245 bool is_mem_read
= (type
== MemoryRequestType_MEMORY_READ
);
246 MemoryNode
thisReq(arrival_time
, message
, addr
, is_mem_read
, !is_mem_read
);
247 enqueueMemRef(thisReq
);
250 // Alternate entry point used when we already have a MemoryNode
253 MemoryControl::enqueueMemRef(MemoryNode
& memRef
)
256 memRef
.m_msg_counter
= m_msg_counter
;
257 Time arrival_time
= memRef
.m_time
;
258 uint64 at
= arrival_time
;
259 bool is_mem_read
= memRef
.m_is_mem_read
;
260 physical_address_t addr
= memRef
.m_addr
;
261 int bank
= getBank(addr
);
263 cprintf("New memory request%7d: %#08x %c arrived at %10d bank = %3x\n",
264 m_msg_counter
, addr
, is_mem_read
? 'R':'W', at
, bank
);
267 m_profiler_ptr
->profileMemReq(bank
);
268 m_input_queue
.push_back(memRef
);
270 g_eventQueue_ptr
->scheduleEvent(this, 1);
275 // dequeue, peek, and isReady are used to transfer completed requests
276 // back to the directory
278 MemoryControl::dequeue()
281 m_response_queue
.pop_front();
285 MemoryControl::peek()
287 MemoryNode node
= peekNode();
288 Message
* msg_ptr
= node
.m_msgptr
.get();
289 assert(msg_ptr
!= NULL
);
294 MemoryControl::peekNode()
297 MemoryNode req
= m_response_queue
.front();
298 uint64 returnTime
= req
.m_time
;
300 cprintf("Old memory request%7d: %#08x %c peeked at %10d\n",
301 req
.m_msg_counter
, req
.m_addr
, req
.m_is_mem_read
? 'R':'W',
308 MemoryControl::isReady()
310 return ((!m_response_queue
.empty()) &&
311 (m_response_queue
.front().m_time
<= g_eventQueue_ptr
->getTime()));
315 MemoryControl::setConsumer(Consumer
* consumer_ptr
)
317 m_consumer_ptr
= consumer_ptr
;
321 MemoryControl::print(ostream
& out
) const
326 MemoryControl::printConfig(ostream
& out
)
328 out
<< "Memory Control " << name() << ":" << endl
;
329 out
<< " Ruby cycles per memory cycle: " << m_mem_bus_cycle_multiplier
331 out
<< " Basic read latency: " << m_mem_ctl_latency
<< endl
;
332 if (m_mem_fixed_delay
) {
333 out
<< " Fixed Latency mode: Added cycles = " << m_mem_fixed_delay
336 out
<< " Bank busy time: " << m_bank_busy_time
<< " memory cycles"
338 out
<< " Memory channel busy time: " << m_basic_bus_busy_time
<< endl
;
339 out
<< " Dead cycles between reads to different ranks: "
340 << m_rank_rank_delay
<< endl
;
341 out
<< " Dead cycle between a read and a write: "
342 << m_read_write_delay
<< endl
;
343 out
<< " tFaw (four-activate) window: " << m_tFaw
<< endl
;
345 out
<< " Banks per rank: " << m_banks_per_rank
<< endl
;
346 out
<< " Ranks per DIMM: " << m_ranks_per_dimm
<< endl
;
347 out
<< " DIMMs per channel: " << m_dimms_per_channel
<< endl
;
348 out
<< " LSB of bank field in address: " << m_bank_bit_0
<< endl
;
349 out
<< " LSB of rank field in address: " << m_rank_bit_0
<< endl
;
350 out
<< " LSB of DIMM field in address: " << m_dimm_bit_0
<< endl
;
351 out
<< " Max size of each bank queue: " << m_bank_queue_size
<< endl
;
352 out
<< " Refresh period (within one bank): " << m_refresh_period
<< endl
;
353 out
<< " Arbitration randomness: " << m_mem_random_arbitrate
<< endl
;
357 MemoryControl::setDebug(int debugFlag
)
363 MemoryControl::clearStats() const
365 m_profiler_ptr
->clearStats();
369 MemoryControl::printStats(ostream
& out
) const
371 m_profiler_ptr
->printStats(out
);
374 // Queue up a completed request to send back to directory
376 MemoryControl::enqueueToDirectory(MemoryNode req
, int latency
)
378 Time arrival_time
= g_eventQueue_ptr
->getTime()
379 + (latency
* m_mem_bus_cycle_multiplier
);
380 req
.m_time
= arrival_time
;
381 m_response_queue
.push_back(req
);
383 // schedule the wake up
384 g_eventQueue_ptr
->scheduleEventAbsolute(m_consumer_ptr
, arrival_time
);
387 // getBank returns an integer that is unique for each
388 // bank across this memory controller.
390 MemoryControl::getBank(physical_address_t addr
)
392 int dimm
= (addr
>> m_dimm_bit_0
) & (m_dimms_per_channel
- 1);
393 int rank
= (addr
>> m_rank_bit_0
) & (m_ranks_per_dimm
- 1);
394 int bank
= (addr
>> m_bank_bit_0
) & (m_banks_per_rank
- 1);
395 return (dimm
* m_ranks_per_dimm
* m_banks_per_rank
)
396 + (rank
* m_banks_per_rank
)
400 // getRank returns an integer that is unique for each rank
401 // and independent of individual bank.
403 MemoryControl::getRank(int bank
)
405 int rank
= (bank
/ m_banks_per_rank
);
406 assert (rank
< (m_ranks_per_dimm
* m_dimms_per_channel
));
410 // queueReady determines if the head item in a bank queue
411 // can be issued this cycle
413 MemoryControl::queueReady(int bank
)
415 if ((m_bankBusyCounter
[bank
] > 0) && !m_mem_fixed_delay
) {
416 m_profiler_ptr
->profileMemBankBusy();
419 printf(" bank %x busy %d\n", bank
, m_bankBusyCounter
[bank
]);
424 if (m_mem_random_arbitrate
>= 2) {
425 if ((random() % 100) < m_mem_random_arbitrate
) {
426 m_profiler_ptr
->profileMemRandBusy();
431 if (m_mem_fixed_delay
)
434 if ((m_ageCounter
> (2 * m_bank_busy_time
)) && !m_oldRequest
[bank
]) {
435 m_profiler_ptr
->profileMemNotOld();
439 if (m_busBusyCounter_Basic
== m_basic_bus_busy_time
) {
440 // Another bank must have issued this same cycle. For
441 // profiling, we count this as an arb wait rather than a bus
442 // wait. This is a little inaccurate since it MIGHT have also
443 // been blocked waiting for a read-write or a read-read
444 // instead, but it's pretty close.
445 m_profiler_ptr
->profileMemArbWait(1);
449 if (m_busBusyCounter_Basic
> 0) {
450 m_profiler_ptr
->profileMemBusBusy();
454 int rank
= getRank(bank
);
455 if (m_tfaw_count
[rank
] >= ACTIVATE_PER_TFAW
) {
456 m_profiler_ptr
->profileMemTfawBusy();
460 bool write
= !m_bankQueues
[bank
].front().m_is_mem_read
;
461 if (write
&& (m_busBusyCounter_Write
> 0)) {
462 m_profiler_ptr
->profileMemReadWriteBusy();
466 if (!write
&& (rank
!= m_busBusy_WhichRank
)
467 && (m_busBusyCounter_ReadNewRank
> 0)) {
468 m_profiler_ptr
->profileMemDataBusBusy();
475 // issueRefresh checks to see if this bank has a refresh scheduled
476 // and, if so, does the refresh and returns true
478 MemoryControl::issueRefresh(int bank
)
480 if (!m_need_refresh
|| (m_refresh_bank
!= bank
))
482 if (m_bankBusyCounter
[bank
] > 0)
484 // Note that m_busBusyCounter will prevent multiple issues during
485 // the same cycle, as well as on different but close cycles:
486 if (m_busBusyCounter_Basic
> 0)
488 int rank
= getRank(bank
);
489 if (m_tfaw_count
[rank
] >= ACTIVATE_PER_TFAW
)
495 uint64 current_time
= g_eventQueue_ptr
->getTime();
496 printf(" Refresh bank %3x at %lld\n", bank
, current_time
);
500 m_profiler_ptr
->profileMemRefresh();
503 if (m_refresh_bank
>= m_total_banks
)
505 m_bankBusyCounter
[bank
] = m_bank_busy_time
;
506 m_busBusyCounter_Basic
= m_basic_bus_busy_time
;
507 m_busBusyCounter_Write
= m_basic_bus_busy_time
;
508 m_busBusyCounter_ReadNewRank
= m_basic_bus_busy_time
;
513 // Mark the activate in the tFaw shift register
515 MemoryControl::markTfaw(int rank
)
518 m_tfaw_shift
[rank
] |= (1 << (m_tFaw
-1));
519 m_tfaw_count
[rank
]++;
523 // Issue a memory request: Activate the bank, reserve the address and
524 // data buses, and queue the request for return to the requesting
525 // processor after a fixed latency.
527 MemoryControl::issueRequest(int bank
)
529 int rank
= getRank(bank
);
530 MemoryNode req
= m_bankQueues
[bank
].front();
531 m_bankQueues
[bank
].pop_front();
533 uint64 current_time
= g_eventQueue_ptr
->getTime();
534 cprintf(" Mem issue request%7d: %#08x %c at %10d "
536 req
.m_msg_counter
, req
.m_addr
, req
.m_is_mem_read
? 'R':'W',
539 if (req
.m_msgptr
) { // don't enqueue L3 writebacks
540 enqueueToDirectory(req
, m_mem_ctl_latency
+ m_mem_fixed_delay
);
542 m_oldRequest
[bank
] = 0;
544 m_bankBusyCounter
[bank
] = m_bank_busy_time
;
545 m_busBusy_WhichRank
= rank
;
546 if (req
.m_is_mem_read
) {
547 m_profiler_ptr
->profileMemRead();
548 m_busBusyCounter_Basic
= m_basic_bus_busy_time
;
549 m_busBusyCounter_Write
= m_basic_bus_busy_time
+ m_read_write_delay
;
550 m_busBusyCounter_ReadNewRank
=
551 m_basic_bus_busy_time
+ m_rank_rank_delay
;
553 m_profiler_ptr
->profileMemWrite();
554 m_busBusyCounter_Basic
= m_basic_bus_busy_time
;
555 m_busBusyCounter_Write
= m_basic_bus_busy_time
;
556 m_busBusyCounter_ReadNewRank
= m_basic_bus_busy_time
;
560 // executeCycle: This function is called once per memory clock cycle
561 // to simulate all the periodic hardware.
563 MemoryControl::executeCycle()
565 // Keep track of time by counting down the busy counters:
566 for (int bank
=0; bank
< m_total_banks
; bank
++) {
567 if (m_bankBusyCounter
[bank
] > 0) m_bankBusyCounter
[bank
]--;
569 if (m_busBusyCounter_Write
> 0)
570 m_busBusyCounter_Write
--;
571 if (m_busBusyCounter_ReadNewRank
> 0)
572 m_busBusyCounter_ReadNewRank
--;
573 if (m_busBusyCounter_Basic
> 0)
574 m_busBusyCounter_Basic
--;
576 // Count down the tFAW shift registers:
577 for (int rank
=0; rank
< m_total_ranks
; rank
++) {
578 if (m_tfaw_shift
[rank
] & 1) m_tfaw_count
[rank
]--;
579 m_tfaw_shift
[rank
] >>= 1;
582 // After time period expires, latch an indication that we need a refresh.
583 // Disable refresh if in mem_fixed_delay mode.
584 if (!m_mem_fixed_delay
) m_refresh_count
--;
585 if (m_refresh_count
== 0) {
586 m_refresh_count
= m_refresh_period_system
;
588 // Are we overrunning our ability to refresh?
589 assert(m_need_refresh
< 10);
593 // If this batch of requests is all done, make a new batch:
596 for (int bank
=0; bank
< m_total_banks
; bank
++) {
597 anyOld
|= m_oldRequest
[bank
];
600 for (int bank
=0; bank
< m_total_banks
; bank
++) {
601 if (!m_bankQueues
[bank
].empty()) m_oldRequest
[bank
] = 1;
606 // If randomness desired, re-randomize round-robin position each cycle
607 if (m_mem_random_arbitrate
) {
608 m_roundRobin
= random() % m_total_banks
;
611 // For each channel, scan round-robin, and pick an old, ready
612 // request and issue it. Treat a refresh request as if it were at
613 // the head of its bank queue. After we issue something, keep
614 // scanning the queues just to gather statistics about how many
615 // are waiting. If in mem_fixed_delay mode, we can issue more
616 // than one request per cycle.
619 for (int i
= 0; i
< m_total_banks
; i
++) {
621 if (m_roundRobin
>= m_total_banks
) m_roundRobin
= 0;
622 issueRefresh(m_roundRobin
);
623 int qs
= m_bankQueues
[m_roundRobin
].size();
625 m_profiler_ptr
->profileMemBankQ(qs
-1);
628 // we're not idle if anything is queued
629 m_idleCount
= IDLECOUNT_MAX_VALUE
;
631 if (queueReady(m_roundRobin
)) {
632 issueRequest(m_roundRobin
);
634 if (m_mem_fixed_delay
) {
635 m_profiler_ptr
->profileMemWaitCycles(m_mem_fixed_delay
);
641 // memWaitCycles is a redundant catch-all for the specific
642 // counters in queueReady
643 m_profiler_ptr
->profileMemWaitCycles(queueHeads
- banksIssued
);
645 // Check input queue and move anything to bank queues if not full.
646 // Since this is done here at the end of the cycle, there will
647 // always be at least one cycle of latency in the bank queue. We
648 // deliberately move at most one request per cycle (to simulate
649 // typical hardware). Note that if one bank queue fills up, other
650 // requests can get stuck behind it here.
651 if (!m_input_queue
.empty()) {
652 // we're not idle if anything is pending
653 m_idleCount
= IDLECOUNT_MAX_VALUE
;
654 MemoryNode req
= m_input_queue
.front();
655 int bank
= getBank(req
.m_addr
);
656 if (m_bankQueues
[bank
].size() < m_bank_queue_size
) {
657 m_input_queue
.pop_front();
658 m_bankQueues
[bank
].push_back(req
);
660 m_profiler_ptr
->profileMemInputQ(m_input_queue
.size());
664 // wakeup: This function is called once per memory controller clock cycle.
666 MemoryControl::wakeup()
668 // execute everything
672 if (m_idleCount
<= 0) {
675 // Reschedule ourselves so that we run every memory cycle:
676 g_eventQueue_ptr
->scheduleEvent(this, m_mem_bus_cycle_multiplier
);
681 RubyMemoryControlParams::create()
683 return new MemoryControl(this);