1 # Tutorial for setting up Microwatt chroot and running simulations
3 Useful Links (External):
5 * <https://codeconstruct.com.au/docs/microwatt-orangecrab/>
6 * <https://shenki.github.io/boot-linux-on-microwatt/>
7 * <https://github.com/gregdavill/OrangeCrab-test-sw>
8 * [Verilator docs, commands](https://verilator.org/guide/latest/exe_verilator.html)
9 * [Verilator runtime command documentation](https://verilator.org/guide/latest/exe_sim.html)
10 * Tutorials for how to work with verilator:
11 [part1](https://www.itsembedded.com/dhd/verilator_1/),
12 [part2](https://www.itsembedded.com/dhd/verilator_2/)
14 Useful links (Libre-SOC):
16 * Libre-SOC page covering our workflow: [[HDL_workflow]]
17 * Devscripts Libre-SOC page: [[devscripts]]
18 * Original Microwatt Libre-SOC page: [[microwatt]]
19 * [Libre-SOC Microwatt repo branch](https://git.libre-soc.org/?p=microwatt.git;a=tree;hb=refs/heads/verilator_trace)
20 * [Libre-SOC devscripts repo](https://git.libre-soc.org/?p=dev-env-setup.git;a=tree)
22 Other Tutorials (Libre-SOC):
24 * First steps for working with PowerISA instructions Libre-SOC page:
27 ## Development environment scripts
29 If you haven't already, clone Libre-SOC's development environment setup scripts.
30 These are bash scripts, and greatly simplify the time it takes to create a:
33 - With all software and libraries at specific versions
34 (which are known to work).
36 These attributes are absolutely critical, and no support will be
37 provided, unless you use these scripts to setup a development environment. This
38 helps us fix any bugs in the scripts, and make sure everyone runs on the same
41 $ git clone https://git.libre-soc.org/git/dev-env-setup.git
44 [code](https://git.libre-soc.org/?p=dev-env-setup.git;a=tree) before running
45 any of those scripts. They may be confusing, however after reading a few you'll
46 start to become more familiar with them.
48 It is expected for you to use Debian, we mostly use 11 (Bullseye) for the host
49 OS, while all the chroots run Debian 10 (Buster).
53 Scripts we will be using for the setup are:
55 * `mk-deb-chroot`, `cp-scripts-to-chroot` for chroot setup
56 * `install-hdl-apt-reqs`, `verilator-install`, `hdl-tools-yosys` for working
59 (*Current limitation for `mk-deb-chroot`, is that you must be the first user on
60 the host machine, having user ID 1000.*)
62 Commands to run in terminal to setup a new chroot environment for microwatt
67 # ./mk-deb-chroot microwatt
68 # ./cp-scripts-to-chroot microwatt
70 $ schroot -c microwatt
71 (microwatt):$ cd dev-env-setup
72 (microwatt):$ sudo bash
73 (microwatt):# ./install-hdl-apt-reqs
74 (microwatt):# ./verilator-install
75 (microwatt):# ./hdl-tools-yosys
77 (microwatt):$ cd ~/src/
78 (microwatt):$ git clone https://git.libre-soc.org/git/microwatt.git
79 (microwatt):$ cd microwatt
80 (microwatt):$ git checkout verilator_trace
82 Make sure verilator binaries in $PATH:
84 (microwatt):$ export PATH=/usr/local/verilator/bin:$PATH
85 (microwatt):$ export GHDLSYNTH=ghdl
87 (GHDLSYNTH needs to be redefined because the Makefile has default `ghdl.so`,
88 but somewhere else '.so' gets appended. You may see the following error if you
90 `ERROR: Can't load module
91 ./ghdl.so':/usr/local/bin/../share/yosys/plugins/**ghdl.so.so**`)
92 [IRC](https://libre-soc.org/irclog/%23libre-soc.2023-01-25.log.html#t2023-01-25T11:10:47)
94 ## Compiling the verilator sim for Microwatt
96 * [Libre-SOC Microwatt repo branch, Makefile](https://git.libre-soc.org/?p=microwatt.git;a=blob;f=Makefile;hb=refs/heads/verilator_trace)
98 Verilator creates a fairly fast simulation by converting the HDL design to C++,
99 and then compiling a binary which the user runs.
101 To compile the verilator simulation, first set verilator as the target for the
104 (microwatt):$ export FPGA_TARGET=verilator
106 Before compiling, you can change the `THREADS` variable in the makefile, which
107 will allow the compiled verilator simulation binary to use more than 1 thread
108 (*make sure to check how many CPU threads you have before changing this!*)
110 To compile the verilator simulation binary, call make with the
111 `microwatt-verilator` rule.
113 (microwatt):$ make microwatt-verilator
115 ## Compiling hello world code
117 We need some code to actually run on the core, so start with the 'hello world'.
118 Instructions assume you're still in the microwatt directory.
120 (microwatt):$ cd hello_world
123 A `hello_world.bin` should be generated (the final binary to be loaded), as
125 [.elf file](https://en.wikipedia.org/wiki/Executable_and_Linkable_Format), and
126 .hex (representing the binary data as hex text strings).
128 To view the symbol table (useful to see where various sections of the binary
131 (microwatt):$ powerpc64le-linux-gnu-objdump -h hello_world.elf
132 (microwatt):$ powerpc64le-linux-gnu-objdump -x hello_world.elf
134 `-h` shows just the section headers, `-x` shows all headers.
136 And to view the disassembly (great for learning about the PowerISA instructions,
137 and for associating the binary hex with actual instructions):
139 (microwatt):$ powerpc64le-linux-gnu-objdump -d hello_world.elf
141 For more information about `objdump` (common utility, not just for PowerISA),
142 see the manual pages.
144 (microwatt):$ man powerpc64le-linux-gnu-objdump
146 The binary is ready to go, now it can be loaded into the simulation.
150 ### Command line args
152 To find out the `microwatt-verilator` arguments, you can check with `-h` arg:
154 (microwatt):$ ./microwatt-verilator -h
156 Some of the arguments are explained in further sections.
160 Run the `microwatt-verilator` binary, with `hello_world/hello_world.bin` as an
163 (microwatt):$ time ./microwatt-verilator hello_world/hello_world.bin
165 `time` is a utility you can use to measure how long it takes to run the sim.
167 A pretty ASCII art of a lightbulb should be printed, and then the user can type
168 any characters, which will be echoed back. To end the simulation press Ctrl+C.
170 If no characters are appearing after about 20 seconds, stop the simulation,
171 as there might be other issues.
173 Single-threaded verilator sim binary, on a 2nd gen intel i5 (sandybridge)
174 takes 53 seconds to print the ASCII lightbulb.
176 On another dev's machine, ASUS KGPE D16, this takes just over a minute.
178 (*You'll find that uart printout is one of the longer parts of the simulation
181 ## Analysing results after simulation
183 The following files will be generated during the sim:
185 - `bram.dump` - Shows the PC address and instruction being executed. If the sim
186 hangs without any printing, view this file, as the processor may have hit an
187 exception etc. Grows in size as the sim runs.
189 - `bram.snapshot.[NUMBER]`, `verilator.save.[NUMBER]` - Snapshot files of the
190 contents of bram and verilator model respectively. Can be used to resume the
191 simulation. The number on the end corresponds to the tick time (i.e.
192 `bram.snapshot.1999990`/`verilator.save.1999990`). First the verilator model is
193 loaded, and then the bram contents are loaded. See lines `#65-108` and
195 [microwatt-verilator.cpp file](https://git.libre-soc.org/?p=microwatt.git;a=blob;f=verilator/microwatt-verilator.cpp;h=a226393f6ba74d5e3e1ffdb729d731d2311d53ad;hb=refs/heads/verilator_trace).
196 Pass the tick number on the end of the filename with the '-s' flag:
198 (microwatt):$ ./microwatt-verilator hello_world/hello_world.bin -s 1999990
200 You'll get a message like this:
202 loading hello_world/hello_world.bin at 0x0 size 0x1888
203 loading bram.snapshot.1999990 at 0x0 size 0x10000000
206 These snapshots are generated at intervals of every 2,000,000 ticks.
208 - `microwatt-verilator.vcd` - GTKWave waveform file, allowing you to look at
209 processor signals and transitions during simulation.
210 Pass `-d` flag to `microwatt-verilator` binary:
212 (microwatt):$ ./microwatt-verilator hello_world/hello_world.bin -d
214 **NOTE**: Trace dumping will generate a large VCD file (about 6GB for the hello
217 If you want GTKWave to load it faster, convert to fst first:
219 (microwatt):$ vcd2fst --vcdname=microwatt-verilator.vcd --fstname=microwatt-verilator.fst
220 (microwatt):$ gtkwave microwatt-verilator.fst
222 Fst files are orders-of-magnitude smaller (about 20MB vs 6GB), but are specific
227 The Microwatt repo comes with a pre-compiled
228 [micropython binary](https://git.libre-soc.org/?p=microwatt.git;a=tree;f=micropython;h=18fa078c8145bdaa75667a0ab04eb0b261245665;hb=refs/heads/verilator_trace)
229 (version 1.12), which you can try out after confirming 'hello world' works.
230 Bear in mind, not all features of python will be available. Such as
231 floating-point numbers.
233 For micropython to work, you'll need to increase the RAM size in the makefile.
234 Go to the microwatt-verilator makefile, and comment out the following lines:
237 RAM_INIT_FILE=hello_world/hello_world.hex
239 And uncomment the following:
242 RAM_INIT_FILE=micropython/firmware.hex
244 This will increase the RAM size from 8KiB to 384KiB. The `RAM_INIT_FILE` in
245 these examples isn't doing anything, however good practice to follow.
247 Clean up generated files, and recompile:
249 (microwatt):$ make clean
250 (microwatt):$ make microwatt-verilator
252 Once the binary has been built, run the same way as before, but point to the
253 micropython firmware binary:
255 (microwatt):$ microwatt-verilator micropython/firmware.bin
257 On the same system as above, with 1 thread, it took 49 seconds to get to the
260 ## Verilator runtime commands
263 # Show the version of verilator being used
264 (microwatt):$ ./microwatt-verilator +verilator+version
266 ## Building `microwatt-verilator` using the Libre-SOC core
269 make microwatt_external_core
270 cp external_core_top.v /path/to/microwatt
271 cd /path/to/microwatt
272 export FPGA_TARGET=verilator
273 export GHDLSYNTH=ghdl
274 make microwatt-verilator
276 ## Running Linux kernel - TODO: Need to check
278 To run Linux on Microwatt, you'll need two binaries:
280 - The `sdram_init.bin`, which is easy to compile (no additional software
283 - The `dtbImage.microwatt` device tree Linux kernel. This can be compiled (see
284 below), or a copy can be downloaded from: <https://ftp.libre-soc.org/dtbImage.microwatt>.
286 ### Building the kernel - TODO:
287 On a POWER9 there is no need to install gcc-powerpc64le-linux-gnu,
288 you can omit CROSS_COMPILE and ARCH in this case
290 apt install gcc-powerpc64le-linux-gnu
291 apt install flex bison lz4
292 git clone -b microwatt-5.7 https://git.kernel.org/pub/scm/linux/kernel/git/joel/microwatt.git
294 wget https://ftp.libre-soc.org/microwatt-linux-5.7.patch
295 patch -p1 < microwatt-linux-5.7.patch
296 wget https://ftp.libre-soc.org/rootfs.cpio
297 CROSS_COMPILE="ccache powerpc64le-linux-gnu-" ARCH=powerpc make -j8 O=microwatt microwatt_defconfig
298 CROSS_COMPILE="ccache powerpc64le-linux-gnu-" ARCH=powerpc make -j8 O=microwatt
300 This will produce a file
301 microwatt/arch/powerpc/boot/dtbImage.microwatt
303 ### Building `sdram_init.bin`
304 This needs gcc-powerpc64le-linux-gnu (already included in the setup step) if
305 cross compilation is used. It is assumed you're already in `~/src/microwatt/`
308 (microwatt):$ cd litedram/gen-src/sdram_init/
311 The resulting binary will be in the `obj/` directory.
313 ### Running the simulation
315 Make sure to return back to `src/microwatt/`.
317 (microwatt):$ cd ~/src/microwatt/
318 (microwatt):$ cp microwatt/arch/powerpc/boot/dtbImage.microwatt
319 (microwatt):$ ./microwatt-verilator sdram_init.bin dtbImage.microwatt
321 This will take some time...
323 ### Sysconn information
325 TODO WIP integrate from <https://libre-soc.org/irclog/%23libre-soc.2022-01-26.log.html>
327 Sysconn is a module which includes information about the SoC, and the info is
328 printed at the start of the simulation.
332 `microwatt-verilator` was compiled with 3 threads for faster simulation.
334 - Time to finish printing Sysconn info: about 1min
335 - Time to allocate bytes to kernel: ?
336 - Time to login prompt: about 1 hour
337 - Time to user shell: ?
341 * https://github.com/shenki/buildroot/commits/microwatt
342 * https://codeconstruct.com.au/docs/microwatt-orangecrab/
344 ## FPGA Development - TODO: Need checking
345 ### Building the bitstring for OrangeCrab
348 export FPGA_TARGET=ORANGE-CRAB
349 export GHDLSYNTH=ghdl
352 ### flashing the bitstring to the OrangeCrab
354 make prog # this will run OpenOCD
358 notes for how to compile for ulx3s
360 git clone https://github.com/kost/fujprog
361 (follow build procedure shown in fujprog README)
362 git clone https://git.libre-soc.org/git/microwatt.git
363 git checkout -b verilator_trace
364 export FPGA_TARGET=ulx3s
366 fujprog microwatt.svf
369 ### Notes for nextpnr-xilinx
371 superceded: see page [[nextpnr-xilinx]] and devscript
372 <https://git.libre-soc.org/?p=dev-env-setup.git;a=blob;f=nextpnr-xilinx-install;hb=HEAD>
374 for compiling nextpnr-xilinx and making it useable for nmigen
375 to compile for the digilent arty-a7-100t, requires a little
376 futzing around, using the symbiflow version of prjxray-db
377 instead of the one recommended as a submodule
379 git clone https://github.com/gatecat/nextpnr-xilinx
381 git checkout cd8b15db6ff5c1a7f10a9e
385 mv prjxray-db prjxray-db-no
386 git clone https://github.com/SymbiFlow/prjxray-db
388 git checkout 0a0addedd73e7
389 cp ./artix7/xc7a100t/*.json \
390 ./artix7/xc7a100tcsg324-1
392 cmake -DARCH=xilinx .
395 python3 xilinx/python/bbaexport.py --device xc7a100tcsg324-1 --bba xilinx/xc7a100t.bba
396 ./bbasm --l xilinx/xc7a100t.bba xilinx/xc7a100t.bin
397 mkdir -p /usr/share/nextpnr/xilinx-chipdb
398 cp xilinx/*.bin /usr/share/nextpnr/xilinx-chipdb
399 cp -aux xilinx/external/prjxray-db /usr/share/nextpnr
401 # Additional Useful Info for UART <-> USB Serial Interface Through OrangeCrab's Built-in USB Interface
403 This uses OrangeCrab's built-in USB interface, rather than needing a
404 separate USB-serial adapter. see the following for further details:
406 * <https://github.com/antonblanchard/microwatt/pull/347#issuecomment-1058800570>
407 * <https://github.com/antonblanchard/microwatt/pull/347#issuecomment-1058834790>
409 # running orangecrab-examples before flashing microwatt
411 See <https://github.com/orangecrab-fpga/orangecrab-hardware/blob/main/contrib/10-orangecrab.rules>
413 If the OrangeCrab is running in DFU mode, lsusb will show:
415 1209:5af0 Generic OrangeCrab r0.2 DFU Bootloader v3.1-6-g62e92e2
417 OrangeCrab has two DFU devices:
419 Found DFU: [1209:5af0] ver=0101, devnum=22, cfg=1, intf=0, path="1-4.2", alt=1, name="0x00100000 RISC-V Firmware", serial="UNKNOWN"
420 Found DFU: [1209:5af0] ver=0101, devnum=22, cfg=1, intf=0, path="1-4.2", alt=0, name="0x00080000 Bitstream", serial="UNKNOWN"
422 Then clone and patch orangecrab-examples:
424 git clone https://github.com/orangecrab-fpga/orangecrab-examples
425 patch -p1 < orangecrab-examples.diff
427 To build and flash the example:
429 pushd orangecrab-examples/nmigen
432 sudo dfu-util -D orangecrab-examples/nmigen/build/top.bit -a 0