1 # Tutorial for setting up Microwatt chroot and running simulations
3 TODO WIP integrate from <https://libre-soc.org/irclog/%23libre-soc.2022-01-26.log.html>
5 Useful Links (External):
7 * <https://codeconstruct.com.au/docs/microwatt-orangecrab/>
8 * <https://shenki.github.io/boot-linux-on-microwatt/>
9 * <https://github.com/gregdavill/OrangeCrab-test-sw>
10 * [Verilator docs, commands](https://verilator.org/guide/latest/exe_verilator.html)
11 * [Verilator runtime command documentation](https://verilator.org/guide/latest/exe_sim.html)
12 * Tutorials for how to work with verilator:
13 [part1](https://www.itsembedded.com/dhd/verilator_1/),
14 [part2](https://www.itsembedded.com/dhd/verilator_2/)
16 Useful links (Libre-SOC):
18 * Libre-SOC page covering our workflow: [[HDL_workflow]]
19 * Devscripts Libre-SOC page: [[devscripts]]
20 * Original Microwatt Libre-SOC page: [[microwatt]]
21 * [Libre-SOC Microwatt repo branch](https://git.libre-soc.org/?p=microwatt.git;a=tree;hb=refs/heads/verilator_trace)
22 * [Libre-SOC devscripts repo](https://git.libre-soc.org/?p=dev-env-setup.git;a=tree)
24 Other Tutorials (Libre-SOC):
26 * First steps for working with PowerISA instructions Libre-SOC page:
29 ## Development environment scripts
31 If you haven't already, clone Libre-SOC's development environment setup scripts.
32 These are bash scripts, and greatly simplify the time it takes to create a:
35 - With all software and libraries at specific versions
36 (which are known to work).
38 These attributes are absolutely critical, and no support will be
39 provided, unless you use these scripts to setup a development environment. This
40 helps us fix any bugs in the scripts, and make sure everyone runs on the same
43 $ git clone https://git.libre-soc.org/git/dev-env-setup.git
46 [code](https://git.libre-soc.org/?p=dev-env-setup.git;a=tree) before running
47 any of those scripts. They may be confusing, however after reading a few you'll
48 start to become more familiar with them.
50 It is expected for you to use Debian, we mostly use 11 (Bullseye) for the host
51 OS, while all the chroots run Debian 10 (Buster).
55 Scripts we will be using for the setup are:
57 * `mk-deb-chroot`, `cp-scripts-to-chroot` for chroot setup
58 * `install-hdl-apt-reqs`, `verilator-install`, `hdl-tools-yosys` for working
61 (*Current limitation for `mk-deb-chroot`, is that you must be the first user on
62 the host machine, having user ID 1000.*)
64 Commands to run in terminal to setup a new chroot environment for microwatt
69 # ./mk-deb-chroot microwatt
70 # ./cp-scripts-to-chroot microwatt
72 $ schroot -c microwatt
73 (microwatt):$ cd dev-env-setup
74 (microwatt):$ sudo bash
75 (microwatt):# ./install-hdl-apt-reqs
76 (microwatt):# ./verilator-install
77 (microwatt):# ./hdl-tools-yosys
79 (microwatt):$ cd ~/src/
80 (microwatt):$ git clone https://git.libre-soc.org/git/microwatt.git
81 (microwatt):$ cd microwatt
82 (microwatt):$ git checkout verilator_trace
84 Make sure verilator binaries in $PATH:
86 (microwatt):$ export PATH=/usr/local/verilator/bin:$PATH
87 (microwatt):$ export GHDLSYNTH=ghdl
89 (GHDLSYNTH needs to be redefined because the Makefile has default `ghdl.so`,
90 but somewhere else '.so' gets appended. You may see the following error if you
92 `ERROR: Can't load module
93 ./ghdl.so':/usr/local/bin/../share/yosys/plugins/**ghdl.so.so**`)
94 [IRC](https://libre-soc.org/irclog/%23libre-soc.2023-01-25.log.html#t2023-01-25T11:10:47)
96 ## Compiling the verilator sim for Microwatt
98 * [Libre-SOC Microwatt repo branch, Makefile](https://git.libre-soc.org/?p=microwatt.git;a=blob;f=Makefile;hb=refs/heads/verilator_trace)
100 Verilator creates a fairly fast simulation by converting the HDL design to C++,
101 and then compiling a binary which the user runs.
103 To compile the verilator simulation, first set verilator as the target for the
106 (microwatt):$ export FPGA_TARGET=verilator
108 Before compiling, you can change the `THREADS` variable in the makefile, which
109 will allow the compiled verilator simulation binary to use more than 1 thread
110 (*make sure to check how many CPU threads you have before changing this!*)
112 To compile the verilator simulation binary, call make with the
113 `microwatt-verilator` rule.
115 (microwatt):$ make microwatt-verilator
117 ## Compiling hello world code
119 We need some code to actually run on the core, so start with the 'hello world'.
120 Instructions assume you're still in the microwatt directory.
122 (microwatt):$ cd hello_world
125 A `hello_world.bin` should be generated (the final binary to be loaded), as
127 [.elf file](https://en.wikipedia.org/wiki/Executable_and_Linkable_Format), and
128 .hex (representing the binary data as hex text strings).
130 To view the symbol table (useful to see where various sections of the binary
133 (microwatt):$ powerpc64le-linux-gnu-objdump -h hello_world.elf
134 (microwatt):$ powerpc64le-linux-gnu-objdump -x hello_world.elf
136 `-h` shows just the section headers, `-x` shows all headers.
138 And to view the disassembly (great for learning about the PowerISA instructions,
139 and for associating the binary hex with actual instructions):
141 (microwatt):$ powerpc64le-linux-gnu-objdump -d hello_world.elf
143 For more information about `objdump` (common utility, not just for PowerISA),
144 see the manual pages.
146 (microwatt):$ man powerpc64le-linux-gnu-objdump
148 The binary is ready to go, now it can be loaded into the simulation.
152 ### Command line args
154 To find out the `microwatt-verilator` arguments, you can check with `-h` arg:
156 (microwatt):$ ./microwatt-verilator -h
158 Some of the arguments are explained in further sections.
162 Run the `microwatt-verilator` binary, with `hello_world/hello_world.bin` as an
165 (microwatt):$ time ./microwatt-verilator hello_world/hello_world.bin
167 `time` is a utility you can use to measure how long it takes to run the sim.
169 A pretty ASCII art of a lightbulb should be printed, and then the user can type
170 any characters, which will be echoed back. To end the simulation press Ctrl+C.
172 If no characters are appearing after about 20 seconds, stop the simulation,
173 as there might be other issues.
175 Single-threaded verilator sim binary, on a 2nd gen intel i5 (sandybridge)
176 takes 53 seconds to print the ASCII lightbulb.
178 On another dev's machine, ASUS KGPE D16, this takes just over a minute.
180 (*You'll find that uart printout is one of the longer parts of the simulation
183 ## Analysing results after simulation
185 The following files will be generated during the sim:
187 - `bram.dump` - Shows the PC address and instruction being executed. If the sim
188 hangs without any printing, view this file, as the processor may have hit an
189 exception etc. Grows in size as the sim runs.
191 - `bram.snapshot.[NUMBER]`, `verilator.save.[NUMBER]` - Snapshot files of the
192 contents of bram and verilator model respectively. Can be used to resume the
193 simulation. The number on the end corresponds to the tick time (i.e.
194 `bram.snapshot.1999990`/`verilator.save.1999990`). First the verilator model is
195 loaded, and then the bram contents are loaded. See lines `#65-108` and
197 [microwatt-verilator.cpp file](https://git.libre-soc.org/?p=microwatt.git;a=blob;f=verilator/microwatt-verilator.cpp;h=a226393f6ba74d5e3e1ffdb729d731d2311d53ad;hb=refs/heads/verilator_trace).
198 Pass the tick number on the end of the filename with the '-s' flag:
200 (microwatt):$ ./microwatt-verilator hello_world/hello_world.bin -s 1999990
202 You'll get a message like this:
204 loading hello_world/hello_world.bin at 0x0 size 0x1888
205 loading bram.snapshot.1999990 at 0x0 size 0x10000000
208 These snapshots are generated at intervals of every 2,000,000 ticks.
210 - `microwatt-verilator.vcd` - GTKWave waveform file, allowing you to look at
211 processor signals and transitions during simulation.
212 Pass `-d` flag to `microwatt-verilator` binary:
214 (microwatt):$ ./microwatt-verilator hello_world/hello_world.bin -d
216 **NOTE**: Trace dumping will generate a large VCD file (about 6GB for the hello
219 If you want GTKWave to load it faster, convert to fst first:
221 (microwatt):$ vcd2fst --vcdname=microwatt-verilator.vcd --fstname=microwatt-verilator.fst
222 (microwatt):$ gtkwave microwatt-verilator.fst
224 Fst files are orders-of-magnitude smaller (about 20MB vs 6GB), but are specific
229 The Microwatt repo comes with a pre-compiled
230 [micropython binary](https://git.libre-soc.org/?p=microwatt.git;a=tree;f=micropython;h=18fa078c8145bdaa75667a0ab04eb0b261245665;hb=refs/heads/verilator_trace)
231 (version 1.12), which you can try out after confirming 'hello world' works.
232 Bear in mind, not all features of python will be available. Such as
233 floating-point numbers.
235 For micropython to work, you'll need to increase the RAM size in the makefile.
236 Go to the microwatt-verilator makefile, and comment out the following lines:
239 RAM_INIT_FILE=hello_world/hello_world.hex
241 And uncomment the following:
244 RAM_INIT_FILE=micropython/firmware.hex
246 This will increase the RAM size from 8KiB to 384KiB. The `RAM_INIT_FILE` in
247 these examples isn't doing anything, however good practice to follow.
249 Clean up generated files, and recompile:
251 (microwatt):$ make clean
252 (microwatt):$ make microwatt-verilator
254 Once the binary has been built, run the same way as before, but point to the
255 micropython firmware binary:
257 (microwatt):$ microwatt-verilator micropython/firmware.bin
259 On the same system as above, with 1 thread, it took 49 seconds to get to the
262 ## Verilator runtime commands
265 # Show the version of verilator being used
266 (microwatt):$ ./microwatt-verilator +verilator+version
268 ## Building `microwatt-verilator` using the Libre-SOC core
271 make microwatt_external_core
272 cp external_core_top.v /path/to/microwatt
273 cd /path/to/microwatt
274 export FPGA_TARGET=verilator
275 export GHDLSYNTH=ghdl
276 make microwatt-verilator
278 ## Running Linux kernel - TODO: Need to check
280 To run Linux on Microwatt, you'll need two binaries:
282 - The `sdram_init.bin`, which is easy to compile (no additional software
285 - The `dtbImage.microwatt` device tree Linux kernel. This can be compiled (see
286 below), or a copy can be downloaded from: <https://ftp.libre-soc.org/dtbImage.microwatt>.
288 ### Building the kernel - TODO:
289 On a POWER9 there is no need to install gcc-powerpc64le-linux-gnu,
290 you can omit CROSS_COMPILE and ARCH in this case
292 apt install gcc-powerpc64le-linux-gnu
293 apt install flex bison lz4
294 git clone -b microwatt-5.7 https://git.kernel.org/pub/scm/linux/kernel/git/joel/microwatt.git
296 wget https://ftp.libre-soc.org/microwatt-linux-5.7.patch
297 patch -p1 < microwatt-linux-5.7.patch
298 wget https://ftp.libre-soc.org/rootfs.cpio
299 CROSS_COMPILE="ccache powerpc64le-linux-gnu-" ARCH=powerpc make -j8 O=microwatt microwatt_defconfig
300 CROSS_COMPILE="ccache powerpc64le-linux-gnu-" ARCH=powerpc make -j8 O=microwatt
302 This will produce a file
303 microwatt/arch/powerpc/boot/dtbImage.microwatt
305 ### Building sdram_init.bin
306 This needs gcc-powerpc64le-linux-gnu (already included in the setup step) if
307 cross compilation is used. It is assumed you're already in `~/src/microwatt/`
310 (microwatt):$ cd litedram/gen-src/sdram_init/
313 The resulting binary will be in the `obj/` directory.
315 ### Running the simulation
317 Make sure to return back to `src/microwatt/`.
319 (microwatt):$ cd ~/src/microwatt/
320 (microwatt):$ cp microwatt/arch/powerpc/boot/dtbImage.microwatt
321 (microwatt):$ ./microwatt-verilator sdram_init.bin dtbImage.microwatt
323 This will take some time...
327 * https://github.com/shenki/buildroot/commits/microwatt
328 * https://codeconstruct.com.au/docs/microwatt-orangecrab/
330 ## FPGA Development - TODO: Need checking
331 ### Building the bitstring for OrangeCrab
334 export FPGA_TARGET=ORANGE-CRAB
335 export GHDLSYNTH=ghdl
338 ### flashing the bitstring to the OrangeCrab
340 make prog # this will run OpenOCD
344 notes for how to compile for ulx3s
346 git clone https://github.com/kost/fujprog
347 (follow build procedure shown in fujprog README)
348 git clone https://git.libre-soc.org/git/microwatt.git
349 git checkout -b verilator_trace
350 export FPGA_TARGET=ulx3s
352 fujprog microwatt.svf
355 ### Notes for nextpnr-xilinx
357 superceded: see page [[nextpnr-xilinx]] and devscript
358 <https://git.libre-soc.org/?p=dev-env-setup.git;a=blob;f=nextpnr-xilinx-install;hb=HEAD>
360 for compiling nextpnr-xilinx and making it useable for nmigen
361 to compile for the digilent arty-a7-100t, requires a little
362 futzing around, using the symbiflow version of prjxray-db
363 instead of the one recommended as a submodule
365 git clone https://github.com/gatecat/nextpnr-xilinx
367 git checkout cd8b15db6ff5c1a7f10a9e
371 mv prjxray-db prjxray-db-no
372 git clone https://github.com/SymbiFlow/prjxray-db
374 git checkout 0a0addedd73e7
375 cp ./artix7/xc7a100t/*.json \
376 ./artix7/xc7a100tcsg324-1
378 cmake -DARCH=xilinx .
381 python3 xilinx/python/bbaexport.py --device xc7a100tcsg324-1 --bba xilinx/xc7a100t.bba
382 ./bbasm --l xilinx/xc7a100t.bba xilinx/xc7a100t.bin
383 mkdir -p /usr/share/nextpnr/xilinx-chipdb
384 cp xilinx/*.bin /usr/share/nextpnr/xilinx-chipdb
385 cp -aux xilinx/external/prjxray-db /usr/share/nextpnr
387 # Additional Useful Info for UART <-> USB Serial Interface Through OrangeCrab's Built-in USB Interface
389 This uses OrangeCrab's built-in USB interface, rather than needing a
390 separate USB-serial adapter. see the following for further details:
392 * <https://github.com/antonblanchard/microwatt/pull/347#issuecomment-1058800570>
393 * <https://github.com/antonblanchard/microwatt/pull/347#issuecomment-1058834790>
395 # running orangecrab-examples before flashing microwatt
397 See <https://github.com/orangecrab-fpga/orangecrab-hardware/blob/main/contrib/10-orangecrab.rules>
399 If the OrangeCrab is running in DFU mode, lsusb will show:
401 1209:5af0 Generic OrangeCrab r0.2 DFU Bootloader v3.1-6-g62e92e2
403 OrangeCrab has two DFU devices:
405 Found DFU: [1209:5af0] ver=0101, devnum=22, cfg=1, intf=0, path="1-4.2", alt=1, name="0x00100000 RISC-V Firmware", serial="UNKNOWN"
406 Found DFU: [1209:5af0] ver=0101, devnum=22, cfg=1, intf=0, path="1-4.2", alt=0, name="0x00080000 Bitstream", serial="UNKNOWN"
408 Then clone and patch orangecrab-examples:
410 git clone https://github.com/orangecrab-fpga/orangecrab-examples
411 patch -p1 < orangecrab-examples.diff
413 To build and flash the example:
415 pushd orangecrab-examples/nmigen
418 sudo dfu-util -D orangecrab-examples/nmigen/build/top.bit -a 0