1 # Tutorial for setting up Microwatt chroot and running simulations
3 Useful Links (External):
5 * <https://codeconstruct.com.au/docs/microwatt-orangecrab/>
6 * <https://shenki.github.io/boot-linux-on-microwatt/>
7 * <https://github.com/gregdavill/OrangeCrab-test-sw>
8 * [Verilator docs, commands](https://verilator.org/guide/latest/exe_verilator.html)
9 * [Verilator runtime command documentation](https://verilator.org/guide/latest/exe_sim.html)
10 * Tutorials for how to work with verilator:
11 [part1](https://www.itsembedded.com/dhd/verilator_1/),
12 [part2](https://www.itsembedded.com/dhd/verilator_2/)
14 Useful links (Libre-SOC):
16 * Libre-SOC page covering our workflow: [[HDL_workflow]]
17 * Devscripts Libre-SOC page: [[devscripts]]
18 * Original Microwatt Libre-SOC page: [[microwatt]]
19 * [Libre-SOC Microwatt repo branch](https://git.libre-soc.org/?p=microwatt.git;a=tree;hb=refs/heads/verilator_trace)
20 * [Libre-SOC devscripts repo](https://git.libre-soc.org/?p=dev-env-setup.git;a=tree)
22 Other Tutorials (Libre-SOC):
24 * First steps for working with PowerISA instructions Libre-SOC page:
29 [43min tutorial](https://youtu.be/02LCl3ang8g) was made and uploaded to
30 Youtube, covering some of the material you'll find on this page.
32 ## Development environment scripts
34 If you haven't already, clone Libre-SOC's development environment setup scripts.
35 These are bash scripts, and greatly simplify the time it takes to create a:
38 - With all software and libraries at specific versions
39 (which are known to work).
41 These attributes are absolutely critical, and no support will be
42 provided, unless you use these scripts to setup a development environment. This
43 helps us fix any bugs in the scripts, and make sure everyone runs on the same
46 $ git clone https://git.libre-soc.org/git/dev-env-setup.git
49 [code](https://git.libre-soc.org/?p=dev-env-setup.git;a=tree) before running
50 any of those scripts. They may be confusing, however after reading a few you'll
51 start to become more familiar with them.
53 It is expected for you to use Debian, we mostly use 11 (Bullseye) for the host
54 OS, while all the chroots run Debian 10 (Buster).
58 Scripts we will be using for the setup are:
60 * `mk-deb-chroot`, `cp-scripts-to-chroot` for chroot setup
61 * `install-hdl-apt-reqs`, `verilator-install`, `hdl-tools-yosys` for working
64 (*Current limitation for `mk-deb-chroot`, is that you must be the first user on
65 the host machine, having user ID 1000.*)
67 Commands to run in terminal to setup a new chroot environment for microwatt
72 # ./mk-deb-chroot microwatt
73 # ./cp-scripts-to-chroot microwatt
75 $ schroot -c microwatt
76 (microwatt):$ cd dev-env-setup
77 (microwatt):$ sudo bash
78 (microwatt):# ./install-hdl-apt-reqs
79 (microwatt):# ./verilator-install
80 (microwatt):# ./hdl-tools-yosys
82 (microwatt):$ cd ~/src/
83 (microwatt):$ git clone https://git.libre-soc.org/git/microwatt.git
84 (microwatt):$ cd microwatt
85 (microwatt):$ git checkout verilator_trace
87 Make sure verilator binaries in $PATH:
89 (microwatt):$ export PATH=/usr/local/verilator/bin:$PATH
90 (microwatt):$ export GHDLSYNTH=ghdl
92 (GHDLSYNTH needs to be redefined because the Makefile has default `ghdl.so`,
93 but somewhere else '.so' gets appended. You may see the following error if you
95 `ERROR: Can't load module
96 ./ghdl.so':/usr/local/bin/../share/yosys/plugins/**ghdl.so.so**`)
97 [IRC](https://libre-soc.org/irclog/%23libre-soc.2023-01-25.log.html#t2023-01-25T11:10:47)
99 ## Compiling the verilator sim for Microwatt
101 * [Libre-SOC Microwatt repo branch, Makefile](https://git.libre-soc.org/?p=microwatt.git;a=blob;f=Makefile;hb=refs/heads/verilator_trace)
103 Verilator creates a fairly fast simulation by converting the HDL design to C++,
104 and then compiling a binary which the user runs.
106 To compile the verilator simulation, first set verilator as the target for the
109 (microwatt):$ export FPGA_TARGET=verilator
111 Before compiling, you can change the `THREADS` variable in the makefile, which
112 will allow the compiled verilator simulation binary to use more than 1 thread
113 (*make sure to check how many CPU threads you have before changing this!*)
115 To compile the verilator simulation binary, call make with the
116 `microwatt-verilator` rule.
118 (microwatt):$ make microwatt-verilator
120 ## Compiling hello world code
122 We need some code to actually run on the core, so start with the 'hello world'.
123 Instructions assume you're still in the microwatt directory.
125 (microwatt):$ cd hello_world
128 A `hello_world.bin` should be generated (the final binary to be loaded), as
130 [.elf file](https://en.wikipedia.org/wiki/Executable_and_Linkable_Format), and
131 .hex (representing the binary data as hex text strings).
133 To view the symbol table (useful to see where various sections of the binary
136 (microwatt):$ powerpc64le-linux-gnu-objdump -h hello_world.elf
137 (microwatt):$ powerpc64le-linux-gnu-objdump -x hello_world.elf
139 `-h` shows just the section headers, `-x` shows all headers.
141 And to view the disassembly (great for learning about the PowerISA instructions,
142 and for associating the binary hex with actual instructions):
144 (microwatt):$ powerpc64le-linux-gnu-objdump -d hello_world.elf
146 For more information about `objdump` (common utility, not just for PowerISA),
147 see the manual pages.
149 (microwatt):$ man powerpc64le-linux-gnu-objdump
151 The binary is ready to go, now it can be loaded into the simulation.
155 ### Command line args
157 To find out the `microwatt-verilator` arguments, you can check with `-h` arg:
159 (microwatt):$ ./microwatt-verilator -h
161 Some of the arguments are explained in further sections.
165 Run the `microwatt-verilator` binary, with `hello_world/hello_world.bin` as an
168 (microwatt):$ time ./microwatt-verilator hello_world/hello_world.bin
170 `time` is a utility you can use to measure how long it takes to run the sim.
172 A pretty ASCII art of a lightbulb should be printed, and then the user can type
173 any characters, which will be echoed back. To end the simulation press Ctrl+C.
175 If no characters are appearing after about 20 seconds, stop the simulation,
176 as there might be other issues.
178 Single-threaded verilator sim binary, on a 2nd gen intel i5 (sandybridge)
179 takes 53 seconds to print the ASCII lightbulb.
181 On another dev's machine, ASUS KGPE D16, this takes just over a minute.
183 (*You'll find that uart printout is one of the longer parts of the simulation
186 ## Analysing results after simulation
188 The following files will be generated during the sim:
190 - `bram.dump` - Shows the PC address and instruction being executed. If the sim
191 hangs without any printing, view this file, as the processor may have hit an
192 exception etc. Grows in size as the sim runs.
194 - `bram.snapshot.[NUMBER]`, `verilator.save.[NUMBER]` - Snapshot files of the
195 contents of bram and verilator model respectively. Can be used to resume the
196 simulation. The number on the end corresponds to the tick time (i.e.
197 `bram.snapshot.1999990`/`verilator.save.1999990`). First the verilator model is
198 loaded, and then the bram contents are loaded. See lines `#65-108` and
200 [microwatt-verilator.cpp file](https://git.libre-soc.org/?p=microwatt.git;a=blob;f=verilator/microwatt-verilator.cpp;h=a226393f6ba74d5e3e1ffdb729d731d2311d53ad;hb=refs/heads/verilator_trace).
201 Pass the tick number on the end of the filename with the '-s' flag:
203 (microwatt):$ ./microwatt-verilator hello_world/hello_world.bin -s 1999990
205 You'll get a message like this:
207 loading hello_world/hello_world.bin at 0x0 size 0x1888
208 loading bram.snapshot.1999990 at 0x0 size 0x10000000
211 These snapshots are generated at intervals of every 2,000,000 ticks.
213 - `microwatt-verilator.vcd` - GTKWave waveform file, allowing you to look at
214 processor signals and transitions during simulation.
215 Pass `-d` flag to `microwatt-verilator` binary:
217 (microwatt):$ ./microwatt-verilator hello_world/hello_world.bin -d
219 **NOTE**: Trace dumping will generate a large VCD file (about 6GB for the hello
222 If you want GTKWave to load it faster, convert to fst first:
224 (microwatt):$ vcd2fst --vcdname=microwatt-verilator.vcd --fstname=microwatt-verilator.fst
225 (microwatt):$ gtkwave microwatt-verilator.fst
227 Fst files are orders-of-magnitude smaller (about 20MB vs 6GB), but are specific
232 The Microwatt repo comes with a pre-compiled
233 [micropython binary](https://git.libre-soc.org/?p=microwatt.git;a=tree;f=micropython;h=18fa078c8145bdaa75667a0ab04eb0b261245665;hb=refs/heads/verilator_trace)
234 (version 1.12), which you can try out after confirming 'hello world' works.
235 Bear in mind, not all features of python will be available. Such as
236 floating-point numbers.
238 For micropython to work, you'll need to increase the RAM size in the makefile.
239 Go to the microwatt-verilator makefile, and comment out the following lines:
242 RAM_INIT_FILE=hello_world/hello_world.hex
244 And uncomment the following:
247 RAM_INIT_FILE=micropython/firmware.hex
249 This will increase the RAM size from 8KiB to 384KiB. The `RAM_INIT_FILE` in
250 these examples isn't doing anything, however good practice to follow.
252 Clean up generated files, and recompile:
254 (microwatt):$ make clean
255 (microwatt):$ make microwatt-verilator
257 Once the binary has been built, run the same way as before, but point to the
258 micropython firmware binary:
260 (microwatt):$ microwatt-verilator micropython/firmware.bin
262 On the same system as above, with 1 thread, it took 49 seconds to get to the
265 ## Verilator runtime commands
268 # Show the version of verilator being used
269 (microwatt):$ ./microwatt-verilator +verilator+version
271 ## Building `microwatt-verilator` using the Libre-SOC core
273 In the Makefile, you need to set `EXTERNAL_CORE` to true, and copy the
274 generated core from soc repo to microwatt. *(If you use a separate chroot to
275 generate Libre-SOC cores, then you'll need to copy from that chroot to
276 microwatt chroot from host.*
279 make microwatt_external_core
280 cp external_core_top.v /path/to/microwatt
282 Then compile verilator sim binary as before:
285 export FPGA_TARGET=verilator
286 export GHDLSYNTH=ghdl
287 make microwatt-verilator
289 ## Running Linux kernel - TODO: Need to check
291 To run Linux on Microwatt, you'll need two binaries:
293 - The `sdram_init.bin`, which is easy to compile (no additional software
296 - The `dtbImage.microwatt` device tree Linux kernel. This can be compiled (see
297 below), or a copy can be downloaded from: <https://ftp.libre-soc.org/dtbImage.microwatt>.
299 ### Building the kernel - TODO:
300 On a POWER9 there is no need to install gcc-powerpc64le-linux-gnu,
301 you can omit CROSS_COMPILE and ARCH in this case
303 apt install gcc-powerpc64le-linux-gnu
304 apt install flex bison lz4
305 git clone -b microwatt-5.7 https://git.kernel.org/pub/scm/linux/kernel/git/joel/microwatt.git
307 wget https://ftp.libre-soc.org/microwatt-linux-5.7.patch
308 patch -p1 < microwatt-linux-5.7.patch
309 wget https://ftp.libre-soc.org/rootfs.cpio
310 CROSS_COMPILE="ccache powerpc64le-linux-gnu-" ARCH=powerpc make -j8 O=microwatt microwatt_defconfig
311 CROSS_COMPILE="ccache powerpc64le-linux-gnu-" ARCH=powerpc make -j8 O=microwatt
313 This will produce a file
314 microwatt/arch/powerpc/boot/dtbImage.microwatt
316 ### Building `sdram_init.bin`
317 This needs gcc-powerpc64le-linux-gnu (already included in the setup step) if
318 cross compilation is used. It is assumed you're already in `~/src/microwatt/`
321 (microwatt):$ cd litedram/gen-src/sdram_init/
324 The resulting binary will be in the `obj/` directory.
326 ### Running the simulation
328 Make sure to return back to `src/microwatt/`.
330 (microwatt):$ cd ~/src/microwatt/
331 (microwatt):$ cp microwatt/arch/powerpc/boot/dtbImage.microwatt
332 (microwatt):$ ./microwatt-verilator sdram_init.bin dtbImage.microwatt
334 This will take some time...
336 ### Sysconn information
338 TODO WIP integrate from <https://libre-soc.org/irclog/%23libre-soc.2022-01-26.log.html>
340 Sysconn is a module which includes information about the SoC, and the info is
341 printed at the start of the simulation.
345 `microwatt-verilator` was compiled with 3 threads for faster simulation.
347 - Time to finish printing Sysconn info: about 1min
348 - Time to allocate bytes to kernel: ?
349 - Time to login prompt: about 1 hour
350 - Time to user shell: ?
354 * https://github.com/shenki/buildroot/commits/microwatt
355 * https://codeconstruct.com.au/docs/microwatt-orangecrab/
357 ## FPGA Development - TODO: Need checking
358 ### Building the bitstring for OrangeCrab
361 export FPGA_TARGET=ORANGE-CRAB
362 export GHDLSYNTH=ghdl
365 ### flashing the bitstring to the OrangeCrab
367 make prog # this will run OpenOCD
371 notes for how to compile for ulx3s
373 git clone https://github.com/kost/fujprog
374 (follow build procedure shown in fujprog README)
375 git clone https://git.libre-soc.org/git/microwatt.git
376 git checkout -b verilator_trace
377 export FPGA_TARGET=ulx3s
379 fujprog microwatt.svf
382 ### Notes for nextpnr-xilinx
384 superceded: see page [[nextpnr-xilinx]] and devscript
385 <https://git.libre-soc.org/?p=dev-env-setup.git;a=blob;f=nextpnr-xilinx-install;hb=HEAD>
387 for compiling nextpnr-xilinx and making it useable for nmigen
388 to compile for the digilent arty-a7-100t, requires a little
389 futzing around, using the symbiflow version of prjxray-db
390 instead of the one recommended as a submodule
392 git clone https://github.com/gatecat/nextpnr-xilinx
394 git checkout cd8b15db6ff5c1a7f10a9e
398 mv prjxray-db prjxray-db-no
399 git clone https://github.com/SymbiFlow/prjxray-db
401 git checkout 0a0addedd73e7
402 cp ./artix7/xc7a100t/*.json \
403 ./artix7/xc7a100tcsg324-1
405 cmake -DARCH=xilinx .
408 python3 xilinx/python/bbaexport.py --device xc7a100tcsg324-1 --bba xilinx/xc7a100t.bba
409 ./bbasm --l xilinx/xc7a100t.bba xilinx/xc7a100t.bin
410 mkdir -p /usr/share/nextpnr/xilinx-chipdb
411 cp xilinx/*.bin /usr/share/nextpnr/xilinx-chipdb
412 cp -aux xilinx/external/prjxray-db /usr/share/nextpnr
414 # Additional Useful Info for UART <-> USB Serial Interface Through OrangeCrab's Built-in USB Interface
416 This uses OrangeCrab's built-in USB interface, rather than needing a
417 separate USB-serial adapter. see the following for further details:
419 * <https://github.com/antonblanchard/microwatt/pull/347#issuecomment-1058800570>
420 * <https://github.com/antonblanchard/microwatt/pull/347#issuecomment-1058834790>
422 # running orangecrab-examples before flashing microwatt
424 See <https://github.com/orangecrab-fpga/orangecrab-hardware/blob/main/contrib/10-orangecrab.rules>
426 If the OrangeCrab is running in DFU mode, lsusb will show:
428 1209:5af0 Generic OrangeCrab r0.2 DFU Bootloader v3.1-6-g62e92e2
430 OrangeCrab has two DFU devices:
432 Found DFU: [1209:5af0] ver=0101, devnum=22, cfg=1, intf=0, path="1-4.2", alt=1, name="0x00100000 RISC-V Firmware", serial="UNKNOWN"
433 Found DFU: [1209:5af0] ver=0101, devnum=22, cfg=1, intf=0, path="1-4.2", alt=0, name="0x00080000 Bitstream", serial="UNKNOWN"
435 Then clone and patch orangecrab-examples:
437 git clone https://github.com/orangecrab-fpga/orangecrab-examples
438 patch -p1 < orangecrab-examples.diff
440 To build and flash the example:
442 pushd orangecrab-examples/nmigen
445 sudo dfu-util -D orangecrab-examples/nmigen/build/top.bit -a 0