1 # Tutorial for setting up Microwatt chroot and running simulations
3 Useful Links (External):
5 * <https://codeconstruct.com.au/docs/microwatt-orangecrab/>
6 * <https://shenki.github.io/boot-linux-on-microwatt/>
7 * <https://github.com/gregdavill/OrangeCrab-test-sw>
8 * [Verilator docs, commands](https://verilator.org/guide/latest/exe_verilator.html)
9 * [Verilator runtime command documentation](https://verilator.org/guide/latest/exe_sim.html)
10 * Tutorials for how to work with verilator:
11 [part1](https://www.itsembedded.com/dhd/verilator_1/),
12 [part2](https://www.itsembedded.com/dhd/verilator_2/)
14 Useful links (Libre-SOC):
16 * Libre-SOC page covering our workflow: [[HDL_workflow]]
17 * Devscripts Libre-SOC page: [[devscripts]]
18 * Original Microwatt Libre-SOC page: [[microwatt]]
19 * [Libre-SOC Microwatt repo branch](https://git.libre-soc.org/?p=microwatt.git;a=tree;hb=refs/heads/verilator_trace)
20 * [Libre-SOC devscripts repo](https://git.libre-soc.org/?p=dev-env-setup.git;a=tree)
22 Other Tutorials (Libre-SOC):
24 * First steps for working with PowerISA instructions Libre-SOC page:
29 [43min tutorial](https://youtu.be/02LCl3ang8g) was made and uploaded to
30 Youtube, covering some of the material you'll find on this page.
32 ## Development environment scripts
34 If you haven't already, clone Libre-SOC's development environment setup scripts.
35 These are bash scripts, and greatly simplify the time it takes to create a:
38 - With all software and libraries at specific versions
39 (which are known to work).
41 These attributes are absolutely critical, and no support will be
42 provided, unless you use these scripts to setup a development environment. This
43 helps us fix any bugs in the scripts, and make sure everyone runs on the same
46 $ git clone https://git.libre-soc.org/git/dev-env-setup.git
50 [code](https://git.libre-soc.org/?p=dev-env-setup.git;a=tree) before running
52 This is your own legal responsibility (to not run
53 arbitrary code off of the internet) and we take no responsibility or accept
54 any liability whatsoever for your use or misuse of them.
56 It is expected for you to use Debian for the host OS (anything else
57 is unsupported: many contributors have repeatedly gotten into trouble by not
58 following this advice), while all the chroots - which are developed
59 very specifically for "reproducible builds" - run Debian 10 (Buster).
63 Scripts we will be using for the setup are:
65 * `mk-deb-chroot`, `cp-scripts-to-chroot` for chroot setup
66 * `install-hdl-apt-reqs`, `verilator-install`, `hdl-tools-yosys` for working
69 (*Current limitation for `mk-deb-chroot`, is that you must be the first user on
70 the host machine, having user ID 1000.*)
72 Commands to run in terminal to setup a new chroot environment for microwatt
77 # ./mk-deb-chroot microwatt
78 # ./cp-scripts-to-chroot microwatt
80 $ schroot -c microwatt
81 (microwatt):$ cd dev-env-setup
82 (microwatt):$ sudo bash
83 (microwatt):# ./install-hdl-apt-reqs
84 (microwatt):# ./verilator-install
85 (microwatt):# ./hdl-tools-yosys
87 (microwatt):$ cd ~/src/
88 (microwatt):$ git clone https://git.libre-soc.org/git/microwatt.git
89 (microwatt):$ cd microwatt
90 (microwatt):$ git checkout verilator_trace
92 Make sure verilator binaries in $PATH:
94 (microwatt):$ export PATH=/usr/local/verilator/bin:$PATH
95 (microwatt):$ export GHDLSYNTH=ghdl
97 (GHDLSYNTH needs to be redefined because the Makefile has default `ghdl.so`,
98 but somewhere else '.so' gets appended. You may see the following error if you
100 `ERROR: Can't load module
101 ./ghdl.so':/usr/local/bin/../share/yosys/plugins/**ghdl.so.so**`)
102 [IRC](https://libre-soc.org/irclog/%23libre-soc.2023-01-25.log.html#t2023-01-25T11:10:47)
104 ## Compiling the verilator sim for Microwatt
106 * [Libre-SOC Microwatt repo branch, Makefile](https://git.libre-soc.org/?p=microwatt.git;a=blob;f=Makefile;hb=refs/heads/verilator_trace)
108 Verilator creates a fairly fast simulation by converting the HDL design to C++,
109 and then compiling a binary which the user runs.
111 To compile the verilator simulation, first set verilator as the target for the
114 (microwatt):$ export FPGA_TARGET=verilator
116 Before compiling, you can change the `THREADS` variable in the makefile, which
117 will allow the compiled verilator simulation binary to use more than 1 thread
118 (*make sure to check how many CPU threads you have before changing this!*)
120 To compile the verilator simulation binary, call make with the
121 `microwatt-verilator` rule.
123 (microwatt):$ make microwatt-verilator
125 ## Compiling hello world code
127 We need some code to actually run on the core, so start with the 'hello world'.
128 Instructions assume you're still in the microwatt directory.
130 (microwatt):$ cd hello_world
133 A `hello_world.bin` should be generated (the final binary to be loaded), as
135 [.elf file](https://en.wikipedia.org/wiki/Executable_and_Linkable_Format), and
136 .hex (representing the binary data as hex text strings).
138 To view the symbol table (useful to see where various sections of the binary
141 (microwatt):$ powerpc64le-linux-gnu-objdump -h hello_world.elf
142 (microwatt):$ powerpc64le-linux-gnu-objdump -x hello_world.elf
144 `-h` shows just the section headers, `-x` shows all headers.
146 And to view the disassembly (great for learning about the PowerISA instructions,
147 and for associating the binary hex with actual instructions), you can view the
148 automatically generated `hello_world.as` file.
150 Command to generate the disassembly:
152 (microwatt):$ powerpc64le-linux-gnu-objdump -D hello_world.elf
154 For more information about `objdump` (common utility, not just for PowerISA),
155 see the manual pages.
157 (microwatt):$ man powerpc64le-linux-gnu-objdump
159 The binary is ready to go, now it can be loaded into the simulation.
163 ### Command line args
165 To find out the `microwatt-verilator` arguments, you can check with `-h` arg:
167 (microwatt):$ ./microwatt-verilator -h
169 Some of the arguments are explained in further sections.
173 Run the `microwatt-verilator` binary, with `hello_world/hello_world.bin` as an
176 (microwatt):$ time ./microwatt-verilator hello_world/hello_world.bin
178 `time` is a utility you can use to measure how long it takes to run the sim.
180 A pretty ASCII art of a lightbulb should be printed, and then the user can type
181 any characters, which will be echoed back. To end the simulation press Ctrl+C.
183 If no characters are appearing after about 20 seconds, stop the simulation,
184 as there might be other issues.
186 Single-threaded verilator sim binary, on a 2nd gen intel i5 (sandybridge)
187 takes 53 seconds to print the ASCII lightbulb.
189 On another dev's machine, ASUS KGPE D16, this takes just over a minute.
191 (*You'll find that uart printout is one of the longer parts of the simulation
194 ## Analysing results after simulation
196 The following files will be generated during the sim:
198 - `bram.dump` - Shows the PC address and instruction being executed. If the sim
199 hangs without any printing, view this file, as the processor may have hit an
200 exception etc. Grows in size as the sim runs.
202 - `bram.snapshot.[NUMBER]`, `verilator.save.[NUMBER]` - Snapshot files of the
203 contents of bram and verilator model respectively. Can be used to resume the
204 simulation. The number on the end corresponds to the tick time (i.e.
205 `bram.snapshot.1999990`/`verilator.save.1999990`). First the verilator model is
206 loaded, and then the bram contents are loaded. See lines `#65-108` and
208 [microwatt-verilator.cpp file](https://git.libre-soc.org/?p=microwatt.git;a=blob;f=verilator/microwatt-verilator.cpp;h=a226393f6ba74d5e3e1ffdb729d731d2311d53ad;hb=refs/heads/verilator_trace).
209 Pass the tick number on the end of the filename with the '-s' flag:
211 (microwatt):$ ./microwatt-verilator hello_world/hello_world.bin -s 1999990
213 You'll get a message like this:
215 loading hello_world/hello_world.bin at 0x0 size 0x1888
216 loading bram.snapshot.1999990 at 0x0 size 0x10000000
219 These snapshots are generated at intervals of every 2,000,000 ticks.
221 - `microwatt-verilator.vcd` - GTKWave waveform file, allowing you to look at
222 processor signals and transitions during simulation.
223 Pass `-d` flag to `microwatt-verilator` binary:
225 (microwatt):$ ./microwatt-verilator hello_world/hello_world.bin -d
227 **NOTE**: Trace dumping will generate a large VCD file (about 6GB for the hello
230 If you want GTKWave to load it faster, convert to fst first:
232 (microwatt):$ vcd2fst --vcdname=microwatt-verilator.vcd --fstname=microwatt-verilator.fst
233 (microwatt):$ gtkwave microwatt-verilator.fst
235 Fst files are orders-of-magnitude smaller (about 20MB vs 6GB), but are specific
240 The Microwatt repo comes with a pre-compiled
241 [micropython binary](https://git.libre-soc.org/?p=microwatt.git;a=tree;f=micropython;h=18fa078c8145bdaa75667a0ab04eb0b261245665;hb=refs/heads/verilator_trace)
242 (version 1.12), which you can try out after confirming 'hello world' works.
243 Bear in mind, not all features of python will be available. Such as
244 floating-point numbers.
246 For micropython to work, you'll need to increase the RAM size in the makefile.
247 Go to the microwatt-verilator makefile, and comment out the following lines:
250 RAM_INIT_FILE=hello_world/hello_world.hex
252 And uncomment the following:
255 RAM_INIT_FILE=micropython/firmware.hex
257 This will increase the RAM size from 8KiB to 384KiB. The `RAM_INIT_FILE` in
258 these examples isn't doing anything, however good practice to follow.
260 Clean up generated files, and recompile:
262 (microwatt):$ make clean
263 (microwatt):$ make microwatt-verilator
265 Once the binary has been built, run the same way as before, but point to the
266 micropython firmware binary:
268 (microwatt):$ microwatt-verilator micropython/firmware.bin
270 On the same system as above, with 1 thread, it took 49 seconds to get to the
273 ## Verilator runtime commands
276 # Show the version of verilator being used
277 (microwatt):$ ./microwatt-verilator +verilator+version
279 ## Building `microwatt-verilator` using the Libre-SOC core
281 In the Makefile, you need to set `EXTERNAL_CORE` to true, and copy the
282 generated core from soc repo to microwatt. *(If you use a separate chroot to
283 generate Libre-SOC cores, then you'll need to copy from that chroot to
284 microwatt chroot from host.*
287 make microwatt_external_core
288 cp external_core_top.v /path/to/microwatt
290 Then compile verilator sim binary as before:
293 export FPGA_TARGET=verilator
294 export GHDLSYNTH=ghdl
295 make microwatt-verilator
297 ## Running Linux kernel
299 To run Linux on Microwatt, you'll need two binaries:
301 - The `sdram_init.bin`, which is easy to compile (no additional software
304 - The `dtbImage.microwatt` device tree Linux kernel. This can be compiled (see
305 below), or a copy can be downloaded from: <https://ftp.libre-soc.org/dtbImage.microwatt>.
307 Like with the micropython example, you'll need to increase the RAM size in the
308 makefile, and recompile the microwatt-verilator binary.
310 Uncomment the following:
312 MEMORY_SIZE=536870912
314 Which will change the RAM size to 512KiB.
316 As there is no `dtbImage.microwatt.hex`, you can leave `RAM_INIT_FILE` unchanged.
318 ### Building the kernel - TODO:
320 *(Please don't build the kernel yourself, until you've tested with the existing kernel linked
323 On a POWER9 there is no need to install gcc-powerpc64le-linux-gnu,
324 you can omit CROSS_COMPILE and ARCH in this case
326 apt install gcc-powerpc64le-linux-gnu
327 apt install flex bison lz4
328 git clone -b microwatt-5.7 https://git.kernel.org/pub/scm/linux/kernel/git/joel/microwatt.git
330 wget https://ftp.libre-soc.org/microwatt-linux-5.7.patch
331 patch -p1 < microwatt-linux-5.7.patch
332 wget https://ftp.libre-soc.org/rootfs.cpio
333 CROSS_COMPILE="ccache powerpc64le-linux-gnu-" ARCH=powerpc make -j8 O=microwatt microwatt_defconfig
334 CROSS_COMPILE="ccache powerpc64le-linux-gnu-" ARCH=powerpc make -j8 O=microwatt
336 This will produce a file
337 microwatt/arch/powerpc/boot/dtbImage.microwatt
339 ### Building `sdram_init.bin`
340 This needs gcc-powerpc64le-linux-gnu (already included in the setup step) if
341 cross compilation is used. It is assumed you're already in `~/src/microwatt/`
344 (microwatt):$ cd litedram/gen-src/sdram_init/
347 The resulting binary will be in the `obj/` directory.
349 ### Running the simulation
351 Make sure to return back to `src/microwatt/`.
353 (microwatt):$ cd ~/src/microwatt/
354 (microwatt):$ cp microwatt/arch/powerpc/boot/dtbImage.microwatt
355 (microwatt):$ ./microwatt-verilator sdram_init.bin dtbImage.microwatt
357 This will take some time...
359 ### Sysconn information
361 TODO WIP integrate from <https://libre-soc.org/irclog/%23libre-soc.2022-01-26.log.html>
363 Sysconn is a module which includes information about the SoC, and the info is
364 printed at the start of the simulation.
368 `microwatt-verilator` was compiled with 3 threads for faster simulation.
370 - Time to finish printing Sysconn info: about 1min
371 - Time to allocate bytes to kernel: ?
372 - Time to login prompt: about 1 hour
373 - Time to user shell: ?
377 * https://github.com/shenki/buildroot/commits/microwatt
378 * https://codeconstruct.com.au/docs/microwatt-orangecrab/
380 ## FPGA Development - TODO: Need checking
381 ### Building the bitstring for OrangeCrab
384 export FPGA_TARGET=ORANGE-CRAB
385 export GHDLSYNTH=ghdl
388 ### flashing the bitstring to the OrangeCrab
390 make prog # this will run OpenOCD
394 notes for how to compile for ulx3s
396 git clone https://github.com/kost/fujprog
397 (follow build procedure shown in fujprog README)
398 git clone https://git.libre-soc.org/git/microwatt.git
399 git checkout -b verilator_trace
400 export FPGA_TARGET=ulx3s
402 fujprog microwatt.svf
405 ### Notes for nextpnr-xilinx
407 superceded: see page [[nextpnr-xilinx]] and devscript
408 <https://git.libre-soc.org/?p=dev-env-setup.git;a=blob;f=nextpnr-xilinx-install;hb=HEAD>
410 for compiling nextpnr-xilinx and making it useable for nmigen
411 to compile for the digilent arty-a7-100t, requires a little
412 futzing around, using the symbiflow version of prjxray-db
413 instead of the one recommended as a submodule
415 git clone https://github.com/gatecat/nextpnr-xilinx
417 git checkout cd8b15db6ff5c1a7f10a9e
421 mv prjxray-db prjxray-db-no
422 git clone https://github.com/SymbiFlow/prjxray-db
424 git checkout 0a0addedd73e7
425 cp ./artix7/xc7a100t/*.json \
426 ./artix7/xc7a100tcsg324-1
428 cmake -DARCH=xilinx .
431 python3 xilinx/python/bbaexport.py --device xc7a100tcsg324-1 --bba xilinx/xc7a100t.bba
432 ./bbasm --l xilinx/xc7a100t.bba xilinx/xc7a100t.bin
433 mkdir -p /usr/share/nextpnr/xilinx-chipdb
434 cp xilinx/*.bin /usr/share/nextpnr/xilinx-chipdb
435 cp -aux xilinx/external/prjxray-db /usr/share/nextpnr
437 # Additional Useful Info for UART <-> USB Serial Interface Through OrangeCrab's Built-in USB Interface
439 This uses OrangeCrab's built-in USB interface, rather than needing a
440 separate USB-serial adapter. see the following for further details:
442 * <https://github.com/antonblanchard/microwatt/pull/347#issuecomment-1058800570>
443 * <https://github.com/antonblanchard/microwatt/pull/347#issuecomment-1058834790>
445 # running orangecrab-examples before flashing microwatt
447 See <https://github.com/orangecrab-fpga/orangecrab-hardware/blob/main/contrib/10-orangecrab.rules>
449 If the OrangeCrab is running in DFU mode, lsusb will show:
451 1209:5af0 Generic OrangeCrab r0.2 DFU Bootloader v3.1-6-g62e92e2
453 OrangeCrab has two DFU devices:
455 Found DFU: [1209:5af0] ver=0101, devnum=22, cfg=1, intf=0, path="1-4.2", alt=1, name="0x00100000 RISC-V Firmware", serial="UNKNOWN"
456 Found DFU: [1209:5af0] ver=0101, devnum=22, cfg=1, intf=0, path="1-4.2", alt=0, name="0x00080000 Bitstream", serial="UNKNOWN"
458 Then clone and patch orangecrab-examples:
460 git clone https://github.com/orangecrab-fpga/orangecrab-examples
461 patch -p1 < orangecrab-examples.diff
463 To build and flash the example:
465 pushd orangecrab-examples/nmigen
468 sudo dfu-util -D orangecrab-examples/nmigen/build/top.bit -a 0