HDL_workflow/microwatt.mdwn

   1 # Tutorial for setting up Microwatt chroot and running simulations
   2
   3 Useful Links (External):
   4
   5 * <https://codeconstruct.com.au/docs/microwatt-orangecrab/>
   6 * <https://shenki.github.io/boot-linux-on-microwatt/>
   7 * <https://github.com/gregdavill/OrangeCrab-test-sw>
   8 * [Verilator docs, commands](https://verilator.org/guide/latest/exe_verilator.html)
   9 * [Verilator runtime command documentation](https://verilator.org/guide/latest/exe_sim.html)
  10 * Tutorials for how to work with verilator:
  11 [part1](https://www.itsembedded.com/dhd/verilator_1/),
  12 [part2](https://www.itsembedded.com/dhd/verilator_2/)
  13
  14 Useful links (Libre-SOC):
  15
  16 * Libre-SOC page covering our workflow: [[HDL_workflow]]
  17 * Devscripts Libre-SOC page: [[devscripts]]
  18 * Original Microwatt Libre-SOC page: [[microwatt]]
  19 * [Libre-SOC Microwatt repo branch](https://git.libre-soc.org/?p=microwatt.git;a=tree;hb=refs/heads/verilator_trace)
  20 * [Libre-SOC devscripts repo](https://git.libre-soc.org/?p=dev-env-setup.git;a=tree)
  21
  22 Other Tutorials (Libre-SOC):
  23
  24 * First steps for working with PowerISA instructions Libre-SOC page:
  25 [[/docs/firststeps]]
  26
  27 ## Video Tutorial
  28
  29 [43min tutorial](https://youtu.be/02LCl3ang8g) was made and uploaded to
  30 Youtube, covering some of the material you'll find on this page.
  31
  32 ## Development environment scripts
  33
  34 If you haven't already, clone Libre-SOC's development environment setup scripts.
  35 These are bash scripts, and greatly simplify the time it takes to create a:
  36
  37 - Stable environment
  38 - With all software and libraries at specific versions
  39 (which are known to work).
  40
  41 These attributes are absolutely critical, and no support will be
  42 provided, unless you use these scripts to setup a development environment. This
  43 helps us fix any bugs in the scripts, and make sure everyone runs on the same
  44 page.
  45
  46     $ git clone https://git.libre-soc.org/git/dev-env-setup.git
  47
  48 Do *look through* the
  49 [code](https://git.libre-soc.org/?p=dev-env-setup.git;a=tree) before running
  50 any of those scripts. They may be confusing, however after reading a few you'll
  51 start to become more familiar with them.
  52
  53 It is expected for you to use Debian, we mostly use 11 (Bullseye) for the host
  54 OS, while all the chroots run Debian 10 (Buster).
  55
  56 ## Setting up chroot
  57
  58 Scripts we will be using for the setup are:
  59
  60 * `mk-deb-chroot`, `cp-scripts-to-chroot` for chroot setup
  61 * `install-hdl-apt-reqs`, `verilator-install`, `hdl-tools-yosys` for working
  62 with Microwatt
  63
  64 (*Current limitation for `mk-deb-chroot`, is that you must be the first user on
  65 the host machine, having user ID 1000.*)
  66
  67 Commands to run in terminal to setup a new chroot environment for microwatt
  68 simulations.
  69
  70     $ cd dev-env-setup
  71     $ sudo bash
  72     # ./mk-deb-chroot microwatt
  73     # ./cp-scripts-to-chroot microwatt
  74     # exit
  75     $ schroot -c microwatt
  76     (microwatt):$ cd dev-env-setup
  77     (microwatt):$ sudo bash
  78     (microwatt):# ./install-hdl-apt-reqs
  79     (microwatt):# ./verilator-install
  80     (microwatt):# ./hdl-tools-yosys
  81     (microwatt):# exit
  82     (microwatt):$ cd ~/src/
  83     (microwatt):$ git clone https://git.libre-soc.org/git/microwatt.git
  84     (microwatt):$ cd microwatt
  85     (microwatt):$ git checkout verilator_trace
  86
  87 Make sure verilator binaries in $PATH:
  88
  89     (microwatt):$ export PATH=/usr/local/verilator/bin:$PATH
  90     (microwatt):$ export GHDLSYNTH=ghdl
  91
  92 (GHDLSYNTH needs to be redefined because the Makefile has default `ghdl.so`,
  93 but somewhere else '.so' gets appended. You may see the following error if you
  94 don't redefine:
  95 `ERROR: Can't load module
  96 ./ghdl.so':/usr/local/bin/../share/yosys/plugins/**ghdl.so.so**`)
  97 [IRC](https://libre-soc.org/irclog/%23libre-soc.2023-01-25.log.html#t2023-01-25T11:10:47)
  98
  99 ## Compiling the verilator sim for Microwatt
 100
 101 * [Libre-SOC Microwatt repo branch, Makefile](https://git.libre-soc.org/?p=microwatt.git;a=blob;f=Makefile;hb=refs/heads/verilator_trace)
 102
 103 Verilator creates a fairly fast simulation by converting the HDL design to C++,
 104 and then compiling a binary which the user runs.
 105
 106 To compile the verilator simulation, first set verilator as the target for the
 107 makefile:
 108
 109     (microwatt):$ export FPGA_TARGET=verilator
 110
 111 Before compiling, you can change the `THREADS` variable in the makefile, which
 112 will allow the compiled verilator simulation binary to use more than 1 thread
 113 (*make sure to check how many CPU threads you have before changing this!*)
 114
 115 To compile the verilator simulation binary, call make with the
 116 `microwatt-verilator` rule.
 117
 118     (microwatt):$ make microwatt-verilator
 119
 120 ## Compiling hello world code
 121
 122 We need some code to actually run on the core, so start with the 'hello world'.
 123 Instructions assume you're still in the microwatt directory.
 124
 125     (microwatt):$ cd hello_world
 126     (microwatt):$ make
 127
 128 A `hello_world.bin` should be generated (the final binary to be loaded), as
 129 well as an
 130 [.elf file](https://en.wikipedia.org/wiki/Executable_and_Linkable_Format), and
 131 .hex (representing the binary data as hex text strings).
 132
 133 To view the symbol table (useful to see where various sections of the binary
 134 begin):
 135
 136     (microwatt):$ powerpc64le-linux-gnu-objdump -h hello_world.elf
 137     (microwatt):$ powerpc64le-linux-gnu-objdump -x hello_world.elf
 138
 139 `-h` shows just the section headers, `-x` shows all headers.
 140
 141 And to view the disassembly (great for learning about the PowerISA instructions,
 142 and for associating the binary hex with actual instructions):
 143
 144     (microwatt):$ powerpc64le-linux-gnu-objdump -d hello_world.elf
 145
 146 For more information about `objdump` (common utility, not just for PowerISA),
 147 see the manual pages.
 148
 149     (microwatt):$ man powerpc64le-linux-gnu-objdump
 150
 151 The binary is ready to go, now it can be loaded into the simulation.
 152
 153 ## Simulation
 154
 155 ### Command line args
 156
 157 To find out the `microwatt-verilator` arguments, you can check with `-h` arg:
 158
 159     (microwatt):$ ./microwatt-verilator -h
 160
 161 Some of the arguments are explained in further sections.
 162
 163 ### Running
 164
 165 Run the `microwatt-verilator` binary, with `hello_world/hello_world.bin` as an
 166 argument:
 167
 168     (microwatt):$ time ./microwatt-verilator hello_world/hello_world.bin
 169
 170 `time` is a utility you can use to measure how long it takes to run the sim.
 171
 172 A pretty ASCII art of a lightbulb should be printed, and then the user can type
 173 any characters, which will be echoed back. To end the simulation press Ctrl+C.
 174
 175 If no characters are appearing after about 20 seconds, stop the simulation,
 176 as there might be other issues.
 177
 178 Single-threaded verilator sim binary, on a 2nd gen intel i5 (sandybridge)
 179 takes 53 seconds to print the ASCII lightbulb.
 180
 181 On another dev's machine, ASUS KGPE D16, this takes just over a minute.
 182
 183 (*You'll find that uart printout is one of the longer parts of the simulation
 184 in general.*)
 185
 186 ## Analysing results after simulation
 187
 188 The following files will be generated during the sim:
 189
 190 - `bram.dump` - Shows the PC address and instruction being executed. If the sim
 191 hangs without any printing, view this file, as the processor may have hit an
 192 exception etc. Grows in size as the sim runs.
 193
 194 - `bram.snapshot.[NUMBER]`, `verilator.save.[NUMBER]` - Snapshot files of the
 195 contents of bram and verilator model respectively. Can be used to resume the
 196 simulation. The number on the end corresponds to the tick time (i.e.
 197 `bram.snapshot.1999990`/`verilator.save.1999990`). First the verilator model is
 198 loaded, and then the bram contents are loaded. See lines `#65-108` and
 199 `#189-195` of the
 200 [microwatt-verilator.cpp file](https://git.libre-soc.org/?p=microwatt.git;a=blob;f=verilator/microwatt-verilator.cpp;h=a226393f6ba74d5e3e1ffdb729d731d2311d53ad;hb=refs/heads/verilator_trace).
 201 Pass the tick number on the end of the filename with the '-s' flag:
 202
 203     (microwatt):$ ./microwatt-verilator hello_world/hello_world.bin -s 1999990
 204
 205 You'll get a message like this:
 206
 207     loading hello_world/hello_world.bin at 0x0 size 0x1888
 208     loading bram.snapshot.1999990 at 0x0 size 0x10000000
 209     restored at 1999990
 210
 211 These snapshots are generated at intervals of every 2,000,000 ticks.
 212
 213 - `microwatt-verilator.vcd` - GTKWave waveform file, allowing you to look at
 214 processor signals and transitions during simulation.
 215 Pass `-d` flag to `microwatt-verilator` binary:
 216
 217     (microwatt):$ ./microwatt-verilator hello_world/hello_world.bin -d
 218
 219 **NOTE**: Trace dumping will generate a large VCD file (about 6GB for the hello
 220 world example)!
 221
 222 If you want GTKWave to load it faster, convert to fst first:
 223
 224     (microwatt):$ vcd2fst --vcdname=microwatt-verilator.vcd --fstname=microwatt-verilator.fst
 225     (microwatt):$ gtkwave microwatt-verilator.fst
 226
 227 Fst files are orders-of-magnitude smaller (about 20MB vs 6GB), but are specific
 228 to the GTKWave tool.
 229
 230 ## Micropython
 231
 232 The Microwatt repo comes with a pre-compiled
 233 [micropython binary](https://git.libre-soc.org/?p=microwatt.git;a=tree;f=micropython;h=18fa078c8145bdaa75667a0ab04eb0b261245665;hb=refs/heads/verilator_trace)
 234 (version 1.12), which you can try out after confirming 'hello world' works.
 235 Bear in mind, not all features of python will be available. Such as
 236 floating-point numbers.
 237
 238 For micropython to work, you'll need to increase the RAM size in the makefile.
 239 Go to the microwatt-verilator makefile, and comment out the following lines:
 240
 241     MEMORY_SIZE=8192
 242     RAM_INIT_FILE=hello_world/hello_world.hex
 243
 244 And uncomment the following:
 245
 246     MEMORY_SIZE=393216
 247     RAM_INIT_FILE=micropython/firmware.hex
 248
 249 This will increase the RAM size from 8KiB to 384KiB. The `RAM_INIT_FILE` in
 250 these examples isn't doing anything, however good practice to follow.
 251
 252 Clean up generated files, and recompile:
 253
 254     (microwatt):$ make clean
 255     (microwatt):$ make microwatt-verilator
 256
 257 Once the binary has been built, run the same way as before, but point to the
 258 micropython firmware binary:
 259
 260     (microwatt):$ microwatt-verilator micropython/firmware.bin
 261
 262 On the same system as above, with 1 thread, it took 49 seconds to get to the
 263 micropython shell.
 264
 265 ## Verilator runtime commands
 266 A few examples:
 267
 268     # Show the version of verilator being used
 269     (microwatt):$ ./microwatt-verilator +verilator+version
 270
 271 ## Building `microwatt-verilator` using the Libre-SOC core
 272
 273         cd /path/to/soc
 274         make microwatt_external_core
 275         cp external_core_top.v /path/to/microwatt
 276         cd /path/to/microwatt
 277         export FPGA_TARGET=verilator
 278         export GHDLSYNTH=ghdl
 279         make microwatt-verilator
 280
 281 ## Running Linux kernel - TODO: Need to check
 282
 283 To run Linux on Microwatt, you'll need two binaries:
 284
 285 - The `sdram_init.bin`, which is easy to compile (no additional software
 286 required).
 287
 288 - The `dtbImage.microwatt` device tree Linux kernel. This can be compiled (see
 289 below), or a copy can be downloaded from: <https://ftp.libre-soc.org/dtbImage.microwatt>.
 290
 291 ### Building the kernel - TODO:
 292 On a POWER9 there is no need to install gcc-powerpc64le-linux-gnu,
 293 you can omit CROSS_COMPILE and ARCH in this case
 294
 295         apt install gcc-powerpc64le-linux-gnu
 296         apt install flex bison lz4
 297         git clone -b microwatt-5.7 https://git.kernel.org/pub/scm/linux/kernel/git/joel/microwatt.git
 298         cd microwatt
 299         wget https://ftp.libre-soc.org/microwatt-linux-5.7.patch
 300         patch -p1 < microwatt-linux-5.7.patch
 301         wget https://ftp.libre-soc.org/rootfs.cpio
 302         CROSS_COMPILE="ccache powerpc64le-linux-gnu-" ARCH=powerpc make -j8 O=microwatt microwatt_defconfig
 303         CROSS_COMPILE="ccache powerpc64le-linux-gnu-" ARCH=powerpc make -j8 O=microwatt
 304
 305 This will produce a file
 306         microwatt/arch/powerpc/boot/dtbImage.microwatt
 307
 308 ### Building `sdram_init.bin`
 309 This needs gcc-powerpc64le-linux-gnu (already included in the setup step) if
 310 cross compilation is used. It is assumed you're already in `~/src/microwatt/`
 311 directory.
 312
 313     (microwatt):$ cd litedram/gen-src/sdram_init/
 314         (microwatt):$ make
 315
 316 The resulting binary will be in the `obj/` directory.
 317
 318 ### Running the simulation
 319
 320 Make sure to return back to `src/microwatt/`.
 321
 322     (microwatt):$ cd ~/src/microwatt/
 323         (microwatt):$ cp microwatt/arch/powerpc/boot/dtbImage.microwatt
 324         (microwatt):$ ./microwatt-verilator sdram_init.bin dtbImage.microwatt
 325
 326 This will take some time...
 327
 328 ### Sysconn information
 329
 330 TODO WIP integrate from <https://libre-soc.org/irclog/%23libre-soc.2022-01-26.log.html>
 331
 332 Sysconn is a module which includes information about the SoC, and the info is
 333 printed at the start of the simulation.
 334
 335 ### Time benchmarks
 336
 337 `microwatt-verilator` was compiled with 3 threads for faster simulation.
 338
 339 - Time to finish printing Sysconn info: about 1min
 340 - Time to allocate bytes to kernel: ?
 341 - Time to login prompt: about 1 hour
 342 - Time to user shell: ?
 343
 344 ### TODO: buildroot
 345
 346 * https://github.com/shenki/buildroot/commits/microwatt
 347 * https://codeconstruct.com.au/docs/microwatt-orangecrab/
 348
 349 ## FPGA Development - TODO: Need checking
 350 ### Building the bitstring for OrangeCrab
 351
 352         cd microwatt
 353         export FPGA_TARGET=ORANGE-CRAB
 354         export GHDLSYNTH=ghdl
 355         make microwatt.bit
 356
 357 ### flashing the bitstring to the OrangeCrab
 358
 359         make prog # this will run OpenOCD
 360
 361 ### Notes for ulx3s
 362
 363 notes for how to compile for ulx3s
 364
 365     git clone https://github.com/kost/fujprog
 366     (follow build procedure shown in fujprog README)
 367     git clone https://git.libre-soc.org/git/microwatt.git
 368     git checkout -b verilator_trace
 369     export FPGA_TARGET=ulx3s
 370     make microwatt.svf
 371     fujprog microwatt.svf
 372
 373
 374 ### Notes for nextpnr-xilinx
 375
 376 superceded: see page [[nextpnr-xilinx]] and devscript
 377 <https://git.libre-soc.org/?p=dev-env-setup.git;a=blob;f=nextpnr-xilinx-install;hb=HEAD>
 378
 379 for compiling nextpnr-xilinx and making it useable for nmigen
 380 to compile for the digilent arty-a7-100t, requires a little
 381 futzing around, using the symbiflow version of prjxray-db
 382 instead of the one recommended as a submodule
 383
 384     git clone https://github.com/gatecat/nextpnr-xilinx
 385     cd nextpnr-xilinx
 386     git checkout cd8b15db6ff5c1a7f10a9e
 387     git submodule init
 388     git submodule update
 389     cd xilinx/external
 390     mv prjxray-db prjxray-db-no
 391     git clone https://github.com/SymbiFlow/prjxray-db
 392     cd prjxray-db
 393     git checkout 0a0addedd73e7
 394     cp ./artix7/xc7a100t/*.json \
 395        ./artix7/xc7a100tcsg324-1
 396     cd ../../..
 397     cmake -DARCH=xilinx .
 398     make
 399     make install
 400     python3 xilinx/python/bbaexport.py --device xc7a100tcsg324-1 --bba xilinx/xc7a100t.bba
 401     ./bbasm --l xilinx/xc7a100t.bba xilinx/xc7a100t.bin
 402     mkdir -p /usr/share/nextpnr/xilinx-chipdb
 403     cp xilinx/*.bin /usr/share/nextpnr/xilinx-chipdb
 404     cp -aux xilinx/external/prjxray-db /usr/share/nextpnr
 405
 406 # Additional Useful Info for UART <-> USB Serial Interface Through OrangeCrab's Built-in USB Interface
 407
 408 This uses OrangeCrab's built-in USB interface, rather than needing a
 409 separate USB-serial adapter. see the following for further details:
 410
 411 * <https://github.com/antonblanchard/microwatt/pull/347#issuecomment-1058800570>
 412 * <https://github.com/antonblanchard/microwatt/pull/347#issuecomment-1058834790>
 413
 414 # running orangecrab-examples before flashing microwatt
 415
 416 See <https://github.com/orangecrab-fpga/orangecrab-hardware/blob/main/contrib/10-orangecrab.rules>
 417
 418 If the OrangeCrab is running in DFU mode, lsusb will show:
 419
 420     1209:5af0 Generic OrangeCrab r0.2 DFU Bootloader v3.1-6-g62e92e2
 421
 422 OrangeCrab has two DFU devices:
 423
 424     Found DFU: [1209:5af0] ver=0101, devnum=22, cfg=1, intf=0, path="1-4.2", alt=1, name="0x00100000 RISC-V Firmware", serial="UNKNOWN"
 425     Found DFU: [1209:5af0] ver=0101, devnum=22, cfg=1, intf=0, path="1-4.2", alt=0, name="0x00080000 Bitstream", serial="UNKNOWN"
 426
 427 Then clone and patch orangecrab-examples:
 428
 429         git clone https://github.com/orangecrab-fpga/orangecrab-examples
 430         patch -p1 < orangecrab-examples.diff
 431
 432 To build and flash the example:
 433
 434         pushd orangecrab-examples/nmigen
 435         python3 blink.py
 436         popd
 437         sudo dfu-util -D orangecrab-examples/nmigen/build/top.bit -a 0