HDL_workflow/microwatt.mdwn

   1 # Tutorial for setting up Microwatt chroot and running simulations
   2
   3 Useful Links (External):
   4
   5 * <https://codeconstruct.com.au/docs/microwatt-orangecrab/>
   6 * <https://shenki.github.io/boot-linux-on-microwatt/>
   7 * <https://github.com/gregdavill/OrangeCrab-test-sw>
   8 * [Verilator docs, commands](https://verilator.org/guide/latest/exe_verilator.html)
   9 * [Verilator runtime command documentation](https://verilator.org/guide/latest/exe_sim.html)
  10 * Tutorials for how to work with verilator:
  11 [part1](https://www.itsembedded.com/dhd/verilator_1/),
  12 [part2](https://www.itsembedded.com/dhd/verilator_2/)
  13
  14 Useful links (Libre-SOC):
  15
  16 * Libre-SOC page covering our workflow: [[HDL_workflow]]
  17 * Devscripts Libre-SOC page: [[devscripts]]
  18 * Original Microwatt Libre-SOC page: [[microwatt]]
  19 * [Libre-SOC Microwatt repo branch](https://git.libre-soc.org/?p=microwatt.git;a=tree;hb=refs/heads/verilator_trace)
  20 * [Libre-SOC devscripts repo](https://git.libre-soc.org/?p=dev-env-setup.git;a=tree)
  21
  22 Other Tutorials (Libre-SOC):
  23
  24 * First steps for working with PowerISA instructions Libre-SOC page:
  25 [[/docs/firststeps]]
  26
  27 ## Development environment scripts
  28
  29 If you haven't already, clone Libre-SOC's development environment setup scripts.
  30 These are bash scripts, and greatly simplify the time it takes to create a:
  31
  32 - Stable environment
  33 - With all software and libraries at specific versions
  34 (which are known to work).
  35
  36 These attributes are absolutely critical, and no support will be
  37 provided, unless you use these scripts to setup a development environment. This
  38 helps us fix any bugs in the scripts, and make sure everyone runs on the same
  39 page.
  40
  41     $ git clone https://git.libre-soc.org/git/dev-env-setup.git
  42
  43 Do *look through* the
  44 [code](https://git.libre-soc.org/?p=dev-env-setup.git;a=tree) before running
  45 any of those scripts. They may be confusing, however after reading a few you'll
  46 start to become more familiar with them.
  47
  48 It is expected for you to use Debian, we mostly use 11 (Bullseye) for the host
  49 OS, while all the chroots run Debian 10 (Buster).
  50
  51 ## Setting up chroot
  52
  53 Scripts we will be using for the setup are:
  54
  55 * `mk-deb-chroot`, `cp-scripts-to-chroot` for chroot setup
  56 * `install-hdl-apt-reqs`, `verilator-install`, `hdl-tools-yosys` for working
  57 with Microwatt
  58
  59 (*Current limitation for `mk-deb-chroot`, is that you must be the first user on
  60 the host machine, having user ID 1000.*)
  61
  62 Commands to run in terminal to setup a new chroot environment for microwatt
  63 simulations.
  64
  65     $ cd dev-env-setup
  66     $ sudo bash
  67     # ./mk-deb-chroot microwatt
  68     # ./cp-scripts-to-chroot microwatt
  69     # exit
  70     $ schroot -c microwatt
  71     (microwatt):$ cd dev-env-setup
  72     (microwatt):$ sudo bash
  73     (microwatt):# ./install-hdl-apt-reqs
  74     (microwatt):# ./verilator-install
  75     (microwatt):# ./hdl-tools-yosys
  76     (microwatt):# exit
  77     (microwatt):$ cd ~/src/
  78     (microwatt):$ git clone https://git.libre-soc.org/git/microwatt.git
  79     (microwatt):$ cd microwatt
  80     (microwatt):$ git checkout verilator_trace
  81
  82 Make sure verilator binaries in $PATH:
  83
  84     (microwatt):$ export PATH=/usr/local/verilator/bin:$PATH
  85     (microwatt):$ export GHDLSYNTH=ghdl
  86
  87 (GHDLSYNTH needs to be redefined because the Makefile has default `ghdl.so`,
  88 but somewhere else '.so' gets appended. You may see the following error if you
  89 don't redefine:
  90 `ERROR: Can't load module
  91 ./ghdl.so':/usr/local/bin/../share/yosys/plugins/**ghdl.so.so**`)
  92 [IRC](https://libre-soc.org/irclog/%23libre-soc.2023-01-25.log.html#t2023-01-25T11:10:47)
  93
  94 ## Compiling the verilator sim for Microwatt
  95
  96 * [Libre-SOC Microwatt repo branch, Makefile](https://git.libre-soc.org/?p=microwatt.git;a=blob;f=Makefile;hb=refs/heads/verilator_trace)
  97
  98 Verilator creates a fairly fast simulation by converting the HDL design to C++,
  99 and then compiling a binary which the user runs.
 100
 101 To compile the verilator simulation, first set verilator as the target for the
 102 makefile:
 103
 104     (microwatt):$ export FPGA_TARGET=verilator
 105
 106 Before compiling, you can change the `THREADS` variable in the makefile, which
 107 will allow the compiled verilator simulation binary to use more than 1 thread
 108 (*make sure to check how many CPU threads you have before changing this!*)
 109
 110 To compile the verilator simulation binary, call make with the
 111 `microwatt-verilator` rule.
 112
 113     (microwatt):$ make microwatt-verilator
 114
 115 ## Compiling hello world code
 116
 117 We need some code to actually run on the core, so start with the 'hello world'.
 118 Instructions assume you're still in the microwatt directory.
 119
 120     (microwatt):$ cd hello_world
 121     (microwatt):$ make
 122
 123 A `hello_world.bin` should be generated (the final binary to be loaded), as
 124 well as an
 125 [.elf file](https://en.wikipedia.org/wiki/Executable_and_Linkable_Format), and
 126 .hex (representing the binary data as hex text strings).
 127
 128 To view the symbol table (useful to see where various sections of the binary
 129 begin):
 130
 131     (microwatt):$ powerpc64le-linux-gnu-objdump -h hello_world.elf
 132     (microwatt):$ powerpc64le-linux-gnu-objdump -x hello_world.elf
 133
 134 `-h` shows just the section headers, `-x` shows all headers.
 135
 136 And to view the disassembly (great for learning about the PowerISA instructions,
 137 and for associating the binary hex with actual instructions):
 138
 139     (microwatt):$ powerpc64le-linux-gnu-objdump -d hello_world.elf
 140
 141 For more information about `objdump` (common utility, not just for PowerISA),
 142 see the manual pages.
 143
 144     (microwatt):$ man powerpc64le-linux-gnu-objdump
 145
 146 The binary is ready to go, now it can be loaded into the simulation.
 147
 148 ## Simulation
 149
 150 ### Command line args
 151
 152 To find out the `microwatt-verilator` arguments, you can check with `-h` arg:
 153
 154     (microwatt):$ ./microwatt-verilator -h
 155
 156 Some of the arguments are explained in further sections.
 157
 158 ### Running
 159
 160 Run the `microwatt-verilator` binary, with `hello_world/hello_world.bin` as an
 161 argument:
 162
 163     (microwatt):$ time ./microwatt-verilator hello_world/hello_world.bin
 164
 165 `time` is a utility you can use to measure how long it takes to run the sim.
 166
 167 A pretty ASCII art of a lightbulb should be printed, and then the user can type
 168 any characters, which will be echoed back. To end the simulation press Ctrl+C.
 169
 170 If no characters are appearing after about 20 seconds, stop the simulation,
 171 as there might be other issues.
 172
 173 Single-threaded verilator sim binary, on a 2nd gen intel i5 (sandybridge)
 174 takes 53 seconds to print the ASCII lightbulb.
 175
 176 On another dev's machine, ASUS KGPE D16, this takes just over a minute.
 177
 178 (*You'll find that uart printout is one of the longer parts of the simulation
 179 in general.*)
 180
 181 ## Analysing results after simulation
 182
 183 The following files will be generated during the sim:
 184
 185 - `bram.dump` - Shows the PC address and instruction being executed. If the sim
 186 hangs without any printing, view this file, as the processor may have hit an
 187 exception etc. Grows in size as the sim runs.
 188
 189 - `bram.snapshot.[NUMBER]`, `verilator.save.[NUMBER]` - Snapshot files of the
 190 contents of bram and verilator model respectively. Can be used to resume the
 191 simulation. The number on the end corresponds to the tick time (i.e.
 192 `bram.snapshot.1999990`/`verilator.save.1999990`). First the verilator model is
 193 loaded, and then the bram contents are loaded. See lines `#65-108` and
 194 `#189-195` of the
 195 [microwatt-verilator.cpp file](https://git.libre-soc.org/?p=microwatt.git;a=blob;f=verilator/microwatt-verilator.cpp;h=a226393f6ba74d5e3e1ffdb729d731d2311d53ad;hb=refs/heads/verilator_trace).
 196 Pass the tick number on the end of the filename with the '-s' flag:
 197
 198     (microwatt):$ ./microwatt-verilator hello_world/hello_world.bin -s 1999990
 199
 200 You'll get a message like this:
 201
 202     loading hello_world/hello_world.bin at 0x0 size 0x1888
 203     loading bram.snapshot.1999990 at 0x0 size 0x10000000
 204     restored at 1999990
 205
 206 These snapshots are generated at intervals of every 2,000,000 ticks.
 207
 208 - `microwatt-verilator.vcd` - GTKWave waveform file, allowing you to look at
 209 processor signals and transitions during simulation.
 210 Pass `-d` flag to `microwatt-verilator` binary:
 211
 212     (microwatt):$ ./microwatt-verilator hello_world/hello_world.bin -d
 213
 214 **NOTE**: Trace dumping will generate a large VCD file (about 6GB for the hello
 215 world example)!
 216
 217 If you want GTKWave to load it faster, convert to fst first:
 218
 219     (microwatt):$ vcd2fst --vcdname=microwatt-verilator.vcd --fstname=microwatt-verilator.fst
 220     (microwatt):$ gtkwave microwatt-verilator.fst
 221
 222 Fst files are orders-of-magnitude smaller (about 20MB vs 6GB), but are specific
 223 to the GTKWave tool.
 224
 225 ## Micropython
 226
 227 The Microwatt repo comes with a pre-compiled
 228 [micropython binary](https://git.libre-soc.org/?p=microwatt.git;a=tree;f=micropython;h=18fa078c8145bdaa75667a0ab04eb0b261245665;hb=refs/heads/verilator_trace)
 229 (version 1.12), which you can try out after confirming 'hello world' works.
 230 Bear in mind, not all features of python will be available. Such as
 231 floating-point numbers.
 232
 233 For micropython to work, you'll need to increase the RAM size in the makefile.
 234 Go to the microwatt-verilator makefile, and comment out the following lines:
 235
 236     MEMORY_SIZE=8192
 237     RAM_INIT_FILE=hello_world/hello_world.hex
 238
 239 And uncomment the following:
 240
 241     MEMORY_SIZE=393216
 242     RAM_INIT_FILE=micropython/firmware.hex
 243
 244 This will increase the RAM size from 8KiB to 384KiB. The `RAM_INIT_FILE` in
 245 these examples isn't doing anything, however good practice to follow.
 246
 247 Clean up generated files, and recompile:
 248
 249     (microwatt):$ make clean
 250     (microwatt):$ make microwatt-verilator
 251
 252 Once the binary has been built, run the same way as before, but point to the
 253 micropython firmware binary:
 254
 255     (microwatt):$ microwatt-verilator micropython/firmware.bin
 256
 257 On the same system as above, with 1 thread, it took 49 seconds to get to the
 258 micropython shell.
 259
 260 ## Verilator runtime commands
 261 A few examples:
 262
 263     # Show the version of verilator being used
 264     (microwatt):$ ./microwatt-verilator +verilator+version
 265
 266 ## Building `microwatt-verilator` using the Libre-SOC core
 267
 268         cd /path/to/soc
 269         make microwatt_external_core
 270         cp external_core_top.v /path/to/microwatt
 271         cd /path/to/microwatt
 272         export FPGA_TARGET=verilator
 273         export GHDLSYNTH=ghdl
 274         make microwatt-verilator
 275
 276 ## Running Linux kernel - TODO: Need to check
 277
 278 To run Linux on Microwatt, you'll need two binaries:
 279
 280 - The `sdram_init.bin`, which is easy to compile (no additional software
 281 required).
 282
 283 - The `dtbImage.microwatt` device tree Linux kernel. This can be compiled (see
 284 below), or a copy can be downloaded from: <https://ftp.libre-soc.org/dtbImage.microwatt>.
 285
 286 ### Building the kernel - TODO:
 287 On a POWER9 there is no need to install gcc-powerpc64le-linux-gnu,
 288 you can omit CROSS_COMPILE and ARCH in this case
 289
 290         apt install gcc-powerpc64le-linux-gnu
 291         apt install flex bison lz4
 292         git clone -b microwatt-5.7 https://git.kernel.org/pub/scm/linux/kernel/git/joel/microwatt.git
 293         cd microwatt
 294         wget https://ftp.libre-soc.org/microwatt-linux-5.7.patch
 295         patch -p1 < microwatt-linux-5.7.patch
 296         wget https://ftp.libre-soc.org/rootfs.cpio
 297         CROSS_COMPILE="ccache powerpc64le-linux-gnu-" ARCH=powerpc make -j8 O=microwatt microwatt_defconfig
 298         CROSS_COMPILE="ccache powerpc64le-linux-gnu-" ARCH=powerpc make -j8 O=microwatt
 299
 300 This will produce a file
 301         microwatt/arch/powerpc/boot/dtbImage.microwatt
 302
 303 ### Building `sdram_init.bin`
 304 This needs gcc-powerpc64le-linux-gnu (already included in the setup step) if
 305 cross compilation is used. It is assumed you're already in `~/src/microwatt/`
 306 directory.
 307
 308     (microwatt):$ cd litedram/gen-src/sdram_init/
 309         (microwatt):$ make
 310
 311 The resulting binary will be in the `obj/` directory.
 312
 313 ### Running the simulation
 314
 315 Make sure to return back to `src/microwatt/`.
 316
 317     (microwatt):$ cd ~/src/microwatt/
 318         (microwatt):$ cp microwatt/arch/powerpc/boot/dtbImage.microwatt
 319         (microwatt):$ ./microwatt-verilator sdram_init.bin dtbImage.microwatt
 320
 321 This will take some time...
 322
 323 ### Sysconn information
 324
 325 TODO WIP integrate from <https://libre-soc.org/irclog/%23libre-soc.2022-01-26.log.html>
 326
 327 Sysconn is a module which includes information about the SoC, and the info is
 328 printed at the start of the simulation.
 329
 330 ### Time benchmarks
 331
 332 `microwatt-verilator` was compiled with 3 threads for faster simulation.
 333
 334 - Time to finish printing Sysconn info: about 1min
 335 - Time to allocate bytes to kernel: ?
 336 - Time to login prompt: about 1 hour
 337 - Time to user shell: ?
 338
 339 ### TODO: buildroot
 340
 341 * https://github.com/shenki/buildroot/commits/microwatt
 342 * https://codeconstruct.com.au/docs/microwatt-orangecrab/
 343
 344 ## FPGA Development - TODO: Need checking
 345 ### Building the bitstring for OrangeCrab
 346
 347         cd microwatt
 348         export FPGA_TARGET=ORANGE-CRAB
 349         export GHDLSYNTH=ghdl
 350         make microwatt.bit
 351
 352 ### flashing the bitstring to the OrangeCrab
 353
 354         make prog # this will run OpenOCD
 355
 356 ### Notes for ulx3s
 357
 358 notes for how to compile for ulx3s
 359
 360     git clone https://github.com/kost/fujprog
 361     (follow build procedure shown in fujprog README)
 362     git clone https://git.libre-soc.org/git/microwatt.git
 363     git checkout -b verilator_trace
 364     export FPGA_TARGET=ulx3s
 365     make microwatt.svf
 366     fujprog microwatt.svf
 367
 368
 369 ### Notes for nextpnr-xilinx
 370
 371 superceded: see page [[nextpnr-xilinx]] and devscript
 372 <https://git.libre-soc.org/?p=dev-env-setup.git;a=blob;f=nextpnr-xilinx-install;hb=HEAD>
 373
 374 for compiling nextpnr-xilinx and making it useable for nmigen
 375 to compile for the digilent arty-a7-100t, requires a little
 376 futzing around, using the symbiflow version of prjxray-db
 377 instead of the one recommended as a submodule
 378
 379     git clone https://github.com/gatecat/nextpnr-xilinx
 380     cd nextpnr-xilinx
 381     git checkout cd8b15db6ff5c1a7f10a9e
 382     git submodule init
 383     git submodule update
 384     cd xilinx/external
 385     mv prjxray-db prjxray-db-no
 386     git clone https://github.com/SymbiFlow/prjxray-db
 387     cd prjxray-db
 388     git checkout 0a0addedd73e7
 389     cp ./artix7/xc7a100t/*.json \
 390        ./artix7/xc7a100tcsg324-1
 391     cd ../../..
 392     cmake -DARCH=xilinx .
 393     make
 394     make install
 395     python3 xilinx/python/bbaexport.py --device xc7a100tcsg324-1 --bba xilinx/xc7a100t.bba
 396     ./bbasm --l xilinx/xc7a100t.bba xilinx/xc7a100t.bin
 397     mkdir -p /usr/share/nextpnr/xilinx-chipdb
 398     cp xilinx/*.bin /usr/share/nextpnr/xilinx-chipdb
 399     cp -aux xilinx/external/prjxray-db /usr/share/nextpnr
 400
 401 # Additional Useful Info for UART <-> USB Serial Interface Through OrangeCrab's Built-in USB Interface
 402
 403 This uses OrangeCrab's built-in USB interface, rather than needing a
 404 separate USB-serial adapter. see the following for further details:
 405
 406 * <https://github.com/antonblanchard/microwatt/pull/347#issuecomment-1058800570>
 407 * <https://github.com/antonblanchard/microwatt/pull/347#issuecomment-1058834790>
 408
 409 # running orangecrab-examples before flashing microwatt
 410
 411 See <https://github.com/orangecrab-fpga/orangecrab-hardware/blob/main/contrib/10-orangecrab.rules>
 412
 413 If the OrangeCrab is running in DFU mode, lsusb will show:
 414
 415     1209:5af0 Generic OrangeCrab r0.2 DFU Bootloader v3.1-6-g62e92e2
 416
 417 OrangeCrab has two DFU devices:
 418
 419     Found DFU: [1209:5af0] ver=0101, devnum=22, cfg=1, intf=0, path="1-4.2", alt=1, name="0x00100000 RISC-V Firmware", serial="UNKNOWN"
 420     Found DFU: [1209:5af0] ver=0101, devnum=22, cfg=1, intf=0, path="1-4.2", alt=0, name="0x00080000 Bitstream", serial="UNKNOWN"
 421
 422 Then clone and patch orangecrab-examples:
 423
 424         git clone https://github.com/orangecrab-fpga/orangecrab-examples
 425         patch -p1 < orangecrab-examples.diff
 426
 427 To build and flash the example:
 428
 429         pushd orangecrab-examples/nmigen
 430         python3 blink.py
 431         popd
 432         sudo dfu-util -D orangecrab-examples/nmigen/build/top.bit -a 0