# Tutorial for setting up Microwatt chroot and running simulations
Useful Links (External):
*
*
*
* [Verilator docs, commands](https://verilator.org/guide/latest/exe_verilator.html)
* [Verilator runtime command documentation](https://verilator.org/guide/latest/exe_sim.html)
* Tutorials for how to work with verilator:
[part1](https://www.itsembedded.com/dhd/verilator_1/),
[part2](https://www.itsembedded.com/dhd/verilator_2/)
Useful links (Libre-SOC):
* Libre-SOC page covering our workflow: [[HDL_workflow]]
* Devscripts Libre-SOC page: [[devscripts]]
* Original Microwatt Libre-SOC page: [[microwatt]]
* [Libre-SOC Microwatt repo branch](https://git.libre-soc.org/?p=microwatt.git;a=tree;hb=refs/heads/verilator_trace)
* [Libre-SOC devscripts repo](https://git.libre-soc.org/?p=dev-env-setup.git;a=tree)
Other Tutorials (Libre-SOC):
* First steps for working with PowerISA instructions Libre-SOC page:
[[/docs/firststeps]]
## Video Tutorial
[43min tutorial](https://youtu.be/02LCl3ang8g) was made and uploaded to
Youtube, covering some of the material you'll find on this page.
## Development environment scripts
If you haven't already, clone Libre-SOC's development environment setup scripts.
These are bash scripts, and greatly simplify the time it takes to create a:
- Stable environment
- With all software and libraries at specific versions
(which are known to work).
These attributes are absolutely critical, and no support will be
provided, unless you use these scripts to setup a development environment. This
helps us fix any bugs in the scripts, and make sure everyone runs on the same
page.
$ git clone https://git.libre-soc.org/git/dev-env-setup.git
Do *look through* the
[code](https://git.libre-soc.org/?p=dev-env-setup.git;a=tree) before running
any of those scripts.
This is your own legal responsibility (to not run
arbitrary code off of the internet) and we take no responsibility or accept
any liability whatsoever for your use or misuse of them.
It is expected for you to use Debian for the host OS (anything else
is unsupported: many contributors have repeatedly gotten into trouble by not
following this advice), while all the chroots - which are developed
very specifically for "reproducible builds" - run Debian 10 (Buster).
## Setting up chroot
Scripts we will be using for the setup are:
* `mk-deb-chroot`, `cp-scripts-to-chroot` for chroot setup
* `install-hdl-apt-reqs`, `verilator-install`, `hdl-tools-yosys` for working
with Microwatt
(*Current limitation for `mk-deb-chroot`, is that you must be the first user on
the host machine, having user ID 1000.*)
Commands to run in terminal to setup a new chroot environment for microwatt
simulations.
$ cd dev-env-setup
$ sudo bash
# ./mk-deb-chroot microwatt
# ./cp-scripts-to-chroot microwatt
# exit
$ schroot -c microwatt
(microwatt):$ cd dev-env-setup
(microwatt):$ sudo bash
(microwatt):# ./install-hdl-apt-reqs
(microwatt):# ./verilator-install
(microwatt):# ./hdl-tools-yosys
(microwatt):# exit
(microwatt):$ cd ~/src/
(microwatt):$ git clone https://git.libre-soc.org/git/microwatt.git
(microwatt):$ cd microwatt
(microwatt):$ git checkout verilator_trace
Make sure verilator binaries in $PATH:
(microwatt):$ export PATH=/usr/local/verilator/bin:$PATH
(microwatt):$ export GHDLSYNTH=ghdl
(GHDLSYNTH needs to be redefined because the Makefile has default `ghdl.so`,
but somewhere else '.so' gets appended. You may see the following error if you
don't redefine:
`ERROR: Can't load module
./ghdl.so':/usr/local/bin/../share/yosys/plugins/**ghdl.so.so**`)
[IRC](https://libre-soc.org/irclog/%23libre-soc.2023-01-25.log.html#t2023-01-25T11:10:47)
## Compiling the verilator sim for Microwatt
* [Libre-SOC Microwatt repo branch, Makefile](https://git.libre-soc.org/?p=microwatt.git;a=blob;f=Makefile;hb=refs/heads/verilator_trace)
Verilator creates a fairly fast simulation by converting the HDL design to C++,
and then compiling a binary which the user runs.
To compile the verilator simulation, first set verilator as the target for the
makefile:
(microwatt):$ export FPGA_TARGET=verilator
Before compiling, you can change the `THREADS` variable in the makefile, which
will allow the compiled verilator simulation binary to use more than 1 thread
(*make sure to check how many CPU threads you have before changing this!*)
To compile the verilator simulation binary, call make with the
`microwatt-verilator` rule.
(microwatt):$ make microwatt-verilator
## Compiling hello world code
We need some code to actually run on the core, so start with the 'hello world'.
Instructions assume you're still in the microwatt directory.
(microwatt):$ cd hello_world
(microwatt):$ make
A `hello_world.bin` should be generated (the final binary to be loaded), as
well as an
[.elf file](https://en.wikipedia.org/wiki/Executable_and_Linkable_Format), and
.hex (representing the binary data as hex text strings).
To view the symbol table (useful to see where various sections of the binary
begin):
(microwatt):$ powerpc64le-linux-gnu-objdump -h hello_world.elf
(microwatt):$ powerpc64le-linux-gnu-objdump -x hello_world.elf
`-h` shows just the section headers, `-x` shows all headers.
And to view the disassembly (great for learning about the PowerISA instructions,
and for associating the binary hex with actual instructions), you can view the
automatically generated `hello_world.as` file.
Command to generate the disassembly:
(microwatt):$ powerpc64le-linux-gnu-objdump -D hello_world.elf
For more information about `objdump` (common utility, not just for PowerISA),
see the manual pages.
(microwatt):$ man powerpc64le-linux-gnu-objdump
The binary is ready to go, now it can be loaded into the simulation.
## Simulation
### Command line args
To find out the `microwatt-verilator` arguments, you can check with `-h` arg:
(microwatt):$ ./microwatt-verilator -h
Some of the arguments are explained in further sections.
### Running
Run the `microwatt-verilator` binary, with `hello_world/hello_world.bin` as an
argument:
(microwatt):$ time ./microwatt-verilator hello_world/hello_world.bin
`time` is a utility you can use to measure how long it takes to run the sim.
A pretty ASCII art of a lightbulb should be printed, and then the user can type
any characters, which will be echoed back. To end the simulation press Ctrl+C.
If no characters are appearing after about 20 seconds, stop the simulation,
as there might be other issues.
Single-threaded verilator sim binary, on a 2nd gen intel i5 (sandybridge)
takes 53 seconds to print the ASCII lightbulb.
On another dev's machine, ASUS KGPE D16, this takes just over a minute.
(*You'll find that uart printout is one of the longer parts of the simulation
in general.*)
## Analysing results after simulation
The following files will be generated during the sim:
- `bram.dump` - Shows the PC address and instruction being executed. If the sim
hangs without any printing, view this file, as the processor may have hit an
exception etc. Grows in size as the sim runs.
- `bram.snapshot.[NUMBER]`, `verilator.save.[NUMBER]` - Snapshot files of the
contents of bram and verilator model respectively. Can be used to resume the
simulation. The number on the end corresponds to the tick time (i.e.
`bram.snapshot.1999990`/`verilator.save.1999990`). First the verilator model is
loaded, and then the bram contents are loaded. See lines `#65-108` and
`#189-195` of the
[microwatt-verilator.cpp file](https://git.libre-soc.org/?p=microwatt.git;a=blob;f=verilator/microwatt-verilator.cpp;h=a226393f6ba74d5e3e1ffdb729d731d2311d53ad;hb=refs/heads/verilator_trace).
Pass the tick number on the end of the filename with the '-s' flag:
(microwatt):$ ./microwatt-verilator hello_world/hello_world.bin -s 1999990
You'll get a message like this:
loading hello_world/hello_world.bin at 0x0 size 0x1888
loading bram.snapshot.1999990 at 0x0 size 0x10000000
restored at 1999990
These snapshots are generated at intervals of every 2,000,000 ticks.
- `microwatt-verilator.vcd` - GTKWave waveform file, allowing you to look at
processor signals and transitions during simulation.
Pass `-d` flag to `microwatt-verilator` binary:
(microwatt):$ ./microwatt-verilator hello_world/hello_world.bin -d
**NOTE**: Trace dumping will generate a large VCD file (about 6GB for the hello
world example)!
If you want GTKWave to load it faster, convert to fst first:
(microwatt):$ vcd2fst --vcdname=microwatt-verilator.vcd --fstname=microwatt-verilator.fst
(microwatt):$ gtkwave microwatt-verilator.fst
Fst files are orders-of-magnitude smaller (about 20MB vs 6GB), but are specific
to the GTKWave tool.
## Micropython
The Microwatt repo comes with a pre-compiled
[micropython binary](https://git.libre-soc.org/?p=microwatt.git;a=tree;f=micropython;h=18fa078c8145bdaa75667a0ab04eb0b261245665;hb=refs/heads/verilator_trace)
(version 1.12), which you can try out after confirming 'hello world' works.
Bear in mind, not all features of python will be available. Such as
floating-point numbers.
For micropython to work, you'll need to increase the RAM size in the makefile.
Go to the microwatt-verilator makefile, and comment out the following lines:
MEMORY_SIZE=8192
RAM_INIT_FILE=hello_world/hello_world.hex
And uncomment the following:
MEMORY_SIZE=393216
RAM_INIT_FILE=micropython/firmware.hex
This will increase the RAM size from 8KiB to 384KiB. The `RAM_INIT_FILE` in
these examples isn't doing anything, however good practice to follow.
Clean up generated files, and recompile:
(microwatt):$ make clean
(microwatt):$ make microwatt-verilator
Once the binary has been built, run the same way as before, but point to the
micropython firmware binary:
(microwatt):$ microwatt-verilator micropython/firmware.bin
On the same system as above, with 1 thread, it took 49 seconds to get to the
micropython shell.
## Verilator runtime commands
A few examples:
# Show the version of verilator being used
(microwatt):$ ./microwatt-verilator +verilator+version
## Building `microwatt-verilator` using the Libre-SOC core
In the Makefile, you need to set `EXTERNAL_CORE` to true, and copy the
generated core from soc repo to microwatt. *(If you use a separate chroot to
generate Libre-SOC cores, then you'll need to copy from that chroot to
microwatt chroot from host.*
cd /path/to/soc
make microwatt_external_core
cp external_core_top.v /path/to/microwatt
Then compile verilator sim binary as before:
cd ~/src/microwatt/
export FPGA_TARGET=verilator
export GHDLSYNTH=ghdl
make microwatt-verilator
## Running Linux kernel
To run Linux on Microwatt, you'll need two binaries:
- The `sdram_init.bin`, which is easy to compile (no additional software
required).
- The `dtbImage.microwatt` device tree Linux kernel. This can be compiled (see
below), or a copy can be downloaded from: .
Like with the micropython example, you'll need to increase the RAM size in the
makefile, and recompile the microwatt-verilator binary.
Uncomment the following:
MEMORY_SIZE=536870912
Which will change the RAM size to 512KiB.
As there is no `dtbImage.microwatt.hex`, you can leave `RAM_INIT_FILE` unchanged.
### Building the kernel - TODO:
*(Please don't build the kernel yourself, until you've tested with the existing kernel linked
above!)*
On a POWER9 there is no need to install gcc-powerpc64le-linux-gnu,
you can omit CROSS_COMPILE and ARCH in this case
apt install gcc-powerpc64le-linux-gnu
apt install flex bison lz4
git clone -b microwatt-5.7 https://git.kernel.org/pub/scm/linux/kernel/git/joel/microwatt.git
cd microwatt
wget https://ftp.libre-soc.org/microwatt-linux-5.7.patch
patch -p1 < microwatt-linux-5.7.patch
wget https://ftp.libre-soc.org/rootfs.cpio
CROSS_COMPILE="ccache powerpc64le-linux-gnu-" ARCH=powerpc make -j8 O=microwatt microwatt_defconfig
CROSS_COMPILE="ccache powerpc64le-linux-gnu-" ARCH=powerpc make -j8 O=microwatt
This will produce a file
microwatt/arch/powerpc/boot/dtbImage.microwatt
### Building `sdram_init.bin`
This needs gcc-powerpc64le-linux-gnu (already included in the setup step) if
cross compilation is used. It is assumed you're already in `~/src/microwatt/`
directory.
(microwatt):$ cd litedram/gen-src/sdram_init/
(microwatt):$ make
The resulting binary will be in the `obj/` directory.
### Running the simulation
Make sure to return back to `src/microwatt/`.
(microwatt):$ cd ~/src/microwatt/
(microwatt):$ cp microwatt/arch/powerpc/boot/dtbImage.microwatt
(microwatt):$ ./microwatt-verilator sdram_init.bin dtbImage.microwatt
This will take some time...
### Sysconn information
TODO WIP integrate from
Sysconn is a module which includes information about the SoC, and the info is
printed at the start of the simulation.
### Time benchmarks
`microwatt-verilator` was compiled with 3 threads for faster simulation.
- Time to finish printing Sysconn info: about 1min
- Time to allocate bytes to kernel: ?
- Time to login prompt: about 1 hour
- Time to user shell: ?
### TODO: buildroot
* https://github.com/shenki/buildroot/commits/microwatt
* https://codeconstruct.com.au/docs/microwatt-orangecrab/
## FPGA Development - TODO: Need checking
### Building the bitstring for OrangeCrab
cd microwatt
export FPGA_TARGET=ORANGE-CRAB
export GHDLSYNTH=ghdl
make microwatt.bit
### flashing the bitstring to the OrangeCrab
make prog # this will run OpenOCD
### Notes for ulx3s
notes for how to compile for ulx3s
git clone https://github.com/kost/fujprog
(follow build procedure shown in fujprog README)
git clone https://git.libre-soc.org/git/microwatt.git
git checkout -b verilator_trace
export FPGA_TARGET=ulx3s
make microwatt.svf
fujprog microwatt.svf
### Notes for nextpnr-xilinx
superceded: see page [[nextpnr-xilinx]] and devscript
for compiling nextpnr-xilinx and making it useable for nmigen
to compile for the digilent arty-a7-100t, requires a little
futzing around, using the symbiflow version of prjxray-db
instead of the one recommended as a submodule
git clone https://github.com/gatecat/nextpnr-xilinx
cd nextpnr-xilinx
git checkout cd8b15db6ff5c1a7f10a9e
git submodule init
git submodule update
cd xilinx/external
mv prjxray-db prjxray-db-no
git clone https://github.com/SymbiFlow/prjxray-db
cd prjxray-db
git checkout 0a0addedd73e7
cp ./artix7/xc7a100t/*.json \
./artix7/xc7a100tcsg324-1
cd ../../..
cmake -DARCH=xilinx .
make
make install
python3 xilinx/python/bbaexport.py --device xc7a100tcsg324-1 --bba xilinx/xc7a100t.bba
./bbasm --l xilinx/xc7a100t.bba xilinx/xc7a100t.bin
mkdir -p /usr/share/nextpnr/xilinx-chipdb
cp xilinx/*.bin /usr/share/nextpnr/xilinx-chipdb
cp -aux xilinx/external/prjxray-db /usr/share/nextpnr
# Additional Useful Info for UART <-> USB Serial Interface Through OrangeCrab's Built-in USB Interface
This uses OrangeCrab's built-in USB interface, rather than needing a
separate USB-serial adapter. see the following for further details:
*
*
# running orangecrab-examples before flashing microwatt
See
If the OrangeCrab is running in DFU mode, lsusb will show:
1209:5af0 Generic OrangeCrab r0.2 DFU Bootloader v3.1-6-g62e92e2
OrangeCrab has two DFU devices:
Found DFU: [1209:5af0] ver=0101, devnum=22, cfg=1, intf=0, path="1-4.2", alt=1, name="0x00100000 RISC-V Firmware", serial="UNKNOWN"
Found DFU: [1209:5af0] ver=0101, devnum=22, cfg=1, intf=0, path="1-4.2", alt=0, name="0x00080000 Bitstream", serial="UNKNOWN"
Then clone and patch orangecrab-examples:
git clone https://github.com/orangecrab-fpga/orangecrab-examples
patch -p1 < orangecrab-examples.diff
To build and flash the example:
pushd orangecrab-examples/nmigen
python3 blink.py
popd
sudo dfu-util -D orangecrab-examples/nmigen/build/top.bit -a 0