From 0bd5570059740f6c8df9654b63ab08c8044f258b Mon Sep 17 00:00:00 2001 From: sadoon Date: Thu, 17 Aug 2023 18:43:10 +0100 Subject: [PATCH] Completed the guide, added notes and issues. Missing links --- HDL_workflow/debian-sffs.mdwn | 406 ++-------------------------------- 1 file changed, 18 insertions(+), 388 deletions(-) diff --git a/HDL_workflow/debian-sffs.mdwn b/HDL_workflow/debian-sffs.mdwn index 7fb040409..fa6a0f114 100644 --- a/HDL_workflow/debian-sffs.mdwn +++ b/HDL_workflow/debian-sffs.mdwn @@ -61,400 +61,30 @@ Scripts we will be using for the setup are: * `create-deb-repo` to create a local dpkg repository of the built packages * `mk-sffs-chroot` to finally create the chroot using the repository of built packages -Commands to run in terminal to setup a new chroot environment for microwatt -simulations. +Commands to run in terminal to rebuild Debian 12 with sffs compliance: $ cd dev-env-setup $ sudo bash - # ./mk-deb-chroot microwatt - # ./cp-scripts-to-chroot microwatt + # ./install-sbuild-apt-reqs + # ./mk-schroot-bookworm # exit - $ schroot -c microwatt - (microwatt):$ cd dev-env-setup - (microwatt):$ sudo bash - (microwatt):# ./install-hdl-apt-reqs - (microwatt):# ./verilator-install - (microwatt):# ./hdl-tools-yosys - (microwatt):# exit - (microwatt):$ cd ~/src/ - (microwatt):$ git clone https://git.libre-soc.org/git/microwatt.git - (microwatt):$ cd microwatt - (microwatt):$ git checkout verilator_trace + $ ./build-bookworm-sffs + $ ./create-deb-repo + $ ./mk-sffs-chroot -Make sure verilator binaries in $PATH: +## Using prebuilt tarball - (microwatt):$ export PATH=/usr/local/verilator/bin:$PATH - (microwatt):$ export GHDLSYNTH=ghdl +If you prefer to use the prebuilt tarball (which will save days of your time), you could simply download and extract the tarball to a convenient location and chroot into it. -(GHDLSYNTH needs to be redefined because the Makefile has default `ghdl.so`, -but somewhere else '.so' gets appended. You may see the following error if you -don't redefine: -`ERROR: Can't load module -./ghdl.so':/usr/local/bin/../share/yosys/plugins/**ghdl.so.so**`) -[IRC](https://libre-soc.org/irclog/%23libre-soc.2023-01-25.log.html#t2023-01-25T11:10:47) + $ curl -O [link to be added] + $ mkdir bookworm-sffs && cd bookworm-sffs + $ tar xpvf bookworm-sffs.tar.xz --xattrs-include='*.*' --numeric-owner + $ cd .. + $ sudo chroot bookworm-sffs -## Compiling the verilator sim for Microwatt +## Notes and issues -* [Libre-SOC Microwatt repo branch, Makefile](https://git.libre-soc.org/?p=microwatt.git;a=blob;f=Makefile;hb=refs/heads/verilator_trace) - -Verilator creates a fairly fast simulation by converting the HDL design to C++, -and then compiling a binary which the user runs. - -To compile the verilator simulation, first set verilator as the target for the -makefile: - - (microwatt):$ export FPGA_TARGET=verilator - -Before compiling, you can change the `THREADS` variable in the makefile, which -will allow the compiled verilator simulation binary to use more than 1 thread -(*make sure to check how many CPU threads you have before changing this!*) - -To compile the verilator simulation binary, call make with the -`microwatt-verilator` rule. - - (microwatt):$ make microwatt-verilator - -## Compiling hello world code - -We need some code to actually run on the core, so start with the 'hello world'. -Instructions assume you're still in the microwatt directory. - - (microwatt):$ cd hello_world - (microwatt):$ make - -A `hello_world.bin` should be generated (the final binary to be loaded), as -well as an -[.elf file](https://en.wikipedia.org/wiki/Executable_and_Linkable_Format), and -.hex (representing the binary data as hex text strings). - -To view the symbol table (useful to see where various sections of the binary -begin): - - (microwatt):$ powerpc64le-linux-gnu-objdump -h hello_world.elf - (microwatt):$ powerpc64le-linux-gnu-objdump -x hello_world.elf - -`-h` shows just the section headers, `-x` shows all headers. - -And to view the disassembly (great for learning about the PowerISA instructions, -and for associating the binary hex with actual instructions), you can view the -automatically generated `hello_world.as` file. - -Command to generate the disassembly: - - (microwatt):$ powerpc64le-linux-gnu-objdump -D hello_world.elf - -For more information about `objdump` (common utility, not just for PowerISA), -see the manual pages. - - (microwatt):$ man powerpc64le-linux-gnu-objdump - -The binary is ready to go, now it can be loaded into the simulation. - -## Simulation - -### Command line args - -To find out the `microwatt-verilator` arguments, you can check with `-h` arg: - - (microwatt):$ ./microwatt-verilator -h - -Some of the arguments are explained in further sections. - -### Running - -Run the `microwatt-verilator` binary, with `hello_world/hello_world.bin` as an -argument: - - (microwatt):$ time ./microwatt-verilator hello_world/hello_world.bin - -`time` is a utility you can use to measure how long it takes to run the sim. - -A pretty ASCII art of a lightbulb should be printed, and then the user can type -any characters, which will be echoed back. To end the simulation press Ctrl+C. - -If no characters are appearing after about 20 seconds, stop the simulation, -as there might be other issues. - -Single-threaded verilator sim binary, on a 2nd gen intel i5 (sandybridge) -takes 53 seconds to print the ASCII lightbulb. - -On another dev's machine, ASUS KGPE D16, this takes just over a minute. - -(*You'll find that uart printout is one of the longer parts of the simulation -in general.*) - -## Analysing results after simulation - -The following files will be generated during the sim: - -- `bram.dump` - Shows the PC address and instruction being executed. If the sim -hangs without any printing, view this file, as the processor may have hit an -exception etc. Grows in size as the sim runs. - -- `bram.snapshot.[NUMBER]`, `verilator.save.[NUMBER]` - Snapshot files of the -contents of bram and verilator model respectively. Can be used to resume the -simulation. The number on the end corresponds to the tick time (i.e. -`bram.snapshot.1999990`/`verilator.save.1999990`). First the verilator model is -loaded, and then the bram contents are loaded. See lines `#65-108` and -`#189-195` of the -[microwatt-verilator.cpp file](https://git.libre-soc.org/?p=microwatt.git;a=blob;f=verilator/microwatt-verilator.cpp;h=a226393f6ba74d5e3e1ffdb729d731d2311d53ad;hb=refs/heads/verilator_trace). -Pass the tick number on the end of the filename with the '-s' flag: - - (microwatt):$ ./microwatt-verilator hello_world/hello_world.bin -s 1999990 - -You'll get a message like this: - - loading hello_world/hello_world.bin at 0x0 size 0x1888 - loading bram.snapshot.1999990 at 0x0 size 0x10000000 - restored at 1999990 - -These snapshots are generated at intervals of every 2,000,000 ticks. - -- `microwatt-verilator.vcd` - GTKWave waveform file, allowing you to look at -processor signals and transitions during simulation. -Pass `-d` flag to `microwatt-verilator` binary: - - (microwatt):$ ./microwatt-verilator hello_world/hello_world.bin -d - -**NOTE**: Trace dumping will generate a large VCD file (about 6GB for the hello -world example)! - -If you want GTKWave to load it faster, convert to fst first: - - (microwatt):$ vcd2fst --vcdname=microwatt-verilator.vcd --fstname=microwatt-verilator.fst - (microwatt):$ gtkwave microwatt-verilator.fst - -Fst files are orders-of-magnitude smaller (about 20MB vs 6GB), but are specific -to the GTKWave tool. - -## Micropython - -The Microwatt repo comes with a pre-compiled -[micropython binary](https://git.libre-soc.org/?p=microwatt.git;a=tree;f=micropython;h=18fa078c8145bdaa75667a0ab04eb0b261245665;hb=refs/heads/verilator_trace) -(version 1.12), which you can try out after confirming 'hello world' works. -Bear in mind, not all features of python will be available. Such as -floating-point numbers. - -For micropython to work, you'll need to increase the RAM size in the makefile. -Go to the microwatt-verilator makefile, and comment out the following lines: - - MEMORY_SIZE=8192 - RAM_INIT_FILE=hello_world/hello_world.hex - -And uncomment the following: - - MEMORY_SIZE=393216 - RAM_INIT_FILE=micropython/firmware.hex - -This will increase the RAM size from 8KiB to 384KiB. The `RAM_INIT_FILE` in -these examples isn't doing anything, however good practice to follow. - -Clean up generated files, and recompile: - - (microwatt):$ make clean - (microwatt):$ make microwatt-verilator - -Once the binary has been built, run the same way as before, but point to the -micropython firmware binary: - - (microwatt):$ microwatt-verilator micropython/firmware.bin - -On the same system as above, with 1 thread, it took 49 seconds to get to the -micropython shell. - -## Verilator runtime commands -A few examples: - - # Show the version of verilator being used - (microwatt):$ ./microwatt-verilator +verilator+version - -## Building `microwatt-verilator` using the Libre-SOC core - -In the Makefile, you need to set `EXTERNAL_CORE` to true, and copy the -generated core from soc repo to microwatt. *(If you use a separate chroot to -generate Libre-SOC cores, then you'll need to copy from that chroot to -microwatt chroot from host.* - - cd /path/to/soc - make microwatt_external_core - cp external_core_top.v /path/to/microwatt - -Then compile verilator sim binary as before: - - cd ~/src/microwatt/ - export FPGA_TARGET=verilator - export GHDLSYNTH=ghdl - make microwatt-verilator - -## Running Linux kernel - -To run Linux on Microwatt, you'll need two binaries: - -- The `sdram_init.bin`, which is easy to compile (no additional software -required). - -- The `dtbImage.microwatt` device tree Linux kernel. This can be compiled (see -below), or a copy can be downloaded from: . - -Like with the micropython example, you'll need to increase the RAM size in the -makefile, and recompile the microwatt-verilator binary. - -Uncomment the following: - - MEMORY_SIZE=536870912 - -Which will change the RAM size to 512KiB. - -As there is no `dtbImage.microwatt.hex`, you can leave `RAM_INIT_FILE` unchanged. - -### Building the kernel - TODO: - -*(Please don't build the kernel yourself, until you've tested with the existing kernel linked -above!)* - -On a POWER9 there is no need to install gcc-powerpc64le-linux-gnu, -you can omit CROSS_COMPILE and ARCH in this case - - apt install gcc-powerpc64le-linux-gnu - apt install flex bison lz4 - git clone -b microwatt-5.7 https://git.kernel.org/pub/scm/linux/kernel/git/joel/microwatt.git - cd microwatt - wget https://ftp.libre-soc.org/microwatt-linux-5.7.patch - patch -p1 < microwatt-linux-5.7.patch - wget https://ftp.libre-soc.org/rootfs.cpio - CROSS_COMPILE="ccache powerpc64le-linux-gnu-" ARCH=powerpc make -j8 O=microwatt microwatt_defconfig - CROSS_COMPILE="ccache powerpc64le-linux-gnu-" ARCH=powerpc make -j8 O=microwatt - -This will produce a file - microwatt/arch/powerpc/boot/dtbImage.microwatt - -### Building `sdram_init.bin` -This needs gcc-powerpc64le-linux-gnu (already included in the setup step) if -cross compilation is used. It is assumed you're already in `~/src/microwatt/` -directory. - - (microwatt):$ cd litedram/gen-src/sdram_init/ - (microwatt):$ make - -The resulting binary will be in the `obj/` directory. - -### Running the simulation - -Make sure to return back to `src/microwatt/`. - - (microwatt):$ cd ~/src/microwatt/ - (microwatt):$ cp microwatt/arch/powerpc/boot/dtbImage.microwatt - (microwatt):$ ./microwatt-verilator sdram_init.bin dtbImage.microwatt - -This will take some time... - -### Sysconn information - -TODO WIP integrate from - -Sysconn is a module which includes information about the SoC, and the info is -printed at the start of the simulation. - -### Time benchmarks - -`microwatt-verilator` was compiled with 3 threads for faster simulation. - -- Time to finish printing Sysconn info: about 1min -- Time to allocate bytes to kernel: ? -- Time to login prompt: about 1 hour -- Time to user shell: ? - -### TODO: buildroot - -* https://github.com/shenki/buildroot/commits/microwatt -* https://codeconstruct.com.au/docs/microwatt-orangecrab/ - -## FPGA Development - TODO: Need checking -### Building the bitstring for OrangeCrab - - cd microwatt - export FPGA_TARGET=ORANGE-CRAB - export GHDLSYNTH=ghdl - make microwatt.bit - -### flashing the bitstring to the OrangeCrab - - make prog # this will run OpenOCD - -### Notes for ulx3s - -notes for how to compile for ulx3s - - git clone https://github.com/kost/fujprog - (follow build procedure shown in fujprog README) - git clone https://git.libre-soc.org/git/microwatt.git - git checkout -b verilator_trace - export FPGA_TARGET=ulx3s - make microwatt.svf - fujprog microwatt.svf - - -### Notes for nextpnr-xilinx - -superceded: see page [[nextpnr-xilinx]] and devscript - - -for compiling nextpnr-xilinx and making it useable for nmigen -to compile for the digilent arty-a7-100t, requires a little -futzing around, using the symbiflow version of prjxray-db -instead of the one recommended as a submodule - - git clone https://github.com/gatecat/nextpnr-xilinx - cd nextpnr-xilinx - git checkout cd8b15db6ff5c1a7f10a9e - git submodule init - git submodule update - cd xilinx/external - mv prjxray-db prjxray-db-no - git clone https://github.com/SymbiFlow/prjxray-db - cd prjxray-db - git checkout 0a0addedd73e7 - cp ./artix7/xc7a100t/*.json \ - ./artix7/xc7a100tcsg324-1 - cd ../../.. - cmake -DARCH=xilinx . - make - make install - python3 xilinx/python/bbaexport.py --device xc7a100tcsg324-1 --bba xilinx/xc7a100t.bba - ./bbasm --l xilinx/xc7a100t.bba xilinx/xc7a100t.bin - mkdir -p /usr/share/nextpnr/xilinx-chipdb - cp xilinx/*.bin /usr/share/nextpnr/xilinx-chipdb - cp -aux xilinx/external/prjxray-db /usr/share/nextpnr - -# Additional Useful Info for UART <-> USB Serial Interface Through OrangeCrab's Built-in USB Interface - -This uses OrangeCrab's built-in USB interface, rather than needing a -separate USB-serial adapter. see the following for further details: - -* -* - -# running orangecrab-examples before flashing microwatt - -See - -If the OrangeCrab is running in DFU mode, lsusb will show: - - 1209:5af0 Generic OrangeCrab r0.2 DFU Bootloader v3.1-6-g62e92e2 - -OrangeCrab has two DFU devices: - - Found DFU: [1209:5af0] ver=0101, devnum=22, cfg=1, intf=0, path="1-4.2", alt=1, name="0x00100000 RISC-V Firmware", serial="UNKNOWN" - Found DFU: [1209:5af0] ver=0101, devnum=22, cfg=1, intf=0, path="1-4.2", alt=0, name="0x00080000 Bitstream", serial="UNKNOWN" - -Then clone and patch orangecrab-examples: - - git clone https://github.com/orangecrab-fpga/orangecrab-examples - patch -p1 < orangecrab-examples.diff - -To build and flash the example: - - pushd orangecrab-examples/nmigen - python3 blink.py - popd - sudo dfu-util -D orangecrab-examples/nmigen/build/top.bit -a 0 +### Some packages will refuse to build with sffs compliant build flags and need workarounds, this will be handled in the future. +### The Linux kernel cannot boot without AltiVec and VSX, even though they are build-time options. The solution for now is to use [microwatt's 5.7 kernel](https://lore.kernel.org/all/CAPweEDw710zFK8KLZY5gsQxEkQKrDiFkNRgABY9HJZ1rxpeVCg@mail.gmail.com/T/) and to incorporate the needed patches upstream in the future. +### Previous note also means this is purely a chroot environment, this page will be updated once it becomes possible to boot a full VM of this build. +### glibc *may* have some issues on sffs, the only way to test this is to run on microwatt which will be done soon. -- 2.30.2