(no commit message)

[libreriscv.git] / HDL_workflow.mdwn

diff --git a/HDL_workflow.mdwn b/HDL_workflow.mdwn
index bd30b9684b47162a49c13ab593a395aef3fe342f..21bc60f45fb53959d15d371e2d78613b393b5227 100644 (file)
--- a/HDL_workflow.mdwn
+++ b/HDL_workflow.mdwn
@@ -1,3 +1,7 @@
+[[!toc ]]
+
+---
+

 # HDL workflow
 
 This section describes the workflow and some best practices for developing
 # HDL workflow
 
 This section describes the workflow and some best practices for developing


@@ -29,16 +33,13 @@ either actively prevents or makes that much harder.
 The main message here: **use the right tool for the right job**.
 
 * mailing list: general communication and discussion.
 The main message here: **use the right tool for the right job**.
 
 * mailing list: general communication and discussion.

-* irc channel #libre-soc: real(ish)-time communication.
+* irc channel #libre-soc on irc.libera.chat: real(ish)-time communication.

 * bugtracker: task-orientated, goal-orientated *focussed* discussion.
 * ikiwiki: document store, information store, and (editable) main website
 * git repositories: code stores (**not binary or auto-generated output store**)
 * ftp server (<https://ftp.libre-soc.org/>): large (temporary,
   auto-generated) file store.
 
 * bugtracker: task-orientated, goal-orientated *focussed* discussion.
 * ikiwiki: document store, information store, and (editable) main website
 * git repositories: code stores (**not binary or auto-generated output store**)
 * ftp server (<https://ftp.libre-soc.org/>): large (temporary,
   auto-generated) file store.
 

-We will add an IRC channel at some point when there are enough people
-to warrant having one (and it will be publicly archived)
-

 Note also the lack of a "forum" in the above list.  this is very
 deliberate.  forums are a serious distraction when it comes to technical
 heavily goal-orientated development. recent internet users may enjoy
 Note also the lack of a "forum" in the above list.  this is very
 deliberate.  forums are a serious distraction when it comes to technical
 heavily goal-orientated development. recent internet users may enjoy


@@ -105,7 +106,8 @@ on the server, that is a high security risk, and i'm not doing it. sorry.)
 ### Mailing list != editable document store
 
 Also, please do not use the mailing list as an "information or document
 ### Mailing list != editable document store
 
 Also, please do not use the mailing list as an "information or document

-store or poor-man's editor". We have the wiki for that.  Edit a page and
+store or poor-man's editor" **including not sending large images**.
+We have the wiki for that.  Edit a page and

 tell people what you did (summarise rather than drop the entire contents
 at the list) and include the link to the page.
 
 tell people what you did (summarise rather than drop the entire contents
 at the list) and include the link to the page.
 


@@ -170,6 +172,8 @@ does a decent job. <https://git.libre-soc.org/>
 tracks changes to files so that previous versions can be got back or
 compared.
 
 tracks changes to files so that previous versions can be got back or
 compared.
 

+Checklist page [[HDL_workflow/git_checklist]]
+

 ## ftp server
 
 <https://ftp.libre-soc.org/> is available for storing large files
 ## ftp server
 
 <https://ftp.libre-soc.org/> is available for storing large files


@@ -276,7 +280,12 @@ he does not want Linux kernel development to become the exclusive
 domain of the "wealthy".  That means **no** to assumptions about
 access to ultra-high resolution screens.
 
 domain of the "wealthy".  That means **no** to assumptions about
 access to ultra-high resolution screens.
 

-# Software prerequisites
+# Software prerequisites<a name="software-prerequisites"></a>
+
+**Please make sure if you install manually that you install dependencies
+in strict order.  Failing to adhere to this will result in pip3 downloading
+unauthorised older software versions.  See
+<http://lists.libre-soc.org/pipermail/libre-soc-dev/2021-September/003666.html>**

 
 Whilst many resources online advocate "`sudo`" in front of all root-level
 commands below, this quickly becomes tiresome. run "`sudo bash`", get a
 
 Whilst many resources online advocate "`sudo`" in front of all root-level
 commands below, this quickly becomes tiresome. run "`sudo bash`", get a


@@ -285,7 +294,7 @@ root prompt, and save yourself some typing.
 * sudo bash
 * apt-get install vim exuberant-ctags
 * apt-get install build-essential
 * sudo bash
 * apt-get install vim exuberant-ctags
 * apt-get install build-essential

-* apt-get install git python3.7 python3.7-dev python-nosetest3
+* apt-get install git python3.7 python3.7-dev python3-nose

 * apt-get install graphviz xdot gtkwave
 * apt-get install python3-venv
 * apt-get install python-virtualenv  # this is an alternative to python3-venv
 * apt-get install graphviz xdot gtkwave
 * apt-get install python3-venv
 * apt-get install python-virtualenv  # this is an alternative to python3-venv


@@ -307,6 +316,8 @@ the process below.
 If you would like just to install only the apt dependencies use
 [install-hdl-apt-reqs](https://git.libre-soc.org/?p=dev-env-setup.git;a=blob;f=install-hdl-apt-reqs;hb=HEAD) instead.
 
 If you would like just to install only the apt dependencies use
 [install-hdl-apt-reqs](https://git.libre-soc.org/?p=dev-env-setup.git;a=blob;f=install-hdl-apt-reqs;hb=HEAD) instead.
 

+This page gives more details and a step by step process : [[HDL_workflow/devscripts]] 
+

 ## git
 
 Look up good tutorials on how to use git effectively.  There are so many
 ## git
 
 Look up good tutorials on how to use git effectively.  There are so many


@@ -326,6 +337,9 @@ relevant unit tests pass 100%.  This ensures that people's work does not
 get "lost" or isolated and out of touch due to major branch diversion,
 and that people communicate and coordinate with each other.
 
 get "lost" or isolated and out of touch due to major branch diversion,
 and that people communicate and coordinate with each other.
 

+This is not a hard rule: under special cirmstances branches can be useful.
+They should not however be considered "routine".
+

 ## yosys
 
 Follow the source code (git clone) instructions here, do **not** use
 ## yosys
 
 Follow the source code (git clone) instructions here, do **not** use


@@ -333,7 +347,7 @@ the "stable" version (do not download the tarball):
 <http://www.clifford.at/yosys/download.html>
 
 Or, alternatively, use the
 <http://www.clifford.at/yosys/download.html>
 
 Or, alternatively, use the

-[yosys-et-al](https://git.libre-soc.org/?p=dev-env-setup.git;a=blob;f=yosys-et-al;hb=HEAD)
+[hdl-tools-yosys](https://git.libre-soc.org/?p=dev-env-setup.git;a=blob;f=hdl-tools-yosys;hb=HEAD)

 script (which also installs symbiyosys and its dependencies)
 
 Do not try to use a fixed revision of yosys (currently 0.9), nmigen is
 script (which also installs symbiyosys and its dependencies)
 
 Do not try to use a fixed revision of yosys (currently 0.9), nmigen is


@@ -365,7 +379,11 @@ flows.
 
 ## nmigen
 
 
 ## nmigen
 

-[nmigen](https://m-labs.hk/gateware/nmigen/) may be installed as follows:
+**PLEASE NOTE: it is critical to install nmigen as the first dependency
+prior to installing any further python-based Libre-SOC HDL repositories.
+If "pip3 list" shows that nmigen has been auto-installed please remove it**
+
+[nmigen](https://nmigen.info/) may be installed as follows:

 
 * mkdir ~/src
 * cd !$
 
 * mkdir ~/src
 * cd !$


@@ -430,20 +448,27 @@ It should print out `Posit8(1.3125)`
 
 As we are doing POWER ISA, POWER ISA compilers, toolchains and
 emulators are required.
 
 As we are doing POWER ISA, POWER ISA compilers, toolchains and
 emulators are required.

+Again, if you want to save yourself some typing, use the dev scripts.
+[install-hdl-apt-reqs](https://git.libre-soc.org/?p=dev-env-setup.git;a=blob;f=install-hdl-apt-reqs;hb=HEAD)
+script will install the qemu;
+[ppc64-gdb-gcc](https://git.libre-soc.org/?p=dev-env-setup.git;a=blob;f=ppc64-gdb-gcc;hb=HEAD)
+script will install the toolchain and the corresponding debugger.
+The steps are provided below only for reference; when in doubt,
+consider checking and running the scripts.

 
 Install powerpc64 gcc:
 
 
 Install powerpc64 gcc:
 

-    apt-get install gcc-9-powerpc64-linux-gnu
+    apt-get install gcc-8-powerpc64-linux-gnu

 
 Install qemu:
 
     apt-get install qemu-system-ppc
 
 Install gdb from source.  Obtain the required tarball matching
 
 Install qemu:
 
     apt-get install qemu-system-ppc
 
 Install gdb from source.  Obtain the required tarball matching

-the version of gcc (9.1) from here <https://ftp.gnu.org/gnu/gdb/>,
+the version of gcc (8.3) from here <https://ftp.gnu.org/gnu/gdb/>,

 unpack it, then:
 
 unpack it, then:
 

-    cd gdb-9.1 (or other location)
+    cd gdb-8.3 (or other location)

     mkdir build
     cd build
      ../configure --srcdir=.. --host=x86_64-linux --target=powerpc64-linux-gnu
     mkdir build
     cd build
      ../configure --srcdir=.. --host=x86_64-linux --target=powerpc64-linux-gnu


@@ -454,37 +479,25 @@ unpack it, then:
 programs.  [qemu](https://www.qemu.org/) emulates processors, you can
 run programs under qemu.
 
 programs.  [qemu](https://www.qemu.org/) emulates processors, you can
 run programs under qemu.
 

-## power_instruction_analyzer (pia)
-
-We have a custom tool built in rust by programmerjake to help analyze
-the power instructions execution on *actual* hardware.
+## power-instruction-analyzer (pia)

 
 

-Note: a very recent version of pip3 is required for this to work.
+We have a custom tool built in Rust by programmerjake to help analyze
+the OpenPower instructions' execution on *actual* hardware.

 
 

-Install rust:
+Install Rust:

 
     curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
 
 
     curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
 

-Make sure we have the correct and up-to-date rust compiler (rustc):
+Make sure we have the correct and up-to-date rust compiler (rustc & cargo):

 
     rustup default stable
     rustup update
 
 
     rustup default stable
     rustup update
 

-Use rust's package manager *cargo* to install the rust-python build
-tool maturin:
-
-    cargo install maturin
-
-Install from git source by doing the following:
+Install the Python extension from git source by doing the following:

 
     git clone https://salsa.debian.org/Kazan-team/power-instruction-analyzer.git pia
     cd pia
 
     git clone https://salsa.debian.org/Kazan-team/power-instruction-analyzer.git pia
     cd pia

-    maturin build --cargo-extra-args=--features=python-extension
-    python3 -m pip install --user target/wheels/*.whl
-
-Note: an ongoing bug in maturin interferes with successful installation.
-This can be worked around by explicitly installing only the `.whl`
-files needed rather than installing everything (`\*.whl`).
+    ./libre-soc-install.sh

 
 ## Chips4Makers JTAG
 
 
 ## Chips4Makers JTAG
 


@@ -524,16 +537,58 @@ source hardware computer bus intended to let the parts of an integrated
 circuit communicate with each other.
 
 ## Coriolis2
 circuit communicate with each other.
 
 ## Coriolis2

-

 See [[HDL_workflow/coriolis2]] page, for those people doing layout work.
 
 ## Nextpnr
 
 A portable FPGA place and route tool.
 
 See [[HDL_workflow/coriolis2]] page, for those people doing layout work.
 
 ## Nextpnr
 
 A portable FPGA place and route tool.
 

-See [[HDL_workflow/nextpnr]] page for installation instructions of nextpnr with ECP5 support for Lattice FPGA ECP5 series.
+See [[HDL_workflow/nextpnr]] page for installation instructions of nextpnr with ECP5 support for Lattice FPGA ECP5 series.  Also see
+[[HDL_workflow/ECP5_FPGA]] for connecting up to JTAG with a ULX3S
+and the Lattice VERSA_ECP5.
+
+## Verilator
+
+The fastest Verilog and SystemVerilog simulator. It compiles Verilog to C++ or SystemC.
+
+Advise use only v4.106 at the moment.
+
+See [[HDL_workflow/verilator]] page for installation instructions.
+
+## GHDL
+
+GHDL is a shorthand for G Hardware Design Language. It is a VHDL analyzer, compiler, simulator and (experimental) synthesizer that can process (nearly) any VHDL design.
+
+VHDL is an acronym for Very High Speed Integrated Circuit (VHSIC) Hardware Description Language (HDL), which is a programming language used to describe a logic circuit by function, data flow behavior, or structure.
+
+Unlike some other simulators, GHDL is a compiler: it directly translates a VHDL file to machine code, without using an intermediary language such as C or C++. Therefore, the compiled code should be faster and the analysis time should be shorter than with a compiler using an intermediary language.
+
+GHDL aims at implementing VHDL as defined by IEEE 1076. It supports the 1987, 1993 and 2002 revisions and, partially, 2008. PSL is also partially supported.

 
 

-# Registering for git repository access
+See [[HDL_workflow/ghdl]] page for installation instructions.
+
+## Icarus Verilog
+
+Icarus Verilog is a Verilog simulation and synthesis tool. It operates as a compiler, compiling source code written in Verilog (IEEE-1364) into some target format.
+
+See [[HDL_workflow/iverilog]] page for installation instructions.
+
+## Cocotb
+
+cocotb is a COroutine based COsimulation TestBench environment for verifying VHDL and SystemVerilog RTL using Python.
+
+See [[HDL_workflow/cocotb]] page for installation instructions.
+
+## Symbiflow
+
+A fully open source toolchain for the development of FPGAs. Currently it targets Xilinx 7-series, Lattice iCE40 and ECP5, Quicklogic EOS S3.
+
+Needed for the Arty A7 100t Digilent FPGA board.
+
+See [[HDL_workflow/symbiflow]] for installation instructions
+and dependencies.
+
+# Registering for git repository access<a name="gitolite3_access"></a>

 
 After going through the onboarding process and having agreed to take
 responsibility for certain tasks, ask on the mailing list for git
 
 After going through the onboarding process and having agreed to take
 responsibility for certain tasks, ask on the mailing list for git


@@ -549,6 +604,14 @@ Create a file `~/.ssh/config` with the following lines:
     Host git.libre-soc.org
     Port 922
 
     Host git.libre-soc.org
     Port 922
 

+Test that you have access with this command:
+
+    ssh -v -p922 gitolite3@git.libre-soc.org
+
+Please note: **DO NOT TYPE A PASSWORD** - the server gets hit by a lot of
+port-scanning, and detection of password failures are used to instantly
+ban IP addresses.
+

 Wait for the Project Admin to confirm that the ssh key has been added
 to the required repositories.  Once confirmed, you can clone any of the
 repos at https://git.libre-soc.org/:
 Wait for the Project Admin to confirm that the ssh key has been added
 to the required repositories.  Once confirmed, you can clone any of the
 repos at https://git.libre-soc.org/:


@@ -559,6 +622,16 @@ Alternatively, the .ssh/config can be skipped and this used:
 
      git clone ssh://gitolite3@git.libre-soc.org:922/REPONAME.git
 
 
      git clone ssh://gitolite3@git.libre-soc.org:922/REPONAME.git
 

+Note: **DO NOT ATTEMPT TO LOG IN TO THE SERVER WITH A PERSONAL ACCOUNT**.
+fail2ban is running and, due to repeated persistent port-scanning spammers
+is set up to instantly ban any unauthorised ssh access for up to two weeks.
+This keeps log file sizes down on the server (which is resource-constrained).
+If you are wondering why this is done, it's a *lot* of port-scans.
+
+Therefore, *only* ssh in to server with the gitolite3 account, *only*
+on port 922, and *only* once the systems administrator has given you
+the all-clear that the ssh key has been added.
+

 # git configuration
 
 Although there are methods online which describe how (and why) these
 # git configuration
 
 Although there are methods online which describe how (and why) these


@@ -586,14 +659,21 @@ Before running the following, install the
 dependencies.  This is easiest done with this script
 <https://git.libre-soc.org/?p=dev-env-setup.git;a=blob;f=install-hdl-apt-reqs;hb=HEAD>
 
 dependencies.  This is easiest done with this script
 <https://git.libre-soc.org/?p=dev-env-setup.git;a=blob;f=install-hdl-apt-reqs;hb=HEAD>
 

+**It is critically important to install these in STRICT order, otherwise
+pip3 interferes and performs unauthorised downloads without informing
+you of what it is doing**.
+

 * mkdir ~/src
 * cd !$
 * mkdir ~/src
 * cd !$

+* git clone gitolite3@git.libre-soc.org:nmigen.git
+* git clone gitolite3@git.libre-soc.org:c4m-jtag.git

 * git clone gitolite3@git.libre-soc.org:nmutil.git
 * git clone gitolite3@git.libre-soc.org:nmutil.git

+* git clone gitolite3@git.libre-soc.org:openpower-isa.git

 * git clone gitolite3@git.libre-soc.org:ieee754fpu.git
 * git clone gitolite3@git.libre-soc.org:nmigen-soc.git
 * git clone gitolite3@git.libre-soc.org:soc.git
 
 * git clone gitolite3@git.libre-soc.org:ieee754fpu.git
 * git clone gitolite3@git.libre-soc.org:nmigen-soc.git
 * git clone gitolite3@git.libre-soc.org:soc.git
 

-In each of these directories, in the order listed, track down the
+In each of these directories, **in the order listed***, track down the

 `setup.py` file, then, as root (`sudo bash`), run the following:
 
 * python3 setup.py develop
 `setup.py` file, then, as root (`sudo bash`), run the following:
 
 * python3 setup.py develop


@@ -604,17 +684,24 @@ for multi-user machine use however it is often just easier to get your
 own machine these days.
 
 The reason for the order is because soc depends on ieee754fpu, and
 own machine these days.
 
 The reason for the order is because soc depends on ieee754fpu, and

-ieee754fpu depends on nmutil
+ieee754fpu depends on nmutil.  If you do not follow the listed order
+pip3 will go off and download an arbitrary version without your
+consent.

 
 If "`python3 setup.py install`" is used it is a pain: edit, then
 install. edit, then install. It gets extremely tedious, hence why
 "develop" was created.
 
 
 If "`python3 setup.py install`" is used it is a pain: edit, then
 install. edit, then install. It gets extremely tedious, hence why
 "develop" was created.
 

+If you prefer you can use this script instead: of course you checked it
+in advance and accept full responsibility.
+<https://git.libre-soc.org/?p=dev-env-setup.git;a=blob;f=hdl-dev-repos;hb=HEAD>
+

 # Development Rules
 
 Team communication:
 
 # Development Rules
 
 Team communication:
 

-* new members, add yourself to the [[about_us]] page and create yourself a home page using someone else's page as a template.
+* new members, add yourself to the [[about_us]] page and create yourself
+  a home page using someone else's page as a template.

 * communicate on the mailing list or the bugtracker an intent to take
   responsibility for a particular task.
 * assign yourself as the bug's owner
 * communicate on the mailing list or the bugtracker an intent to take
   responsibility for a particular task.
 * assign yourself as the bug's owner


@@ -656,7 +743,8 @@ for actual code development
 
 * commit often. several times a day, and "git push" it.  this is
   collaboration. if something is left even overnight uncommitted and not
 
 * commit often. several times a day, and "git push" it.  this is
   collaboration. if something is left even overnight uncommitted and not

-  pushed so that other people can see it, it is a red flag.  if you find
+  pushed so that other people can see it, it is a red flag.
+* if you find

   yourself thinking "i'll commit it when it's finished" or "i don't want to
   commit something that people might criticise" *this is not collaboration*,
   it is making yourself a bottleneck.  pair-programming is supposed to help
   yourself thinking "i'll commit it when it's finished" or "i don't want to
   commit something that people might criticise" *this is not collaboration*,
   it is making yourself a bottleneck.  pair-programming is supposed to help


@@ -669,6 +757,16 @@ This is important.  "`git pull`" will merge in changes.  If you then
 arbitrarily save a file without re-loading it, you risk destroying
 other people's work.
 
 arbitrarily save a file without re-loading it, you risk destroying
 other people's work.
 

+You can avoid damaging the repositories by following some simple procedures:
+
+    run appropriate unit tests
+    git pull
+    run appropriate unit tests again (checks other people's work)
+    git diff    # and actually read and review the output
+    git status  # check for any missing files
+    git commit  # with appropriate arguments and message
+    git push    # always always always do this
+

 ### Absolutely no auto-generated output
 
 * **do not commit autogenerated output**. write a shell script and commit
 ### Absolutely no auto-generated output
 
 * **do not commit autogenerated output**. write a shell script and commit


@@ -676,10 +774,13 @@ other people's work.
   **do not** add the actual output of **any** command to the repository.
   ever.  this is really important.  even if it is a human-readable file
   rather than a binary object file.
   **do not** add the actual output of **any** command to the repository.
   ever.  this is really important.  even if it is a human-readable file
   rather than a binary object file.

-  it is very common to add pdfs (the result of running `latex2pdf`) or
+* it is very common to add PDFs (the result of running `latex2pdf`) or

   configure.in (the result of running `automake`), they are an absolute
   nuisance and interfere hugely with git diffs, as well as waste hard
   disk space *and* network bandwidth. don't do it.
   configure.in (the result of running `automake`), they are an absolute
   nuisance and interfere hugely with git diffs, as well as waste hard
   disk space *and* network bandwidth. don't do it.

+* do not add multi-megabyte or multi-gigabyte "test data".
+  use shell scripts and commit that, which automatically downloads the
+  "test data" from a well-known known-good reliable location instead.

 
 ### Write commands that do tasks and commit those
 
 
 ### Write commands that do tasks and commit those
 


@@ -981,10 +1082,10 @@ Find appropriate tutorials for nmigen and yosys, as well as symbiyosys.
 * Although a verilog example this is very useful to do
   <https://symbiyosys.readthedocs.io/en/latest/quickstart.html#first-step-a-simple-bmc-example>
 * This tutorial looks pretty good and will get you started
 * Although a verilog example this is very useful to do
   <https://symbiyosys.readthedocs.io/en/latest/quickstart.html#first-step-a-simple-bmc-example>
 * This tutorial looks pretty good and will get you started

-  <http://blog.lambdaconcept.com/doku.php?id=nmigen:nmigen_install> and
-  walks not just through simulation, it takes you through using gtkwave
-  as well.
+  <https://web.archive.org/web/20210123052724/http://blog.lambdaconcept.com/doku.php?id=nmigen:nmigen_install>
+  and walks not just through simulation, it takes you through using
+  gtkwave as well.

 * There exist several nmigen examples which are also executable
 * There exist several nmigen examples which are also executable

-  <https://github.com/m-labs/nmigen/tree/master/examples/> exactly as
+  <https://github.com/nmigen/nmigen/tree/master/examples/> exactly as

   described in the above tutorial (python3 filename.py -h)
   described in the above tutorial (python3 filename.py -h)

-
+* More nmigen tutorials at [[learning_nmigen]]