HDL_workflow.mdwn

   1 # HDL workflow
   2
   3 This section describes the workflow for developing the LibreSoC.  We use nmigen, yosys and symbiyosys, and this page is intended not just to help you get set up, it is intended to help advise you of some tricks and practices that will help you become effective team contributors.
   4
   5 It is particularly important to bear in mind that we are not just "developing code", here: we are creating a "lasting legacy educational resource" for other people to learn from, and for businesses and students alike to be able to use, learn from and augment for their own purposes.
   6
   7 It is also important to appreciate and respect that we are funded under NLNet's Privacy and Enhanced Trust Programme <http://nlnet.nl/PET>. Full transparency, readability, documentation, effective team communication and formal mathematical proofs for all code at all levels is therefore paramount.
   8
   9 # Hardware
  10
  11 RAM is the biggest requirement. Minimum 16GB, the more the better (32 or 64GB starts to reach "acceptable" levels.  Disk space is not hugely critical: 256GB SSD should be more than adequate.  Simulations and FPGA compilations however are where raw processing power is a must.  High end Graphics Cards are nonessential.
  12
  13 What is particularly useful is to have not only hi-res screens (curved is *strongly* recommended if the LCD is over 24in wide, to avoid eyeballs going "prism" through longterm use), and to have several of them: the more the better.  Either a DisplayLink UD160A (or more modern variant) or simply using a second (lower spec hardware) machine is really effective.
  14
  15 Also it is really recommended to have a UHD monitor (4k - 3840x2160), or at least 2560x1200.  If given a choice, 4:3 aspect ratio is better than 16:9 particularly when using several of them. However, caveat (details below): please when editing do not assume that everyone will have access to such high resolution screens.
  16
  17 # Operating System
  18
  19 First install and become familiar with Debian (ubuntu if you absolutely must) for standardisation cross-team and so that toolchain installation is greatly simplified.  yosys in particular warns that trying to use Windows, *BSD or MacOS will get you into a world of pain.
  20
  21 Only a basic GUI desktop is necessary: fvwm2, xfce4, lxde are perfectly sufficient (alongside wicd-gtk for network management). Other more complex desktops can be used however may consume greater resources.
  22
  23 # editors and editing
  24
  25 Whilst this is often a personal choice, the fact that many editors are GUI based and run fullscreen with the entire right hand side *and* middle *and* the majority of the left side of the hi-res screen entirely unused and bereft of text leaves experienced developers both amused and puzzled.
  26
  27 At the point where such fullscreen users commit code with line lengths well over 160 characters, that amusement quickly evaporates.
  28
  29 Where the problems occur with fullscreen editor usage is when a project is split into dozens if not hundreds of small files (as this one is). At that point it becomes pretty much essential to have as many as six to eight files open *and on-screen* at once, without overlaps i.e. not in hidden tabs, next to at least two if not three additional free and clear terminals into which commands are regularly and routinely  typed (make, git commit, nosetests3 etc).
  30
  31 (hint/tip: fvwm2 set up with "mouse-over to raise focus, rather than additionally requiring a mouseclick, can save a huge amount of cumulative development time here, switching between editor terminal(s) and the command terminals).
  32
  33 Once this becomes necessary, it it turn implies that having greater than 80 chars per line - and running editors fullscreen -is a severe hindance to an essential *and highly effective* workflow technique.
  34
  35 Additionally, care should be taken to respect that not everyone will have 200+ column editor windows. They may only have a 1280 x 800 laptop which barely fits 2 80x53 xterms side by side.  Consequently, having excessively long functions is also a hindrance to others, as such developers with limited screen resources would need to continuously page-up and page-down to read the code even of a single function, in full.
  36
  37 This helps explain in part, below, why compliance with pep8 is enforced, including its 80 character limit.  In short: not everyone has the same "modern" GUI workflow or has access to the same computing resources as you, so please do respect that.
  38
  39 # Software prerequisites
  40
  41 Whilst many resources online advocate "sudo" in front of all root-level commands below, this quickly becomes tiresome. run "sudo bash", get a root prompt, and save yourself some typing.
  42
  43 * sudo bash
  44 * apt-get install vim exuberant-ctags
  45 * apt-get install build-essential
  46 * apt-get install git python3.7 python3.7-dev python-nosetest3
  47 * apt-get install graphviz xdot
  48 * return to user prompt (ctrl-d)
  49
  50 ## yosys
  51
  52 Follow the source code (git clone) instructions here: <http://www.clifford.at/yosys/download.html>
  53
  54 Do not try to use a fixed revision (currently 0.9), nmigen is evolving and frequently interacts with yosys
  55
  56 ## symbiyosys
  57
  58 Follow the instructions here: <https://symbiyosys.readthedocs.io/en/latest/quickstart.html#installing>
  59
  60 You do not have to install all of those (avy, boolector can be left out if desired) however the more that are installed the more effective the formal proof scripts will be (less resource utilisation in certain circumstances).
  61
  62 ## nmigen
  63
  64 nmigen may be installed as follows:
  65
  66 * mkdir ~/src
  67 * cd !$
  68 * git clone https://github.com/m-labs/nmigen.git
  69 * cd nmigen
  70 * sudo bash
  71 * python3 setup.py develop
  72 * ctrl-d
  73
  74 testing can then be carried out with "python3 setp.py test"
  75
  76 # Registering for git repository access
  77
  78 After going through the onboarding process and having agreed to take responsibility for certain tasks, ask on the mailing list for git repository access, sending in a public key (id_rsa.pub). If you do not have one generate it with ssh-keygen -t rsa.
  79
  80 NEVER SEND ANYONE THE PRIVATE KEY.  By contrast the public key, on account of being public, is perfectly fine to make... err... public.
  81
  82 Create a file ~/.ssh/config with the following lines:
  83
  84     Host git.libre-riscv.org
  85     Port 922
  86
  87 Wait for the Project Admin to confirm that the ssh key has been added to the required repositories.  Once confirmed,
  88 you can clone any of the repos at http://git.libre-riscv.org:
  89
  90     git clone gitolite3@git.libre-riscv.org:REPONAME.git
  91
  92 # Development Rules
  93
  94 team communication:
  95
  96 * communicate on the mailing list or the bugtracker an intent to take responsibility for a particular task.
  97 * assign yourself as the bug's owner
  98 * *keep in touch* about what you are doing, and why.
  99 * if you cannot do sonething that you gave taken responsibility for, then unless it is a dire emergency please say so, on-list. we won't mind. we'll help sort it out.
 100
 101 for actual code development:
 102
 103 * **do not commit autogenerated output**. write a shell script and commit that, or add a Makefile to run the command that generates the output, but **do not** add the actual output of **any** command to the repository.  ever.  this is really important.  even if it is a human-readable file rather than a binary object file.
 104 * if the command needed to create any given autogenerated output is not currently in the list of dependencies, first consult on the list if it is okay to make that command become a hard dependency of the project (hint: java, node.js php and .NET commands will cause delays in responses due to list participants laughing hysterically too much), and after a decision is made, document the dependency and how its source code is obtained and built.
 105 * if you find yourself repeating commands regularly, chances are high that someone else will need to run them, too. therefore, put them into a .sh shell script (and/or a Makefile) and document them at the very minimum in README or INSTALL.txt or somewhere in a docs folder as appopriate.  if unsure, ask on the mailing list for advice.
 106 * plan in advance to write not just code but a full test suite for that code.  **this is not optional**. large python projects that do not have unit tests **FAIL**.
 107 * edit files making minimal *single purpose* modifications (even if it involves multiple files. Good extreme example: globally changing a function name across an entire codebase is one purpose, one commit, yet hundreds of files).
 108 * prior to committing make sure that relevant unit tests pass, or that the change is a zero-impact addition.
 109 * commit no more than 5 to 10 lines at a time, with a CLEAR message (no "added this" or "changed that").
 110 * if as you write you find that the commit message involves a *list* of changes or the word "and", then STOP. do not proceed: it is a "red flag" that the commit has not been properly broken down into separate-purpose commits. ask for advice on-list.
 111 * if it is essential to commit large amounts of code, ensure that it is **not** in use **anywhere** by any other code.
 112
 113 the reason for the above is because python is a weakly typed language.  make one tiny change at the base level of the class hierarchy and the effect may be disastrous.
 114
 115 therefore, if you are absolutely certain that a new addition (new file, new class, new function) is not going to have any side-effects, committing it (a large amount of code) is perfectly fine.
 116
 117 as a general rule, however, do not use this an an excuse to write code first then write unit tests as an afterthought.  write *less* code *in conjunction* with its (more basic) unit tests, instead.
 118
 119 the reason for separating out commits to single purpose only becomes obvious (and regretted if not followed) when, months later, a mistake has to be tracked down and reverted.  if the commit does not have an easy-to-find message, it cannot even be located, and once found, if the commit confuses several unrelated changes, not only the diff is larger than it should be, the reversion process becomes extremely painful.
 120
 121 * all code needs to conform to pep8.  use either pep8checker or better run autopep8.  however whenever committing whitespace changes, *make a separate commit* with a commit message "whitespace" or "autopep8 cleanup".
 122 * pep8 REQUIRES no more than 80 chars per line. this is non-negotiable. if you think you need greater than 80 chars, it *fundamentally* indicates poor code design. split the code down further into smaller classes and functions.
 123 * TBD there is a docstring checker.  at the minimum make sure to have an SPD license header, module header docstring, class docstring and function docstrings on at least non-obvious functions.
 124 * make liberal but not excessive use of comments.  describe a group of lines of code, with terse but useful comments describing the purpose, documenting any side-effects, and anything that could trip you or other developers up.  unusual coding techniques should *definitely* contain a warning.
 125 * unless they are very closely related, only have one module (one class) per file. a file only 25 lines long including imports and docstrings is perfectly fine however don't force yourself. again, if unsure, ask on-list.
 126 * *keep files short and simple*. see below as to why
 127 * create a decent directory hierarchy but do not go mad. ask for advice if unsure
 128 * please do not use "from module import *". it is extremely bad practice, causes unnecessary resource utilisation, makes code readability extremely difficult, and results in unintended side-effects.
 129 * try to keep both filenames and variable names short but not ridiculously obtuse. an interesting compromise on imports is "from ridiculousfilename import longsillyname as lsn", and to assign variables as well: "comb = m.d.comb" followed by multiple "comb += nmigen_stmt" lines is a good trick that can reduce code indentation by 6 characters without reducing clarity.
 130
 131 regarding code structure: we decided to go with small modules that are both easy to analyse, as well as fit onto a single page and be readable when displayed as a visual graph on a full UHD monitor.  this is done as follows:
 132
 133 * using the capability of nmigen (TODO crossref to example) output the module to a yosys ilang (.il) file
 134 * in a separate terminal window, run yosys
 135 * at the yosys prompt type "read_ilang modulename.il"
 136 * type "show top" and a graphviz window should appear. note that typing show, then space, then pressing the tab key twice will give a full list of submodules (one of which will be "top")
 137
 138 you can now fullsize the graphviz window and scroll around.  if it looks reasonably obvious at 100% zoom, i.e the connections can be clearly related in your mind back to the actual code (by matching the graph names against signals and modules in the original nmigen code) and the words are not tiny when zoomed out, and connections are not total incomprehensible spaghetti, then congratulations, you have well-designed code. If not, then this indicates a need to split the code further into submodules and do a bit more work.
 139
 140 The reasons for doing a proper modularisation job are several-fold:
 141
 142 * firstly, we will not be doing a full automated layout-and-hope using alliance/coriolis2, we will be doing leaf-node thru tree node half-automated half-manual layout, finally getting to the floorplan, then revising and iteratively adjusting.
 143 * secondly, examining modules at the gate level (or close to it) is just good practice.  poor design creeps in by *not* knowing what the tools are actually doing (word to experienced developers: yes, we know that the yosys graph != final netlist).
 144 * thirdly, unit testing, particularly formal proofs, is far easier on small sections of code, and complete in a reasonable time.
 145
 146 # TODO Tutorials
 147
 148 Find appropriate tutorials for nmigen and yosys, as well as symbiyosys.
 149
 150 * Although a verilog example this is very useful to do <https://symbiyosys.readthedocs.io/en/latest/quickstart.html#first-step-a-simple-bmc-example>