# HDL workflow
This section describes the workflow and some best practices for developing
-the LibreSoC hardware. We use nmigen, yosys and symbiyosys, and this
+the Libre-SOC hardware. We use nmigen, yosys and symbiyosys, and this
page is intended not just to help you get set up, it is intended to
help advise you of some tricks and practices that will help you become
effective team contributors.
# Collaboration resources
+The main message here: **use the right tool for the right job**.
+
+* mailing list: general communication and discussion.
+* bugtracker: task-orientated, goal-orientated *focussed* discussion.
+* ikiwiki: document store, information store, and (editable) main website
+* git repositories: code stores (**not binary or auto-generated output store**)
+* ftp server (<http://ftp.libre-riscv.org>): large file store.
+
+we will add an IRC channel at some point when there are enough people
+to warrant having one.
+
+note also the lack of a "forum" in the above list. this is very deliberate. forums are a serious distraction when it comes to technical heavily goal-orientated development. recent internet users may enjoy looking up the "AOL metoo postings" meme.
+
+note also the complete lack of "social platforms". if we wanted to tell everybody how much better each of us are than anyone else in the team, how many times we made a commit (look at me, look at me, i'm so clever), and how many times we went to the bathroom, we would have installed a social media based project "management" system.
+
## Main contact method: mailing list
To respect the transparency requirements, conversations need to be
using phone at airport flight soon, v. quick reply: ....")
* Always trim context but do not cut excessively to the point where people
cannot follow the discussion. Especially do not cut the attribution
- ("On monday xxx wrote")
+ ("On monday xxx wrote") of something that you are actually replying
+ to.
* Use inline replies i.e. reply at the point in the relevant part of
the conversation, as if you were actually having a conversation.
* Follow standard IETF reply formatting, using ">" for cascaded
the link. Which should not require any kind of login to access. ask the
listadmin if you don't have anywhere suitable: FTP access can be arranged.
+### Actionable items from mailing list
+
If discussions result in any actionable items, it is important not to
lose track of them. Create a bugreport, find the discussion in the
archives <http://lists.libre-riscv.org/pipermail/libre-riscv-dev/>,
and put the link actually in the bugtracker as one of the comments.
-At some point it may become better to use <http://bugs.libre-riscv.org>
+At some point in any discussion, the sudden realisation may dawn on one
+or more people that this is an "actionable" discussion. at that point
+it may become better to use <http://bugs.libre-riscv.org>
itself to continue the discussion rather than to keep on dropping copies
of links into the bugtracker. The bugtracker sends copies of comments
*to* the list however this is 'one-way' (note from lkcl: because this
involves running an automated perl script from email, on every email,
on the server, that is a high security risk, and i'm not doing it. sorry.)
+### Mailing list != editable document store
+
Also, please do not use the mailing list as an "information or document
store or poor-man's editor". We have the wiki for that. Edit a page and
tell people what you did (summarise rather than drop the entire contents
source tree browser and post that (in the bugtracker or mailing list)
See <http://git.libre-riscv.org>
+### gmail "spam"ifying the list
+
+see <https://blog.kittycooper.com/2014/05/keeping-my-mailing-list-emails-out-of-gmails-spam-folder/>
+
+basically it is possible to select any message from the list, create a "filter" (under "More"),
+and, on the 2nd dialog box, click the "never send this to Spam" option.
+
## Bugtracker
bugzilla. old and highly effective. sign up in the usual way. any
gitweb is provided which does a decent job. <http://git.libre-riscv.org>
+## ftp server
+
+<http://ftp.libre-riscv.org> is available for storing large files
+that do not belong in a git repository, if we have (or ever need)
+any. images (etc.) if small and appropriate should go into the
+wiki, however .tgz archives (etc.) and, at some point, binaries,
+should be on the ftp server.
+
+ask on the list if you have a file that belongs on the ftp server.
+
## server
as an aside: all this is "old school" and run on a single core 512MB
mythic-beasts and means that the project is in no way dependent on anyone
else - not microsoft, not google, not facebook, not amazon.
-we tried gitlab. it didn't go well.
+we tried gitlab. it didn't go well. please don't ask to replace the
+above extremely resource-efficient services with it.
# Hardware
hidden tabs, next to at least two if not three additional free and clear
terminals into which commands are regularly and routinely typed (make,
git commit, nosetests3 etc). Illustrated with the following 3840x2160
-screenshot (click to view full image):
+screenshot (click to view full image), where *every one* of those 80x70
+xterm windows is *relevant to the task at hand*.
-[[!img defaults size=640x 2020-01-24_11-56.png]]
+[[!img 2020-01-24_11-56.png size=640x ]]
(hint/tip: fvwm2 set up with "mouse-over to raise focus, rather than
additionally requiring a mouseclick, can save a huge amount of cumulative
These are a test suite dependency for the ieee754fpu library, and
will be changed in the future to use Jacob's algorithmic numeric
-library. In the meantime the README describing the process is here:
-<https://git.libre-riscv.org/?p=ieee754fpu.git;a=blob;f=README.md;h=d219864a341e4b656680de476e385b6a7f70fb9b;hb=HEAD>
+library. In the meantime, sfpy can be built as follows:
+
+ git clone --recursive https://github.com/billzorn/sfpy.git
+ cd sfpy
+ cd SoftPosit
+ git apply ../softposit_sfpy_build.patch
+ git apply /path/to/ieee754fpu/SoftPosit.patch
+ cd ../berkely-softfloat-3
+ # Note: Do not apply the patch included in sfpy for berkely-softfloat,
+ # it contains the same changes as this one
+ git apply /path/to/ieee754fpu/berkeley-softfloat.patch
+ cd ..
+
+ # install dependencies
+ python3 -m venv .env
+ . .env/bin/activate
+ pip3 install --upgrade -r requirements.txt
+
+ # build
+ make lib -j8
+ make cython
+ make inplace -j8
+ make wheel
+
+ # install
+ deactivate # deactivates venv, optional
+ pip3 install dist/sfpy*.whl
+
+You can test your installation by doing the following:
+
+ python3
+ >>> from sfpy import *
+ >>> Posit8(1.3)
+
+It should print out `Posit8(1.3125)`
+
+## Coriolis2
+
+See [[coriolis2]] page, for those people doing layout work.
# Registering for git repository access
git clone gitolite3@git.libre-riscv.org:REPONAME.git
+Alternatively, the .ssh/config can be skipped and this used:
+
+ git clone ssh://gitolite3@git.libre-riscv.org:922/REPONAME.git
+
+# git configuration
+
+Although there are methods online which describe how (and why) these
+settings are normally done, honestly it is simpler and easier to open
+~/.gitconfig and add them by hand.
+
+core.autocrlf is a good idea to ensure that anyone adding DOS-formatted
+files they don't become a pain. pull.rebase is something that is greatly
+preferred for this project because it avoids the mess of "multiple
+extra merge git tree entries", and branch.autosetuprebase=always will,
+if you want it, always ensure that a new git checkout is set up with rebase.
+
+ [core]
+ autocrlf = input
+ [push]
+ default = simple
+ [pull]
+ rebase = true
+ [branch]
+ autosetuprebase = always
+
# Checking out the HDL repositories
* mkdir ~/src
* cd !$
-* git clone gitolite3@git.libre-riscv.org:soc.git
+* git clone gitolite3@git.libre-riscv.org:nmutil.git
* git clone gitolite3@git.libre-riscv.org:ieee754fpu.git
+* git clone gitolite3@git.libre-riscv.org:soc.git
-In each of these directories, track down the setup.py file, then, as root
-(sudo bash) run the following:
+In each of these directories, in the order listed, track down the
+setup.py file, then, as root (sudo bash), run the following:
* python3 setup.py develop
for multi-user machine use however it is often just easier to get your
own machine these days.
+The reason for the order is because soc depends on ieee754fpu, and
+ieee754fpu depends on nmutil
+
If "python3 setup.py install" is used it is a pain: edit, then
install. edit, then install. It gets extremely tedious, hence why
"develop" was created.
team communication:
+* new members, add yourself to the [[about_us]] page and create yourself a home page using someone else's page as a template.
* communicate on the mailing list or the bugtracker an intent to take
responsibility for a particular task.
* assign yourself as the bug's owner
* *keep in touch* about what you are doing, and why you are doing it.
+* edit your home page regularly, particularly to track tasks so that they can be paid by NLNet.
* if you cannot do something that you have taken responsibility for,
then unless it is a dire personal emergency please say so, on-list. we
won't mind. we'll help sort it out.
as an effective organisation. It's *not* about "setting rules and meting
out punishment".
-for actual code development:
+## Coding
+
+for actual code development
+
+### Plan unit tests
* plan in advance to write not just code but a full test suite for
that code. **this is not optional**. large python projects that do not
have unit tests **FAIL** (see separate section below).
+* Prioritise writing formal proofs and a single clear unit test that is more like a "worked example".
+ We receive NLNet funds for writing formal proofs, plus they
+cover corner cases and take far less time to write
+
+### Commit tested or zero-dependent code
+
* only commit code that has been tested (or is presently unused). other
people will be depending on you, so do take care not to screw up.
not least because, as it says in the [[charter]] it will be your
responsibility to fix. that said, do not feel intimidated: ask for help
and advice, and you'll get it straight away.
+
+### Commit often
+
* commit often. several times a day, and "git push" it. this is
collaboration. if something is left even overnight uncommitted and not
pushed so that other people can see it, it is a red flag. if you find
it is making yourself a bottleneck. pair-programming is supposed to help
avoid this kind of thing however pair-programming is difficult to organise
for remote collaborative libre projects (suggestions welcomed here)
+
+### Absolutely no auto-generated output
+
* **do not commit autogenerated output**. write a shell script and commit
that, or add a Makefile to run the command that generates the output, but
**do not** add the actual output of **any** command to the repository.
ever. this is really important. even if it is a human-readable file
rather than a binary object file.
+ it is very common to add pdfs (the result of running latex2pdf) or configure.in (the result of running automake), they are an absolute nuisance and interfere hugely with git diffs, as well as waste hard disk space *and* network bandwidth. don't do it.
+
+### Write commands that do tasks and commit those
+
* if the command needed to create any given autogenerated output is not
currently in the list of known project dependencies, first consult on
the list if it is okay to make that command become a hard dependency of
the repository and document them at the very minimum in the README,
INSTALL.txt or somewhere in a docs folder as appropriate. if unsure,
ask on the mailing list for advice.
+
+### Keep commits single-purpose
+
* edit files making minimal *single purpose* modifications (even if
it involves multiple files. Good extreme example: globally changing
a function name across an entire codebase is one purpose, one commit,
yet hundreds of files. miss out one of those files, requiring multiple
commits, and it actually becomes a nuisance).
+
+### Run unit tests prior to commits
+
* prior to committing make sure that relevant unit tests pass, or that
the change is a zero-impact addition (no unit tests fail at the minimum)
+
+### Do not break existing code
+
+* keep working code working **at all times**. find ways to ensure that this is the case. examples include writing alternative classes that replace existing functionality and adding runtime optiobs to select between old and new code.
+
+### Small commits with relevant commit message
+
* commit no more than around 5 to 10 lines at a time, with a CLEAR message
(no "added this" or "changed that").
* if as you write you find that the commit message involves a *list* of
changes or the word "and", then STOP. do not proceed: it is a "red flag"
that the commit has not been properly broken down into separate-purpose
commits. ask for advice on-list on how to proceed.
+
+### Exceptions to small commit: atomic single purpose commit
+
* if it is essential to commit large amounts of code, ensure that it
is **not** in use **anywhere** by any other code. then make a *small*
(single purpose) followup commit which actually puts that code into use.
change which integrates untested code: just commit the new code (only)
and follow up the next day *after* running the full relevant unit tests.
+### Why such strict rules?
+
the reason for all the above is because python is a weakly typed language.
make one tiny change at the base level of the class hierarchy and the
effect may be disastrous.
as a general rule, however, do not use this an an excuse to write code
first then write unit tests as an afterthought. write *less* code *in
-conjunction* with its (more basic) unit tests, instead.
+conjunction* with its (more basic) unit tests, instead. then, folliw up with
+additions and improvements.
the reason for separating out commits to single purpose only becomes
obvious (and regretted if not followed) when, months later, a mistake
commit confuses several unrelated changes, not only the diff is larger
than it should be, the reversion process becomes extremely painful.
+### PEP8 format
+
* all code needs to conform to pep8. use either pep8checker or better
run autopep8. however whenever committing whitespace changes, *make a
separate commit* with a commit message "whitespace" or "autopep8 cleanup".
you think you need greater than 80 chars, it *fundamentally* indicates
poor code design. split the code down further into smaller classes
and functions.
+
+### Docstring checker
+
* TBD there is a docstring checker. at the minimum make sure to have
an SPD license header, module header docstring, class docstring and
function docstrings on at least non-obvious functions.
+
+### Clear code commenting and docstrings
+
* make liberal but not excessive use of comments. describe a group of
lines of code, with terse but useful comments describing the purpose,
documenting any side-effects, and anything that could trip you or other
developers up. unusual coding techniques should *definitely* contain
a warning.
+
+### Only one class per module (ish)
+
* unless they are very closely related, only have one module (one class)
per file. a file only 25 lines long including imports and docstrings
is perfectly fine however don't force yourself. again, if unsure,
ask on-list.
+
+### File and Directory hierarchy
+
* *keep files short and simple*. see below as to why
* create a decent directory hierarchy but do not go mad. ask for advice
if unsure
+
+### No import star!
+
* please do not use "from module import \*". it is extremely bad practice,
causes unnecessary resource utilisation, makes code readability and
tracking extremely difficult, and results in unintended side-effects.
+
+### Keep file and variables short but clear
+
* try to keep both filenames and variable names short but not ridiculously
obtuse. an interesting compromise on imports is "from ridiculousfilename
import longsillyname as lsn", and to assign variables as well: "comb =
m.d.comb" followed by multiple "comb += nmigen_stmt" lines is a good trick
that can reduce code indentation by 6 characters without reducing clarity.
+### Reasons for code structure
+
regarding code structure: we decided to go with small modules that are
both easy to analyse, as well as fit onto a single page and be readable
when displayed as a visual graph on a full UHD monitor. this is done
* thirdly, unit testing, particularly formal proofs, is far easier on
small sections of code, and complete in a reasonable time.
+## Special warning / alert to vim users!
+
+Some time around the beginning of 2019 some bright spark decided that
+an "auto-recommend-completion-of-stuff" option would be a nice, shiny
+idea to enable by default from that point onwards.
+
+This incredibly annoying "feature" results in tabs (or spaces) being
+inserted "on your behalf" when you press return on one line, for your
+"convenience" of not needing to type lots of spaces/tabs just to get
+to the same indentation level.
+
+Of course, this "feature", if you press return on one line in edit
+mode and then press "escape", leaves a bundle-of-joy extraneous
+whitespace **exactly** where you don't want it, and didn't ask for it,
+pooped all over your file.
+
+Therefore, *please*: **before** running "git commit", get into the
+habit of always running "git diff", and at the very minimum
+speed-skim the entire diff, looking for tell-tale "red squares"
+(these show up under bash diff colour-syntax-highlighting) that
+inform you that, without your knowledge or consent, vim has
+"helpfully" inserted extraneous whitespace.
+
+Remove them **before** git committing because they are not part
+of the actual desired code-modifications, and committing them
+is a major and constant distraction for reviewers about actual
+important things like "the code that actually *usefully* was
+modified for that commit"
+
+This has the useful side-effect of ensuring that, right before
+the commit, you've got the actual diff right in front of you
+in the xterm window, on which you can base the "commit message".
+
## Unit tests
This deserves its own special section. It is extremely important to
* There exist several nmigen examples which are also executable
<https://github.com/m-labs/nmigen/tree/master/examples/> exactly as
described in the above tutorial (python3 filename.py -h)
+* Robert Baruch's nmigen tutorials look really good:
+ <https://github.com/RobertBaruch/nmigen-tutorial>
projects / libreriscv.git / blobdiff