7 This section describes the workflow and some best practices for developing
8 the Libre-SOC hardware. We use nmigen, yosys and symbiyosys, and this
9 page is intended not just to help you get set up, it is intended to
10 help advise you of some tricks and practices that will help you become
11 effective team contributors.
13 It is particularly important to bear in mind that we are not just
14 "developing code", here: we are creating a "lasting legacy educational
15 resource" for other people to learn from, and for businesses and students
16 alike to be able to use, learn from and augment for their own purposes.
18 It is also important to appreciate and respect that we are funded under
19 NLNet's Privacy and Enhanced Trust Programme <http://nlnet.nl/PET>. Full
20 transparency, readability, documentation, effective team communication
21 and formal mathematical proofs for all code at all levels is therefore
24 Therefore, we need not only to be "self-sufficient" (absolutely
25 under no circumstances critically reliant on somebody else's servers
26 **or protocols**) we also need to ensure that everything (including
27 all communication such as the mailing list archives) are recorded,
28 replicable, and accessible in perpetuity. Use of slack or a "forum"
29 either actively prevents or makes that much harder.
31 # Collaboration resources
33 The main message here: **use the right tool for the right job**.
35 * mailing list: general communication and discussion.
36 * irc channel #libre-soc: real(ish)-time communication.
37 * bugtracker: task-orientated, goal-orientated *focussed* discussion.
38 * ikiwiki: document store, information store, and (editable) main website
39 * git repositories: code stores (**not binary or auto-generated output store**)
40 * ftp server (<https://ftp.libre-soc.org/>): large (temporary,
41 auto-generated) file store.
43 We will add an IRC channel at some point when there are enough people
44 to warrant having one (and it will be publicly archived)
46 Note also the lack of a "forum" in the above list. this is very
47 deliberate. forums are a serious distraction when it comes to technical
48 heavily goal-orientated development. recent internet users may enjoy
49 looking up the "AOL metoo postings" meme.
51 Note also the complete lack of "social platforms". if we wanted to tell
52 everybody how much better each of us are than anyone else in the team,
53 how many times we made a commit (look at me, look at me, i'm so clever),
54 and how many times we went to the bathroom, we would have installed a
55 social media based project "management" system.
57 ## Main contact method: mailing list
59 To respect the transparency requirements, conversations need to be
60 public and archived (i.e not skype, not telegram, not discord,
61 and anyone seriously suggesting slack will be thrown to the
62 lions). Therefore we have a mailing list. Everything goes through
63 there. <https://lists.libre-soc.org/mailman/listinfo/libre-soc-dev>
64 therefore please do google "mailing list etiquette" and at the very
65 minimum look up and understand the following:
67 * This is a technical mailing list with complex topics. Top posting
68 is completely inappropriate. Don't do it unless you have mitigating
69 circumstances, and even then please apologise and explain ("hello sorry
70 using phone at airport flight soon, v. quick reply: ....")
71 * Always trim context but do not cut excessively to the point where people
72 cannot follow the discussion. Especially do not cut the attribution
73 ("On monday xxx wrote") of something that you are actually replying
75 * Use inline replies i.e. reply at the point in the relevant part of
76 the conversation, as if you were actually having a conversation.
77 * Follow standard IETF reply formatting, using ">" for cascaded
78 indentation of other people's replies. If using gmail, please: SWITCH
79 OFF RICH TEXT EDITING.
80 * Please for god's sake do not use "my replies are in a different
81 colour". Only old and highly regarded people still using AOL are allowed
82 to get away with that (such as Mitch).
83 * Start a new topic with a relevant subject line. If an existing
84 discussion changes direction, change the subject line to reflect the
85 new topic (or start a new conversation entirely, without using the
87 * DMARC is a pain on the neck. Try to avoid GPG signed messages. sigh.
88 * Don't send massive attachments. Put them online (no, not on facebook or
89 google drive or anywhere else that demands privacy violations) and provide
90 the link. Which should not require any kind of login to access. ask the
91 listadmin if you don't have anywhere suitable: FTP access can be arranged.
93 ### Actionable items from mailing list
95 If discussions result in any actionable items, it is important not to
96 lose track of them. Create a bugreport, find the discussion in the
97 archives <https://lists.libre-soc.org/pipermail/libre-soc-dev/>,
98 and put the link actually in the bugtracker as one of the comments.
100 At some point in any discussion, the sudden realisation may dawn on one
101 or more people that this is an "actionable" discussion. at that point
102 it may become better to use <https://bugs.libre-soc.org/>
103 itself to continue the discussion rather than to keep on dropping copies
104 of links into the bugtracker. The bugtracker sends copies of comments
105 *to* the list however this is 'one-way' (note from lkcl: because this
106 involves running an automated perl script from email, on every email,
107 on the server, that is a high security risk, and i'm not doing it. sorry.)
109 ### Mailing list != editable document store
111 Also, please do not use the mailing list as an "information or document
112 store or poor-man's editor". We have the wiki for that. Edit a page and
113 tell people what you did (summarise rather than drop the entire contents
114 at the list) and include the link to the page.
116 Or, if it is more appropriate, commit a document (or source code)
117 into the relevant git repository then look up the link in the gitweb
118 source tree browser and post that (in the bugtracker or mailing list)
119 See <https://git.libre-soc.org/>
121 ### gmail "spam"ifying the list
123 See <https://blog.kittycooper.com/2014/05/keeping-my-mailing-list-emails-out-of-gmails-spam-folder/>
125 Basically it is possible to select any message from the list, create a
126 "filter" (under "More"), and, on the 2nd dialog box, click the "never
127 send this to Spam" option.
131 bugzilla. old and highly effective. sign up in the usual way. any
132 problems, ask on the list.
134 Please do not ask for the project to be transferred to github or other
135 proprietary nonfree service "because it's soooo convenient", as the
136 lions are getting wind and gout from overfeeding on that one.
140 Runs the main libre-soc.org site (including this page). effective,
141 stunningly light on resources, and uses a git repository not a database.
142 That means it can be edited offline.
144 Usual deal: register an account and you can start editing and contributing
147 Hint: to create a new page, find a suitable page that would link to it,
148 first, then put the link in of the page you want to create, as if the
149 page already exists. Save that page, and you will find a question mark
150 next to the new link you created. click that link, and it will fire up a
151 "create new page" editor.
153 Wiki pages are formatted in [[markdown|ikiwiki/markdown]] syntax.
155 Hint again: the wiki is backed by a git repository. Don't go overboard
156 but at the same time do not be afraid that you might "damage" or "lose"
157 pages. Although it would be a minor pain, the pages can always be
158 reverted or edited by the sysadmins to restore things if you get in a tiz.
160 Assistance in creating a much better theme greatly appreciated. e.g.
161 <http://www.math.cmu.edu/~gautam/sj/blog/20140720-ikiwiki-navbar.html>
165 We use git. More on this below. We also use
166 [gitolite3](https://gitolite.com/gitolite/) running on a dedicated server.
167 again, it is extremely effective and low resource utilisation. Reminder:
168 lions are involved if github is mentioned.
170 [gitweb](https://git.wiki.kernel.org/index.php/Gitweb) is provided which
171 does a decent job. <https://git.libre-soc.org/>
173 [Git](https://en.wikipedia.org/wiki/Git) does version control, ie it
174 tracks changes to files so that previous versions can be got back or
177 Checklist page [[HDL_workflow/git_checklist]]
181 <https://ftp.libre-soc.org/> is available for storing large files
182 that do not belong in a git repository, if we have (or ever need)
183 any. Images (etc.) if small and appropriate should go into the
184 wiki, however .tgz archives (etc.) and, at some point, binaries,
185 should be on the ftp server.
187 Ask on the list if you have a file that belongs on the ftp server.
191 As an aside: all this is "old school" and run on a single core 512MB
192 VM with only a 20GB HDD allocation. it costs only 8 GBP per month from
193 mythic-beasts and means that the project is in no way dependent on anyone
194 else - not microsoft, not google, not facebook, not amazon.
196 We tried [gitlab](https://about.gitlab.com/). it didn't go well. please
197 don't ask to replace the above extremely resource-efficient services
202 RAM is the biggest requirement. Minimum 16GB, the more the better (32
203 or 64GB starts to reach "acceptable" levels. Disk space is not hugely
204 critical: 256GB SSD should be more than adequate. Simulations and
205 FPGA compilations however are where raw processing power is a must.
206 High end Graphics Cards are nonessential.
208 What is particularly useful is to have hi-res screens (curved is
209 *strongly* recommended if the LCD is over 24in wide, to avoid eyeballs
210 going "prism" through long term use), and to have several of them: the
211 more the better. Either a DisplayLink UD160A (or more modern variant)
212 or simply using a second machine (lower spec hardware because it will
213 run editors) is really effective.
215 Also it is really recommended to have a UHD monitor (4k - 3840x2160),
216 or at least 2560x1200. If given a choice, 4:3 aspect ratio is better
217 than 16:9 particularly when using several of them. However, caveat
218 (details below): please when editing do not assume that everyone will
219 have access to such high resolution screens.
223 First install and become familiar with
224 [Debian](https://www.debian.org/) ([Ubuntu](https://ubuntu.com/)
226 must) for standardisation cross-team and so that toolchain installation
227 is greatly simplified. yosys in particular warns that trying to use
228 Windows, BSD or MacOS will get you into a world of pain.
230 Only a basic GUI desktop is necessary: fvwm2, xfce4, lxde are perfectly
231 sufficient (alongside wicd-gtk for network management). Other more
232 complex desktops can be used however may consume greater resources.
234 # editors and editing
236 Whilst this is often a personal choice, the fact that many editors are
237 GUI based and run full-screen with the entire right hand side *and* middle
238 *and* the majority of the left side of the hi-res screen entirely unused
239 and bereft of text leaves experienced developers both amused and puzzled.
241 At the point where such full-screen users commit code with line lengths
242 well over 160 characters, that amusement quickly evaporates.
244 Where the problems occur with full-screen editor usage is when a project
245 is split into dozens if not hundreds of small files (as this one is). At
246 that point it becomes pretty much essential to have as many as six to
247 eight files open *and on-screen* at once, without overlaps i.e. not in
248 hidden tabs, next to at least two if not three additional free and clear
249 terminals into which commands are regularly and routinely typed (make,
250 git commit, nosetests3 etc). Illustrated with the following 3840x2160
251 screenshot (click to view full image), where *every one* of those 80x70
252 xterm windows is *relevant to the task at hand*.
254 [[!img 2020-01-24_11-56.png size=640x ]]
256 (hint/tip: fvwm2 set up with "mouse-over to raise focus, rather than
257 additionally requiring a mouse click, can save a huge amount of cumulative
258 development time here, switching between editor terminal(s) and the
261 Once this becomes necessary, it it turn implies that having greater
262 than 80 chars per line - and running editors full-screen - is a severe
263 hinderance to an essential *and highly effective* workflow technique.
265 Additionally, care should be taken to respect that not everyone will have
266 200+ column editor windows and the eyesight of a hawk. They may only have
267 a 1280 x 800 laptop which barely fits two 80x53 xterms side by side.
268 Consequently, having excessively long functions is also a hindrance to
269 others, as such developers with limited screen resources would need to
270 continuously page-up and page-down to read the code even of a single
273 This helps explain in part, below, why compliance with
274 [pep8](https://pep8.org/) is enforced, including its 80 character limit.
275 In short: not everyone has the same "modern" GUI workflow or has access
276 to the same computing resources as you, so please do respect that.
278 More on this concept is
279 [here](https://www.linuxjournal.com/content/line-length-limits).
280 Note *very pointedly* that Linus Torvalds *specifically* states that
281 he does not want Linux kernel development to become the exclusive
282 domain of the "wealthy". That means **no** to assumptions about
283 access to ultra-high resolution screens.
285 # Software prerequisites<a name="software-prerequisites"></a>
287 Whilst many resources online advocate "`sudo`" in front of all root-level
288 commands below, this quickly becomes tiresome. run "`sudo bash`", get a
289 root prompt, and save yourself some typing.
292 * apt-get install vim exuberant-ctags
293 * apt-get install build-essential
294 * apt-get install git python3.7 python3.7-dev python-nosetest3
295 * apt-get install graphviz xdot gtkwave
296 * apt-get install python3-venv
297 * apt-get install python-virtualenv # this is an alternative to python3-venv
298 * apt-get install tcl-dev libreadline-dev bison flex libffi-dev iverilog
299 * return to user prompt (ctrl-d)
301 (The above assumes that you are running Debian.)
303 This will get you python3 and other tools that are
304 needed. [graphviz](https://graphviz.org/) is essential
305 for showing the interconnections between cells, and
306 [gtkwave](http://gtkwave.sourceforge.net/) is essential for debugging.
308 If you would like to save yourself a lot more typing, check out the
309 [dev-env-setup](https://git.libre-soc.org/?p=dev-env-setup.git;a=summary)
310 repository, examine the scripts there and use them to automate much of
313 If you would like just to install only the apt dependencies use
314 [install-hdl-apt-reqs](https://git.libre-soc.org/?p=dev-env-setup.git;a=blob;f=install-hdl-apt-reqs;hb=HEAD) instead.
316 This page gives more details and a step by step process : [[HDL_workflow/devscripts]]
320 Look up good tutorials on how to use git effectively. There are so many
321 it is hard to recommend one. This is however essential. If you are not
322 comfortable with git, and you let things stay that way, it will seriously
323 impede development progress.
325 If working all day you should expect to be making at least two commits per
326 hour, so should become familiar with it very quickly. If you are *not*
327 doing around 2 commits per hour, something is wrong and you should read
328 the workflow instructions below more carefully, and also ask for advice
331 Worth noting: *this project does not use branches*. All code is committed
332 to master and we *require* that it be either zero-impact additions or that
333 relevant unit tests pass 100%. This ensures that people's work does not
334 get "lost" or isolated and out of touch due to major branch diversion,
335 and that people communicate and coordinate with each other.
339 Follow the source code (git clone) instructions here, do **not** use
340 the "stable" version (do not download the tarball):
341 <http://www.clifford.at/yosys/download.html>
343 Or, alternatively, use the
344 [yosys-et-al](https://git.libre-soc.org/?p=dev-env-setup.git;a=blob;f=yosys-et-al;hb=HEAD)
345 script (which also installs symbiyosys and its dependencies)
347 Do not try to use a fixed revision of yosys (currently 0.9), nmigen is
348 evolving and frequently interacts with yosys.
350 [Yosys](http://www.clifford.at/yosys/) is a framework for Verilog RTL.
351 [Verilog](https://en.wikipedia.org/wiki/Verilog) is a hardware description
353 RTL [Register Transfer
354 Level](https://en.wikipedia.org/wiki/Register-transfer_level)
355 models how data moves between
356 [registers](https://en.wikipedia.org/wiki/Hardware_register).
360 To install follow the [instructions
361 here](https://symbiyosys.readthedocs.io/en/latest/install.html)
362 Once done look at [A simple BMC
363 example](https://symbiyosys.readthedocs.io/en/latest/quickstart.html)
365 You do not have to install all of those (avy, boolector can be left
366 out if desired) however the more that are installed the more effective
367 the formal proof scripts will be (less resource utilisation in certain
370 [SymbiYosys](https://symbiyosys.readthedocs.io/en/latest/) (sby) is a
371 front-end driver program for Yosys-based formal hardware verification
376 [nmigen](https://m-labs.hk/gateware/nmigen/) may be installed as follows:
380 * git clone https://github.com/nmigen/nmigen.git
383 * python3 setup.py develop
386 Testing can then be carried out with "python3 setup.py test"
388 nmigen is a Python toolbox for building complex digital hardware.
390 ## Softfloat and sfpy
392 These are a test suite dependency for the
393 [ieee754fpu](https://www.gaisler.com/index.php/products/ipcores/ieee754fpu)
394 library, and will be changed in the future to use Jacob's
395 [simple-soft-float](https://crates.io/crates/simple-soft-float) library.
396 In the meantime, sfpy can be built as follows:
398 git clone --recursive https://github.com/billzorn/sfpy.git
401 git apply ../softposit_sfpy_build.patch
402 git apply /path/to/ieee754fpu/SoftPosit.patch
403 cd ../berkely-softfloat-3
404 # Note: Do not apply the patch included in sfpy for berkely-softfloat,
405 # it contains the same changes as this one
406 git apply /path/to/ieee754fpu/berkeley-softfloat.patch
409 # prepare a virtual environment for building
412 # or, if you prefer the old way:
413 # virtualenv -p python3 .env
415 # install dependencies
416 source .env/bin/activate
417 pip3 install --upgrade -r requirements.txt
422 make inplace -j$(nproc)
426 deactivate # deactivates venv, optional
427 pip3 install dist/sfpy*.whl
429 You can test your installation by doing the following:
432 >>> from sfpy import *
435 It should print out `Posit8(1.3125)`
437 ## qemu, cross-compilers, gdb
439 As we are doing POWER ISA, POWER ISA compilers, toolchains and
440 emulators are required.
442 Install powerpc64 gcc:
444 apt-get install gcc-9-powerpc64-linux-gnu
448 apt-get install qemu-system-ppc
450 Install gdb from source. Obtain the required tarball matching
451 the version of gcc (9.1) from here <https://ftp.gnu.org/gnu/gdb/>,
454 cd gdb-9.1 (or other location)
457 ../configure --srcdir=.. --host=x86_64-linux --target=powerpc64-linux-gnu
461 [gdb](https://en.wikipedia.org/wiki/GNU_Debugger) lets you debug running
462 programs. [qemu](https://www.qemu.org/) emulates processors, you can
463 run programs under qemu.
465 ## power_instruction_analyzer (pia)
467 We have a custom tool built in rust by programmerjake to help analyze
468 the power instructions execution on *actual* hardware.
470 Note: a very recent version of pip3 is required for this to work.
474 curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
476 Make sure we have the correct and up-to-date rust compiler (rustc):
478 rustup default stable
481 Use rust's package manager *cargo* to install the rust-python build
484 cargo install maturin
486 Install from git source by doing the following:
488 git clone https://salsa.debian.org/Kazan-team/power-instruction-analyzer.git pia
490 maturin build --cargo-extra-args=--features=python-extension
491 python3 -m pip install --user target/wheels/*.whl
493 Note: an ongoing bug in maturin interferes with successful installation.
494 This can be worked around by explicitly installing only the `.whl`
495 files needed rather than installing everything (`\*.whl`).
499 As this is an actual ASIC, we do not rely on an FPGA's JTAG TAP
500 interface, instead require a full complete independent implementation
501 of JTAG. Staf Verhaegen has one, with a full test suite, and it is
502 superb and well-written. The Libre-SOC version includes DMI (Debug
505 git clone https://git.libre-soc.org/git/c4m-jtag.git/
507 python3 setup.py develop
509 Included is an IDCODE tap point, Wishbone Master (for direct memory read
510 and write, fully independent of the core), IOPad redirection and testing,
511 and general purpose shift register capability for any custom use.
513 We added a DMI to JTAG bridge in LibreSOC which is
514 directly connected to the core, to access registers and
515 to be able to start and stop the core and change the PC.
516 In combination with the JTAG Wishbone interface the test
517 [ASIC](https://en.wikipedia.org/wiki/Application-specific_integrated_circuit)
518 can have a bootloader uploaded directly into onboard
519 [SRAM](https://en.wikipedia.org/wiki/Static_random-access_memory) and
522 [Chips4Makers](https://chips4makers.io/) make it possible for makers
523 and hobbyists to make their own open source chips.
525 [JTAG](https://en.wikipedia.org/wiki/JTAG) (Joint Test Action Group) is
526 an industry standard for verifying designs and testing printed circuit
527 boards after manufacture.
530 bus](https://en.wikipedia.org/wiki/Wishbone_%28computer_bus%29) is an open
531 source hardware computer bus intended to let the parts of an integrated
532 circuit communicate with each other.
535 See [[HDL_workflow/coriolis2]] page, for those people doing layout work.
539 A portable FPGA place and route tool.
541 See [[HDL_workflow/nextpnr]] page for installation instructions of nextpnr with ECP5 support for Lattice FPGA ECP5 series.
545 The fastest Verilog and SystemVerilog simulator. It compiles Verilog to C++ or SystemC.
547 Advise use only v4.106 at the moment.
549 See [[HDL_workflow/verilator]] page for installation instructions.
553 GHDL is a shorthand for G Hardware Design Language. It is a VHDL analyzer, compiler, simulator and (experimental) synthesizer that can process (nearly) any VHDL design.
555 VHDL is an acronym for Very High Speed Integrated Circuit (VHSIC) Hardware Description Language (HDL), which is a programming language used to describe a logic circuit by function, data flow behavior, or structure.
557 Unlike some other simulators, GHDL is a compiler: it directly translates a VHDL file to machine code, without using an intermediary language such as C or C++. Therefore, the compiled code should be faster and the analysis time should be shorter than with a compiler using an intermediary language.
559 GHDL aims at implementing VHDL as defined by IEEE 1076. It supports the 1987, 1993 and 2002 revisions and, partially, 2008. PSL is also partially supported.
561 See [[HDL_workflow/ghdl]] page for installation instructions.
565 Icarus Verilog is a Verilog simulation and synthesis tool. It operates as a compiler, compiling source code written in Verilog (IEEE-1364) into some target format.
567 See [[HDL_workflow/iverilog]] page for installation instructions.
571 cocotb is a COroutine based COsimulation TestBench environment for verifying VHDL and SystemVerilog RTL using Python.
573 See [[HDL_workflow/cocotb]] page for installation instructions.
575 # Registering for git repository access
577 After going through the onboarding process and having agreed to take
578 responsibility for certain tasks, ask on the mailing list for git
579 repository access, sending in a public key (`id_rsa.pub`). If you do
580 not have one then generate it with `ssh-keygen -t rsa`. You will find it
583 NEVER SEND ANYONE THE PRIVATE KEY. By contrast the public key, on
584 account of being public, is perfectly fine to make... err... public.
586 Create a file `~/.ssh/config` with the following lines:
588 Host git.libre-soc.org
591 Wait for the Project Admin to confirm that the ssh key has been added
592 to the required repositories. Once confirmed, you can clone any of the
593 repos at https://git.libre-soc.org/:
595 git clone gitolite3@git.libre-soc.org:REPONAME.git
597 Alternatively, the .ssh/config can be skipped and this used:
599 git clone ssh://gitolite3@git.libre-soc.org:922/REPONAME.git
603 Although there are methods online which describe how (and why) these
604 settings are normally done, honestly it is simpler and easier to open
605 ~/.gitconfig and add them by hand.
607 core.autocrlf is a good idea to ensure that anyone adding DOS-formatted
608 files they don't become a pain. pull.rebase is something that is greatly
609 preferred for this project because it avoids the mess of "multiple
610 extra merge git tree entries", and branch.autosetuprebase=always will,
611 if you want it, always ensure that a new git checkout is set up with rebase.
620 autosetuprebase = always
622 # Checking out the HDL repositories
624 Before running the following, install the
625 dependencies. This is easiest done with this script
626 <https://git.libre-soc.org/?p=dev-env-setup.git;a=blob;f=install-hdl-apt-reqs;hb=HEAD>
630 * git clone gitolite3@git.libre-soc.org:nmigen.git
631 * git clone gitolite3@git.libre-soc.org:c4m-jtag.git
632 * git clone gitolite3@git.libre-soc.org:nmutil.git
633 * git clone gitolite3@git.libre-soc.org:openpower-isa.git
634 * git clone gitolite3@git.libre-soc.org:ieee754fpu.git
635 * git clone gitolite3@git.libre-soc.org:nmigen-soc.git
636 * git clone gitolite3@git.libre-soc.org:soc.git
638 In each of these directories, in the order listed, track down the
639 `setup.py` file, then, as root (`sudo bash`), run the following:
641 * python3 setup.py develop
643 The reason for using "develop" mode is that the code may be edited
644 in-place yet still imported "globally". There are variants on this theme
645 for multi-user machine use however it is often just easier to get your
646 own machine these days.
648 The reason for the order is because soc depends on ieee754fpu, and
649 ieee754fpu depends on nmutil
651 If "`python3 setup.py install`" is used it is a pain: edit, then
652 install. edit, then install. It gets extremely tedious, hence why
653 "develop" was created.
655 If you prefer you can use this script instead: of course you checked it
656 in advance and accept full responsibility.
657 <https://git.libre-soc.org/?p=dev-env-setup.git;a=blob;f=hdl-dev-repos;hb=HEAD>
663 * new members, add yourself to the [[about_us]] page and create yourself
664 a home page using someone else's page as a template.
665 * communicate on the mailing list or the bugtracker an intent to take
666 responsibility for a particular task.
667 * assign yourself as the bug's owner
668 * *keep in touch* about what you are doing, and why you are doing it.
669 * edit your home page regularly, particularly to track tasks so that
670 they can be paid by NLNet.
671 * if you cannot do something that you have taken responsibility for,
672 then unless it is a dire personal emergency please say so, on-list. we
673 won't mind. we'll help sort it out.
675 Regarding the above it is important that you read, understand, and agree
676 to the [[charter]] because the charter is about ensuring that we operate
677 as an effective organisation. It's *not* about "setting rules and meting
682 for actual code development
686 * plan in advance to write not just code but a full test suite for
687 that code. **this is not optional**. large python projects that do not
688 have unit tests **FAIL** (see separate section below).
689 * Prioritise writing formal proofs and a single clear unit test that is more
690 like a "worked example".
691 We receive NLNet funds for writing formal proofs, plus they
692 cover corner cases and take far less time to write
694 ### Commit tested or zero-dependent code
696 * only commit code that has been tested (or is presently unused). other
697 people will be depending on you, so do take care not to screw up.
698 not least because, as it says in the [[charter]] it will be your
699 responsibility to fix. that said, do not feel intimidated: ask for help
700 and advice, and you'll get it straight away.
704 * commit often. several times a day, and "git push" it. this is
705 collaboration. if something is left even overnight uncommitted and not
706 pushed so that other people can see it, it is a red flag.
708 yourself thinking "i'll commit it when it's finished" or "i don't want to
709 commit something that people might criticise" *this is not collaboration*,
710 it is making yourself a bottleneck. pair-programming is supposed to help
711 avoid this kind of thing however pair-programming is difficult to organise
712 for remote collaborative libre projects (suggestions welcomed here)
714 ### Enable editor auto-detection of file changes by external programs
716 This is important. "`git pull`" will merge in changes. If you then
717 arbitrarily save a file without re-loading it, you risk destroying
720 You can avoid damaging the repositories by following some simple procedures:
722 run appropriate unit tests
724 run appropriate unit tests again (checks other people's work)
725 git diff # and actually read and review the output
726 git status # check for any missing files
727 git commit # with appropriate arguments and message
728 git push # always always always do this
730 ### Absolutely no auto-generated output
732 * **do not commit autogenerated output**. write a shell script and commit
733 that, or add a `Makefile` to run the command that generates the output, but
734 **do not** add the actual output of **any** command to the repository.
735 ever. this is really important. even if it is a human-readable file
736 rather than a binary object file.
737 * it is very common to add PDFs (the result of running `latex2pdf`) or
738 configure.in (the result of running `automake`), they are an absolute
739 nuisance and interfere hugely with git diffs, as well as waste hard
740 disk space *and* network bandwidth. don't do it.
741 * do not add multi-megabyte or multi-gigabyte "test data".
742 use shell scripts and commit that, which automatically downloads the
743 "test data" from a well-known known-good reliable location instead.
745 ### Write commands that do tasks and commit those
747 * if the command needed to create any given autogenerated output is not
748 currently in the list of known project dependencies, first consult on
749 the list if it is okay to make that command become a hard dependency of
750 the project (hint: java, node.js php and .NET commands may cause delays
751 in response time due to other list participants laughing hysterically),
752 and after a decision is made, document the dependency and how its source
753 code is obtained and built (hence why it has to be discussed carefully)
754 * if you find yourself repeating commands regularly, chances are high
755 that someone else will need to run them, too. clearly this includes
756 yourself, therefore, to make everyone's lives easier including your own,
757 put them into a `.sh` shell script (and/or a `Makefile`), commit them to
758 the repository and document them at the very minimum in the README,
759 INSTALL.txt or somewhere in a docs folder as appropriate. if unsure,
760 ask on the mailing list for advice.
762 ### Keep commits single-purpose
764 * edit files making minimal *single purpose* modifications (even if
765 it involves multiple files. Good extreme example: globally changing
766 a function name across an entire codebase is one purpose, one commit,
767 yet hundreds of files. miss out one of those files, requiring multiple
768 commits, and it actually becomes a nuisance).
770 ### Run unit tests prior to commits
772 * prior to committing make sure that relevant unit tests pass, or that
773 the change is a zero-impact addition (no unit tests fail at the minimum)
775 ### Do not break existing code
777 * keep working code working **at all times**. find ways to ensure that
778 this is the case. examples include writing alternative classes that
779 replace existing functionality and adding runtime options to select
780 between old and new code.
782 ### Small commits with relevant commit message
784 * commit no more than around 5 to 10 lines at a time, with a CLEAR message
785 (no "added this" or "changed that").
786 * if as you write you find that the commit message involves a *list* of
787 changes or the word "and", then STOP. do not proceed: it is a "red flag"
788 that the commit has not been properly broken down into separate-purpose
789 commits. ask for advice on-list on how to proceed.
791 ### Exceptions to small commit: atomic single purpose commit
793 * if it is essential to commit large amounts of code, ensure that it
794 is **not** in use **anywhere** by any other code. then make a *small*
795 (single purpose) followup commit which actually puts that code into use.
797 This last rule is kinda flexible, because if you add the code *and* add
798 the unit test *and* added it into the main code *and* ran all relevant
799 unit tests on all cascade-impacted areas by that change, that's perfectly
800 fine too. however if it is the end of a day, and you need to stop and
801 do not have time to run the necessary unit tests, do *not* commit the
802 change which integrates untested code: just commit the new code (only)
803 and follow up the next day *after* running the full relevant unit tests.
805 ### Why such strict rules?
807 The reason for all the above is because python is a dynamically typed
808 language. make one tiny change at the base level of the class hierarchy
809 and the effect may be disastrous.
811 It is therefore worth reiterating: make absolutely certain that you *only*
812 commit working code or zero-impact code.
814 Therefore, if you are absolutely certain that a new addition (new file,
815 new class, new function) is not going to have any side-effects, committing
816 it (a large amount of code) is perfectly fine.
818 As a general rule, however, do not use this an an excuse to write code
819 first then write unit tests as an afterthought. write *less* code *in
820 conjunction* with its (more basic) unit tests, instead. then, folliw up with
821 additions and improvements.
823 The reason for separating out commits to single purpose only becomes
824 obvious (and regretted if not followed) when, months later, a mistake
825 has to be tracked down and reverted. if the commit does not have an
826 easy-to-find message, it cannot even be located, and once found, if the
827 commit confuses several unrelated changes, not only the diff is larger
828 than it should be, the reversion process becomes extremely painful.
832 * all code needs to conform to pep8. use either pep8checker or better
833 run autopep8. however whenever committing whitespace changes, *make a
834 separate commit* with a commit message "whitespace" or "autopep8 cleanup".
835 * pep8 REQUIRES no more than 80 chars per line. this is non-negotiable. if
836 you think you need greater than 80 chars, it *fundamentally* indicates
837 poor code design. split the code down further into smaller classes
840 ### Docstring checker
842 * TBD there is a docstring checker. at the minimum make sure to have
843 an SPD license header, module header docstring, class docstring and
844 function docstrings on at least non-obvious functions.
846 ### Clear code commenting and docstrings
848 * make liberal but not excessive use of comments. describe a group of
849 lines of code, with terse but useful comments describing the purpose,
850 documenting any side-effects, and anything that could trip you or other
851 developers up. unusual coding techniques should *definitely* contain
854 ### Only one class per module (ish)
856 * unless they are very closely related, only have one module (one class)
857 per file. a file only 25 lines long including imports and docstrings
858 is perfectly fine however don't force yourself. again, if unsure,
861 ### File and Directory hierarchy
863 * *keep files short and simple*. see below as to why
864 * create a decent directory hierarchy but do not go mad. ask for advice
869 * please do not use "from module import \*". it is extremely bad practice,
870 causes unnecessary resource utilisation, makes code readability and
871 tracking extremely difficult, and results in unintended side-effects.
873 Example: often you want to find the code from which a class was imported.
874 nirmally you go to the top of the file, check the imports, and you know
875 exactly which file has the class because of the import path. by using
876 wildcards, you have absolutely *no clue* which wildcard imported which
879 Example: sometimes you may accidentally have duplicate code maintained
880 in two or more places. editing one of them you find, puzzlingly, that
881 the code behaves in some files with the old behaviour, but in others it
882 works. after a massive amount of investigation, you find that the working
883 files happen to have a wildcard import of the newer accidental duplicate
884 class **after** the wildcard import of the older class with exactly the
885 same name. if you had used explicit imports, you would have spotted
886 the double import of the class from two separate locations, immediately.
888 Really. don't. use. wildcards.
890 ### Keep file and variables short but clear
892 * try to keep both filenames and variable names short but not ridiculously
893 obtuse. an interesting compromise on imports is "from ridiculousfilename
894 import longsillyname as lsn", and to assign variables as well: "comb =
895 m.d.comb" followed by multiple "comb += nmigen_stmt" lines is a good trick
896 that can reduce code indentation by 6 characters without reducing clarity.
898 Additionally, use comments just above an obtuse variable in order to
899 help explain what it is for. In combination with keeping the the module
900 itself short, other readers will not need to scroll back several pages
901 in order to understand the code.
903 Yes it is tempting to actually use the variables as
904 self-explanatory-comments and generally this can be extremely good
905 practice. the problem comes when the variable is so long that a function
906 with several parameters csn no longer fit on a single line, and takes
907 up five to ten lines rather than one or two. at that point, the length
908 of the code is adversely affected and thus so is readability by forcing
909 readers to scroll through reams of pages.
911 It is a tricky balance: basically use your common sense, or just ask
912 someone else, "can you understand this code?"
914 ### Reasons for code structure
916 Regarding code structure: we decided to go with small modules that are
917 both easy to analyse, as well as fit onto a single page and be readable
918 when displayed as a visual graph on a full UHD monitor. this is done
921 * using the capability of nmigen (TODO crossref to example) output the
922 module to a yosys ilang (.il) file
923 * in a separate terminal window, run yosys
924 * at the yosys prompt type "read_ilang modulename.il"
925 * type "show top" and a graphviz window should appear. note that typing
926 show, then space, then pressing the tab key twice will give a full list
927 of submodules (one of which will be "top")
929 You can now fullsize the graphviz window and scroll around. if it looks
930 reasonably obvious at 100% zoom, i.e the connections can be clearly
931 related in your mind back to the actual code (by matching the graph names
932 against signals and modules in the original nmigen code) and the words are
933 not tiny when zoomed out, and connections are not total incomprehensible
934 spaghetti, then congratulations, you have well-designed code. If not,
935 then this indicates a need to split the code further into submodules
936 and do a bit more work.
938 The reasons for doing a proper modularisation job are several-fold:
940 * firstly, we will not be doing a full automated layout-and-hope
941 using alliance/coriolis2, we will be doing leaf-node thru tree node
942 half-automated half-manual layout, finally getting to the floorplan,
943 then revising and iteratively adjusting.
944 * secondly, examining modules at the gate level (or close to it) is just
945 good practice. poor design creeps in by *not* knowing what the tools
946 are actually doing (word to experienced developers: yes, we know that
947 the yosys graph != final netlist).
948 * thirdly, unit testing, particularly formal proofs, is far easier on
949 small sections of code, and complete in a reasonable time.
951 ## Special warning / alert to vim users!
953 Some time around the beginning of 2019 some bright spark decided that
954 an "auto-recommend-completion-of-stuff" option would be a nice, shiny
955 idea to enable by default from that point onwards.
957 This incredibly annoying "feature" results in tabs (or spaces) being
958 inserted "on your behalf" when you press return on one line, for your
959 "convenience" of not needing to type lots of spaces/tabs just to get
960 to the same indentation level.
962 Of course, this "feature", if you press return on one line in edit
963 mode and then press "escape", leaves a bundle-of-joy extraneous
964 whitespace **exactly** where you don't want it, and didn't ask for it,
965 pooped all over your file.
967 Therefore, *please*: **before** running "git commit", get into the
968 habit of always running "git diff", and at the very minimum
969 speed-skim the entire diff, looking for tell-tale "red squares"
970 (these show up under bash diff colour-syntax-highlighting) that
971 inform you that, without your knowledge or consent, vim has
972 "helpfully" inserted extraneous whitespace.
974 Remove them **before** git committing because they are not part
975 of the actual desired code-modifications, and committing them
976 is a major and constant distraction for reviewers about actual
977 important things like "the code that actually *usefully* was
978 modified for that commit"
980 This has the useful side-effect of ensuring that, right before
981 the commit, you've got the actual diff right in front of you
982 in the xterm window, on which you can base the "commit message".
986 For further reading, see the wikipedia page on
987 [Test-driven Development](https://en.wikipedia.org/wiki/Test-driven_development)
989 This deserves its own special section. It is extremely important to
990 appreciate that without unit tests, python projects are simply unviable.
991 Python itself has over 25,000 individual tests.
993 This can be quite overwhelming to a beginner developer, especially one
994 used to writing scripts of only 100 lines in length.
996 Thanks to Samuel Falvo we learned that writing unit tests as a formal
997 proof is not only shorter, it's also far more readable and also, if
998 written properly, provides 100% coverage of corner-cases that would
999 otherwise be overlooked or require tens to hundreds of thousands of
1002 No this is not a joke or even remotely hypothetical, this is an actual
1005 The ieee754fpu requires several hundreds of thousands of tests to be
1006 run (currently needing several days to run them all), and even then we
1007 cannot be absolutely certain that all possible combinations of input have
1008 been tested. With 2^128 permutations to try with 2 64 bit FP numbers
1009 it is simply impossible to even try.
1011 This is where formal proofs come into play.
1013 Samuel illustrated to us that "ordinary" unit tests can then be written
1014 to *augment* the formal ones, serving the purpose of illustrating how
1015 to use the module, more than anything.
1017 However it is appreciated that writing formal proofs is a bit of a
1018 black art. This is where team collaboration particularly kicks in,
1019 so if you need help, ask on the mailing list.
1021 ## Don't comment out unit tests: add them first (as failures) and fix code later
1023 Unit tests serve an additional critical purpose of keeping track of code
1024 that needs to be written. In many cases, you write the unit test *first*,
1025 despite knowing full well that the code doesn't even exist or is completely
1026 broken. The unit test then serves as a constant and important reminder
1027 to actually fix (or write) the code.
1029 Therefore, *do not* comment out unit tests just because they "don't work".
1030 If you absolutely must stop a unit test from running, **do not delete it**.
1031 Simply mark it with an appropriate
1032 ["skip" decorator](https://docs.python.org/3/library/unittest.html#skipping-tests-and-expected-failures),
1033 preferably with a link to a URL in the [bugtracker](https://bugs.libre-soc.org/)
1034 with further details as to why the unit test should not be run.
1038 Find appropriate tutorials for nmigen and yosys, as well as symbiyosys.
1040 * Robert Baruch's nmigen tutorials look really good:
1041 <https://github.com/RobertBaruch/nmigen-tutorial>
1042 * Although a verilog example this is very useful to do
1043 <https://symbiyosys.readthedocs.io/en/latest/quickstart.html#first-step-a-simple-bmc-example>
1044 * This tutorial looks pretty good and will get you started
1045 <http://blog.lambdaconcept.com/doku.php?id=nmigen:nmigen_install> and
1046 walks not just through simulation, it takes you through using gtkwave
1048 * There exist several nmigen examples which are also executable
1049 <https://github.com/m-labs/nmigen/tree/master/examples/> exactly as
1050 described in the above tutorial (python3 filename.py -h)