3 This section describes the workflow and some best practices for developing
4 the Libre-SOC hardware. We use nmigen, yosys and symbiyosys, and this
5 page is intended not just to help you get set up, it is intended to
6 help advise you of some tricks and practices that will help you become
7 effective team contributors.
9 It is particularly important to bear in mind that we are not just
10 "developing code", here: we are creating a "lasting legacy educational
11 resource" for other people to learn from, and for businesses and students
12 alike to be able to use, learn from and augment for their own purposes.
14 It is also important to appreciate and respect that we are funded under
15 NLNet's Privacy and Enhanced Trust Programme <http://nlnet.nl/PET>. Full
16 transparency, readability, documentation, effective team communication
17 and formal mathematical proofs for all code at all levels is therefore
20 Therefore, we need not only to be "self-sufficient" (absolutely under no circumstances critically reliant on somebody else's servers **or protocols**) we also need to ensure that everything (including all communication such as the mailing list archives) are recorded, replicable, and accessible in perpetuity. Use of slack or a "forum" either actively prevents or makes that much harder.
22 # Collaboration resources
24 The main message here: **use the right tool for the right job**.
26 * mailing list: general communication and discussion.
27 * irc channel #libre-soc: real(ish)-time communication.
28 * bugtracker: task-orientated, goal-orientated *focussed* discussion.
29 * ikiwiki: document store, information store, and (editable) main website
30 * git repositories: code stores (**not binary or auto-generated output store**)
31 * ftp server (<https://ftp.libre-soc.org/>): large (temporary,
32 auto-generated) file store.
34 we will add an IRC channel at some point when there are enough people
35 to warrant having one (and it will be publicly archived)
37 note also the lack of a "forum" in the above list. this is very deliberate. forums are a serious distraction when it comes to technical heavily goal-orientated development. recent internet users may enjoy looking up the "AOL metoo postings" meme.
39 note also the complete lack of "social platforms". if we wanted to tell everybody how much better each of us are than anyone else in the team, how many times we made a commit (look at me, look at me, i'm so clever), and how many times we went to the bathroom, we would have installed a social media based project "management" system.
41 ## Main contact method: mailing list
43 To respect the transparency requirements, conversations need to be
44 public and archived (i.e not skype, not telegram, not discord,
45 and anyone seriously suggesting slack will be thrown to the
46 lions). Therefore we have a mailing list. Everything goes through
47 there. <https://lists.libre-soc.org/mailman/listinfo/libre-soc-dev>
48 therefore please do google "mailing list etiquette" and at the very
49 minimum look up and understand the following:
51 * This is a technical mailing list with complex topics. Top posting
52 is completely inappropriate. Don't do it unless you have mitigating
53 circumstances, and even then please apologise and explain ("hello sorry
54 using phone at airport flight soon, v. quick reply: ....")
55 * Always trim context but do not cut excessively to the point where people
56 cannot follow the discussion. Especially do not cut the attribution
57 ("On monday xxx wrote") of something that you are actually replying
59 * Use inline replies i.e. reply at the point in the relevant part of
60 the conversation, as if you were actually having a conversation.
61 * Follow standard IETF reply formatting, using ">" for cascaded
62 indentation of other people's replies. If using gmail, please: SWITCH
63 OFF RICH TEXT EDITING.
64 * Please for god's sake do not use "my replies are in a different
65 colour". Only old and highly regarded people still using AOL are allowed
66 to get away with that (such as Mitch).
67 * Start a new topic with a relevant subject line. If an existing
68 discussion changes direction, change the subject line to reflect the
69 new topic (or start a new conversation entirely, without using the
71 * DMARC is a pain on the neck. Try to avoid GPG signed messages. sigh.
72 * Don't send massive attachments. Put them online (no, not on facebook or
73 google drive or anywhere else that demands privacy violations) and provide
74 the link. Which should not require any kind of login to access. ask the
75 listadmin if you don't have anywhere suitable: FTP access can be arranged.
77 ### Actionable items from mailing list
79 If discussions result in any actionable items, it is important not to
80 lose track of them. Create a bugreport, find the discussion in the
81 archives <https://lists.libre-soc.org/pipermail/libre-soc-dev/>,
82 and put the link actually in the bugtracker as one of the comments.
84 At some point in any discussion, the sudden realisation may dawn on one
85 or more people that this is an "actionable" discussion. at that point
86 it may become better to use <https://bugs.libre-soc.org/>
87 itself to continue the discussion rather than to keep on dropping copies
88 of links into the bugtracker. The bugtracker sends copies of comments
89 *to* the list however this is 'one-way' (note from lkcl: because this
90 involves running an automated perl script from email, on every email,
91 on the server, that is a high security risk, and i'm not doing it. sorry.)
93 ### Mailing list != editable document store
95 Also, please do not use the mailing list as an "information or document
96 store or poor-man's editor". We have the wiki for that. Edit a page and
97 tell people what you did (summarise rather than drop the entire contents
98 at the list) and include the link to the page.
100 Or, if it is more appropriate, commit a document (or source code)
101 into the relevant git repository then look up the link in the gitweb
102 source tree browser and post that (in the bugtracker or mailing list)
103 See <https://git.libre-soc.org/>
105 ### gmail "spam"ifying the list
107 see <https://blog.kittycooper.com/2014/05/keeping-my-mailing-list-emails-out-of-gmails-spam-folder/>
109 basically it is possible to select any message from the list, create a "filter" (under "More"),
110 and, on the 2nd dialog box, click the "never send this to Spam" option.
114 bugzilla. old and highly effective. sign up in the usual way. any
115 problems, ask on the list.
117 Please do not ask for the project to be transferred to github or other
118 proprietary nonfree service "because it's soooo convenient", as the
119 lions are getting wind and gout from overfeeding on that one.
123 Runs the main libre-soc.org site (including this page). effective,
124 stunningly light on resources, and uses a git repository not a database.
125 That means it can be edited offline.
127 Usual deal: register an account and you can start editing and contributing
130 Hint: to create a new page, find a suitable page that would link to it,
131 first, then put the link in of the page you want to create, as if the
132 page already exists. Save that page, and you will find a questionmark
133 next to the new link you created. click that link, and it will fire up a
134 "create new page" editor.
136 Hint again: the wiki is backed by a git repository. Don't go overboard
137 but at the same time do not be afraid that you might "damage" or "lose"
138 pages. Although it would be a minor pain, the pages can always be
139 reverted or edited by the sysadmins to restore things if you get in a tiz.
141 Assistance in creating a much better theme greatly appreciated. e.g.
142 <http://www.math.cmu.edu/~gautam/sj/blog/20140720-ikiwiki-navbar.html>
146 we use git. more on this below. we also use gitolite3 running on a
147 dedicated server. again, it is extremely effective and low resource
148 utilisation. reminder: lions are involved if github is mentioned.
150 gitweb is provided which does a decent job. <https://git.libre-soc.org/>
154 <https://ftp.libre-soc.org/ is available for storing large files
155 that do not belong in a git repository, if we have (or ever need)
156 any. images (etc.) if small and appropriate should go into the
157 wiki, however .tgz archives (etc.) and, at some point, binaries,
158 should be on the ftp server.
160 ask on the list if you have a file that belongs on the ftp server.
164 as an aside: all this is "old school" and run on a single core 512MB
165 VM with only a 20GB HDD allocation. it costs only 8 GBP per month from
166 mythic-beasts and means that the project is in no way dependent on anyone
167 else - not microsoft, not google, not facebook, not amazon.
169 we tried gitlab. it didn't go well. please don't ask to replace the
170 above extremely resource-efficient services with it.
174 RAM is the biggest requirement. Minimum 16GB, the more the better (32
175 or 64GB starts to reach "acceptable" levels. Disk space is not hugely
176 critical: 256GB SSD should be more than adequate. Simulations and
177 FPGA compilations however are where raw processing power is a must.
178 High end Graphics Cards are nonessential.
180 What is particularly useful is to have hi-res screens (curved is
181 *strongly* recommended if the LCD is over 24in wide, to avoid eyeballs
182 going "prism" through longterm use), and to have several of them: the
183 more the better. Either a DisplayLink UD160A (or more modern variant)
184 or simply using a second machine (lower spec hardware because it will
185 run editors) is really effective.
187 Also it is really recommended to have a UHD monitor (4k - 3840x2160),
188 or at least 2560x1200. If given a choice, 4:3 aspect ratio is better
189 than 16:9 particularly when using several of them. However, caveat
190 (details below): please when editing do not assume that everyone will
191 have access to such high resolution screens.
195 First install and become familiar with Debian (ubuntu if you absolutely
196 must) for standardisation cross-team and so that toolchain installation
197 is greatly simplified. yosys in particular warns that trying to use
198 Windows, BSD or MacOS will get you into a world of pain.
200 Only a basic GUI desktop is necessary: fvwm2, xfce4, lxde are perfectly
201 sufficient (alongside wicd-gtk for network management). Other more
202 complex desktops can be used however may consume greater resources.
204 # editors and editing
206 Whilst this is often a personal choice, the fact that many editors are
207 GUI based and run fullscreen with the entire right hand side *and* middle
208 *and* the majority of the left side of the hi-res screen entirely unused
209 and bereft of text leaves experienced developers both amused and puzzled.
211 At the point where such fullscreen users commit code with line lengths
212 well over 160 characters, that amusement quickly evaporates.
214 Where the problems occur with fullscreen editor usage is when a project
215 is split into dozens if not hundreds of small files (as this one is). At
216 that point it becomes pretty much essential to have as many as six to
217 eight files open *and on-screen* at once, without overlaps i.e. not in
218 hidden tabs, next to at least two if not three additional free and clear
219 terminals into which commands are regularly and routinely typed (make,
220 git commit, nosetests3 etc). Illustrated with the following 3840x2160
221 screenshot (click to view full image), where *every one* of those 80x70
222 xterm windows is *relevant to the task at hand*.
224 [[!img 2020-01-24_11-56.png size=640x ]]
226 (hint/tip: fvwm2 set up with "mouse-over to raise focus, rather than
227 additionally requiring a mouseclick, can save a huge amount of cumulative
228 development time here, switching between editor terminal(s) and the
231 Once this becomes necessary, it it turn implies that having greater
232 than 80 chars per line - and running editors fullscreen - is a severe
233 hindance to an essential *and highly effective* workflow technique.
235 Additionally, care should be taken to respect that not everyone will have
236 200+ column editor windows and the eyesight of a hawk. They may only have
237 a 1280 x 800 laptop which barely fits two 80x53 xterms side by side.
238 Consequently, having excessively long functions is also a hindrance to
239 others, as such developers with limited screen resources would need to
240 continuously page-up and page-down to read the code even of a single
243 This helps explain in part, below, why compliance with pep8 is enforced,
244 including its 80 character limit. In short: not everyone has the same
245 "modern" GUI workflow or has access to the same computing resources as
246 you, so please do respect that.
248 More on this concept is
249 [here](https://www.linuxjournal.com/content/line-length-limits).
250 Note *very pointedly* that Linus Torvalds *specifically* states that
251 he does not want Linux kernel development to become the exclusive
252 domain of the "wealthy". That means **no** to assumptions about
253 access to ultra-high resolution screens.
255 # Software prerequisites
257 Whilst many resources online advocate "sudo" in front of all root-level
258 commands below, this quickly becomes tiresome. run "sudo bash", get a
259 root prompt, and save yourself some typing.
262 * apt-get install vim exuberant-ctags
263 * apt-get install build-essential
264 * apt-get install git python3.7 python3.7-dev python-nosetest3
265 * apt-get install graphviz xdot gtkwave
266 * apt-get install python3-venv
267 * apt-get install python-virtualenv # this is an alternative to python3-venv
268 * return to user prompt (ctrl-d)
270 This will get you python3 and other tools that are needed. graphviz is
271 essential for showing the interconnections between cells, and gtkwave
272 is essential for debugging.
274 If you would like to save yourself a lot more typing, check out the
275 [dev-env-setup](https://git.libre-soc.org/?p=dev-env-setup.git;a=summary)
276 repository, examine the scripts there and use them to automate much of
281 Look up good tutorials on how to use git effectively. There are so many
282 it is hard to recommend one. This is however essential. If you are not
283 comfortable with git, and you let things stay that way, it will seriously
284 impede development progress.
286 If working all day you should expect to be making at least two commits per
287 hour, so should become familiar with it very quickly. If you are *not*
288 doing around 2 commits per hour, something is wrong and you should read
289 the workflow instructions below more carefully, and also ask for advice
292 Worth noting: *this project does not use branches*. All code is committed
293 to master and we *require* that it be either zero-impact additions or that
294 relevant unit tests pass 100%. This ensures that people's work does not
295 get "lost" or isolated and out of touch due to major branch diversion,
296 and that people communicate and coordinate with each other.
300 Follow the source code (git clone) instructions here:
301 <http://www.clifford.at/yosys/download.html>
303 Or, alternatively, use the
304 [yosys-et-al](https://git.libre-soc.org/?p=dev-env-setup.git;a=blob;f=yosys-et-al;hb=HEAD)
305 script (which also installs symbiyosys and its dependencies)
307 Do not try to use a fixed revision of yosys (currently 0.9), nmigen is evolving
308 and frequently interacts with yosys.
312 Follow the instructions here:
313 <https://symbiyosys.readthedocs.io/en/latest/quickstart.html#installing>
315 You do not have to install all of those (avy, boolector can be left
316 out if desired) however the more that are installed the more effective
317 the formal proof scripts will be (less resource utilisation in certain
322 nmigen may be installed as follows:
326 * git clone https://github.com/nmigen/nmigen.git
329 * python3 setup.py develop
332 testing can then be carried out with "python3 setup.py test"
334 ## Softfloat and sfpy
336 These are a test suite dependency for the ieee754fpu library, and
337 will be changed in the future to use Jacob's [simple-soft-float](https://crates.io/crates/simple-soft-float)
338 library. In the meantime, sfpy can be built as follows:
340 git clone --recursive https://github.com/billzorn/sfpy.git
343 git apply ../softposit_sfpy_build.patch
344 git apply /path/to/ieee754fpu/SoftPosit.patch
345 cd ../berkely-softfloat-3
346 # Note: Do not apply the patch included in sfpy for berkely-softfloat,
347 # it contains the same changes as this one
348 git apply /path/to/ieee754fpu/berkeley-softfloat.patch
351 # prepare a virtual environment for building
354 # or, if you prefer the old way:
355 # virtualenv -p python3 .env
357 # install dependencies
358 source .env/bin/activate
359 pip3 install --upgrade -r requirements.txt
364 make inplace -j$(nproc)
368 deactivate # deactivates venv, optional
369 pip3 install dist/sfpy*.whl
371 You can test your installation by doing the following:
374 >>> from sfpy import *
377 It should print out `Posit8(1.3125)`
379 ## qemu, cross-compilers, gdb
381 As we are doing POWER ISA, POWER ISA compilers, toolchains and
382 emulators are required.
384 Install powerpc64 gcc:
386 apt-get install gcc-9-powerpc64-linux-gnu
390 apt-get install qemu-system-ppc
392 Install gdb from source. Obtain the latest tarball, unpack it, then:
394 cd gdb-9.1 (or other location)
397 ../configure --srcdir=.. --host=x86_64-linux --target=powerpc64-linux-gnu
401 ## power_instruction_analyzer (pia)
403 We have a custom tool built in rust by programmerjake to help analyze
404 the power instructions execution on *actual* hardware.
406 Note: a very recent version of pip3 is required for this to work.
410 curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
412 Make sure we have the correct and up-to-date rust compiler (rustc):
414 rustup default stable
417 Use rust's package manager *cargo* to install the rust-python build tool maturin:
419 cargo install maturin
421 Install from git source by doing the following:
423 git clone https://salsa.debian.org/Kazan-team/power-instruction-analyzer.git pia
425 maturin build --cargo-extra-args=--features=python-extension
426 python3 -m pip install --user target/wheels/*.whl
428 Note: an ongoing bug in maturin interferes with successful installation. This can be worked around by explicitly installing only the .whl files needed rather than installing everything (*.whl).
432 See [[HDL_workflow/coriolis2]] page, for those people doing layout work.
436 As this is an actual ASIC, we do not rely on an FPGA's JTAG TAP interface, instead require a full complete independent implementation of JTAG. Staf Verhaegen has one, with a full test suite, and it is superb and well-written. The Libre-SOC version includes DMI (Debug Memory Interface):
438 git clone https://git.libre-soc.org/c4m-jtag.git
440 Included is an IDCODE tap point, Wishbone Master (for direct memory read and write, fully independent of the core), IOPad redirection and testing, and general purpose shift register capability for any custom use.
442 We added a DMI to JTAG bridge in LibreSOC which is directly connected to the core, to access registers and to be able to start and stop the core and change the PC. In combination with the JTAG Wishbone interface the test ASIC can have a bootloader uploaded directly into onboard SRAM and execution begun.
444 # Registering for git repository access
446 After going through the onboarding process and having agreed to take
447 responsibility for certain tasks, ask on the mailing list for git
448 repository access, sending in a public key (id_rsa.pub). If you do
449 not have one then generate it with ssh-keygen -t rsa. You will find it
452 NEVER SEND ANYONE THE PRIVATE KEY. By contrast the public key, on
453 account of being public, is perfectly fine to make... err... public.
455 Create a file ~/.ssh/config with the following lines:
457 Host git.libre-soc.org
460 Wait for the Project Admin to confirm that the ssh key has been added
461 to the required repositories. Once confirmed, you can clone any of the
462 repos at https://git.libre-soc.org/:
464 git clone gitolite3@git.libre-soc.org:REPONAME.git
466 Alternatively, the .ssh/config can be skipped and this used:
468 git clone ssh://gitolite3@git.libre-soc.org:922/REPONAME.git
472 Although there are methods online which describe how (and why) these
473 settings are normally done, honestly it is simpler and easier to open
474 ~/.gitconfig and add them by hand.
476 core.autocrlf is a good idea to ensure that anyone adding DOS-formatted
477 files they don't become a pain. pull.rebase is something that is greatly
478 preferred for this project because it avoids the mess of "multiple
479 extra merge git tree entries", and branch.autosetuprebase=always will,
480 if you want it, always ensure that a new git checkout is set up with rebase.
489 autosetuprebase = always
491 # Checking out the HDL repositories
495 * git clone gitolite3@git.libre-soc.org:nmutil.git
496 * git clone gitolite3@git.libre-soc.org:ieee754fpu.git
497 * git clone gitolite3@git.libre-soc.org:nmigen-soc.git
498 * git clone gitolite3@git.libre-soc.org:soc.git
500 In each of these directories, in the order listed, track down the
501 setup.py file, then, as root (sudo bash), run the following:
503 * python3 setup.py develop
505 The reason for using "develop" mode is that the code may be edited
506 in-place yet still imported "globally". There are variants on this theme
507 for multi-user machine use however it is often just easier to get your
508 own machine these days.
510 The reason for the order is because soc depends on ieee754fpu, and
511 ieee754fpu depends on nmutil
513 If "python3 setup.py install" is used it is a pain: edit, then
514 install. edit, then install. It gets extremely tedious, hence why
515 "develop" was created.
521 * new members, add yourself to the [[about_us]] page and create yourself a home page using someone else's page as a template.
522 * communicate on the mailing list or the bugtracker an intent to take
523 responsibility for a particular task.
524 * assign yourself as the bug's owner
525 * *keep in touch* about what you are doing, and why you are doing it.
526 * edit your home page regularly, particularly to track tasks so that they can be paid by NLNet.
527 * if you cannot do something that you have taken responsibility for,
528 then unless it is a dire personal emergency please say so, on-list. we
529 won't mind. we'll help sort it out.
531 regarding the above it is important that you read, understand, and agree
532 to the [[charter]] because the charter is about ensuring that we operate
533 as an effective organisation. It's *not* about "setting rules and meting
538 for actual code development
542 * plan in advance to write not just code but a full test suite for
543 that code. **this is not optional**. large python projects that do not
544 have unit tests **FAIL** (see separate section below).
545 * Prioritise writing formal proofs and a single clear unit test that is more like a "worked example".
546 We receive NLNet funds for writing formal proofs, plus they
547 cover corner cases and take far less time to write
549 ### Commit tested or zero-dependent code
551 * only commit code that has been tested (or is presently unused). other
552 people will be depending on you, so do take care not to screw up.
553 not least because, as it says in the [[charter]] it will be your
554 responsibility to fix. that said, do not feel intimidated: ask for help
555 and advice, and you'll get it straight away.
559 * commit often. several times a day, and "git push" it. this is
560 collaboration. if something is left even overnight uncommitted and not
561 pushed so that other people can see it, it is a red flag. if you find
562 yourself thinking "i'll commit it when it's finished" or "i don't want to
563 commit something that people might criticise" *this is not collaboration*,
564 it is making yourself a bottleneck. pair-programming is supposed to help
565 avoid this kind of thing however pair-programming is difficult to organise
566 for remote collaborative libre projects (suggestions welcomed here)
568 ### Enable editor auto-detection of file changes by external programs
570 This is important. "git pull" will merge in changes. If you then
571 arbitrarily save a file without re-loading it, you risk destroying
574 ### Absolutely no auto-generated output
576 * **do not commit autogenerated output**. write a shell script and commit
577 that, or add a Makefile to run the command that generates the output, but
578 **do not** add the actual output of **any** command to the repository.
579 ever. this is really important. even if it is a human-readable file
580 rather than a binary object file.
581 it is very common to add pdfs (the result of running latex2pdf) or configure.in (the result of running automake), they are an absolute nuisance and interfere hugely with git diffs, as well as waste hard disk space *and* network bandwidth. don't do it.
583 ### Write commands that do tasks and commit those
585 * if the command needed to create any given autogenerated output is not
586 currently in the list of known project dependencies, first consult on
587 the list if it is okay to make that command become a hard dependency of
588 the project (hint: java, node.js php and .NET commands may cause delays
589 in response time due to other list participants laughing hysterically),
590 and after a decision is made, document the dependency and how its source
591 code is obtained and built (hence why it has to be discussed carefully)
592 * if you find yourself repeating commands regularly, chances are high
593 that someone else will need to run them, too. clearly this includes
594 yourself, therefore, to make everyone's lives easier including your own,
595 put them into a .sh shell script (and/or a Makefile), commit them to
596 the repository and document them at the very minimum in the README,
597 INSTALL.txt or somewhere in a docs folder as appropriate. if unsure,
598 ask on the mailing list for advice.
600 ### Keep commits single-purpose
602 * edit files making minimal *single purpose* modifications (even if
603 it involves multiple files. Good extreme example: globally changing
604 a function name across an entire codebase is one purpose, one commit,
605 yet hundreds of files. miss out one of those files, requiring multiple
606 commits, and it actually becomes a nuisance).
608 ### Run unit tests prior to commits
610 * prior to committing make sure that relevant unit tests pass, or that
611 the change is a zero-impact addition (no unit tests fail at the minimum)
613 ### Do not break existing code
615 * keep working code working **at all times**. find ways to ensure that this is the case. examples include writing alternative classes that replace existing functionality and adding runtime options to select between old and new code.
617 ### Small commits with relevant commit message
619 * commit no more than around 5 to 10 lines at a time, with a CLEAR message
620 (no "added this" or "changed that").
621 * if as you write you find that the commit message involves a *list* of
622 changes or the word "and", then STOP. do not proceed: it is a "red flag"
623 that the commit has not been properly broken down into separate-purpose
624 commits. ask for advice on-list on how to proceed.
626 ### Exceptions to small commit: atomic single purpose commit
628 * if it is essential to commit large amounts of code, ensure that it
629 is **not** in use **anywhere** by any other code. then make a *small*
630 (single purpose) followup commit which actually puts that code into use.
632 this last rule is kinda flexible, because if you add the code *and* add
633 the unit test *and* added it into the main code *and* ran all relevant
634 unit tests on all cascade-impacted areas by that change, that's perfectly
635 fine too. however if it is the end of a day, and you need to stop and
636 do not have time to run the necessary unit tests, do *not* commit the
637 change which integrates untested code: just commit the new code (only)
638 and follow up the next day *after* running the full relevant unit tests.
640 ### Why such strict rules?
642 the reason for all the above is because python is a dynamically typed language.
643 make one tiny change at the base level of the class hierarchy and the
644 effect may be disastrous.
646 it is therefore worth reiterating: make absolutely certain that you *only*
647 commit working code or zero-impact code.
649 therefore, if you are absolutely certain that a new addition (new file,
650 new class, new function) is not going to have any side-effects, committing
651 it (a large amount of code) is perfectly fine.
653 as a general rule, however, do not use this an an excuse to write code
654 first then write unit tests as an afterthought. write *less* code *in
655 conjunction* with its (more basic) unit tests, instead. then, folliw up with
656 additions and improvements.
658 the reason for separating out commits to single purpose only becomes
659 obvious (and regretted if not followed) when, months later, a mistake
660 has to be tracked down and reverted. if the commit does not have an
661 easy-to-find message, it cannot even be located, and once found, if the
662 commit confuses several unrelated changes, not only the diff is larger
663 than it should be, the reversion process becomes extremely painful.
667 * all code needs to conform to pep8. use either pep8checker or better
668 run autopep8. however whenever committing whitespace changes, *make a
669 separate commit* with a commit message "whitespace" or "autopep8 cleanup".
670 * pep8 REQUIRES no more than 80 chars per line. this is non-negotiable. if
671 you think you need greater than 80 chars, it *fundamentally* indicates
672 poor code design. split the code down further into smaller classes
675 ### Docstring checker
677 * TBD there is a docstring checker. at the minimum make sure to have
678 an SPD license header, module header docstring, class docstring and
679 function docstrings on at least non-obvious functions.
681 ### Clear code commenting and docstrings
683 * make liberal but not excessive use of comments. describe a group of
684 lines of code, with terse but useful comments describing the purpose,
685 documenting any side-effects, and anything that could trip you or other
686 developers up. unusual coding techniques should *definitely* contain
689 ### Only one class per module (ish)
691 * unless they are very closely related, only have one module (one class)
692 per file. a file only 25 lines long including imports and docstrings
693 is perfectly fine however don't force yourself. again, if unsure,
696 ### File and Directory hierarchy
698 * *keep files short and simple*. see below as to why
699 * create a decent directory hierarchy but do not go mad. ask for advice
704 * please do not use "from module import \*". it is extremely bad practice,
705 causes unnecessary resource utilisation, makes code readability and
706 tracking extremely difficult, and results in unintended side-effects.
708 example: often you want to find the code from which a class was imported.
709 nirmally you go to the top of the file, check the imports, and you know
710 exactly which file has the class because of the import path. by using
711 wildcards, you have absolutely *no clue* which wildcard imported which
714 example: sometimes you may accidentally have duplicate code maintained
715 in two or more places. editing one of them you find, puzzlingly, that
716 the code behaves in some files with the old behaviour, but in others it
717 works. after a nassive amount of investigation, you find that the working
718 files happen to have a wildcard import of the newer accidental duplicate
719 class **after** the wildcard import of the older class with exactly the
720 same name. if you had used explicit imports, you would have spotted
721 the double import of the class from two separate locations, immediately.
723 really. don't. use. wildcards.
725 ### Keep file and variables short but clear
727 * try to keep both filenames and variable names short but not ridiculously
728 obtuse. an interesting compromise on imports is "from ridiculousfilename
729 import longsillyname as lsn", and to assign variables as well: "comb =
730 m.d.comb" followed by multiple "comb += nmigen_stmt" lines is a good trick
731 that can reduce code indentation by 6 characters without reducing clarity.
733 Additionally, use comments just above an obtuse variable in order to
734 help explain what it is for. In combination with keeping the the module
735 itself short, other readers will not need to scroll back several pages
736 in order to understand the code.
738 Yes it is tempting to actually use the variables as
739 self-explanatory-comments and generally this can be extremely good
740 practice. the problem comes when the variable is so long that a function
741 with several parameters csn no longer fit on a single line, and takes
742 up five to ten lines rather than one or two. at that point, the length
743 of the code is adversely affected and thus so is readability by forcing
744 readers to scroll through reams of pages.
746 it is a tricky balance: basically use your common sense, or just ask
747 someone else, "can you understand this code?"
749 ### Reasons for code structure
751 regarding code structure: we decided to go with small modules that are
752 both easy to analyse, as well as fit onto a single page and be readable
753 when displayed as a visual graph on a full UHD monitor. this is done
756 * using the capability of nmigen (TODO crossref to example) output the
757 module to a yosys ilang (.il) file
758 * in a separate terminal window, run yosys
759 * at the yosys prompt type "read_ilang modulename.il"
760 * type "show top" and a graphviz window should appear. note that typing
761 show, then space, then pressing the tab key twice will give a full list
762 of submodules (one of which will be "top")
764 you can now fullsize the graphviz window and scroll around. if it looks
765 reasonably obvious at 100% zoom, i.e the connections can be clearly
766 related in your mind back to the actual code (by matching the graph names
767 against signals and modules in the original nmigen code) and the words are
768 not tiny when zoomed out, and connections are not total incomprehensible
769 spaghetti, then congratulations, you have well-designed code. If not,
770 then this indicates a need to split the code further into submodules
771 and do a bit more work.
773 The reasons for doing a proper modularisation job are several-fold:
775 * firstly, we will not be doing a full automated layout-and-hope
776 using alliance/coriolis2, we will be doing leaf-node thru tree node
777 half-automated half-manual layout, finally getting to the floorplan,
778 then revising and iteratively adjusting.
779 * secondly, examining modules at the gate level (or close to it) is just
780 good practice. poor design creeps in by *not* knowing what the tools
781 are actually doing (word to experienced developers: yes, we know that
782 the yosys graph != final netlist).
783 * thirdly, unit testing, particularly formal proofs, is far easier on
784 small sections of code, and complete in a reasonable time.
786 ## Special warning / alert to vim users!
788 Some time around the beginning of 2019 some bright spark decided that
789 an "auto-recommend-completion-of-stuff" option would be a nice, shiny
790 idea to enable by default from that point onwards.
792 This incredibly annoying "feature" results in tabs (or spaces) being
793 inserted "on your behalf" when you press return on one line, for your
794 "convenience" of not needing to type lots of spaces/tabs just to get
795 to the same indentation level.
797 Of course, this "feature", if you press return on one line in edit
798 mode and then press "escape", leaves a bundle-of-joy extraneous
799 whitespace **exactly** where you don't want it, and didn't ask for it,
800 pooped all over your file.
802 Therefore, *please*: **before** running "git commit", get into the
803 habit of always running "git diff", and at the very minimum
804 speed-skim the entire diff, looking for tell-tale "red squares"
805 (these show up under bash diff colour-syntax-highlighting) that
806 inform you that, without your knowledge or consent, vim has
807 "helpfully" inserted extraneous whitespace.
809 Remove them **before** git committing because they are not part
810 of the actual desired code-modifications, and committing them
811 is a major and constant distraction for reviewers about actual
812 important things like "the code that actually *usefully* was
813 modified for that commit"
815 This has the useful side-effect of ensuring that, right before
816 the commit, you've got the actual diff right in front of you
817 in the xterm window, on which you can base the "commit message".
821 For further reading, see the wikipedia page on
822 [Test-driven Development](https://en.wikipedia.org/wiki/Test-driven_development)
824 This deserves its own special section. It is extremely important to
825 appreciate that without unit tests, python projects are simply unviable.
826 Python itself has over 25,000 individual tests.
828 This can be quite overwhelming to a beginner developer, especially one
829 used to writing scripts of only 100 lines in length.
831 Thanks to Samuel Falvo we learned that writing unit tests as a formal
832 proof is not only shorter, it's also far more readable and also, if
833 written properly, provides 100% coverage of corner-cases that would
834 otherwise be overlooked or require tens to hundreds of thousands of
837 No this is not a joke or even remotely hypothetical, this is an actual
840 The ieee754fpu requires several hundreds of thousands of tests to be
841 run (currently needing several days to run them all), and even then we
842 cannot be absolutely certain that all possible combinations of input have
843 been tested. With 2^128 permutations to try with 2 64 bit FP numbers
844 it is simply impossible to even try.
846 This is where formal proofs come into play.
848 Samuel illustrated to us that "ordinary" unit tests can then be written
849 to *augment* the formal ones, serving the purpose of illustrating how
850 to use the module, more than anything.
852 However it is appreciated that writing formal proofs is a bit of a
853 black art. This is where team collaboration particularly kicks in,
854 so if you need help, ask on the mailing list.
856 ## Don't comment out unit tests: add them first (as failures) and fix code later
858 Unit tests serve an additional critical purpose of keeping track of code
859 that needs to be written. In many cases, you write the unit test *first*,
860 despite knowing full well that the code doesn't even exist or is completely
861 broken. The unit test then serves as a constant and important reminder
862 to actually fix (or write) the code.
864 Therefore, *do not* comment out unit tests just because they "don't work".
865 If you absolutely must stop a unit test from running, **do not delete it**.
866 Simply mark it with an appropriate
867 ["skip" decorator](https://docs.python.org/3/library/unittest.html#skipping-tests-and-expected-failures),
868 preferably with a link to a URL in the [bugtracker](https://bugs.libre-soc.org/)
869 with further details as to why the unit test should not be run.
873 Find appropriate tutorials for nmigen and yosys, as well as symbiyosys.
875 * Robert Baruch's nmigen tutorials look really good:
876 <https://github.com/RobertBaruch/nmigen-tutorial>
877 * Although a verilog example this is very useful to do
878 <https://symbiyosys.readthedocs.io/en/latest/quickstart.html#first-step-a-simple-bmc-example>
879 * This tutorial looks pretty good and will get you started
880 <http://blog.lambdaconcept.com/doku.php?id=nmigen:nmigen_install> and
881 walks not just through simulation, it takes you through using gtkwave
883 * There exist several nmigen examples which are also executable
884 <https://github.com/m-labs/nmigen/tree/master/examples/> exactly as
885 described in the above tutorial (python3 filename.py -h)