(no commit message)
[libreriscv.git] / HDL_workflow.mdwn
1 # HDL workflow
2
3 This section describes the workflow and some best practices for developing
4 the Libre-SoC hardware. We use nmigen, yosys and symbiyosys, and this
5 page is intended not just to help you get set up, it is intended to
6 help advise you of some tricks and practices that will help you become
7 effective team contributors.
8
9 It is particularly important to bear in mind that we are not just
10 "developing code", here: we are creating a "lasting legacy educational
11 resource" for other people to learn from, and for businesses and students
12 alike to be able to use, learn from and augment for their own purposes.
13
14 It is also important to appreciate and respect that we are funded under
15 NLNet's Privacy and Enhanced Trust Programme <http://nlnet.nl/PET>. Full
16 transparency, readability, documentation, effective team communication
17 and formal mathematical proofs for all code at all levels is therefore
18 paramount.
19
20 # Collaboration resources
21
22 The main message here: **use the right tool for the right job**.
23
24 * mailing list: general communication and discussion.
25 * bugtracker: task-orientated, goal-orientated *focussed* discussion.
26 * ikiwiki: document store, information store, and (editable) main website
27 * git repositories: code stores (**not binary or auto-generated output store**)
28 * ftp server (<http://ftp.libre-riscv.org>): large file store.
29
30 we will add an IRC channel at some point when there are enough people
31 to warrant having one.
32
33 ## Main contact method: mailing list
34
35 To respect the transparency requirements, conversations need to be
36 public and archived (i.e not skype, not telegram, not discord,
37 and anyone seriously suggesting slack will be thrown to the
38 lions). Therefore we have a mailing list. Everything goes through
39 there. <http://lists.libre-riscv.org/mailman/listinfo/libre-riscv-dev>
40 therefore please do google "mailing list etiquette" and at the very
41 minimum look up and understand the following:
42
43 * This is a technical mailing list with complex topics. Top posting
44 is completely inappropriate. Don't do it unless you have mitigating
45 circumstances, and even then please apologise and explain ("hello sorry
46 using phone at airport flight soon, v. quick reply: ....")
47 * Always trim context but do not cut excessively to the point where people
48 cannot follow the discussion. Especially do not cut the attribution
49 ("On monday xxx wrote")
50 * Use inline replies i.e. reply at the point in the relevant part of
51 the conversation, as if you were actually having a conversation.
52 * Follow standard IETF reply formatting, using ">" for cascaded
53 indentation of other people's replies. If using gmail, please: SWITCH
54 OFF RICH TEXT EDITING.
55 * Please for god's sake do not use "my replies are in a different
56 colour". Only old and highly regarded people still using AOL are allowed
57 to get away with that (such as Mitch).
58 * Start a new topic with a relevant subject line. If an existing
59 discussion changes direction, change the subject line to reflect the
60 new topic (or start a new conversation entirely, without using the
61 "reply" button)
62 * DMARC is a pain on the neck. Try to avoid GPG signed messages. sigh.
63 * Don't send massive attachments. Put them online (no, not on facebook or
64 google drive or anywhere else that demands privacy violations) and provide
65 the link. Which should not require any kind of login to access. ask the
66 listadmin if you don't have anywhere suitable: FTP access can be arranged.
67
68 ### Actionable items from mailing list
69
70 If discussions result in any actionable items, it is important not to
71 lose track of them. Create a bugreport, find the discussion in the
72 archives <http://lists.libre-riscv.org/pipermail/libre-riscv-dev/>,
73 and put the link actually in the bugtracker as one of the comments.
74
75 At some point in any discussion, the sudden realisation may dawn on one
76 or more people that this is an "actionable" discussion. at that point
77 it may become better to use <http://bugs.libre-riscv.org>
78 itself to continue the discussion rather than to keep on dropping copies
79 of links into the bugtracker. The bugtracker sends copies of comments
80 *to* the list however this is 'one-way' (note from lkcl: because this
81 involves running an automated perl script from email, on every email,
82 on the server, that is a high security risk, and i'm not doing it. sorry.)
83
84 ### Mailing list != editable document store
85
86 Also, please do not use the mailing list as an "information or document
87 store or poor-man's editor". We have the wiki for that. Edit a page and
88 tell people what you did (summarise rather than drop the entire contents
89 at the list) and include the link to the page.
90
91 Or, if it is more appropriate, commit a document (or source code)
92 into the relevant git repository then look up the link in the gitweb
93 source tree browser and post that (in the bugtracker or mailing list)
94 See <http://git.libre-riscv.org>
95
96 ### gmail "spam"ifying the list
97
98 see <https://blog.kittycooper.com/2014/05/keeping-my-mailing-list-emails-out-of-gmails-spam-folder/>
99
100 basically it is possible to select any message from the list, create a "filter" (under "More"),
101 and, on the 2nd dialog box, click the "never send this to Spam" option.
102
103 ## Bugtracker
104
105 bugzilla. old and highly effective. sign up in the usual way. any
106 problems, ask on the list.
107
108 Please do not ask for the project to be transferred to github or other
109 proprietary nonfree service "because it's soooo convenient", as the
110 lions are getting wind and gout from overfeeding on that one.
111
112 ## ikiwiki
113
114 Runs the main libre-riscv.org site (including this page). effective,
115 stunningly light on resources, and uses a git repository not a database.
116 That means it can be edited offline.
117
118 Usual deal: register an account and you can start editing and contributing
119 straight away.
120
121 Hint: to create a new page, find a suitable page that would link to it,
122 first, then put the link in of the page you want to create, as if the
123 page already exists. Save that page, and you will find a questionmark
124 next to the new link you created. click that link, and it will fire up a
125 "create new page" editor.
126
127 Hint again: the wiki is backed by a git repository. Don't go overboard
128 but at the same time do not be afraid that you might "damage" or "lose"
129 pages. Although it would be a minor pain, the pages can always be
130 reverted or edited by the sysadmins to restore things if you get in a tiz.
131
132 Assistance in creating a much better theme greatly appreciated. e.g.
133 <http://www.math.cmu.edu/~gautam/sj/blog/20140720-ikiwiki-navbar.html>
134
135 ## git
136
137 we use git. more on this below. we also use gitolite3 running on a
138 dedicated server. again, it is extremely effective and low resource
139 utilisation. reminder: lions are involved if github is mentioned.
140
141 gitweb is provided which does a decent job. <http://git.libre-riscv.org>
142
143 ## ftp server
144
145 <http://ftp.libre-riscv.org> is available for storing large files
146 that do not belong in a git repository, if we have (or ever need)
147 any. images (etc.) if small and appropriate should go into the
148 wiki, however .tgz archives (etc.) and, at some point, binaries,
149 should be on the ftp server.
150
151 ask on the list if you have a file that belongs on the ftp server.
152
153 ## server
154
155 as an aside: all this is "old school" and run on a single core 512MB
156 VM with only a 20GB HDD allocation. it costs only 8 GBP per month from
157 mythic-beasts and means that the project is in no way dependent on anyone
158 else - not microsoft, not google, not facebook, not amazon.
159
160 we tried gitlab. it didn't go well. please don't ask to replace the
161 above extremely resource-efficient services with it.
162
163 # Hardware
164
165 RAM is the biggest requirement. Minimum 16GB, the more the better (32
166 or 64GB starts to reach "acceptable" levels. Disk space is not hugely
167 critical: 256GB SSD should be more than adequate. Simulations and
168 FPGA compilations however are where raw processing power is a must.
169 High end Graphics Cards are nonessential.
170
171 What is particularly useful is to have hi-res screens (curved is
172 *strongly* recommended if the LCD is over 24in wide, to avoid eyeballs
173 going "prism" through longterm use), and to have several of them: the
174 more the better. Either a DisplayLink UD160A (or more modern variant)
175 or simply using a second machine (lower spec hardware because it will
176 run editors) is really effective.
177
178 Also it is really recommended to have a UHD monitor (4k - 3840x2160),
179 or at least 2560x1200. If given a choice, 4:3 aspect ratio is better
180 than 16:9 particularly when using several of them. However, caveat
181 (details below): please when editing do not assume that everyone will
182 have access to such high resolution screens.
183
184 # Operating System
185
186 First install and become familiar with Debian (ubuntu if you absolutely
187 must) for standardisation cross-team and so that toolchain installation
188 is greatly simplified. yosys in particular warns that trying to use
189 Windows, BSD or MacOS will get you into a world of pain.
190
191 Only a basic GUI desktop is necessary: fvwm2, xfce4, lxde are perfectly
192 sufficient (alongside wicd-gtk for network management). Other more
193 complex desktops can be used however may consume greater resources.
194
195 # editors and editing
196
197 Whilst this is often a personal choice, the fact that many editors are
198 GUI based and run fullscreen with the entire right hand side *and* middle
199 *and* the majority of the left side of the hi-res screen entirely unused
200 and bereft of text leaves experienced developers both amused and puzzled.
201
202 At the point where such fullscreen users commit code with line lengths
203 well over 160 characters, that amusement quickly evaporates.
204
205 Where the problems occur with fullscreen editor usage is when a project
206 is split into dozens if not hundreds of small files (as this one is). At
207 that point it becomes pretty much essential to have as many as six to
208 eight files open *and on-screen* at once, without overlaps i.e. not in
209 hidden tabs, next to at least two if not three additional free and clear
210 terminals into which commands are regularly and routinely typed (make,
211 git commit, nosetests3 etc). Illustrated with the following 3840x2160
212 screenshot (click to view full image), where *every one* of those 80x70
213 xterm windows is *relevant to the task at hand*.
214
215 [[!img 2020-01-24_11-56.png size=640x ]]
216
217 (hint/tip: fvwm2 set up with "mouse-over to raise focus, rather than
218 additionally requiring a mouseclick, can save a huge amount of cumulative
219 development time here, switching between editor terminal(s) and the
220 command terminals).
221
222 Once this becomes necessary, it it turn implies that having greater
223 than 80 chars per line - and running editors fullscreen - is a severe
224 hindance to an essential *and highly effective* workflow technique.
225
226 Additionally, care should be taken to respect that not everyone will have
227 200+ column editor windows and the eyesight of a hawk. They may only have
228 a 1280 x 800 laptop which barely fits two 80x53 xterms side by side.
229 Consequently, having excessively long functions is also a hindrance to
230 others, as such developers with limited screen resources would need to
231 continuously page-up and page-down to read the code even of a single
232 function, in full.
233
234 This helps explain in part, below, why compliance with pep8 is enforced,
235 including its 80 character limit. In short: not everyone has the same
236 "modern" GUI workflow or has access to the same computing resources as
237 you, so please do respect that.
238
239 # Software prerequisites
240
241 Whilst many resources online advocate "sudo" in front of all root-level
242 commands below, this quickly becomes tiresome. run "sudo bash", get a
243 root prompt, and save yourself some typing.
244
245 * sudo bash
246 * apt-get install vim exuberant-ctags
247 * apt-get install build-essential
248 * apt-get install git python3.7 python3.7-dev python-nosetest3
249 * apt-get install graphviz xdot gtkwave
250 * return to user prompt (ctrl-d)
251
252 This will get you python3 and other tools that are needed. graphviz is
253 essential for showing the interconnections between cells, and gtkwave
254 is essential for debugging.
255
256 ## git
257
258 Look up good tutorials on how to use git effectively. There are so many
259 it is hard to recommend one. This is however essential. If you are not
260 comfortable with git, and you let things stay that way, it will seriously
261 impede development progress.
262
263 If working all day you should expect to be making at least two commits per
264 hour, so should become familiar with it very quickly. If you are *not*
265 doing around 2 commits per hour, something is wrong and you should read
266 the workflow instructions below more carefully, and also ask for advice
267 on the mailing list.
268
269 Worth noting: *this project does not use branches*. All code is committed
270 to master and we *require* that it be either zero-impact additions or that
271 relevant unit tests pass 100%. This ensures that people's work does not
272 get "lost" or isolated and out of touch due to major branch diversion,
273 and that people communicate and coordinate with each other.
274
275 ## yosys
276
277 Follow the source code (git clone) instructions here:
278 <http://www.clifford.at/yosys/download.html>
279
280 Do not try to use a fixed revision (currently 0.9), nmigen is evolving
281 and frequently interacts with yosys
282
283 ## symbiyosys
284
285 Follow the instructions here:
286 <https://symbiyosys.readthedocs.io/en/latest/quickstart.html#installing>
287
288 You do not have to install all of those (avy, boolector can be left
289 out if desired) however the more that are installed the more effective
290 the formal proof scripts will be (less resource utilisation in certain
291 circumstances).
292
293 ## nmigen
294
295 nmigen may be installed as follows:
296
297 * mkdir ~/src
298 * cd !$
299 * git clone https://github.com/m-labs/nmigen.git
300 * cd nmigen
301 * sudo bash
302 * python3 setup.py develop
303 * ctrl-d
304
305 testing can then be carried out with "python3 setup.py test"
306
307 ## Softfloat and sfpy
308
309 These are a test suite dependency for the ieee754fpu library, and
310 will be changed in the future to use Jacob's algorithmic numeric
311 library. In the meantime, sfpy can be built as follows:
312
313 git clone --recursive https://github.com/billzorn/sfpy.git
314 cd sfpy
315 cd SoftPosit
316 git apply ../softposit_sfpy_build.patch
317 git apply /path/to/ieee754fpu/SoftPosit.patch
318 cd ../berkely-softfloat-3
319 # Note: Do not apply the patch included in sfpy for berkely-softfloat,
320 # it contains the same changes as this one
321 git apply /path/to/ieee754fpu/berkeley-softfloat.patch
322 cd ..
323
324 # install dependencies
325 python3 -m venv .env
326 . .env/bin/activate
327 pip3 install --upgrade -r requirements.txt
328
329 # build
330 make lib -j8
331 make cython
332 make inplace -j8
333 make wheel
334
335 # install
336 deactivate # deactivates venv, optional
337 pip3 install dist/sfpy*.whl
338
339 You can test your installation by doing the following:
340
341 python3
342 >>> from sfpy import *
343 >>> Posit8(1.3)
344
345 It should print out `Posit8(1.3125)`
346
347 ## Coriolis2
348
349 See [[coriolis2]] page, for those people doing layout work.
350
351 # Registering for git repository access
352
353 After going through the onboarding process and having agreed to take
354 responsibility for certain tasks, ask on the mailing list for git
355 repository access, sending in a public key (id_rsa.pub). If you do
356 not have one then generate it with ssh-keygen -t rsa. You will find it
357 in ~/.ssh
358
359 NEVER SEND ANYONE THE PRIVATE KEY. By contrast the public key, on
360 account of being public, is perfectly fine to make... err... public.
361
362 Create a file ~/.ssh/config with the following lines:
363
364 Host git.libre-riscv.org
365 Port 922
366
367 Wait for the Project Admin to confirm that the ssh key has been added
368 to the required repositories. Once confirmed, you can clone any of the
369 repos at http://git.libre-riscv.org:
370
371 git clone gitolite3@git.libre-riscv.org:REPONAME.git
372
373 Alternatively, the .ssh/config can be skipped and this used:
374
375 git clone ssh://gitolite3@git.libre-riscv.org:922/REPONAME.git
376
377 # git configuration
378
379 Although there are methods online which describe how (and why) these
380 settings are normally done, honestly it is simpler and easier to open
381 ~/.gitconfig and add them by hand.
382
383 core.autocrlf is a good idea to ensure that anyone adding DOS-formatted
384 files they don't become a pain. pull.rebase is something that is greatly
385 preferred for this project because it avoids the mess of "multiple
386 extra merge git tree entries", and branch.autosetuprebase=always will,
387 if you want it, always ensure that a new git checkout is set up with rebase.
388
389 [core]
390 autocrlf = input
391 [push]
392 default = simple
393 [pull]
394 rebase = true
395 [branch]
396 autosetuprebase = always
397
398 # Checking out the HDL repositories
399
400 * mkdir ~/src
401 * cd !$
402 * git clone gitolite3@git.libre-riscv.org:nmutil.git
403 * git clone gitolite3@git.libre-riscv.org:ieee754fpu.git
404 * git clone gitolite3@git.libre-riscv.org:soc.git
405
406 In each of these directories, in the order listed, track down the
407 setup.py file, then, as root (sudo bash), run the following:
408
409 * python3 setup.py develop
410
411 The reason for using "develop" mode is that the code may be edited
412 in-place yet still imported "globally". There are variants on this theme
413 for multi-user machine use however it is often just easier to get your
414 own machine these days.
415
416 The reason for the order is because soc depends on ieee754fpu, and
417 ieee754fpu depends on nmutil
418
419 If "python3 setup.py install" is used it is a pain: edit, then
420 install. edit, then install. It gets extremely tedious, hence why
421 "develop" was created.
422
423 # Development Rules
424
425 team communication:
426
427 * new members, add yourself to the [[about_us]] page and create yourself a home page using someone else's page as a template.
428 * communicate on the mailing list or the bugtracker an intent to take
429 responsibility for a particular task.
430 * assign yourself as the bug's owner
431 * *keep in touch* about what you are doing, and why you are doing it.
432 * edit your home page regularly, particularly to track tasks so that they can be paid by NLNet.
433 * if you cannot do something that you have taken responsibility for,
434 then unless it is a dire personal emergency please say so, on-list. we
435 won't mind. we'll help sort it out.
436
437 regarding the above it is important that you read, understand, and agree
438 to the [[charter]] because the charter is about ensuring that we operate
439 as an effective organisation. It's *not* about "setting rules and meting
440 out punishment".
441
442 ## Coding
443
444 for actual code development
445
446 ### Plan unit tests
447
448 * plan in advance to write not just code but a full test suite for
449 that code. **this is not optional**. large python projects that do not
450 have unit tests **FAIL** (see separate section below).
451 * Prioritise writing formal proofs and a single clear unit test that is more like a "worked example".
452 We receive NLNet funds for writing formal proofs, plus they
453 cover corner cases and take far less time to write
454
455 ### Commit tested or zero-dependent code
456
457 * only commit code that has been tested (or is presently unused). other
458 people will be depending on you, so do take care not to screw up.
459 not least because, as it says in the [[charter]] it will be your
460 responsibility to fix. that said, do not feel intimidated: ask for help
461 and advice, and you'll get it straight away.
462
463 ### Commit often
464
465 * commit often. several times a day, and "git push" it. this is
466 collaboration. if something is left even overnight uncommitted and not
467 pushed so that other people can see it, it is a red flag. if you find
468 yourself thinking "i'll commit it when it's finished" or "i don't want to
469 commit something that people might criticise" *this is not collaboration*,
470 it is making yourself a bottleneck. pair-programming is supposed to help
471 avoid this kind of thing however pair-programming is difficult to organise
472 for remote collaborative libre projects (suggestions welcomed here)
473
474 ### Absolutely no auto-generated output
475
476 * **do not commit autogenerated output**. write a shell script and commit
477 that, or add a Makefile to run the command that generates the output, but
478 **do not** add the actual output of **any** command to the repository.
479 ever. this is really important. even if it is a human-readable file
480 rather than a binary object file.
481 it is very common to add pdfs (the result of running latex2pdf) or configure.in (the result of running automake), they are an absolute nuisance and interfere hugely with git diffs, as well as waste hard disk space *and* network bandwidth. don't do it.
482
483 ### Write commands that do tasks and commit those
484
485 * if the command needed to create any given autogenerated output is not
486 currently in the list of known project dependencies, first consult on
487 the list if it is okay to make that command become a hard dependency of
488 the project (hint: java, node.js php and .NET commands may cause delays
489 in response time due to other list participants laughing hysterically),
490 and after a decision is made, document the dependency and how its source
491 code is obtained and built (hence why it has to be discussed carefully)
492 * if you find yourself repeating commands regularly, chances are high
493 that someone else will need to run them, too. clearly this includes
494 yourself, therefore, to make everyone's lives easier including your own,
495 put them into a .sh shell script (and/or a Makefile), commit them to
496 the repository and document them at the very minimum in the README,
497 INSTALL.txt or somewhere in a docs folder as appropriate. if unsure,
498 ask on the mailing list for advice.
499
500 ### Keep commits single-purpose
501
502 * edit files making minimal *single purpose* modifications (even if
503 it involves multiple files. Good extreme example: globally changing
504 a function name across an entire codebase is one purpose, one commit,
505 yet hundreds of files. miss out one of those files, requiring multiple
506 commits, and it actually becomes a nuisance).
507
508 ### Run unit tests prior to commits
509
510 * prior to committing make sure that relevant unit tests pass, or that
511 the change is a zero-impact addition (no unit tests fail at the minimum)
512
513 ### Do not break existing code
514
515 * keep working code working **at all times**. find ways to ensure that this is the case. examples include writing alternative classes that replace existing functionality and adding runtime optiobs to select between old and new code.
516
517 ### Small commits with relevant commit message
518
519 * commit no more than around 5 to 10 lines at a time, with a CLEAR message
520 (no "added this" or "changed that").
521 * if as you write you find that the commit message involves a *list* of
522 changes or the word "and", then STOP. do not proceed: it is a "red flag"
523 that the commit has not been properly broken down into separate-purpose
524 commits. ask for advice on-list on how to proceed.
525
526 ### Exceptions to small commit: atomic single purpose commit
527
528 * if it is essential to commit large amounts of code, ensure that it
529 is **not** in use **anywhere** by any other code. then make a *small*
530 (single purpose) followup commit which actually puts that code into use.
531
532 this last rule is kinda flexible, because if you add the code *and* add
533 the unit test *and* added it into the main code *and* ran all relevant
534 unit tests on all cascade-impacted areas by that change, that's perfectly
535 fine too. however if it is the end of a day, and you need to stop and
536 do not have time to run the necessary unit tests, do *not* commit the
537 change which integrates untested code: just commit the new code (only)
538 and follow up the next day *after* running the full relevant unit tests.
539
540 ### Why such strict rules?
541
542 the reason for all the above is because python is a weakly typed language.
543 make one tiny change at the base level of the class hierarchy and the
544 effect may be disastrous.
545
546 it is therefore worth reiterating: make absolutely certain that you *only*
547 commit working code or zero-impact code.
548
549 therefore, if you are absolutely certain that a new addition (new file,
550 new class, new function) is not going to have any side-effects, committing
551 it (a large amount of code) is perfectly fine.
552
553 as a general rule, however, do not use this an an excuse to write code
554 first then write unit tests as an afterthought. write *less* code *in
555 conjunction* with its (more basic) unit tests, instead. then, folliw up with
556 additions and improvements.
557
558 the reason for separating out commits to single purpose only becomes
559 obvious (and regretted if not followed) when, months later, a mistake
560 has to be tracked down and reverted. if the commit does not have an
561 easy-to-find message, it cannot even be located, and once found, if the
562 commit confuses several unrelated changes, not only the diff is larger
563 than it should be, the reversion process becomes extremely painful.
564
565 ### PEP8 format
566
567 * all code needs to conform to pep8. use either pep8checker or better
568 run autopep8. however whenever committing whitespace changes, *make a
569 separate commit* with a commit message "whitespace" or "autopep8 cleanup".
570 * pep8 REQUIRES no more than 80 chars per line. this is non-negotiable. if
571 you think you need greater than 80 chars, it *fundamentally* indicates
572 poor code design. split the code down further into smaller classes
573 and functions.
574
575 ### Docstring checker
576
577 * TBD there is a docstring checker. at the minimum make sure to have
578 an SPD license header, module header docstring, class docstring and
579 function docstrings on at least non-obvious functions.
580
581 ### Clear code commenting and docstrings
582
583 * make liberal but not excessive use of comments. describe a group of
584 lines of code, with terse but useful comments describing the purpose,
585 documenting any side-effects, and anything that could trip you or other
586 developers up. unusual coding techniques should *definitely* contain
587 a warning.
588
589 ### Only one class per module (ish)
590
591 * unless they are very closely related, only have one module (one class)
592 per file. a file only 25 lines long including imports and docstrings
593 is perfectly fine however don't force yourself. again, if unsure,
594 ask on-list.
595
596 ### File and Directory hierarchy
597
598 * *keep files short and simple*. see below as to why
599 * create a decent directory hierarchy but do not go mad. ask for advice
600 if unsure
601
602 ### No import star!
603
604 * please do not use "from module import \*". it is extremely bad practice,
605 causes unnecessary resource utilisation, makes code readability and
606 tracking extremely difficult, and results in unintended side-effects.
607
608 ### Keep file and variables short but clear
609
610 * try to keep both filenames and variable names short but not ridiculously
611 obtuse. an interesting compromise on imports is "from ridiculousfilename
612 import longsillyname as lsn", and to assign variables as well: "comb =
613 m.d.comb" followed by multiple "comb += nmigen_stmt" lines is a good trick
614 that can reduce code indentation by 6 characters without reducing clarity.
615
616 ### Reasons for code structure
617
618 regarding code structure: we decided to go with small modules that are
619 both easy to analyse, as well as fit onto a single page and be readable
620 when displayed as a visual graph on a full UHD monitor. this is done
621 as follows:
622
623 * using the capability of nmigen (TODO crossref to example) output the
624 module to a yosys ilang (.il) file
625 * in a separate terminal window, run yosys
626 * at the yosys prompt type "read_ilang modulename.il"
627 * type "show top" and a graphviz window should appear. note that typing
628 show, then space, then pressing the tab key twice will give a full list
629 of submodules (one of which will be "top")
630
631 you can now fullsize the graphviz window and scroll around. if it looks
632 reasonably obvious at 100% zoom, i.e the connections can be clearly
633 related in your mind back to the actual code (by matching the graph names
634 against signals and modules in the original nmigen code) and the words are
635 not tiny when zoomed out, and connections are not total incomprehensible
636 spaghetti, then congratulations, you have well-designed code. If not,
637 then this indicates a need to split the code further into submodules
638 and do a bit more work.
639
640 The reasons for doing a proper modularisation job are several-fold:
641
642 * firstly, we will not be doing a full automated layout-and-hope
643 using alliance/coriolis2, we will be doing leaf-node thru tree node
644 half-automated half-manual layout, finally getting to the floorplan,
645 then revising and iteratively adjusting.
646 * secondly, examining modules at the gate level (or close to it) is just
647 good practice. poor design creeps in by *not* knowing what the tools
648 are actually doing (word to experienced developers: yes, we know that
649 the yosys graph != final netlist).
650 * thirdly, unit testing, particularly formal proofs, is far easier on
651 small sections of code, and complete in a reasonable time.
652
653 ## Special warning / alert to vim users!
654
655 Some time around the beginning of 2019 some bright spark decided that
656 an "auto-recommend-completion-of-stuff" option would be a nice, shiny
657 idea to enable by default from that point onwards.
658
659 This incredibly annoying "feature" results in tabs (or spaces) being
660 inserted "on your behalf" when you press return on one line, for your
661 "convenience" of not needing to type lots of spaces/tabs just to get
662 to the same indentation level.
663
664 Of course, this "feature", if you press return on one line in edit
665 mode and then press "escape", leaves a bundle-of-joy extraneous
666 whitespace **exactly** where you don't want it, and didn't ask for it,
667 pooped all over your file.
668
669 Therefore, *please*: **before** running "git commit", get into the
670 habit of always running "git diff", and at the very minimum
671 speed-skim the entire diff, looking for tell-tale "red squares"
672 (these show up under bash diff colour-syntax-highlighting) that
673 inform you that, without your knowledge or consent, vim has
674 "helpfully" inserted extraneous whitespace.
675
676 Remove them **before** git committing because they are not part
677 of the actual desired code-modifications, and committing them
678 is a major and constant distraction for reviewers about actual
679 important things like "the code that actually *usefully* was
680 modified for that commit"
681
682 This has the useful side-effect of ensuring that, right before
683 the commit, you've got the actual diff right in front of you
684 in the xterm window, on which you can base the "commit message".
685
686 ## Unit tests
687
688 This deserves its own special section. It is extremely important to
689 appreciate that without unit tests, python projects are simply unviable.
690 Python itself has over 25,000 individual tests.
691
692 This can be quite overwhelming to a beginner developer, especially one
693 used to writing scripts of only 100 lines in length.
694
695 Thanks to Samuel Falvo we learned that writing unit tests as a formal
696 proof is not only shorter, it's also far more readable and also, if
697 written properly, provides 100% coverage of corner-cases that would
698 otherwise be overlooked or require tens to hundreds of thousands of
699 tests to be run.
700
701 No this is not a joke or even remotely hypothetical, this is an actual
702 real-world problem.
703
704 The ieee754fpu requires several hundreds of thousands of tests to be
705 run (currently needing several days to run them all), and even then we
706 cannot be absolutely certain that all possible combinations of input have
707 been tested. With 2^128 permutations to try with 2 64 bit FP numbers
708 it is simply impossible to even try.
709
710 This is where formal proofs come into play.
711
712 Samuel illustrated to us that "ordinary" unit tests can then be written
713 to *augment* the formal ones, serving the purpose of illustrating how
714 to use the module, more than anything.
715
716 However it is appreciated that writing formal proofs is a bit of a
717 black art. This is where team collaboration particularly kicks in,
718 so if you need help, ask on the mailing list.
719
720 # TODO Tutorials
721
722 Find appropriate tutorials for nmigen and yosys, as well as symbiyosys.
723
724 * Although a verilog example this is very useful to do
725 <https://symbiyosys.readthedocs.io/en/latest/quickstart.html#first-step-a-simple-bmc-example>
726 * This tutorial looks pretty good and will get you started
727 <http://blog.lambdaconcept.com/doku.php?id=nmigen:nmigen_install> and
728 walks not just through simulation, it takes you through using gtkwave
729 as well.
730 * There exist several nmigen examples which are also executable
731 <https://github.com/m-labs/nmigen/tree/master/examples/> exactly as
732 described in the above tutorial (python3 filename.py -h)