From: Jean-Paul Chaput Date: Sat, 28 Mar 2020 16:44:48 +0000 (+0100) Subject: Re: [libre-riscv-dev] extremely busy crowdsupply update started X-Git-Url: https://git.libre-soc.org/?a=commitdiff_plain;h=3363fb549b29cfb6a3ddfe404b930a66ef590286;p=libre-riscv-dev.git Re: [libre-riscv-dev] extremely busy crowdsupply update started --- diff --git a/bd/9d03d2708abd2097c9819c6105d1e4f54bcec5 b/bd/9d03d2708abd2097c9819c6105d1e4f54bcec5 new file mode 100644 index 0000000..5b34216 --- /dev/null +++ b/bd/9d03d2708abd2097c9819c6105d1e4f54bcec5 @@ -0,0 +1,467 @@ +Return-path: +Envelope-to: publicinbox@libre-riscv.org +Delivery-date: Sat, 28 Mar 2020 16:45:03 +0000 +Received: from localhost ([::1] helo=libre-riscv.org) + by libre-riscv.org with esmtp (Exim 4.89) + (envelope-from ) + id 1jIEZy-0006Ul-P3; Sat, 28 Mar 2020 16:45:02 +0000 +Received: from isis.lip6.fr ([132.227.60.2]) + by libre-riscv.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) + (Exim 4.89) (envelope-from ) + id 1jIEZx-0006UZ-8g + for libre-riscv-dev@lists.libre-riscv.org; Sat, 28 Mar 2020 16:45:01 +0000 +Received: from asim.lip6.fr (asim.lip6.fr [132.227.86.2]) + by isis.lip6.fr (8.15.2/8.15.2) with ESMTPS id 02SGiwPJ010739 + (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) + for ; + Sat, 28 Mar 2020 17:44:58 +0100 (CET) +Received: from localhost (melon [132.227.103.13]) + by asim.lip6.fr (8.15.2/8.14.4) with ESMTP id 02SGiwdM026328 + for ; + Sat, 28 Mar 2020 17:44:58 +0100 (MET) +Message-ID: +From: Jean-Paul Chaput +To: libre-riscv-dev@lists.libre-riscv.org +Date: Sat, 28 Mar 2020 17:44:48 +0100 +In-Reply-To: <8fb9dbcc812d3d7050c6e336dc9bf68f88536888.camel@gmail.com> +References: + <8fb9dbcc812d3d7050c6e336dc9bf68f88536888.camel@gmail.com> +Organization: UPMC +X-Mailer: Evolution 3.28.5 (3.28.5-5.el7) +Mime-Version: 1.0 +X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 + (isis.lip6.fr [132.227.60.2]); Sat, 28 Mar 2020 17:44:58 +0100 (CET) +X-Scanned-By: MIMEDefang 2.78 on 132.227.60.2 +X-Content-Filtered-By: Mailman/MimeDel 2.1.23 +Subject: Re: [libre-riscv-dev] extremely busy crowdsupply update started +X-BeenThere: libre-riscv-dev@lists.libre-riscv.org +X-Mailman-Version: 2.1.23 +Precedence: list +List-Id: Libre-RISCV General Development + +List-Unsubscribe: , + +List-Archive: +List-Post: +List-Help: +List-Subscribe: , + +Reply-To: Libre-RISCV General Development + +Content-Type: multipart/mixed; boundary="===============8672095046792038340==" +Errors-To: libre-riscv-dev-bounces@lists.libre-riscv.org +Sender: "libre-riscv-dev" + + +--===============8672095046792038340== +Content-Type: multipart/signed; micalg="pgp-sha1"; protocol="application/pgp-signature"; + boundary="=-p5NYohsGMR04AKFC8NUj" + + +--=-p5NYohsGMR04AKFC8NUj +Content-Type: text/plain; charset="UTF-8" +Content-Transfer-Encoding: quoted-printable + + +Hello Jock, + + +First, thank you for the time you spend analysing the Coriolis code and +that detailed report. I will try to make a comprehensive answer about why +Coriolis is the way it is.=20 + +* My job description at the University is not only to develop Coriolis, + but be the system admistrator of a Linux network (150 desktop & + servers) and manage industrial ASIC toolchains (Cadence,Mentor,Synopys). + So I usually can only spend 70% of my time on Coriolis. + +* One problem in a tool like Coriolis is that you need three kind of + skills to really be efficient: + + 1. A good understanding of the specifics of making an ASIC. + 2. Algorithmic knowledge to tackle efficiently the problems in (1). + 3. Programming skills in at least two language C++ & Python, + plus a bit of software engineering. + + So, I have knowledge in all three, but cannot became a top notch + expert in all of them. This also makes it difficult to have interns. + +* In some parts, namely the detailed router, the algorithm is a genuine + one, developing and testing it is a full time job by itself. It is not + even finished yet. But, when finished it will allow lots of fancy things + like seamless analog/digital designs and taking into account fancy + constraints to avoid side-channel attacks on crypto chips (for example). + +* It has a past... Some decisions where taken more than 10 years ago in + a diffrent context and I simply cannot afford to rewrite the software + every times something shift. + +* Nevertheless, I'm deeply commited to write software as cleanly as possibl= +e, + and will make improvements when I can. So I will very carefully read your + observations. But, unless you want to commit yourself for the next ten ye= +ars, + I need a code that I intimately understand. + + +> Detailed version (The Rant) +> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= +=3D=3D=3D +>=20 +> 1. Installation +>=20 +> Coriolis 2 can be built and installed two ways: +>=20 +> a) inside the home directory (=E2=80=9CFixed directory tree=E2=80=9D), = +using a helper script [1], +> b) as a package, either RPM or DEB [2]. +>=20 +> I followed the (a) route, and it actually went well except for a couple +> of quirks, that is considering the number of dependencies. As I +> understand, the (b) way is not well maintained for now. + + Coriolis is too unstable yet. Using packages would meant to rebuild them + at every git push, and having the users do not forget to reinstall it + afterwards. I think Staf Verhaegen is making some trials. + Ruben Undheim, an official Debian packager, is making DEB of Coriolis, + you may contact him to see if you can set up some continuous integration + there. + My goal is to have something "stable" after the november run in 180nm. + + +> The method (a) can be used per se, at least on Debian GNU/Linux, or +> within chroot or LXC environments. +>=20 +> Unfortunately, the Python extensions are coupled with the rest of the +> Coriolis, so it is impossible to use the advantages of 'virtualenv' or +> similar environments with Coriolis. + + I'm not familiar with virtualenv and I don't understand how the + C++ Python modules can be separated from the Python extension themselves. + +> Thus, entering the development with Coriolis for pythonistas is +> complicated by the combination of two factors: +>=20 +> - lack of native packages for their OSes, +> - lack of PyPI package or pip/virtualenv support. +>=20 +> I understand that Coriolis is a niche product, so we can not really +> expect much in terms of packaging for popular OSes. So it may be better +> for Coriolis to take another route: +>=20 +> - decouple Python extensions from the main product, +> - provide Python extensions as a pip package (via PyPI or git), +> - supply a script and/or instructions for building the main product +> with the pip package. + + Breaking things in multiple components introduce complexity that you + must manage afterwards. I can see a lot of troubles when people will + have mismatched versions of the C++ core and the PyPI extensions. + And lots of nigthmarish debug problems. + Native packages would be better, and can be done, once it is stable + enough. + =20 + +> This is actually not a unique approach. For example, xapian-haystack +> ships a script for building a whole Xapian search engine (C++) in +> PYTHONPATH [3]. +>=20 +> 2. Documentation +> . +> All the basics are covered by Coriolis User's Guide [4] (general +> workflow and GUI) and Python Tutorial [5], that are available online +> and offline, in PDF and HTML formats. +>=20 +> Unfortunately, Coriolis Python API don't have its own reference guide. +> There is a good reference on Hurricane C++ API [6] though. Since Python +> API just copies the C++ API (which is a terrible desicion by itself, +> see the explanation in 3.2), this reference alone has the great part of +> both C++ and Python APIs covered. But other parts of the Python API +> (most importantly, Etesian and Katana) stays undocumented beyond the +> basics. + + For Etesian & Katana, this is just due to the lack of time. Also mixed + with a lack of stability. It is very frustrating to write a doc which + is no longer relevant three month later. The basic doc of Katana is + in fact in Kite. Kite is the digital only version of the router while + Katana is the mixed one. And it is in flux, so I postponed the port + of the doc. + =20 + + +> 3. API +>=20 +> 3.1. Supported Python versions +>=20 +> The Coriolis Python API is a Python 2 extension. +>=20 +> Python 2.7 was released in 2010, and it was meant to be the last 2.x +> release back then. In the following years Python 2 had no new features +> other than backports from Python 3. As of January 2020 Python 2 is +> considered unsupported by PSF. [7] +>=20 +> Still clinging to outdated and abandoned language and not even having a +> (publicly available, at least) transition plan seems a bit scary to me. + + I understand very well. I'm not clinging to it, I just have had not time + yet to make the transition. If nothing unexpected happens I hope to do + the migration sometimes in the next three monthes. Here I must admit + I'm not proud of the way the C++/Python wrapper is written, but it do + works exactly like I want to... Here again with lots of unusual constrain= +ts + that I see no way to make swig or the like to understand. + + +> 3.2. Best practices of Python APIs +>=20 +> To have a new API mimic the old API written in another programming +> language seems like an excellent idea at first. You don't have to +> design anything. Don't have to think of naming. You can reuse +> documentation and tests. The new API is easy to use, if you already got +> used to the old one. Everybody's happy, right? +>=20 +> Well, everybody except the end users whom the new API is created for. + + Not so fast here, we need a definition of "end users". In my lab, + the people using Coriolis are mostly ASIC people, that is electrical + engineers. They use the minimal and simplest possible subset of any + programming language. + You may look at the relationship between C++ and SystmeC. SystemC + is comparable to nMigen but under the form of a C++ library. They + don't event tell them it's C++. SystemC is usually presented as a + language on it's own (sometimes they talk about C). + In fact, and this is a trend I've seen emerge with the rise of FPGA + for some time, you have *two* kind of audience. + =20 + * Electrical engineers or guys comming "from the hardware", with limited + programming skills. They describe their design as series of blocks. + They want to tightly control how and what they generate. + * Computer Science types, or guys coming "from the software", with + advanced programming skills. They tend to see their design in term + of classes and functions, with lot of implicit mechanisms. + Typical example is Chisel or SpinalHDL. + + At this point I have no clear answer about what the "best" language + would be. But I can tell you for sure that many "hardware guys" are + literally repelled by languages like Chisel. + It's a discussion we are currently having at the lab. Like the one + that did take place in this list a little while ago. + =20 + +> Each programming language has its own best practices, and most of the +> time they either contradict the ones accepted by another language users +> or just don't have analogues in that language. Your new users will see +> your API as alien, cumbersome, or just poorly designed. They may have +> to cram their way around the least expected features. They may have to +> write wrappers around your API to reduce the astonishment factor [8] +> for the others. But most likely they will simply walk away. + + Another point to take into account, is that, at first, the Python + extention was made to be able to do "rapid prototyping". That is + quicly develop and test under Python, then, convert back to C++. + So keeping both interfaces in synch as much as possible was to + reduce that work. + And, as if for myself only, I do constant back and forth between + Python and C++, so I have one less "translation" to manage. + =20 + +> Here I'll try to give a run-down of Coriolis API features that scares +> Python developers (=E2=80=9Cbad=E2=80=9D) along with their attractive and= + comfy +> alternatives (=E2=80=9Cgood=E2=80=9D). +>=20 +> 3.2.1. Singletons +>=20 +> You don't need a special methods to create and access a unique object +> in Python, since you can just import it. +>=20 +> Bad: af =3D CRL.AllianceFramework.get() +>=20 +> Good: from CRL import af + + OK. + =20 + +> 3.2.2. Getters and setters +>=20 +> Getters and setters are considered an antipattern (not only in Python), +> so they should be replaced by properties or collection-like interfaces. +>=20 +> Bad: +> env =3D af.getEnvironment() +> env.setCLOCK('^ck$|m_clock|^clk$') +>=20 +> Good: +> af.env['CLOCK'] =3D '^ck$|m_clock|^clk$' + + OK. But introduce some rework at Python/C++ level. + =20 + +> 3.2.3. Data types +>=20 +> Python use duck typing, so you should avoid any direct type indication +> or querying in Python. +>=20 +> Bad: Cfg.getParamInt('katana.vTracksReservedLocal').setInt(6) +>=20 +> Good: Cfg['katana.vTracksReservedLocal'] =3D 6 + + The Cfg module is a shame. Is is unecessarily separated from the rest + and the only one using boost. Creating one of the most painful + dependency only for it. + As those variables are created then stored in C++, I need to be + able to guess their type right. integer, float, sometimes enum, + and percentage... If you can show me an elegant solution, I will + take it. + =20 + +> 3.2.4. Syntactic sugar +>=20 +> It's easier to account for the certain things that goes in pairs +> (open/close, create/destroy) with context managers or decorators. +>=20 +> Bad: +> UpdateSession.open() +> do_my_thing() +> UpdateSession.close() +>=20 +> Good: +> with UpdateSession(): +> do_my_thing() +>=20 +> Also good in some cases: +> @update_session +> def do_my_thing(): +> ... + + OK. I will look into that. + + +> 3.2.5. Self-documenting +>=20 +> Python have a built-in 'help()' function, that returns a docstring for +> a given object. Python C extensions can emulate a docstring for use +> with 'help()'. Static syntax checkers and IDEs also rely on this API. +>=20 +> It is very important that this emulated docstring contains not only the +> description of the function or method, but also a signature (parameters +> and return value) in reST format, as recommended by PEP287 [9], or in +> older Epytext format at worst. This will greatly facilitate the use of +> Python C extension with modern IDEs. +>=20 +> The rules have slightly changed since Python 3.4, but the idea remains. + + The main reason here is still the lack of time. But there may be + another problem. My python interface allow to mirror in Python + the C++ function overload, but Python only allows one set of + parameters, is that true or I'm lagging on Python features again? + So how to describe a function with more than one parameter set? + =20 + +> 3.2.6. Naming conventions +>=20 +> The Python API must adhere to naming conventions accepted in Python. I +> can not stress enough the importance of this point, but let us address +> the styling issues in general in 3.3. +>=20 +> 3.3. Styling +>=20 +> I have to admit that Coriolis Python style is quite consistent, but it +> looks like the only purpose of this consistency is to contradict the +> PEP 8 [10] in any possible way. [11] +>=20 +> You might think that style is not that important, but after 15 minutes +> spent in search of the 'ChipRoute' class definition, diagnosing and +> restarting IDE, checking Moon phase and whatnot, find that 'ChipRoute'= +=20 +> is not a class, but a module=E2=80=A6 How well the end user could perform= + with +> that in mind? + + Quite the opposite, I do think style is important. The goal of style + is readability and it is mine. But I don't think you can say that there + is *one* good style and all others are bad. And even the reference style + may evolve... It is easy to slip from "the style I'm accustomed to" and + "the right style". I understand the advantages of using a style shared + by many peoples, but I won't blindly comply (I'm both French and a bit + stubborn). I will try to find time to read the PEP8, but I have a lot + of work to do... + I've already seen this debate about C++ coding, and there are + definetely good and bad styles (or lack of) but I won't say there is + "the good style". Among good styles, there are different tradeoffs + that may or may not suits you. + +=20 +> Another thing is that it's nearly impossible to lure a junior developer +> into a project that neglects styling. In modern development, +> proprietary and opensource alike, an ability to write a consistent, +> readable, uniform code is very much appreciated, and junior developers +> expect to acquire good professional habits in their first projects. And +> style is the first thing they learn to distinguish in someone else's +> code. + + Maybe it has changed in the company world, but for what I've seen + in the ASIC world, code cleanliness is still a long way home... + But again, it may be different company we are talking about. + I talk of Cadence/Mentor/Synopsys, if you have the opportunity, + take a look. + + A last reflexion about PyCharm, it is very well for people working + only on Python code. But for my part, I edit C++, C, Python, ReST, + cmake, LaTeX, Yaml and whatnot, so I use Emacs (vim could also do). + It would be inefficient to have one IDE/editor per language. + + =20 + Anyway, this is very valuable input, and I will take it into + account as soon as I can, given my current workload. + + + Best regards, + =20 + +> [1] http://coriolis.lip6.fr/pages/users-guide.html#id16 +> [2] http://coriolis.lip6.fr/pages/users-guide.html#id19 +> [3] https://github.com/notanumber/xapian-haystack +> [4] http://coriolis.lip6.fr/pages/users-guide.html +> [5] http://coriolis.lip6.fr/pages/python-tutorial.html +> [6] http://coriolis.lip6.fr/doc/hurricane/index.html +> [7] https://www.python.org/dev/peps/pep-0373/ +> [8] https://en.wikipedia.org/wiki/Principle_of_least_astonishment +> [9] https://www.python.org/dev/peps/pep-0287/ +> [10] https://www.python.org/dev/peps/pep-0008/ +> [11]=20 +>=20 +https://www.opensourceshakespeare.org/views/plays/play_view.php?WorkID=3Dha= +mlet&Act=3D2&Scene=3D2&Scope=3Dscene&LineHighlight=3D1307#1307 + + +--=20 + + .-. J e a n - P a u l C h a p u t / Administrateur Systeme + /v\ Jean-Paul.Chaput@lip6.fr + /(___)\ work: (33) 01.44.27.53.99 =20 + ^^ ^^ cell: 06.66.25.35.55 home: 09.65.29.83.38 + + U P M C Universite Pierre & Marie Curie + L I P 6 Laboratoire d'Informatique de Paris VI + S o C System On Chip + +--=-p5NYohsGMR04AKFC8NUj-- + + + +--===============8672095046792038340== +Content-Type: text/plain; charset="utf-8" +MIME-Version: 1.0 +Content-Transfer-Encoding: base64 +Content-Disposition: inline + +X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KbGlicmUtcmlz +Y3YtZGV2IG1haWxpbmcgbGlzdApsaWJyZS1yaXNjdi1kZXZAbGlzdHMubGlicmUtcmlzY3Yub3Jn +Cmh0dHA6Ly9saXN0cy5saWJyZS1yaXNjdi5vcmcvbWFpbG1hbi9saXN0aW5mby9saWJyZS1yaXNj +di1kZXYK + +--===============8672095046792038340==-- + + +