X-Git-Url: https://git.libre-soc.org/?a=blobdiff_plain;f=vpu.mdwn;h=a646c8609fb9202acae21c6c8d6f9e07c8487c9a;hb=2af9622e16e4664f4769eb72f70619ef1d132526;hp=6781866c36c0639a43fec8f001dbfca08382ebda;hpb=435ae220865a48a9c96bfb452d4f4d95d8c9b603;p=libreriscv.git

diff --git a/vpu.mdwn b/vpu.mdwn
index 6781866c3..a646c8609 100644
--- a/vpu.mdwn
+++ b/vpu.mdwn
@@ -1,3 +1,48 @@
 # Video Processing and Acceleration
 
-TODO
+Here's some preliminary planning for the video part. Top-level headings.
+Any actual work will need to know the ISA.
+
+Decoding is prioritized over encoding. In general, faster decoding
+speeds up a part of encoding.
+
+Codecs to target:
+
+* Audio: MP3, AC3, Vorbis, Opus
+* Video: MPEG1/2, MPEG4 ASP (xvid), H.264, H.265, VP8, VP9, AV1
+
+YUV-RGB conversion for the most common formats (there may be a hw block for this, if so skip):
+
+* rgb/bgr24,
+* rgbx/bgrx/xrgb/xbgr32,
+* nv12,
+* nv21
+
+Most of these are DCT-based, meaning the instructions will likely
+benefit image formats like JPEG as well. However image formats are not
+a part of this project, though MJPEG could be considered a video codec?
+Should we include it and thus speed up JPEG decoding?
+
+Audio was not explicitly mentioned in the proposal, but fast audio
+decoding is necessary for smooth video playback. Otherwise a 300MHz
+chip may not even play Opus in realtime.
+
+The entropy part of codecs similarly has overlap with general
+compression.
+
+Any speedups should really be compared to C (what gcc generates), not a
+hand-made simple asm implementation. Both for no use in trying to match
+the compiler for simple loops, and for potential new instructions
+possibly needing entirely different approaches. We'll also need some
+guideline on what parallelism to target in the simulator (how many ops
+we can assume will be completed at once).
+
+The software part will be ffmpeg and binutils forks until the standards
+are set and  hw exists; then submission; then other projects
+(gstreamer, lib*).
+
+# Links
+
+* Cocotb co-simulation of JPEG <https://github.com/chiggs/oc_jpegencode/>
+* Top level bugreport: <http://bugs.libre-riscv.org/show_bug.cgi?id=137>
+* [[nlnet_2019_video]]