vpu.mdwn

   1 # Video Processing and Acceleration
   2
   3 Here's some preliminary planning for the video part. Top-level headings.
   4 Any actual work will need to know the ISA.
   5
   6 Decoding is prioritized over encoding. In general, faster decoding
   7 speeds up a part of encoding.
   8
   9 Codecs to target:
  10
  11 * Audio: MP3, AC3, Vorbis, Opus
  12 * Video: MPEG1/2, MPEG4 ASP (xvid), H.264, H.265, VP8, VP9, AV1
  13
  14 YUV-RGB conversion for the most common formats (there may be a hw block for this, if so skip):
  15
  16 * rgb/bgr24,
  17 * rgbx/bgrx/xrgb/xbgr32,
  18 * nv12,
  19 * nv21
  20
  21 Most of these are DCT-based, meaning the instructions will likely
  22 benefit image formats like JPEG as well. However image formats are not
  23 a part of this project, though MJPEG could be considered a video codec?
  24 Should we include it and thus speed up JPEG decoding?
  25
  26 Audio was not explicitly mentioned in the proposal, but fast audio
  27 decoding is necessary for smooth video playback. Otherwise a 300MHz
  28 chip may not even play Opus in realtime.
  29
  30 The entropy part of codecs similarly has overlap with general
  31 compression.
  32
  33 Any speedups should really be compared to C (what gcc generates), not a
  34 hand-made simple asm implementation. Both for no use in trying to match
  35 the compiler for simple loops, and for potential new instructions
  36 possibly needing entirely different approaches. We'll also need some
  37 guideline on what parallelism to target in the simulator (how many ops
  38 we can assume will be completed at once).
  39
  40 The software part will be ffmpeg and binutils forks until the standards
  41 are set and  hw exists; then submission; then other projects
  42 (gstreamer, lib*).
  43
  44 # Links
  45
  46 * Cocotb co-simulation of JPEG <https://github.com/chiggs/oc_jpegencode/>
  47 * Top level bugreport: <http://bugs.libre-riscv.org/show_bug.cgi?id=137>
  48 * [[nlnet_2019_video]]