docs/ci/LAVA.rst

   1 LAVA CI
   2 =======
   3
   4 `LAVA <https://lavasoftware.org/>`_ is a system for functional testing
   5 of boards including deploying custom bootloaders and kernels.  This is
   6 particularly relevant to testing Mesa because we often need to change
   7 kernels for UAPI changes (and this lets us do full testing of a new
   8 kernel during development), and our workloads can easily take down
   9 boards when mistakes are made (kernel oopses, OOMs that take out
  10 critical system services).
  11
  12 Mesa-LAVA software architecture
  13 -------------------------------
  14
  15 The gitlab-runner will run on some host that has access to the LAVA
  16 lab, with tags like "lava-mesa-boardname" to control only taking in
  17 jobs for the hardware that the LAVA lab contains.  The gitlab-runner
  18 spawns a docker container with lava-cli in it, and connects to the
  19 LAVA lab using a predefined token to submit jobs under a specific
  20 device type.
  21
  22 The LAVA instance manages scheduling those jobs to the boards present.
  23 For a job, it will deploy the kernel, device tree, and the ramdisk
  24 containing the CTS.
  25
  26 Deploying a new Mesa-LAVA lab
  27 -----------------------------
  28
  29 You'll want to start with setting up your LAVA instance and getting
  30 some boards booting using test jobs.  Start with the stock QEMU
  31 examples to make sure your instance works at all.  Then, you'll need
  32 to define your actual boards.
  33
  34 The device type in lava-gitlab-ci.yml is the device type you create in
  35 your LAVA instance, which doesn't have to match the board's name in
  36 ``/etc/lava-dispatcher/device-types``.  You create your boards under
  37 that device type and the Mesa jobs will be scheduled to any of them.
  38 Instantiate your boards by creating them in the UI or at the command
  39 line attached to that device type, then populate their dictionary
  40 (using an "extends" line probably referencing the board's template in
  41 ``/etc/lava-dispatcher/device-types``).  Now, go find a relevant
  42 healthcheck job for your board as a test job definition, or cobble
  43 something together from a board that boots using the same boot_method
  44 and some public images, and figure out how to get your boards booting.
  45
  46 Once you can boot your board using a custom job definition, it's time
  47 to connect Mesa CI to it.  Install gitlab-runner and register as a
  48 shared runner (you'll need a gitlab admin for help with this).  The
  49 runner *must* have a tag (like "mesa-lava-db410c") to restrict the
  50 jobs it takes or it will grab random jobs from tasks across fd.o, and
  51 your runner isn't ready for that.
  52
  53 The runner will be running an ARM docker image (we haven't done any
  54 x86 LAVA yet, so that isn't documented).  If your host for the
  55 gitlab-runner is x86, then you'll need to install qemu-user-static and
  56 the binfmt support.
  57
  58 The docker image will need access to the lava instance.  If it's on a
  59 public network it should be fine.  If you're running the LAVA instance
  60 on localhost, you'll need to set ``network_mode="host"`` in
  61 ``/etc/gitlab-runner/config.toml`` so it can access localhost.  Create a
  62 gitlab-runner user in your LAVA instance, log in under that user on
  63 the web interface, and create an API token.  Copy that into a
  64 ``lavacli.yaml``:
  65
  66 .. code-block:: yaml
  67
  68   default:
  69     token: <token contents>
  70     uri: <url to the instance>
  71     username: gitlab-runner
  72
  73 Add a volume mount of that ``lavacli.yaml`` to
  74 ``/etc/gitlab-runner/config.toml`` so that the docker container can
  75 access it.  You probably have a ``volumes = ["/cache"]`` already, so now it would be::
  76
  77     volumes = ["/home/anholt/lava-config/lavacli.yaml:/root/.config/lavacli.yaml", "/cache"]
  78
  79 Note that this token is visible to anybody that can submit MRs to
  80 Mesa!  It is not an actual secret.  We could just bake it into the
  81 gitlab CI yml, but this way the current method of connecting to the
  82 LAVA instance is separated from the Mesa branches (particularly
  83 relevant as we have many stable branches all using CI).
  84
  85 Now it's time to define your test runner in
  86 ``.gitlab-ci/lava-gitlab-ci.yml``.