ci: Make a simple little bare-metal fastboot mode for db410c.

This supports powering up the device (using an external tool you
provide based on your particular lab), talking over serial to wait for
the fastboot prompt, and then booting a fastboot image on a target
device.

I was previously relying on LAVA for this, but that ran afoul of
corporate policies related to the AGPL.  However, LAVA wasn't doing
too much for us, given that gitlab already has a job scheduler and
tagging and runners.  We were spending a lot of engineering on making
the two systems match up, when we can just have gitlab do it directly.

Lightly-reviewed-by: Kristian H. Kristensen <hoegsberg@google.com>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4076>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4076>

.gitlab-ci.yml

@@ -161,7 +161,7 @@ arm_build:
     - .debian@container-ifnot-exists@arm64v8
     - .container
   variables:
-    DEBIAN_TAG: &arm_build "2020-02-26"
+    DEBIAN_TAG: &arm_build "2020-03-09"
 
 .use-arm_build:
   variables:
@@ -701,13 +701,39 @@ arm64_a630_gles3:
   variables:
     DEQP_VER: gles3
 
+.baremetal-test:
+  extends:
+    - .ci-run-policy
+  stage: test
+
 arm64_a306_gles2:
-  extends: arm64_a630_gles2
+  extends:
+    - .baremetal-test
+    - .use-arm_build
+  stage: freedreno
   variables:
+    BM_KERNEL: /lava-files/Image
+    BM_DTB: /lava-files/apq8016-sbc.dtb
+    BM_ROOTFS: /lava-files/rootfs-arm64
+    BM_CMDLINE: "ip=dhcp console=ttyMSM0,115200n8"
     DEQP_EXPECTED_FAILS: deqp-freedreno-a307-fails.txt
-    DEQP_SKIPS: deqp-default-skips.txt
+    DEQP_SKIPS: deqp-freedreno-a307-skips.txt
+    DEQP_VER: gles2
+    DEQP_PARALLEL: 4
+  script:
+    - .gitlab-ci/bare-metal/fastboot.sh
+  needs:
+    - meson-arm64
   tags:
-    - db410c
+    - google-freedreno-db410c
+
+# Disabled due to flaky results
+.arm64_a306_gles3:
+  extends:
+    - arm64_a306_gles2
+  parallel: 8
+  variables:
+    DEQP_VER: gles3
 
 # RADV CI
 .test-radv:

.gitlab-ci/README.md

@@ -5,11 +5,12 @@ testing of Mesa drivers on various platforms, so that we can ensure no
 regressions are merged, as long as developers are merging code using
 marge-bot.
 
-There are currently 3 main automated testing systems deployed for
-Mesa.  LAVA and gitlab-runner on the DUTs are used in pre-merge
-testing and are described in this document, while Intel has a
-jenkins-based CI system with restricted access that isn't connected to
-gitlab.
+There are currently 4 automated testing systems deployed for Mesa.
+LAVA and gitlab-runner on the DUTs are used in pre-merge testing and
+are described in this document.  Managing bare metal using
+gitlab-runner is described under [bare-metal/README.md].  Intel also
+has a jenkins-based CI system with restricted access that isn't
+connected to gitlab.
 
 ## Mesa testing using LAVA
 

.gitlab-ci/bare-metal/.editorconfig

@@ -0,0 +1,2 @@
+[*.sh]
+indent_size = 2

.gitlab-ci/bare-metal/README.md

@@ -0,0 +1,80 @@
+# bare-metal Mesa testing
+
+Testing Mesa with gitlab-runner running on the devices being tested
+(DUTs) proved to be too unstable, so this set of scripts is for
+running Mesa testing on bare-metal boards connected to a separate
+system using gitlab-runner.  Currently only "fastboot" devices are
+supported.
+
+In comparison with LAVA, this doesn't involve maintaining a separate
+webservice with its own job scheduler and replicating jobs between the
+two.  It also places more of the board support in git, instead of
+webservice configuration.  Most importantly, it doesn't download the
+rootfs as artifacts on each job, so we can avoid traffic to
+freedesktop.org.  On the other hand, the serial interactions and
+bootloader support are more primitive.
+
+## Requirements
+
+This testing requires power control of the DUTs by the gitlab-runner
+machine, since this is what we use to reset the system and get back to
+a pristine state at the start of testing.
+
+We require access to the console output from the gitlb-runner system,
+since that is how we get the final results back from te tests.  You
+should probably have the console on a serial connection, so that you
+can see bootloader progress.
+
+The boards need to be able to have a kernel/initramfs supplied by the
+gitlab-runner system, since the initramfs is what contains the Mesa
+testing payload.  Currently only "fastboot" devices are supported.
+
+The boards should have networking, so that (in a future iteration of
+this code) we can extract the dEQP .xml results to artifacts on
+gitlab.
+
+## Setup
+
+Each board will be registered in fd.o gitlab.  You'll want something
+like this to register:
+
+```
+sudo gitlab-runner register \
+     --url https://gitlab.freedesktop.org \
+     --registration-token $1 \
+     --name MY_BOARD_NAME \
+     --tag-list MY_BOARD_TAG \
+     --executor docker \
+     --docker-image "alpine:latest" \
+     --docker-volumes "/dev:/dev" \
+     --docker-network-mode "host" \
+     --docker-privileged \
+     --non-interactive
+```
+
+The registration token has to come from a fd.o gitlab admin going to
+https://gitlab.freedesktop.org/admin/runners
+
+The name scheme for Google's lab is google-freedreno-boardname-nn, and
+our tag is google-freedreno-db410c.  The tag is what identifies a
+board type so that board-specific jobs can be dispatched into that
+pool.
+
+We need privileged mode and the /dev bind mount in order to get at the
+serial console and fastboot USB devices (--device arguments don't
+apply to devices that show up after container start, which is the case
+with fastboot).  We use host network mode so that we can (in the
+future) spin up a server to collect XML results.
+
+Once you've added your boards, you're going to need to specify the
+board-specific env vars, adding something like this `environment` line
+to each runner in `/etc/gitlab-runner/config.toml`
+
+```
+[[runners]]
+  name = "google-freedreno-db410c-01"
+  environment = ["BM_SERIAL=/dev/ttyDB410c8", "BM_POWERUP=google-power-up.sh 8", "BM_FASTBOOT_SERIAL=15e9e390"]
+```
+
+Once you've updated your runners' configs, restart with `sudo service
+gitlab-runner restart`

.gitlab-ci/bare-metal/expect-output.sh

@@ -0,0 +1,9 @@
+#!/bin/bash
+
+set -e
+
+echo "Waiting for $1 to say '$2'"
+
+while ! grep -q "$2" $1; do
+  sleep 2
+done

.gitlab-ci/bare-metal/fastboot.sh

@@ -0,0 +1,99 @@
+#!/bin/bash
+
+BM=$CI_PROJECT_DIR/.gitlab-ci/bare-metal
+
+if [ -z "$BM_SERIAL" ]; then
+  echo "Must set BM_SERIAL in your gitlab-runner config.toml [[runners]] environment"
+  echo "This is the serial device to talk to for waiting for fastboot to be ready and logging from the kernel."
+  exit 1
+fi
+
+if [ -z "$BM_POWERUP" ]; then
+  echo "Must set BM_POWERUP in your gitlab-runner config.toml [[runners]] environment"
+  echo "This is a shell script that should reset the device and begin its boot sequence"
+  echo "such that it pauses at fastboot."
+  exit 1
+fi
+
+if [ -z "$BM_FASTBOOT_SERIAL" ]; then
+  echo "Must set BM_FASTBOOT_SERIAL in your gitlab-runner config.toml [[runners]] environment"
+  echo "This must be the a stable-across-resets fastboot serial number."
+  exit 1
+fi
+
+if [ -z "$BM_KERNEL" ]; then
+  echo "Must set BM_KERNEL to your board's kernel Image in the job's variables:"
+  exit 1
+fi
+
+if [ -z "$BM_DTB" ]; then
+  echo "Must set BM_DTB to your board's DTB file in the job's variables:"
+  exit 1
+fi
+
+if [ -z "$BM_ROOTFS" ]; then
+  echo "Must set BM_ROOTFS to your board's rootfs directory in the job's variables:"
+  exit 1
+fi
+
+set -ex
+
+# Copy the rootfs to a temporary for our setup, as I believe changes to the
+# container can end up impacting future runs.
+cp -Rp $BM_ROOTFS rootfs
+
+# Set up the init script that brings up the system.
+cp $BM/init.sh rootfs/init
+sed -i "s|DEQP_VER_REPLACE|$DEQP_VER|g" rootfs/init
+sed -i "s|DEQP_PARALLEL_REPLACE|$DEQP_PARALLEL|g" rootfs/init
+sed -i "s|CI_NODE_INDEX_REPLACE|$CI_NODE_INDEX|g" rootfs/init
+sed -i "s|CI_NODE_TOTAL_REPLACE|$CI_NODE_TOTAL|g" rootfs/init
+
+# Add the Mesa drivers we built, and make a consistent symlink to them.
+mkdir -p rootfs/$CI_PROJECT_DIR
+tar -C rootfs/$CI_PROJECT_DIR/ -xf $CI_PROJECT_DIR/artifacts/install.tar
+ln -sf $CI_PROJECT_DIR/install rootfs/install
+
+# Copy the deqp runner script and metadata.
+cp .gitlab-ci/deqp-runner.sh rootfs/deqp/.
+mkdir -p rootfs/artifacts/deqp
+cp .gitlab-ci/$DEQP_SKIPS rootfs/artifacts/deqp-skips.txt
+if [ -n "$DEQP_EXPECTED_FAILS" ]; then
+  cp .gitlab-ci/$DEQP_EXPECTED_FAILS rootfs/artifacts/deqp-expected-fails.txt
+fi
+
+# Finally, pack it up into a cpio rootfs.
+pushd rootfs
+  find -H | cpio -H newc -o | xz --check=crc32 -T4 - > $CI_PROJECT_DIR/rootfs.cpio.gz
+popd
+
+gzip -c $BM_KERNEL > Image.gz
+cat Image.gz $BM_DTB > Image.gz-dtb
+abootimg \
+  --create artifacts/fastboot.img \
+  -k Image.gz-dtb \
+  -r rootfs.cpio.gz \
+  -c cmdline="$BM_CMDLINE"
+rm Image.gz Image.gz-dtb
+
+# Start watching serial, and power up the device.
+$BM/serial-buffer.py $BM_SERIAL | tee artifacts/serial-output.txt &
+while [ ! -e artifacts/serial-output.txt ]; do
+  sleep 1
+done
+PATH=$BM:$PATH $BM_POWERUP
+
+# Once fastboot is ready, boot our image.
+$BM/expect-output.sh artifacts/serial-output.txt "fastboot: processing commands"
+fastboot boot -s $BM_FASTBOOT_SERIAL artifacts/fastboot.img
+
+# Wait for the device to complete the deqp run
+$BM/expect-output.sh artifacts/serial-output.txt "DEQP RESULT"
+
+set +e
+if grep -q "DEQP RESULT: pass" artifacts/serial-output.txt; then
+   exit 0
+else
+   exit 1
+fi
+

.gitlab-ci/bare-metal/google-power-relay.py

@@ -0,0 +1,19 @@
+#!/usr/bin/python3
+
+import sys
+import serial
+
+mode = sys.argv[1]
+relay = sys.argv[2]
+
+# our relays are "off" means "board is powered".
+mode_swap = {
+     "on" : "off",
+     "off" : "on",
+}
+mode = mode_swap[mode]
+
+ser = serial.Serial('/dev/ttyACM0', 115200, timeout=2)
+command = "relay {} {}\n\r".format(mode, relay)
+ser.write(command.encode())
+ser.close()

.gitlab-ci/bare-metal/google-power-up.sh

@@ -0,0 +1,12 @@
+#!/bin/bash
+
+relay=$1
+
+if [ -z "$relay" ]; then
+    echo "Must supply a relay arg"
+    exit 1
+fi
+
+$CI_PROJECT_DIR/.gitlab-ci/bare-metal/google-power-relay.py off $relay
+sleep 5
+$CI_PROJECT_DIR/.gitlab-ci/bare-metal/google-power-relay.py on $relay

.gitlab-ci/bare-metal/init.sh

@@ -0,0 +1,28 @@
+#!/bin/sh
+
+mount -t proc none /proc
+mount -t sysfs none /sys
+mount -t devtmpfs none /dev || echo possibly already mounted
+mkdir -p /dev/pts
+mount -t devpts devpts /dev/pts
+
+export DEQP_NO_SAVE_RESULTS=1
+export DEQP_RUNNER_OPTIONS="--compact-display false"
+export DEQP_VER=DEQP_VER_REPLACE
+export DEQP_PARALLEL=DEQP_PARALLEL_REPLACE
+export CI_NODE_INDEX=CI_NODE_INDEX_REPLACE
+export CI_NODE_TOTAL=CI_NODE_TOTAL_REPLACE
+export DEQP_SKIPS=deqp-skips.txt
+if [ -e /artifacts/deqp-expected-fails.txt ]; then
+  export DEQP_EXPECTED_FAILS=deqp-expected-fails.txt
+fi
+
+if sh /deqp/deqp-runner.sh; then
+    echo "DEQP RESULT: pass"
+else
+    echo "DEQP RESULT: fail"
+fi
+
+# Wait until the job would have timed out anyway, so we don't spew a "init
+# exited" panic.
+sleep 6000

.gitlab-ci/bare-metal/serial-buffer.py

@@ -0,0 +1,46 @@
+#!/usr/bin/python3
+
+# Copyright © 2020 Google LLC
+#
+# Permission is hereby granted, free of charge, to any person obtaining a
+# copy of this software and associated documentation files (the "Software"),
+# to deal in the Software without restriction, including without limitation
+# the rights to use, copy, modify, merge, publish, distribute, sublicense,
+# and/or sell copies of the Software, and to permit persons to whom the
+# Software is furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice (including the next
+# paragraph) shall be included in all copies or substantial portions of the
+# Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+# IN THE SOFTWARE.
+
+# Tiny script to read bytes from serial, and write the output to stdout, with a
+# buffer in between so we don't lose serial output from its buffer.
+#
+# We don't use 'cu' because it requires stdin to be hooked up and I never
+# managed to make that work without getting blocked somewhere.  We don't use
+# 'conserver' because it's non-free.
+
+import sys
+import serial
+import select
+import os
+import posix
+
+dev=sys.argv[1]
+
+ser = serial.Serial(dev, 115200, timeout=10)
+
+while True:
+    bytes = ser.read()
+    sys.stdout.buffer.write(bytes)
+    sys.stdout.flush()
+
+ser.close()

.gitlab-ci/container/arm_build.sh

@@ -19,6 +19,7 @@ apt-get -y install \
 	cpio \
 	crossbuild-essential-armhf \
 	debootstrap \
+	fastboot \
 	flex \
 	g++ \
 	gettext \
@@ -45,8 +46,10 @@ apt-get -y install \
 	python \
 	python3-distutils \
 	python3-mako \
+	python3-serial \
 	unzip \
 	wget \
+	xz-utils \
 	zlib1g-dev
 
 # dependencies where we want a specific version

.gitlab-ci/create-rootfs.sh

@@ -3,6 +3,7 @@
 set -ex
 
 apt-get -y install --no-install-recommends \
+    ca-certificates \
     initramfs-tools \
     libpng16-16 \
     strace \

.gitlab-ci/lava-gitlab-ci.yml

@@ -153,22 +153,3 @@ lima-mali450-test:arm64:
     ENV_VARS: "DEQP_PARALLEL=6"
   tags:
     - lava-meson-gxl-s905x-libretech-cc
-
-.freedreno-a307-gles2:
-  extends:
-    - .lava-test:arm64
-    - .freedreno-rules
-  variables:
-    DEVICE_TYPE: apq8016-sbc
-    GPU_VERSION: freedreno-a307
-    ENV_VARS: "DEQP_PARALLEL=4"
-    KERNEL_IMAGE_NAME: db410c.img
-    BOOT_METHOD: fastboot
-  tags:
-    - lava-mesa-db410c
-
-.freedreno-a307-gles3:
-  parallel: 6
-  extends: .freedreno-a307-gles2
-  variables:
-    DEQP_VERSION: gles3

.gitlab-ci/prepare-artifacts.sh

@@ -57,17 +57,6 @@ if [ -d /lava-files ]; then
     find -H  |  cpio -H newc -o | gzip -c - > $CI_PROJECT_DIR/artifacts/lava-rootfs-${CROSS:-arm64}.cpio.gz
     popd
 
-    if [ -z "$CROSS" ]; then
-        gzip -c artifacts/Image > Image.gz
-        cat Image.gz artifacts/apq8016-sbc.dtb > Image.gz-dtb
-        abootimg \
-            --create artifacts/db410c.img \
-            -k Image.gz-dtb \
-            -r artifacts/lava-rootfs-${CROSS:-arm64}.cpio.gz \
-            -c cmdline="ip=dhcp console=ttyMSM0,115200n8"
-        rm Image.gz Image.gz-dtb
-    fi
-
     # Store job ID so the test stage can build URLs to the artifacts
     echo $CI_JOB_ID > artifacts/build_job_id.txt