freedreno: Introduce gitlab-based CI.

Since freedreno's kernel and GPU reset seem to be totally solid, we
don't need to have the complexity of the LAVA setup that panfrost has.
Instead, we can register some boards as shared gitlab runners and have
the jobs run out of a docker container just like we do for llvmpipe.
Just make sure that the DRI device node is passed through to the
containers in the gitlab config ('devices = ["/dev/dri"]' under
runners.docker).

If a runner fails (networking dies, kernel panic, etc.) it'll take out
one build but the rest can keep going since gitlab-runner is what
pulls jobs.  Since the runner pulls jobs, it also means that they can
live behind firewalls instead of needing some public address to be
accessed by gitlab.fd.o.

For now, enable it just on db410c (A307) and cheza (A630) as those are
the hardware that I have plenty of.  A307 is only testing GLES2 since
running all of GLES3 takes too long for the number of boards I've
brought up.

Acked-by: Rob Clark <robdclark@chromium.org>
Acked-by: Kenneth Graunke <kenneth@whitecape.org>
This commit is contained in:
Eric Anholt 2019-06-28 16:35:32 -07:00
parent 0b6b0c09f4
commit 6f0dc087b7
6 changed files with 410 additions and 4 deletions

View File

@ -15,12 +15,15 @@
variables:
UPSTREAM_REPO: mesa/mesa
DEBIAN_TAG: "2019-09-11"
DEBIAN_ARM64_TAG: "arm64v8-2019-08-09"
DEBIAN_VERSION: stretch-slim
DEBIAN_ARM64_VERSION: buster-slim
DEBIAN_IMAGE: "$CI_REGISTRY_IMAGE/debian/$DEBIAN_VERSION:$DEBIAN_TAG"
DEBIAN_ARM64_IMAGE: "$CI_REGISTRY_IMAGE/debian/$DEBIAN_ARM64_VERSION:$DEBIAN_ARM64_TAG"
include:
- project: 'wayland/ci-templates'
ref: c73dae8b84697ef18e2dbbf4fed7386d9652b0cd
ref: 1f7f57c6
file: '/templates/debian.yml'
stages:
@ -49,8 +52,7 @@ stages:
# https://gitlab.com/gitlab-org/gitlab-ce/commit/8788fb925706cad594adf6917a6c5f6587dd1521
- artifacts
# CONTAINERS
# Builds the normal CI native and cross-build docker image.
debian:
extends: .debian@container-ifnot-exists
stage: containers-build
@ -59,6 +61,16 @@ debian:
GIT_STRATEGY: none # no need to pull the whole tree for rebuilding the image
DEBIAN_EXEC: 'bash .gitlab-ci/debian-install.sh'
# Builds a Docker image with the native environment and VK-GL-CTS for testing.
test-container:arm64:
extends: .debian@container-ifnot-exists@arm64v8
<<: *ci-run-policy
stage: containers-build
variables:
DEBIAN_TAG: "$DEBIAN_ARM64_TAG"
DEBIAN_VERSION: "$DEBIAN_ARM64_VERSION"
GIT_STRATEGY: none # no need to pull the whole tree for rebuilding the image
DEBIAN_EXEC: 'bash .gitlab-ci/debian-test-install.sh'
# BUILD
@ -252,7 +264,7 @@ meson-vulkan:
DRI_LOADERS: >
-D glx=disabled
-D gbm=false
-D egl=false
-D egl=true
-D platforms=surfaceless
-D osmesa=none
GALLIUM_ST: >
@ -291,6 +303,7 @@ meson-arm64:
-D build-tests=false
-D I-love-half-baked-turnips=true
-D vulkan-overlay-layer=true
BUILDTYPE: "debugoptimized"
# While the main point of this build is testing the i386 cross build,
# we also use this one to test some other options that are exclusive
@ -382,3 +395,48 @@ test-softpipe-gles3-limited:
dependencies:
- meson-main
needs: [meson-main]
arm64_a306_gles2:
extends: .deqp-test
parallel: 4
image: $DEBIAN_ARM64_IMAGE
variables:
DEQP_VER: gles2
DEQP_EXPECTED_FAILS: deqp-freedreno-a307-fails.txt
NIR_VALIDATE: 0
DEQP_RENDERER_MATCH: "FD307"
tags:
- db410c
dependencies:
- meson-arm64
.cheza-test:
image: $DEBIAN_ARM64_IMAGE
extends: .deqp-test
variables:
DEQP_RENDERER_MATCH: "FD630"
DEQP_EXPECTED_FAILS: deqp-freedreno-a630-fails.txt
DEQP_SKIPS: deqp-freedreno-a630-skips.txt
NIR_VALIDATE: 0
tags:
- mesa-cheza
dependencies:
- meson-arm64
needs: [ meson-arm64 ]
arm64_a630_gles2:
extends: .cheza-test
variables:
DEQP_VER: gles2
arm64_a630_gles3:
parallel: 6
extends: .cheza-test
variables:
DEQP_VER: gles3
arm64_a630_gles31:
parallel: 4
extends: .cheza-test
variables:
DEQP_VER: gles31

122
.gitlab-ci/README.md Normal file
View File

@ -0,0 +1,122 @@
## Mesa testing using gitlab-runner
The goal of the "test" stage of the .gitlab-ci.yml is to do pre-merge
testing of Mesa drivers on various platforms, so that we can ensure no
regressions are merged, as long as developers are merging code using
the "Merge when pipeline completes" button.
This document only covers the CI from .gitlab-ci.yml and this
directory. For other CI systems, see Intel's [Mesa
CI](https://gitlab.freedesktop.org/Mesa_CI) or panfrost's LAVA-based
CI (`src/gallium/drivers/panfrost/ci/`)
### Software architecture
For freedreno and llvmpipe CI, we're using gitlab-runner on the test
devices (DUTs), cached docker containers with VK-GL-CTS, and the
normal shared x86_64 runners to build the Mesa drivers to be run
inside of those containers on the DUTs.
The docker containers are rebuilt from the debian-install.sh script
when DEBIAN\_TAG is changed in .gitlab-ci.yml, and
debian-test-install.sh when DEBIAN\_ARM64\_TAG is changed in
.gitlab-ci.yml. The resulting images are around 500MB, and are
expected to change approximately weekly (though an individual
developer working on them may produce many more images while trying to
come up with a working MR!).
gitlab-runner is a client that polls gitlab.freedesktop.org for
available jobs, with no inbound networking requirements. Jobs can
have tags, so we can have DUT-specific jobs that only run on runners
with that tag marked in the gitlab UI.
Since dEQP takes a long time to run, we mark the job as "parallel" at
some level, which spawns multiple jobs from one definition, and then
deqp-runner.sh takes the corresponding fraction of the test list for
that job.
To reduce dEQP runtime (or avoid tests with unreliable results), a
deqp-runner.sh invocation can provide a list of tests to skip. If
your driver is not yet conformant, you can pass a list of expected
failures, and the job will only fail on tests that aren't listed (look
at the job's log for which specific tests failed).
### DUT requirements
#### DUTs must have a stable kernel and GPU reset.
If the system goes down during a test run, that job will eventually
time out and fail (default 1 hour). However, if the kernel can't
reliably reset the GPU on failure, bugs in one MR may leak into
spurious failures in another MR. This would be an unacceptable impact
on Mesa developers working on other drivers.
#### DUTs must be able to run docker
The Mesa gitlab-runner based test architecture is built around docker,
so that we can cache the debian package installation and CTS build
step across multiple test runs. Since the images are large and change
approximately weekly, the DUTs also need to be running some script to
prune stale docker images periodically in order to not run out of disk
space as we rev those containers (perhaps [this
script](https://gitlab.com/gitlab-org/gitlab-runner/issues/2980#note_169233611)).
Note that docker doesn't allow containers to be stored on NFS, and
doesn't allow multiple docker daemons to interact with the same
network block device, so you will probably need some sort of physical
storage on your DUTs.
#### DUTs must be public
By including your device in .gitlab-ci.yml, you're effectively letting
anyone on the internet run code on your device. docker containers may
provide some limited protection, but how much you trust that and what
you do to mitigate hostile access is up to you.
#### DUTs must expose the dri device nodes to the containers.
Obviously, to get access to the HW, we need to pass the render node
through. This is done by adding `devices = ["/dev/dri"]` to the
`runners.docker` section of /etc/gitlab-runner/config.toml.
### HW CI farm expectations
To make sure that testing of one vendor's drivers doesn't block
unrelated work by other vendors, we require that a given driver's test
farm produces a spurious failure no more than once a week. If every
driver had CI and failed once a week, we would be seeing someone's
code getting blocked on a spurious failure daily, which is an
unacceptable cost to the project.
Additionally, the test farm needs to be able to provide a short enough
turnaround time that people can regularly use the "Merge when pipeline
succeeds" button successfully (until we get
[marge-bot](https://github.com/smarkets/marge-bot) in place on
freedesktop.org). As a result, we require that the test farm be able
to handle a whole pipeline's worth of jobs in less than 5 minutes (to
compare, the build stage is about 10 minutes, if you could get all
your jobs scheduled on the shared runners in time.).
If a test farm is short the HW to provide these guarantees, consider
dropping tests to reduce runtime.
`VK-GL-CTS/scripts/log/bottleneck_report.py` can help you find what
tests were slow in a `results.qpa` file. Or, you can have a job with
no `parallel` field set and:
```
variables:
CI_NODE_INDEX: 1
CI_NODE_TOTAL: 10
```
to just run 1/10th of the test list.
If a HW CI farm goes offline (network dies and all CI pipelines end up
stalled) or its runners are consistenly spuriously failing (disk
full?), and the maintainer is not immediately available to fix the
issue, please push through an MR disabling that farm's jobs by adding
'.' to the front of the jobs names until the maintainer can bring
things back up. If this happens, the farm maintainer should provide a
report to mesa-dev@lists.freedesktop.org after the fact explaining
what happened and what the mitigation plan is for that failure next
time.

View File

@ -0,0 +1,104 @@
#!/bin/bash
set -e
set -o xtrace
############### Install packages for building
apt-get update
apt-get -y install ca-certificates
apt-get -y install --no-install-recommends \
g++ \
git \
pkg-config \
python \
python3-pip \
python3-setuptools \
bison \
flex \
gettext \
cmake \
ninja-build \
bc \
bzip2 \
libssl-dev \
curl \
unzip \
wget \
procps \
libexpat1 \
libelf1 \
zlib1g-dev \
libpng-dev \
libgbm-dev \
libgles2-mesa-dev
export LIBDRM_VERSION=libdrm-2.4.99
pip3 install meson
############### Build libdrm
wget https://dri.freedesktop.org/libdrm/$LIBDRM_VERSION.tar.bz2
tar -xvf $LIBDRM_VERSION.tar.bz2 && rm $LIBDRM_VERSION.tar.bz2
cd $LIBDRM_VERSION; meson build/ -Detnaviv=true; ninja -C build/ install; cd ..
rm -rf $LIBDRM_VERSION
############### Build dEQP
git config --global user.email "mesa@example.com"
git config --global user.name "Mesa CI"
# XXX: Use --depth 1 once we can drop the cherry-picks.
git clone \
https://github.com/KhronosGroup/VK-GL-CTS.git \
-b opengl-es-cts-3.2.5.1 \
/VK-GL-CTS
cd /VK-GL-CTS
# Fix surfaceless build
git cherry-pick -x 22f41e5e321c6dcd8569c4dad91bce89f06b3670
git cherry-pick -x 1daa8dff73161ea60ead965bd6c9f2a0a2165648
# surfaceless links against libkms and such despite not using it.
sed -i '/gbm/d' targets/surfaceless/surfaceless.cmake
sed -i '/libkms/d' targets/surfaceless/surfaceless.cmake
sed -i '/libgbm/d' targets/surfaceless/surfaceless.cmake
# --insecure is due to SSL cert failures hitting sourceforge for zlib and
# libpng (sigh). The archives get their checksums checked anyway, and git
# always goes through ssh or https.
python3 external/fetch_sources.py --insecure
mkdir -p /deqp
cd /deqp
cmake -G Ninja \
-DDEQP_TARGET=surfaceless \
-DCMAKE_BUILD_TYPE=Release \
/VK-GL-CTS
ninja
# Copy out the mustpass lists we want from a bunch of other junk.
mkdir /deqp/mustpass
for gles in gles2 gles3 gles31; do
cp \
/deqp/external/openglcts/modules/gl_cts/data/mustpass/gles/aosp_mustpass/3.2.5.x/$gles-master.txt \
/deqp/mustpass/$gles-master.txt
done
rm -rf /deqp/external
rm -rf /deqp/modules/internal
rm -rf /deqp/executor
rm -rf /deqp/execserver
rm -rf /deqp/modules/egl
rm -rf /deqp/framework
du -sh *
rm -rf /VK-GL-CTS
############### Uninstall the build software
apt-get purge -y \
cmake \
git \
gcc \
g++ \
bison \
flex \
ninja-build

View File

@ -0,0 +1,101 @@
dEQP-GLES2.functional.clipping.line.wide_line_clip_viewport_center
dEQP-GLES2.functional.clipping.line.wide_line_clip_viewport_corner
dEQP-GLES2.functional.clipping.point.wide_point_clip
dEQP-GLES2.functional.clipping.point.wide_point_clip_viewport_center
dEQP-GLES2.functional.clipping.point.wide_point_clip_viewport_corner
dEQP-GLES2.functional.clipping.triangle_vertex.clip_three.clip_neg_x_neg_z_and_pos_x_pos_z_and_neg_x_neg_y_pos_z
dEQP-GLES2.functional.fbo.render.recreate_depthbuffer.rebind_rbo_rgb565_depth_component16
dEQP-GLES2.functional.fbo.render.recreate_depthbuffer.rebind_rbo_rgb5_a1_depth_component16
dEQP-GLES2.functional.fbo.render.recreate_depthbuffer.rebind_rbo_rgba4_depth_component16
dEQP-GLES2.functional.fbo.render.recreate_depthbuffer.rebind_tex2d_rgba_depth_component16
dEQP-GLES2.functional.fbo.render.recreate_depthbuffer.rebind_tex2d_rgb_depth_component16
dEQP-GLES2.functional.fbo.render.recreate_stencilbuffer.rebind_rbo_rgb565_stencil_index8
dEQP-GLES2.functional.fbo.render.recreate_stencilbuffer.rebind_rbo_rgb5_a1_stencil_index8
dEQP-GLES2.functional.fbo.render.recreate_stencilbuffer.rebind_rbo_rgba4_stencil_index8
dEQP-GLES2.functional.fbo.render.recreate_stencilbuffer.rebind_tex2d_rgba_stencil_index8
dEQP-GLES2.functional.fbo.render.recreate_stencilbuffer.rebind_tex2d_rgb_stencil_index8
dEQP-GLES2.functional.polygon_offset.fixed16_displacement_with_units
dEQP-GLES2.functional.texture.filtering.2d.linear_nearest_clamp_etc1
dEQP-GLES2.functional.texture.filtering.2d.linear_nearest_clamp_l8_npot
dEQP-GLES2.functional.texture.filtering.2d.linear_nearest_clamp_l8_pot
dEQP-GLES2.functional.texture.filtering.2d.linear_nearest_clamp_rgb888_npot
dEQP-GLES2.functional.texture.filtering.2d.linear_nearest_clamp_rgb888_pot
dEQP-GLES2.functional.texture.filtering.2d.linear_nearest_clamp_rgba4444_npot
dEQP-GLES2.functional.texture.filtering.2d.linear_nearest_clamp_rgba4444_pot
dEQP-GLES2.functional.texture.filtering.2d.linear_nearest_clamp_rgba8888_npot
dEQP-GLES2.functional.texture.filtering.2d.linear_nearest_clamp_rgba8888_pot
dEQP-GLES2.functional.texture.filtering.2d.linear_nearest_mirror_etc1
dEQP-GLES2.functional.texture.filtering.2d.linear_nearest_mirror_l8_pot
dEQP-GLES2.functional.texture.filtering.2d.linear_nearest_mirror_rgb888_pot
dEQP-GLES2.functional.texture.filtering.2d.linear_nearest_mirror_rgba4444_pot
dEQP-GLES2.functional.texture.filtering.2d.linear_nearest_mirror_rgba8888_pot
dEQP-GLES2.functional.texture.filtering.2d.linear_nearest_repeat_etc1
dEQP-GLES2.functional.texture.filtering.2d.linear_nearest_repeat_l8_pot
dEQP-GLES2.functional.texture.filtering.2d.linear_nearest_repeat_rgb888_pot
dEQP-GLES2.functional.texture.filtering.2d.linear_nearest_repeat_rgba4444_pot
dEQP-GLES2.functional.texture.filtering.2d.linear_nearest_repeat_rgba8888_pot
dEQP-GLES2.functional.texture.filtering.2d.nearest_linear_clamp_etc1
dEQP-GLES2.functional.texture.filtering.2d.nearest_linear_clamp_l8_npot
dEQP-GLES2.functional.texture.filtering.2d.nearest_linear_clamp_l8_pot
dEQP-GLES2.functional.texture.filtering.2d.nearest_linear_clamp_rgb888_npot
dEQP-GLES2.functional.texture.filtering.2d.nearest_linear_clamp_rgb888_pot
dEQP-GLES2.functional.texture.filtering.2d.nearest_linear_clamp_rgba4444_npot
dEQP-GLES2.functional.texture.filtering.2d.nearest_linear_clamp_rgba4444_pot
dEQP-GLES2.functional.texture.filtering.2d.nearest_linear_clamp_rgba8888_npot
dEQP-GLES2.functional.texture.filtering.2d.nearest_linear_clamp_rgba8888_pot
dEQP-GLES2.functional.texture.filtering.2d.nearest_linear_mirror_etc1
dEQP-GLES2.functional.texture.filtering.2d.nearest_linear_mirror_l8_pot
dEQP-GLES2.functional.texture.filtering.2d.nearest_linear_mirror_rgb888_pot
dEQP-GLES2.functional.texture.filtering.2d.nearest_linear_mirror_rgba4444_pot
dEQP-GLES2.functional.texture.filtering.2d.nearest_linear_mirror_rgba8888_pot
dEQP-GLES2.functional.texture.filtering.2d.nearest_linear_repeat_etc1
dEQP-GLES2.functional.texture.filtering.2d.nearest_linear_repeat_l8_pot
dEQP-GLES2.functional.texture.filtering.2d.nearest_linear_repeat_rgb888_pot
dEQP-GLES2.functional.texture.filtering.2d.nearest_linear_repeat_rgba4444_pot
dEQP-GLES2.functional.texture.filtering.2d.nearest_linear_repeat_rgba8888_pot
dEQP-GLES2.functional.texture.filtering.cube.linear_nearest_clamp_etc1
dEQP-GLES2.functional.texture.filtering.cube.linear_nearest_clamp_l8_npot
dEQP-GLES2.functional.texture.filtering.cube.linear_nearest_clamp_l8_pot
dEQP-GLES2.functional.texture.filtering.cube.linear_nearest_clamp_rgb888_npot
dEQP-GLES2.functional.texture.filtering.cube.linear_nearest_clamp_rgb888_pot
dEQP-GLES2.functional.texture.filtering.cube.linear_nearest_clamp_rgba4444_npot
dEQP-GLES2.functional.texture.filtering.cube.linear_nearest_clamp_rgba4444_pot
dEQP-GLES2.functional.texture.filtering.cube.linear_nearest_clamp_rgba8888_npot
dEQP-GLES2.functional.texture.filtering.cube.linear_nearest_clamp_rgba8888_pot
dEQP-GLES2.functional.texture.filtering.cube.linear_nearest_mirror_l8_pot
dEQP-GLES2.functional.texture.filtering.cube.linear_nearest_mirror_rgb888_pot
dEQP-GLES2.functional.texture.filtering.cube.linear_nearest_mirror_rgba4444_pot
dEQP-GLES2.functional.texture.filtering.cube.linear_nearest_mirror_rgba8888_pot
dEQP-GLES2.functional.texture.filtering.cube.linear_nearest_repeat_l8_pot
dEQP-GLES2.functional.texture.filtering.cube.linear_nearest_repeat_rgb888_pot
dEQP-GLES2.functional.texture.filtering.cube.linear_nearest_repeat_rgba4444_pot
dEQP-GLES2.functional.texture.filtering.cube.linear_nearest_repeat_rgba8888_pot
dEQP-GLES2.functional.texture.filtering.cube.nearest_linear_clamp_etc1
dEQP-GLES2.functional.texture.filtering.cube.nearest_linear_clamp_l8_npot
dEQP-GLES2.functional.texture.filtering.cube.nearest_linear_clamp_l8_pot
dEQP-GLES2.functional.texture.filtering.cube.nearest_linear_clamp_rgb888_npot
dEQP-GLES2.functional.texture.filtering.cube.nearest_linear_clamp_rgb888_pot
dEQP-GLES2.functional.texture.filtering.cube.nearest_linear_clamp_rgba4444_npot
dEQP-GLES2.functional.texture.filtering.cube.nearest_linear_clamp_rgba4444_pot
dEQP-GLES2.functional.texture.filtering.cube.nearest_linear_clamp_rgba8888_npot
dEQP-GLES2.functional.texture.filtering.cube.nearest_linear_clamp_rgba8888_pot
dEQP-GLES2.functional.texture.filtering.cube.nearest_linear_mirror_l8_pot
dEQP-GLES2.functional.texture.filtering.cube.nearest_linear_mirror_rgb888_pot
dEQP-GLES2.functional.texture.filtering.cube.nearest_linear_mirror_rgba4444_pot
dEQP-GLES2.functional.texture.filtering.cube.nearest_linear_mirror_rgba8888_pot
dEQP-GLES2.functional.texture.filtering.cube.nearest_linear_repeat_l8_pot
dEQP-GLES2.functional.texture.filtering.cube.nearest_linear_repeat_rgb888_pot
dEQP-GLES2.functional.texture.filtering.cube.nearest_linear_repeat_rgba4444_pot
dEQP-GLES2.functional.texture.filtering.cube.nearest_linear_repeat_rgba8888_pot
dEQP-GLES2.functional.texture.vertex.2d.filtering.linear_nearest_clamp
dEQP-GLES2.functional.texture.vertex.2d.filtering.linear_nearest_mirror
dEQP-GLES2.functional.texture.vertex.2d.filtering.linear_nearest_repeat
dEQP-GLES2.functional.texture.vertex.2d.filtering.nearest_linear_clamp
dEQP-GLES2.functional.texture.vertex.2d.filtering.nearest_linear_mirror
dEQP-GLES2.functional.texture.vertex.2d.filtering.nearest_linear_repeat
dEQP-GLES2.functional.texture.vertex.cube.filtering.linear_nearest_clamp
dEQP-GLES2.functional.texture.vertex.cube.filtering.linear_nearest_mirror
dEQP-GLES2.functional.texture.vertex.cube.filtering.linear_nearest_repeat
dEQP-GLES2.functional.texture.vertex.cube.filtering.nearest_linear_clamp
dEQP-GLES2.functional.texture.vertex.cube.filtering.nearest_linear_mirror
dEQP-GLES2.functional.texture.vertex.cube.filtering.nearest_linear_repeat

View File

@ -0,0 +1,6 @@
dEQP-GLES2.functional.clipping.triangle_vertex.clip_three.clip_neg_x_neg_z_and_pos_x_pos_z_and_neg_x_neg_y_pos_z
dEQP-GLES3.functional.draw.random.146
dEQP-GLES3.functional.draw.random.71
dEQP-GLES3.functional.fbo.msaa.2_samples.stencil_index8
dEQP-GLES31.functional.stencil_texturing.render.depth24_stencil8_clear
dEQP-GLES31.functional.stencil_texturing.render.depth24_stencil8_draw

View File

@ -0,0 +1,15 @@
# Note: skips lists for CI are just a list of lines that, when
# non-zero-length and not starting with '#', will regex match to
# delete lines from the test list. Be careful.
# Skip the perf/stress tests to keep runtime manageable
dEQP-GLES[0-9]*.performance
dEQP-GLES[0-9]*.stress
# These are really slow on tiling architectures (including llvmpipe).
dEQP-GLES[0-9]*.functional.flush_finish
# Unstable test results
dEQP-GLES3.functional.fragment_out.random.*
dEQP-GLES3.functional.transform_feedback.*
dEQP-GLES31.functional.primitive_bounding_box.*