Commit Graph

136 Commits

Author SHA1 Message Date
Juan A. Suarez Romero 3f1c375581 ci/broadcom: allow custom kernels
So far, testing VC4 and V3D/V3DV requires the CI runners having access
to a Raspberry Pi 3/4 kernel, and the correspondent modules and
bootloader files. If a different kernel must be used, it means touching
the runners to provide them.

This commit adds the option to define an URL pointing to a (compressed)
tarball containing such files, without requiring dealing with the
runners. This link is provided through the `BM_BOOTFS` job variable.

The tarball must contain two directories in the root: a `/boot`
directory (containing the kernel, DTBs and bootloader files), and a
`/lib/modules` (or `/usr/lib/modules`) with the kernel modules.

Reviewed-by: Eric Anholt <eric@anholt.net>
Signed-off-by: Juan A. Suarez Romero <jasuarez@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9527>
2021-03-12 11:03:17 +00:00
Christian Gmeiner 8413de5db0 ci/bare-metal: fix fastboot
Only copy results from NFS if BM_FASTBOOT_NFSROOT is set.

Signed-off-by: Christian Gmeiner <christian.gmeiner@gmail.com>
Reviewed-by: Eric Anholt <eric@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9482>
2021-03-09 23:09:10 +00:00
Eric Anholt bcdfee3bcd ci/freedreno: Switch the fastboot boards to using nfsroot.
This saves time in packing the rootfs, allows for larger rootfses, and
avoids the need for webdav.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9314>
2021-03-03 21:05:39 +00:00
Juan A. Suarez Romero e0b0507635 ci/broadcom: retry always when serial log timeout
So far we were retrying the testing (through device rebooting) if we did
not detect the boot sequence.

But found a couple of times that the serial log can also be "lost"
during the testing process. In all those times a manual retry of the job
was enough to complete the test.

Thus, let's apply the retry once automatically in this case.

Signed-off-by: Juan A. Suarez Romero <jasuarez@igalia.com>
Reviewed-by: Eric Anholt <eric@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9335>
2021-03-01 18:22:24 +00:00
Juan A. Suarez Romero e45d372968 ci/baremetal: highlight message errors
Highlight in red errors from the baremetal run, so user is more aware of
what happened.

Signed-off-by: Juan A. Suarez Romero <jasuarez@igalia.com>
Reviewed-by: Eric Anholt <eric@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9335>
2021-03-01 18:22:24 +00:00
Juan A. Suarez Romero e814e23f59 ci/piglit: allow parallel piglit jobs
This allows to split a piglit job in several parallel jobs, to speed up
the execution.

Due piglit restrictions, this only works for single profiles. Otherwise
an error will be shown in the runner.

Also, a new gitlab job variable `PIGLIT_TESTS` is introduced that
contains the excluded/included tests with `-x` or `-n`. The rest of the
piglit options go to `PIGLIT_OPTIONS` (like `--timeout n`).

v2 (Andres):
 - Replay profile is supported in parallel jobs.
 - Bail out inmediately if parallel jobs is tried with multiple
profiles.
 - Use testlist only when doing parallel jobs.
 - Do not drop pass tests when filtering executed tests.
 - Get rid of PIGLIT_FRACTION.

v4:
 - uncommit unrelated change (Andres).

Signed-off-by: Juan A. Suarez Romero <jasuarez@igalia.com>
Reviewed-by: Andres Gomez <agomez@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9022>
2021-02-24 09:41:33 +01:00
Eric Anholt 9500341847 ci/freedreno: Add a fractional gles31 run with asan enabled.
We have to disable the GLSL unit tests because with asan it runs way too
much code under qemu and times out.  Those unit tests have coverage on
x86, anyway.

I also included a vulkan run, which is disabled by default due to timeouts
that I need to sort out still.  It should be a useful tool for turnip
devs, though.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9070>
2021-02-18 00:49:00 +00:00
Andres Gomez 7393303637 ci: correct artifacts location for piglit's runner messages
We are now using pages.

v2:
  - Define a helper variable for the artifacts base URL (Juan).

Signed-off-by: Andres Gomez <agomez@igalia.com>
Acked-by: Eric Anholt <eric@anholt.net> [v1]
Reviewed-by: Juan A. Suarez <jasuarez@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9092>
2021-02-17 14:37:35 +00:00
Juan A. Suarez Romero d8236e32de ci/v3d: Add V3D and V3DV testing
Add OpenGL and Vulkan testing for V3D and V3DV respectively.

Add also a couple of manual piglit jobs for V3D.

v2:
 - Replace custom mustpass with running fraction of tests (Eric)

Signed-off-by: Juan A. Suarez Romero <jasuarez@igalia.com>
Reviewed-by: Eric Anholt <eric@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8745>
2021-02-15 15:47:31 +00:00
Juan A. Suarez Romero 5d0c96a8c2 ci: add option to overwrite CPU arch
When loading Vulkan ICD file, it uses the CPU machine identifier to
load the correct one, in case multiple versions are installed.

This is fine if the machine where Mesa has been built and the machine
where the test is run are exactly the same. But this is not always the
case. As example, for armhf architecture, the machine where Mesa is
built is identified as `arm7hlf`, but the Raspberry Pi 4 is identified
as `armv7l`, so it will fail to load the ICD file, though both are
totally compatible.

This allow to define the architecture instead.

Signed-off-by: Juan A. Suarez Romero <jasuarez@igalia.com>
Reviewed-by: Eric Anholt <eric@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8745>
2021-02-15 15:47:31 +00:00
Eric Anholt 48dd9b7e34 ci/deqp: Bump runner to 0.5.1 for recent runtime perf improvements.
3 commits in 0.5.0:

- 20-40s savings on many of our CI runs by dropping the clever test size
  scaling code.

- Even bigger savings (especially on deqp-vk runs) by increasing maximuim
  test group size (~1/4 of runtime was spawning deqp on cheza, that cost
  is cut by ~75%)

- No more needing to manually set MESA_DEBUG=silent

2 commits in 0.5.1:

- Fixed automatic thread pool sizing to keep all CPUs busy (thanks for
  catching that Bas!).

- Automatically size down test groups on short test lists and many CPUs,
  so split the list evenly between CPUs (such as on freedreno -options
  jobs).

Acked-by: Daniel Stone <daniel@fooishbar.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8787>
2021-02-07 21:42:39 -08:00
Eric Anholt 23350b5939 ci/freedreno: Do our piglit runs against Xorg.
We were using surfaceless, which misses out on some useful coverage we'd
like to have in the GLX/EGL piglit tests, but more importantly prevented
many traces from running.

Reviewed-by: Daniel Stone <daniel@fooishbar.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8727>
2021-02-01 18:32:01 +00:00
Eric Anholt 2af6b53588 ci/freedreno: Use the http cache for artifacts downloads, too.
The gitlab artifacts handling has been slow in the past as we hit
gitlab.fdo from multiple runners, and it costs fd.o egress bandwidth.  Use
the local http cache against the packet.net minio to cut that downloads
cost.

Closes: #3249
Reviewed-by: Daniel Stone <daniel@fooishbar.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8727>
2021-02-01 18:32:01 +00:00
Eric Anholt ce1bb26b06 ci/freedreno: Detect cheza HFI errors and restart the run.
These are intermittent (~1/day), seem to be around GPU faults (so
hopefully will go away once we clean up piglit's fault errors), and are
probably also related to our vintage firmware.  Until we can get new
hardware in the farm, just restart the flaked job.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8722>
2021-01-26 19:17:27 +00:00
Juan A. Suarez Romero ea88e1c820 ci/vc4: allow custom timeout values for activity
The script that monitors activity in the serial assumes that something
was wrong if it does not detect activity in 60 seconds, rebooting the
device and re-trying the test again.

While this timeout is enough for most cases, in some cases it is not
enough. For instance, when executing piglit testsuite it takes quite a
few time to generate the results after the test is done.

This allow to setup a custom timeout (`BM_POE_TIMEOUT`) in the proper
jobs.

Signed-off-by: Juan A. Suarez Romero <jasuarez@igalia.com>
Acked-by: Andres Gomez <agomez@igalia.com>
Reviewed-by: Eric Anholt <eric@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8702>
2021-01-26 09:45:29 +00:00
Juan A. Suarez Romero 8588fb65d6 vc4/ci: Replace expect script by python script
Replace the expect-based script to turn on/off the Raspberry Pi devices
using a python-based script.

v2:
 - Fix small nitpicks (Juan)
 - Limit line length (Andres)

v3:
 - Bump image tags (Eric, Andres)

v4:
 - Bump image tags (Eric)

Signed-off-by: Juan A. Suarez Romero <jasuarez@igalia.com>
Acked-by: Andres Gomez <agomez@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8362>
2021-01-23 05:35:24 +00:00
Eric Anholt 9ef2c44ce6 ci: Add a530 and a630 piglit runs.
These bring a whole lot of new coverage to these drivers, since dEQP is
bad at desktop GL feature coverage around early GL 3.x.  piglit also gets
at a lot of MSAA, fast clearing, and texture layout issues that dEQP
doesn't do much with.

Reviewed-by: Christian Gmeiner <christian.gmeiner@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7370>
2021-01-02 02:43:21 +00:00
Eric Anholt cab48ba9f5 ci/bare-metal: Pass through FDO_CI_CONCURRENT on bare-metal runners.
I've set it up in the gitlab-runer config on all the freedreno boards.
This means that for piglit, where the run.sh always choose either this
variable or 4 threads otherwise, we'll have the right number of parallel
tasks.

Reviewed-by: Christian Gmeiner <christian.gmeiner@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7370>
2021-01-02 02:43:21 +00:00
Christian Gmeiner 6e093a4493 ci/bare-metal: pass thorugh PIGLIT env vars
Signed-off-by: Christian Gmeiner <christian.gmeiner@gmail.com>
Reviewed-by: Eric Anholt <eric@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7370>
2021-01-02 02:43:21 +00:00
Christian Gmeiner b020feb4d1 ci/fastboot: exclude either deqp or piglit
Keep the generated initramfs image as small as possible.

Signed-off-by: Christian Gmeiner <christian.gmeiner@gmail.com>
Reviewed-by: Eric Anholt <eric@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7370>
2021-01-02 02:43:21 +00:00
Christian Gmeiner 12647ac193 ci/bare-metal: build full piglit for baremetal ARM targets.
ARM64 had it for traces only, upgrade it to a full build so we can test
a630.  We also add it for armhf, as we'll want it on both rpi and etnaviv.

Bumped the LAVA tag as well, since the script changes a bit and it does
impact the final image (even if we aren't pulling in full piglit there
yet).  Note I also had to drop the "v" on the tarring of their rootfs, as
the verbosity on baremetal was exceeding job log size.

Signed-off-by: Christian Gmeiner <christian.gmeiner@gmail.com>
Reviewed-by: Eric Anholt <eric@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7370>
2021-01-02 02:43:21 +00:00
Andres Gomez 54bdec63ef ci: add piglit job to baremetal and remove tracie ones
v2:
  - Squashed the commit to remove tracie jobs (Eric).

v3:
  - Rename *-piglit-traces jobs with *-traces (Eric).

Signed-off-by: Andres Gomez <agomez@igalia.com>
Reviewed-by: Eric Anholt <eric@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6388>
2020-12-22 18:31:01 +00:00
Andres Gomez 654bfb0012 ci: build piglit inside baremetal and LAVA's rootfs
v2:
  - Updated the ci-fairy commit to use.

Signed-off-by: Andres Gomez <agomez@igalia.com>
Reviewed-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6388>
2020-12-22 18:31:01 +00:00
Eric Anholt 463dbbffa8 ci/deqp: Make sure that we pull in all board-specific xfail/skip/flake files.
When introducing/removing these files, it's easy to forget to update the
yml to point to them.  Instead of requiring the separate update, just have
the runner script pick the right one from a single per-gpu variable.

As a result, we now pick up the new deqp-lvp-skips.txt that was added but
not conected.  This also required moving some bypass flakes from the
shared a630 flakes list to a separate list, which is a feature because now
we'd notice the introduction of flakes to the gmem path.

Fixes: ab79e6b8e3 ("ci: skip failing test on lavapipe")
Acked-by: Daniel Stone <daniels@collabora.com>
Reviewed-by: Juan A. Suarez <jasuarez@igalia.com>
Reviewed-by: Christian Gmeiner <christian.gmeiner@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8147>
2020-12-21 18:44:43 +00:00
Eric Anholt 5fca7cd8b8 ci/freedreno: Detect the cheza power management bus error and restart.
This is an issue on the cheza platform, the theory is due to some old
firmware bug that will be fixed in future platforms.  Given that cheza was
a target that didn't get released and we expect future platforms to be
fixed, just detect the issue and restart.

I've noticed this error in my CI monitoring less than once a week.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7993>
2020-12-08 23:31:17 +00:00
Juan A. Suarez Romero 72b68bd2a6 ci: add testing for VC4 drivers (Raspberry Pi 3)
This tests OpenGL ES 2.0 CTS suite with VC4 drivers, through baremetal
Raspberry Pi 3 devices.

The devices are connected to a switch that supports Power over Ethernet
(PoE), so the devices can be started/stopped through the switch, and
also to a host that runs the GitLab runner through serial-to-USB cables,
to monitor the devices to know when the testing finishes.

The Raspberries uses a network boot, using NFS and TFTP. For the root
filesystem, they use the one created in the armhf container. For the
kernel/modules case, this is handled externally. Currently it is using
the same kernel/modules that come with the Raspberry Pi OS. In future we
could build them in the same armhf container.

At this moment we only test armhf architecture, as this is the default
one suggested by the Raspberry Pi Foundation. In future we could also
add testing for arm64 architecture.

Finally, for the very rare ocassions where the Raspberry Pi 3 device is
booted but no data is received, it retries the testing for a second
time, powering off and on the device in the process.

v2:
 - Remove commit that exists capture devcoredump (Eric)
 - Squash remaining commits in one (Andres)

v3:
 - Add missing boot timeout check (Juan)

v4:
 - Use locks when running the PoE on/off script (Eric)
 - Use a timeout for serial read (Eric)

v5:
 - Rename stage to "raspberrypi" (Eric)
 - Bump up arm64_test tag (Eric)

v6:
 - Make serial buffer timeout optional (Juan)

Reviewed-by: Eric Anholt <eric@anholt.net>
Signed-off-by: Juan A. Suarez Romero <jasuarez@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7628>
2020-12-04 19:05:19 +01:00
Eric Anholt 6bc35c00e2 ci/deqp: Allow specifying the caselist fraction separate from CI_NODE_INDEX.
To increase our VK coverage on a630, we want to have two jobs in parallel,
but we still can't hit full coverage so we need the fractional setting to
be separate from gitlab CI's flags for setting up parallel jobs.

Reviewed-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6971>
2020-11-11 17:22:47 +00:00
Eric Anholt 2998a0b055 ci/freedreno: Group the short a630 dEQP runs into one test job.
This saves the minute and a half boot time on each of these minute-or-less
test jobs.  The whole job was 3.5 minutes in my last run.

Acked-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6971>
2020-11-11 17:22:47 +00:00
Eric Anholt bf29daa1b5 ci/deqp: Switch to a new dEQP runner written in Rust.
I found the C++ runner hard to develop on, and we had stability issues and
outstanding feature needs that made me want something I felt good about
hacking on.  Thus, Rewrite It In Rust of the deqp runner.

The new runner includes:

- Skip lists don't reshuffle the test list.
- Known-flake handling without resorting to skip lists (fixing our main CI
  reliability issue on a3xx right now).
- Per-thread Vulkan shader caches should speed up VK CI runtime.
- Tracking of crashes separate from fails (so we can see progress on that
  front).
- Logging of deqp stderr spam (particularly assertion failures!) in the CI
  log.
- Integrated QPA filtering so we don't have bash perf issues for it.
- Logging of what caselist to go look at for a given error report (in red,
  so it's easier to find in your CI log).
- The code is 1/3 unit tests, and easy to extend for more coverage.
- Non-LAVA CI runs create a failures.csv in artifacts that you can check
  in as your deqp-*-fails.txt file.
- Test runtime is included in results.csv so you can debug how to speed up
  your CI job.
- Pretty summary at the end of the run of slow/flaky/failed tests.

Since this is a new runner with a different RNG, the test groups are
shuffled one more time.  This seems to result in some panfrost T720
stability issues (See its new deqp-panfrost-t720-flakes.txt), and one new
flake in freedreno a630.

Reviewed-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7434>
2020-11-06 19:48:39 +00:00
Eric Anholt fe61230b38 ci/bare-metal: Reset colors at the end of a line of serial output.
We don't want the next line of our timestamp and other context to inherit
colors set by the serial command (visible with the new dEQP runner)

Reviewed-by: Christian Gmeiner <christian.gmeiner@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7434>
2020-11-06 19:48:39 +00:00
Eric Anholt ff6741728d ci/bare-metal: Apply autopep8 to the bare-metal scripts.
Let's follow proper python formatting (easy now that vscode does it for
me)

Reviewed-by: Christian Gmeiner <christian.gmeiner@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7434>
2020-11-06 19:48:39 +00:00
Christian Gmeiner 915f2919f6 ci/bare-metal: suppress 'No such file or directory'
It fills the serial log with unimportant messages.

Signed-off-by: Christian Gmeiner <christian.gmeiner@gmail.com>
Reviewed-by: Eric Anholt <eric@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7347>
2020-10-28 18:04:22 +00:00
Eric Anholt e3c7748b2e ci/bare-metal: Move the "POWER_GOOD not seen in time" check to the right time.
The poweron failure happens before we get to the bootloader
("load_archive: loading locale_en.bin") not after we're trying to boot the
kernel and we're waiting for the deqp run to complete.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6970>
2020-10-02 02:41:37 +00:00
Rob Clark aee1c08c06 ci/deqp-runner: Allow overriding width/height/config
This will allow adding multi-sample caselists, and jobs with larger
surface size.

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6553>
2020-09-29 19:58:50 +00:00
Eric Anholt 0f61f0142a ci/bare-metal: Allow wget of the kernel/dtb for kernel development.
It's useful for kernel dev to be able throw all of our testing
infrastructure at a risky kernel change, but it's expensive (time and
bandwidth) to roll new containers every time your rev your kernel.  Make
it so you can just point the env vars to your personal build you've
uploaded.

Reviewed-by: Kristian H. Kristensen <hoegsberg@google.com>
Reviewed-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6592>
2020-09-09 17:25:38 +00:00
Eric Anholt fd2ee49b21 ci/bare-metal: Use python for handling fastboot booting and parsing
Modeling after what I did for cros_servo_run.py, this gives us easy
support for restarting the test run a530 when we detect a spontaneous
reboot.  I had to touch up serial_buffer.py to handle buffering in from a
file instead of a serial device, to support the upcoming etnaviv CI
(tested by running it against a serial log from db410c and seeing it step
to calling "fastboot")

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6529>
2020-09-03 23:22:44 +00:00
Eric Anholt 0453a46f66 ci/bare-metal: Fix capturing of serial output as job artifacts.
I tried to put them in the wrong directory -- everything needs to go in
results/, which we want clean and ready before we start our job.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6529>
2020-09-03 23:22:43 +00:00
Eric Anholt 24f5f11719 ci/bare-metal: Log why our run restarts when it does.
It would be confusing to see a job quietly restart itself in the middle.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6529>
2020-09-03 23:22:43 +00:00
Eric Anholt 785d3cace4 ci/bare-metal: Include a timestamp in our serial reads.
gitlab CI doesn't include timestamps in its logs by default, but it's
really useful for finding delays in our CI so stuff one in on the lines
coming in from serial and being output to the gitlab log.  The artifacts
file is still the raw serial output.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6529>
2020-09-03 23:22:42 +00:00
Eric Anholt ff42b7e804 ci/bare-metal: Fix detection of "POWER_GOOD not seen in time" fails
We were only reading from the CPU serial, not EC, so we'd never notice
these sources of job timeouts.  I couldn't find a cleaner solution, so I
spawned two threads to do the blocking reads from our serial line fifos
and merge them together in a single queue to read.

Closes: #3470
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6529>
2020-09-03 23:22:41 +00:00
Eric Anholt b7787ce18d ci/bare-metal: Use re.search() instead re.match() for our line matching.
match() looks for the start of the line to match our regex, while search
just looks for the regex anywhere in the line.  I messed this up when
converting our greps in shell to python, which was part of breaking the
POWER_GOOD flake detection.  Most of our matches worked, but let's
consistently use this one so we don't mess this up in the future.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6529>
2020-09-03 23:22:40 +00:00
Eric Anholt 2da1178bf3 ci/bare-metal: Try rebooting chezas again if they get stuck during tftp.
Occasionally something goes weird in the network and a group of chezas
will produce streams of these errors during the tftp process, eventually
timing out after 60 minutes in the job.  By the time we notice, the next
jobs seem to go through fine, so watch for them and try rebooting the
cheza to see if that gets our jobs to pass again.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6398>
2020-08-21 20:10:18 +00:00
Eric Anholt c27075e9e1 ci/bare-metal: Retry booting chezas instead of failing when !POWER_GOOD
If we get this error, we can just try rebooting again and see if it comes
up then.  The POWER_GOOD failures are clustered in time, but it's better
to retry a few times in a row in one job (which has its own 60min timeout)
than to spuriously fail someone's pipeline.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6398>
2020-08-21 20:10:18 +00:00
Eric Anholt c63648121e ci/bare-metal: Convert the main cros-servo boot code to python
Switching this part to python makes the code clearer and cleans up our
logs as well.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6398>
2020-08-21 20:10:18 +00:00
Eric Anholt b4374080d7 ci/bare-metal: Use a new serial buffer tool.
This one uses python threads to move some of our logic from shell
pipelines to python, and opens the door to doing better serial output
tracking in the future (the SerialBuffer.lines() method)

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6398>
2020-08-21 20:10:18 +00:00
Eric Anholt 30da82c74c ci/deqp-runner: Add a post-deqp-run filter list for known flakes.
So far, we've been putting our known flakes that intermittently fail CI
into the skips list.  This has two downsides:

1) You don't know when the flakes stop happening and when to delist them
   from skips, unless you go do a bunch of manual runs with the skips list
   cleared.

2) If the flake was because the previous test left some broken state in
   the HW, you may just move your intermittent to a new test.

With this new path, you can list your flakes in the flakes file to keep
them from erroring out people's pipelines.  They still get run and
reported as is.

Reviewed-by: Christian Gmeiner <christian.gmeiner@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6392>
2020-08-20 23:59:50 +00:00
Eric Engestrom b5d36e9cb6 gitlab-ci: fix quoting of variables passed down to bare-metal runners
Eric Anholt identified the issue when merging one of my MRs: the
variable contained words in '`' backticks, which caused them to be
executed by the bare metal runner's shell.

Quote the value printed using bash's shell expansion feature to make
sure anything in the future will be properly quoted.

Signed-off-by: Eric Engestrom <eric@engestrom.ch>
Reviewed-by: Eric Anholt <eric@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6389>
2020-08-19 21:53:31 +00:00
Eric Engestrom 72fac11ca3 gitlab-ci: fix testing whether a variable with a given name is set or not
The previous code considered unset variables the same as set-but-empty;
sometimes setting a variable as something empty is meaningful, so let's
pass them through properly.

Signed-off-by: Eric Engestrom <eric@engestrom.ch>
Reviewed-by: Eric Anholt <eric@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6389>
2020-08-19 21:53:31 +00:00
Tomeu Vizoso 7d5f4d3f70 ci: Split traces.yml file per driver
As drivers have been tested with more and more traces, the yml file is
becoming a bit unwieldy. As more drivers are going to be tested with
traces, and more traces will be used, split them in per-driver files so
the size stays manageable.

Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Reviewed-by: Reviewed-By: Rohan Garg <rohan.garg@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6185>
2020-08-19 07:47:48 +02:00
Tomeu Vizoso 0b2478381f ci: Actually upload trace artifacts to MinIO for baremetal
Baremetal jobs filter the variables they get from .gitlab-ci.yml, and
TRACIE_UPLOAD_TO_MINIO and others weren't being let through.

Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Acked-by: Daniel Stone <daniels@collabora.com>
Fixes: d4ca45eca2 ("ci: Upload traces' reference and actual images to MinIO")
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6171>
2020-08-04 12:58:56 +00:00
Tomeu Vizoso d4ca45eca2 ci: Upload traces' reference and actual images to MinIO
Now that the devices have sane dates, we can upload to MinIO.

Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Reviewed-by: Eric Anholt <eric@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6136>
2020-08-04 08:18:51 +02:00
Eric Anholt cb82274538 ci/bare-metal: Capture the first devcoredump a job produces.
Connor recently ran into an issue where the chezas were hanging where his
GPUs weren't, and was blocked on getting some feedback on what was
happening.  A devcoredump will help non-cheza-having devs debug (or
hopefully with other intermittent fails).

Closes: #3187
Reviewed-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6036>
2020-07-30 11:41:57 +00:00
Tomeu Vizoso 292882f6bc ci: Fix the overwriting of traces.yml for baremetal
When the lava files were moved out of the container, this stopped
working which caused the traces job for Freedreno to not run any traces
at all.

Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Fixes: dcd171f5e9 ("gitlab-ci: More stable URL for kernel and ramdisk artifacts, for LAVA")
Acked-by: Andres Gomez <agomez@igalia.com>
Acked-by: Eric Anholt <eric@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6021>
2020-07-22 18:07:31 +00:00
Eric Anholt 8b3452a556 ci: Set XDG_CACHE_HOME to tmpfs for bare-metal runners to avoid NFS.
We don't want these files shared between builds (it'll get blown away by
the next rsync), and NFS will just increase our latency for hitting the
cache.

Drops a630 gles31 run from 11-17 minutes to 5.5.  Maximum cache size on a
run I've seen is 153M, which it seems we can easily spare.

Fixes: f97acb4bb4 ("freedreno/ir3: disk-cache support")
Reviewed-by: Christian Gmeiner <christian.gmeiner@gmail.com>
Reviewed-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5998>
2020-07-21 11:04:14 +00:00
Christian Gmeiner 096adbe369 ci: bare-metal: use nginx to get results from DUT
Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/2655
Signed-off-by: Christian Gmeiner <christian.gmeiner@gmail.com>
Reviewed-by: Eric Anholt <eric@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5661>
2020-07-20 20:21:12 +00:00
Eric Anholt 27c9272c3d docs: Move the gitlab-ci docs to RST.
I tried not to edit too much meaning in the process, but I did shuffle
some stuff around to work as structured documentation.

Reviewed-by: Eric Engestrom <eric@engestrom.ch>
Reviewed-by: Erik Faye-Lund <erik.faye-lund@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5510>
2020-07-08 20:13:11 +00:00
Eric Anholt 63805ccd3f ci/bare-metal: Fail early when we get stuck powering on a cheza.
I think I've seen about 3 of this error total so far, but waiting 60
minutes for the scripts to give up wastes marge time.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5651>
2020-06-26 16:54:07 +00:00
Eric Anholt b88c46fa11 ci: Add a freedreno a630 tracie run.
This job runs in about one minute on the current set of traces, and has
successfully revealed some bugs in our current rendering.  Takes about 7
minutes currently.

Reviewed-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5433>
2020-06-25 17:33:28 +00:00
Rohan Garg 7406d627c8 ci: Include trace replay support in ARM rootfses.
Builds the renderdoc and apitrace programs so we can replay GL traces on
DUTs.

[Separated out from 5472's commit that also enabled the jobs in LAVA,
dropped unnecessary python packages from arm_build, fixed up arm64_test
build, traces-db in baremetal, new commit message by anholt]

Signed-off-by: Rohan Garg <rohan.garg@collabora.com>
Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Reviewed-by: Eric Anholt <eric@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5433>
2020-06-25 17:33:28 +00:00
Eric Anholt 9079b53987 ci/bare-metal: Skip setting of unset variables at startup.
It's silly to be setting (and logging the setting of!) all the env vars we
*didn't* set in a job.

Reviewed-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5433>
2020-06-25 17:33:28 +00:00
Eric Anholt 32143cba4d ci/bare-metal: Terminate the job with an error on kernel panic.
Otherwise, we'll time out after 60 minutes of waiting for the run to
complete.

Reviewed-by: Christian Gmeiner <christian.gmeiner@gmail.com>
Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/2651
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5393>
2020-06-12 23:34:44 +00:00
Eric Anholt 72fe7b98ea ci/bare-metal: Stop fetching the git tree.
Like for LAVA, make the tradeoff of moving the test scripts and data (55k)
into the artifacts in order to make the per-build jobs not have to pull
down the git tree (hundreds of MB when you don't hit a cached container
for your specific user, which I see happen multiple times a day in my CI
runs).

To do this, we have to be a bit more careful in some places about our
working directory potentially being dirty.

Reviewed-by: Christian Gmeiner <christian.gmeiner@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5393>
2020-06-12 23:34:44 +00:00
Eric Anholt 109816b518 ci/bare-metal: Use the deqp-runner bits straight out of the artifacts.
We've already uploaded and downloaded them from fd.o and put them in the
rootfs, so we can clean up the extra prep work.

Our test job now extends from .test so that the artifacts' install dir
with all the scripts is extracted.  This required moving the dependency on
meson-testing to the x86 test-gl/test-vk job blocks.

Reviewed-by: Christian Gmeiner <christian.gmeiner@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5393>
2020-06-12 23:34:44 +00:00
Eric Anholt 445f3eb0ea ci/bare-metal: Make which test to run configurable.
I'll use this to run tracie in a new job I'm working on.

Reviewed-by: Christian Gmeiner <christian.gmeiner@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5393>
2020-06-12 23:34:44 +00:00
Eric Anholt a13209bdec ci/bare-metal: Reword the final output of the init script on the board.
I'm going to be adding tracie, which isn't deqp.

Reviewed-by: Christian Gmeiner <christian.gmeiner@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5393>
2020-06-12 23:34:44 +00:00
Eric Anholt 3d5429d646 ci: Use rsync for initial nfsroot population on cheza.
rm -rf and then copying over all the contents again is a waste of time
when we'll almost always be using the same rootfs.  Saves about 30s of job
time.

Closes: #3065
Reviewed-by: Christian Gmeiner <christian.gmeiner@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5266>
2020-06-09 18:28:18 +00:00
Eric Anholt 9e11cce517 ci: Enable pre-merge fractional vulkan CTS runs on the turnip driver.
Test 1/50th of the CTS on a630 pre-merge, since we've got hardware that
can do it and infrastructure that should handle instability with a
less-mature driver.

Reviewed-by: Christian Gmeiner <christian.gmeiner@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5266>
2020-06-09 18:28:18 +00:00
Eric Anholt dd167788ec ci: Build the full VK CTS for baremetal testing.
I'm going to enable the VK CTS on cheza, so swap the deqp we have in the
container.  build-deqp-vk already included GLES deqp binaries and data,
and is a newer branch than the last opengl-es-cts tag.

This brings a few things back over from build-deqp-gl for testlog
extraction, and copyes out the GLES mustpass lists.

Reviewed-by: Christian Gmeiner <christian.gmeiner@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5266>
2020-06-09 18:28:18 +00:00
Christian Gmeiner 4b7de75b4b ci: add U-Boot specific fetch strings
U-Boot's fastboot over udp generates the following output:
  Listening for fastboot command on x.y.z.w

Also add a general 'data abort' error string seen with an
too old U-Boot version:
  https://github.com/u-boot/u-boot/commit/95712af

Signed-off-by: Christian Gmeiner <christian.gmeiner@gmail.com>
Reviewed-by: Eric Anholt <eric@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5258>
2020-06-05 11:02:35 +00:00
Christian Gmeiner 06d8171994 ci: extend expect-output.sh
We need to support different fastboot fetch strings for different
bootloader solutions. Lets extend expect-output.sh to support
multiple fetch strings (-f) and add support for error catch
strings (-e) to stop the CI run early.

Signed-off-by: Christian Gmeiner <christian.gmeiner@gmail.com>
Reviewed-by: Eric Anholt <eric@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5258>
2020-06-05 11:02:35 +00:00
Christian Gmeiner 2dfc241e36 ci: bare-metal: make it possible to use a script for serial
Makes it possible to use e.g. a ser2net script to talk to the devices.

Signed-off-by: Christian Gmeiner <christian.gmeiner@gmail.com>
Reviewed-by: Eric Anholt <eric@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5268>
2020-06-02 22:20:46 +00:00
Eric Anholt c89a749f66 ci: Add scripts for controlling bare-metal chezas.
This will let us:

- deploy kernels for testing code depending on new kernel featuers
- Ensure a pristine state in the HW before starting our tests
- Avoid disk rot on the chezas taking them out (we'd lost 3/9 in a few
  months).

Reviewed-by: Kristian H. Kristensen <hoegsberg@google.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5247>
2020-05-29 16:46:44 +00:00
Eric Anholt 46d9b500f4 ci: Move baremetal DEQP_NO_SAVE_RESULTS setup to the yml.
I'm going to want it unset (artifacts enabled) for the cheza jobs.

Reviewed-by: Kristian H. Kristensen <hoegsberg@google.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5225>
2020-05-28 16:23:59 +00:00
Eric Anholt 38f32372aa ci: Improve baremetal's logging of the job env var passthrough.
Trying to read the sh -x script output was rough, just cat the file once
we're done setting it up.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5089>
2020-05-22 16:44:46 +00:00
Eric Anholt ae442c3598 ci: Enable a fractional run with UBO-to-constbuf disabled on a3xx.
This gets us coverage of an important case in the HW that the CTS
otherwise basically doesn't hit.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5089>
2020-05-22 16:44:46 +00:00
Eric Anholt b4bccbde36 ci: Don't forget to set NIR_VALIDATE in baremetal runs.
Given that a530 doesn't have cpufreq, we really don't have the time to be
running the validator on all of deqp.  This also helps explain why I had
to go to such a small fraction on the a3xx gles3 run (which we can now
increase).  However, a3xx gles2 seems to be fast enough that we can leave
it enabled and get coverage for older chips.

Because we run more tests now, clear out some stale xfails from the a3xx
list.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5089>
2020-05-22 16:44:46 +00:00
Eric Anholt 2637961d29 ci: Fix the nick used in IRC reporting.
robclark found that we needed unique IDs when multiple runners were trying
to report flakes at the same time, but it turns out due to nick limits (16
chars on freenode) we were just getting all the runners appended with
"-142" (or whatever the prefix of the pipelines are these days).  And, for
the new flake reporting from baremetal, all the runners ended up being
just "google-freedreno".

Reviewed-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4896>
2020-05-06 18:34:38 +00:00
Eric Anholt 2c50176dfe ci: Improve the flakes reports on IRC.
We were incorrectly taking the merge-request on non-MR pipelines (the
master build after merge) due to a missing '$'.  And, for those pipelines,
it would be nice to note whether they're for master or a stable branch.

Reviewed-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4896>
2020-05-06 18:34:38 +00:00
Eric Anholt 3b5e71cb18 ci: Enable IRC flake reporting on freedreno baremetal boards.
The IRC channel is useful for me to track and ban flaky tests before they
irritate people too much.

Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/2654
Reviewed-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4896>
2020-05-06 18:34:38 +00:00
Eric Anholt c7bbc211d6 ci: Clean up setup of the job-specific env vars in baremetal testing.
Avoids copy and paste errors when adding more vars.

Reviewed-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4896>
2020-05-06 18:34:38 +00:00
Christian Gmeiner 7d476a1360 ci: bare-metal: power down device after tests
Helps to save electricity.

Signed-off-by: Christian Gmeiner <christian.gmeiner@gmail.com>
Reviewed-by: Eric Anholt <eric@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4754>
2020-04-28 07:17:24 +00:00
Eric Anholt 8b221e0914 ci: Add sanity checking that dEQP gets the expected GL_RENDERER.
It's easy to get something wrong in the driver build or container or
something that results in falling back to swrast, and then your only clue
was runtime and how your failure cases suspiciously match a swrast
driver's.

Reviewed-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4621>
2020-04-27 22:10:10 +00:00
Eric Anholt a9e6a3ecc7 ci: Enable --compact-display false on all dEQP runs.
We always want to see status updates happening in the logs, otherwise it
can like maybe your machine hung until the run actually completes.

Reviewed-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4621>
2020-04-27 22:10:10 +00:00
Tomeu Vizoso 92f3c51560 gitlab-ci: Place files from the Mesa repo into the build tarball
There's some files from the .gitlab-ci directory that are needed in the
test stage and that, because the Mesa repository isn't checked out in
that stage, need to be made available through other means.

Because those files are going to be needed in LAVA devices, place them
ino the tarball containing the built files so it's available to both
gitlab-ci runners and LAVA devices.

Before those files were passed in the artifacts of the Gitlab CI job,
but this commit places them into the built tarball so scripts later in
the pipeline don't need to account for this discrepancy.

Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Reviewed-by: Eric Anholt <eric@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4295>
2020-03-26 09:30:48 +00:00
Eric Anholt 4bc15e78fa ci: Enable testing GLES2-3 on a530 (Dragonboard 820c).
Following on from the db410c conversion to baremetal testing, reuse the
same scripts in the same rack to run 7 db820c boards (#4/8 is failing in
the bootloader for unknown reasons).

Reviewed-by: Kristian H. Kristensen <hoegsberg@google.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4177>
2020-03-17 11:11:51 -07:00
Eric Anholt cf5ba9d409 ci: Make a simple little bare-metal fastboot mode for db410c.
This supports powering up the device (using an external tool you
provide based on your particular lab), talking over serial to wait for
the fastboot prompt, and then booting a fastboot image on a target
device.

I was previously relying on LAVA for this, but that ran afoul of
corporate policies related to the AGPL.  However, LAVA wasn't doing
too much for us, given that gitlab already has a job scheduler and
tagging and runners.  We were spending a lot of engineering on making
the two systems match up, when we can just have gitlab do it directly.

Lightly-reviewed-by: Kristian H. Kristensen <hoegsberg@google.com>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4076>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4076>
2020-03-11 21:36:47 +00:00