The poweron failure happens before we get to the bootloader
("load_archive: loading locale_en.bin") not after we're trying to boot the
kernel and we're waiting for the deqp run to complete.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6970>
It's useful for kernel dev to be able throw all of our testing
infrastructure at a risky kernel change, but it's expensive (time and
bandwidth) to roll new containers every time your rev your kernel. Make
it so you can just point the env vars to your personal build you've
uploaded.
Reviewed-by: Kristian H. Kristensen <hoegsberg@google.com>
Reviewed-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6592>
Modeling after what I did for cros_servo_run.py, this gives us easy
support for restarting the test run a530 when we detect a spontaneous
reboot. I had to touch up serial_buffer.py to handle buffering in from a
file instead of a serial device, to support the upcoming etnaviv CI
(tested by running it against a serial log from db410c and seeing it step
to calling "fastboot")
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6529>
gitlab CI doesn't include timestamps in its logs by default, but it's
really useful for finding delays in our CI so stuff one in on the lines
coming in from serial and being output to the gitlab log. The artifacts
file is still the raw serial output.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6529>
We were only reading from the CPU serial, not EC, so we'd never notice
these sources of job timeouts. I couldn't find a cleaner solution, so I
spawned two threads to do the blocking reads from our serial line fifos
and merge them together in a single queue to read.
Closes: #3470
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6529>
match() looks for the start of the line to match our regex, while search
just looks for the regex anywhere in the line. I messed this up when
converting our greps in shell to python, which was part of breaking the
POWER_GOOD flake detection. Most of our matches worked, but let's
consistently use this one so we don't mess this up in the future.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6529>
Occasionally something goes weird in the network and a group of chezas
will produce streams of these errors during the tftp process, eventually
timing out after 60 minutes in the job. By the time we notice, the next
jobs seem to go through fine, so watch for them and try rebooting the
cheza to see if that gets our jobs to pass again.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6398>
If we get this error, we can just try rebooting again and see if it comes
up then. The POWER_GOOD failures are clustered in time, but it's better
to retry a few times in a row in one job (which has its own 60min timeout)
than to spuriously fail someone's pipeline.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6398>
This one uses python threads to move some of our logic from shell
pipelines to python, and opens the door to doing better serial output
tracking in the future (the SerialBuffer.lines() method)
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6398>
So far, we've been putting our known flakes that intermittently fail CI
into the skips list. This has two downsides:
1) You don't know when the flakes stop happening and when to delist them
from skips, unless you go do a bunch of manual runs with the skips list
cleared.
2) If the flake was because the previous test left some broken state in
the HW, you may just move your intermittent to a new test.
With this new path, you can list your flakes in the flakes file to keep
them from erroring out people's pipelines. They still get run and
reported as is.
Reviewed-by: Christian Gmeiner <christian.gmeiner@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6392>
Eric Anholt identified the issue when merging one of my MRs: the
variable contained words in '`' backticks, which caused them to be
executed by the bare metal runner's shell.
Quote the value printed using bash's shell expansion feature to make
sure anything in the future will be properly quoted.
Signed-off-by: Eric Engestrom <eric@engestrom.ch>
Reviewed-by: Eric Anholt <eric@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6389>
The previous code considered unset variables the same as set-but-empty;
sometimes setting a variable as something empty is meaningful, so let's
pass them through properly.
Signed-off-by: Eric Engestrom <eric@engestrom.ch>
Reviewed-by: Eric Anholt <eric@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6389>
As drivers have been tested with more and more traces, the yml file is
becoming a bit unwieldy. As more drivers are going to be tested with
traces, and more traces will be used, split them in per-driver files so
the size stays manageable.
Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Reviewed-by: Reviewed-By: Rohan Garg <rohan.garg@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6185>
Baremetal jobs filter the variables they get from .gitlab-ci.yml, and
TRACIE_UPLOAD_TO_MINIO and others weren't being let through.
Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Acked-by: Daniel Stone <daniels@collabora.com>
Fixes: d4ca45eca2 ("ci: Upload traces' reference and actual images to MinIO")
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6171>
Connor recently ran into an issue where the chezas were hanging where his
GPUs weren't, and was blocked on getting some feedback on what was
happening. A devcoredump will help non-cheza-having devs debug (or
hopefully with other intermittent fails).
Closes: #3187
Reviewed-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6036>
When the lava files were moved out of the container, this stopped
working which caused the traces job for Freedreno to not run any traces
at all.
Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Fixes: dcd171f5e9 ("gitlab-ci: More stable URL for kernel and ramdisk artifacts, for LAVA")
Acked-by: Andres Gomez <agomez@igalia.com>
Acked-by: Eric Anholt <eric@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6021>
We don't want these files shared between builds (it'll get blown away by
the next rsync), and NFS will just increase our latency for hitting the
cache.
Drops a630 gles31 run from 11-17 minutes to 5.5. Maximum cache size on a
run I've seen is 153M, which it seems we can easily spare.
Fixes: f97acb4bb4 ("freedreno/ir3: disk-cache support")
Reviewed-by: Christian Gmeiner <christian.gmeiner@gmail.com>
Reviewed-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5998>
I tried not to edit too much meaning in the process, but I did shuffle
some stuff around to work as structured documentation.
Reviewed-by: Eric Engestrom <eric@engestrom.ch>
Reviewed-by: Erik Faye-Lund <erik.faye-lund@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5510>
This job runs in about one minute on the current set of traces, and has
successfully revealed some bugs in our current rendering. Takes about 7
minutes currently.
Reviewed-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5433>
Builds the renderdoc and apitrace programs so we can replay GL traces on
DUTs.
[Separated out from 5472's commit that also enabled the jobs in LAVA,
dropped unnecessary python packages from arm_build, fixed up arm64_test
build, traces-db in baremetal, new commit message by anholt]
Signed-off-by: Rohan Garg <rohan.garg@collabora.com>
Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Reviewed-by: Eric Anholt <eric@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5433>
Like for LAVA, make the tradeoff of moving the test scripts and data (55k)
into the artifacts in order to make the per-build jobs not have to pull
down the git tree (hundreds of MB when you don't hit a cached container
for your specific user, which I see happen multiple times a day in my CI
runs).
To do this, we have to be a bit more careful in some places about our
working directory potentially being dirty.
Reviewed-by: Christian Gmeiner <christian.gmeiner@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5393>
We've already uploaded and downloaded them from fd.o and put them in the
rootfs, so we can clean up the extra prep work.
Our test job now extends from .test so that the artifacts' install dir
with all the scripts is extracted. This required moving the dependency on
meson-testing to the x86 test-gl/test-vk job blocks.
Reviewed-by: Christian Gmeiner <christian.gmeiner@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5393>
rm -rf and then copying over all the contents again is a waste of time
when we'll almost always be using the same rootfs. Saves about 30s of job
time.
Closes: #3065
Reviewed-by: Christian Gmeiner <christian.gmeiner@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5266>
Test 1/50th of the CTS on a630 pre-merge, since we've got hardware that
can do it and infrastructure that should handle instability with a
less-mature driver.
Reviewed-by: Christian Gmeiner <christian.gmeiner@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5266>
I'm going to enable the VK CTS on cheza, so swap the deqp we have in the
container. build-deqp-vk already included GLES deqp binaries and data,
and is a newer branch than the last opengl-es-cts tag.
This brings a few things back over from build-deqp-gl for testlog
extraction, and copyes out the GLES mustpass lists.
Reviewed-by: Christian Gmeiner <christian.gmeiner@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5266>
We need to support different fastboot fetch strings for different
bootloader solutions. Lets extend expect-output.sh to support
multiple fetch strings (-f) and add support for error catch
strings (-e) to stop the CI run early.
Signed-off-by: Christian Gmeiner <christian.gmeiner@gmail.com>
Reviewed-by: Eric Anholt <eric@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5258>
Makes it possible to use e.g. a ser2net script to talk to the devices.
Signed-off-by: Christian Gmeiner <christian.gmeiner@gmail.com>
Reviewed-by: Eric Anholt <eric@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5268>
This will let us:
- deploy kernels for testing code depending on new kernel featuers
- Ensure a pristine state in the HW before starting our tests
- Avoid disk rot on the chezas taking them out (we'd lost 3/9 in a few
months).
Reviewed-by: Kristian H. Kristensen <hoegsberg@google.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5247>
Given that a530 doesn't have cpufreq, we really don't have the time to be
running the validator on all of deqp. This also helps explain why I had
to go to such a small fraction on the a3xx gles3 run (which we can now
increase). However, a3xx gles2 seems to be fast enough that we can leave
it enabled and get coverage for older chips.
Because we run more tests now, clear out some stale xfails from the a3xx
list.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5089>
robclark found that we needed unique IDs when multiple runners were trying
to report flakes at the same time, but it turns out due to nick limits (16
chars on freenode) we were just getting all the runners appended with
"-142" (or whatever the prefix of the pipelines are these days). And, for
the new flake reporting from baremetal, all the runners ended up being
just "google-freedreno".
Reviewed-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4896>
We were incorrectly taking the merge-request on non-MR pipelines (the
master build after merge) due to a missing '$'. And, for those pipelines,
it would be nice to note whether they're for master or a stable branch.
Reviewed-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4896>
It's easy to get something wrong in the driver build or container or
something that results in falling back to swrast, and then your only clue
was runtime and how your failure cases suspiciously match a swrast
driver's.
Reviewed-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4621>
Eric Anholt
Rob Clark
Eric Engestrom
Tomeu Vizoso
Christian Gmeiner
Rohan Garg