This brings in some major new features in the runner:
- piglit tests now include subtest reporting
- "-t" support for quick include-filtering of tests.
- piglit tests that crash after their result report are considered crashes.
- throws a nice error if you try to annotate the same failure twice
(e.g. lvp's dEQP-VK.glsl.builtin.precision.pow.highp.vec2,Fail)
Since the runner catches piglit test bugs where the same subtest is run
twice, we also uprev piglit to pull in the fixes for those.
Reviewed-by: Juan A. Suarez <jasuarez@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11283>
We no longer name the template by the test suite being run.
Fixes: 93ec399b28 ("ci: Use a single template for LAVA jobs")
Acked-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Acked-by: Daniel Stone <daniels@collabora.com>
Reviewed-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11293>
LAVA doesn't consider failure to download a kernel/initramfs as an
infrastructure error, rather just a user error for supplying a broken
URL. We know our URLs aren't broken (because we're perfect), so assume
that failures in download validation are network issues and retry when
we hit them.
LAVA itself has been fixed to retry internally, so we'll get that when
upgrade in a couple of weeks, but gloss over it for now.
Signed-off-by: Daniel Stone <daniels@collabora.com>
Reviewed-by: Gustavo Padovan <gustavo.padovan@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11218>
Each step in a LAVA job returns separate results; a successfully-retired
job can have about 12 entries. Make sure we iterate through all of them
when we're looking for infrastructure errors.
Signed-off-by: Daniel Stone <daniels@collabora.com>
Reviewed-by: Gustavo Padovan <gustavo.padovan@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11218>
We added lava_job_submitter.py to improve our robusteness, but
the final result reporting was not handled correctly by the script.
This change fix it by properly calling sys.exit() on failures.
Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
Reviewed-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Reviewed-by: Emma Anholt <emma@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11218>
And also add the required bits to the x86_64 kernel.
syslogd is needed by Crosvm.
iptables is needed to route packets in and out the VM.
Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10862>
hang-detection is a vulkan-based lightweight wrapper from
parallel-deqp-runner that periodically submits empty command buffers
and waits for their completions. If the completion never happens, the
GPU is considered hung, the wrapped script is killed, and the job
should get aborted.
This should have no negative impact on the runtime of dEQP/traces/...,
but will allow saving time when the GPU gets hung as we can abort the
job immediately rather than waiting for the timeout.
In the case of B2C, we are using this tool's error message as a way to
trigger the reboot of the test machine and start again.
v2:
- Use hang-detection already with some jobs (Martin).
Signed-off-by: Andres Gomez <agomez@igalia.com>
Reviewed-by: Martin Peres <martin.peres@mupuf.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11087>
When we remove the contents of the results directory, we `cd` into it.
The script expects that $PWD is /piglit, and $OLDPWD is the Mesa build
directory, however the cd into the results directory will make $OLDPWD
be $BUILDDIR/results.
This means that Piglit emits into results/results/ which looks weird,
but more importantly also fails OpenCL Piglit execution, because we
can't find our baseline result expectations.
Fix it by using an explicit variable rather than relying on history.
Fixes: 683ddf19dc ("ci: remove results directory content only with piglit runners")
Ref: https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10856
Signed-off-by: Daniel Stone <daniels@collabora.com>
Reviewed-by: Martin Peres <martin.peres@mupuf.org>
Reviewed-by: Andres Gomez <agomez@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11126>
So we don't need to provision aarch64 servers, which are these days
rarer than x8_64.
In the switch to the new runner tags, switch to one which contains the
device type, so we can dimension the runner jobs taking into account the
number of DUTs available.
Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Acked-by: Daniel Stone <daniels@collabora.com>
Reviewed-by: Emma Anholt <emma@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11108>
Following the rest of our channels, move CI reporting over. Seems to
still work fine. This affects freedreno and iris.
Reviewed-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11020>
Now, flakes that aren't in the *-flakes.txt get a "NEW" in their report so
I can watch for them.
The bash was unwieldy and made debugging hard, so I switched to python.
Reviewed-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11020>
Covert the job submission process to a python script for more
robustness and control. allowing easier manipulation of job data.
As a result, it adds retry logic to deal with Infrastructure Errors in LAVA.
_call_proxy() is equipped with a robust retry logic, which I have been
using already in the past few weeks in stress testing to run hundreds
of jobs.
Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11079>
Lock CPU frequency scaling to max to speed up test execution and lower
the variation of frame times from performance replay jobs.
Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Reviewed-by: Corentin Noël <corentin.noel@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11101>
This reverts commit eef5409df4.
I suspect this has caused a lot of the CI instability today -- some flakes
were already added, but the a630-traces job is still flaking. Revert
until a fix makes it stable.
Reviewed-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11024>
Use Piglit's replay profile to measure and store the time that frames
take to render in the GPU.
This job won't run automatically in regular pipelines, but will be
triggered automatically by a script for every successful pre-merge
pipeline.
This is because we want to generate performance data for every relevant
commit merged in main, but we don't want to keep a device busy during
the pre-merge run.
Signed-off-by: Antonio Caggiano <antonio.caggiano@collabora.com>
Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Reviewed-By: Rohan Garg <rohan.garg@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7987>
Lock CPU and GPU frequency scaling on devices so to speed up test
execution and lower the variation of frame times from performance replay
jobs.
Also disable autosuspend of the GPU device.
Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Reviewed-By: Rohan Garg <rohan.garg@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7987>
In order to reduce the amount of building work and network traffic, we
use docker caching. For that, we use the MESA_IMAGE_TAG and
MESA_BASE_TAG env variables which build the MESA_IMAGE variable to
identify different containers.
We are also using these tags to identify the cached artifacts produced
by other containers when those are part of the underlying OS to run
directly in DUTs through the DISTRIBUTION_TAG env variable.
The undesirable collateral effect is that we cannot combine a test job
using a container which would like to make use of some of the cached
artifacts created by another container. In other words, we cannot have
a job using a DISTRIBUTION_TAG and a MESA_IMAGE using a different
MESA_[IMAGE|BASE]_TAG variables.
Now, we split the usage in the DISTRIBUTION_TAG through the definition
of MESA_ARTIFACTS_TAG AND MESA_ARTIFACTS_BASE_TAG.
Signed-off-by: Andres Gomez <agomez@igalia.com>
Reviewed-by: Martin Peres <martin.peres@mupuf.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10977>
Instead of installing the distribution package, build and install
locally, including the tests.
Signed-off-by: Andres Gomez <agomez@igalia.com>
Reviewed-by: Martin Peres <martin.peres@mupuf.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10870>
ci-fairy minio ls will try to list files in the path given, which for
trace buckets is generally forbidden. We don't really need to do any
listing in this case, so use wget instead to check that the reference
image doesn't exist yet.
Previous to this patch, trace jobs would re-upload all reference images
to minio every time because they wouldn't be able to verify that the
reference image was already there. Jobs would often take up to 4 minutes
needlessly re-uploading these files.
Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Reviewed-by: Emma Anholt <emma@anholt.net>
Reviewed-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10953>
Changes to Panfrost or Freedreno should not trigger ppc64 rebuilds.
Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Reviewed-by: Emma Anholt <emma@anholt.net>
Reviewed-by: Adam Jackson <ajax@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10926>
Panfrost has two compilers, one for Midgard GPUs and one for Bifrost
GPUs. The respective compilers are src/panfrost/midgard and
src/panfrost/bifrost. Changes internal to just one compiler (or
disassembler) cannot affect the other hardware, so there's no need to
run extra jobs in these cases.
Also split out common vs Gallium panfrost so we can do the right thing
for panvk builds in the imminent future.
Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10924>
Run part of the VK CTS in 3 devices with Stoney Ridge graphics.
Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Reviewed-by: Rohan Garg <rohan.garg@collabora.com>
Reviewed-by: Corentin Noël <corentin.noel@collabora.com>
Acked-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10853>
Removing the directory itself can be problematic with certain runner
strategies (B2C).
v2:
- Better deleting pattern matching since the previously used one was
problematic and not pointed out by /bin/sh, as noticed by Emma.
v3:
- Check that the results directory exists before attempting to
delete its content.
Signed-off-by: Andres Gomez <agomez@igalia.com>
Reviewed-by: Martin Peres <martin.peres@mupuf.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10856>
And enable it in the radv's Raven traces job.
v2:
- Adapt to changes in the start-x.sh script.
v3:
- Not deleting any more the non-existent by now
".gitlab-ci/common/start-x.sh" script (Martin).
Signed-off-by: Andres Gomez <agomez@igalia.com>
Reviewed-by: Martin Peres <martin.peres@mupuf.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10856>
Only the AMD video drivers for xorg are added since there are no other
expected users by now.
v2:
- Remove the start/stop logic from the x.sh script. We don't care
about stopping since that's already managed by gitlab-ci (Emma).
v3:
- Remove mistakenly added ".gitlab-ci/common/start-x.sh"
script (Martin).
Signed-off-by: Andres Gomez <agomez@igalia.com>
Reviewed-by: Martin Peres <martin.peres@mupuf.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10856>
This covers dEQP-EGL, GL, and GLES on GLK, APL (BXT), and AML-Y
Chromebooks. The GLES tests are fractional runs on some boards to keep
the pipeline time down. Flakes are reported to #intel-ci on freenode.
Reviewed-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10438>
This pulls in a fix for replayer to make the backends send their
stderr to sys.stderr always.
v2:
- Updated piglit results for llvmpipe, softpipe, zink and
radeonsi-stoney.
Signed-off-by: Andres Gomez <agomez@igalia.com>
Reviewed-by: Tomeu Vizoso <tomeu.vizoso@collabora.com> [v1]
Reviewed-by: Martin Peres <martin.peres@mupuf.org> [v1]
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10854>
This version adds support for some Chromebooks with the MT8183 SoC that
we want to use to test Panfrost.
Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Reviewed-by: Emma Anholt <emma@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10785>
Emma Anholt
Tomeu Vizoso
Daniel Stone
Gustavo Padovan
Andres Gomez
Tapani Pälli
Antonio Caggiano
Alyssa Rosenzweig
Eric Anholt
Juan A. Suarez Romero