Now, every rootfs based device has bash installed, so we can use bash
shebang instead of dash to enforce the use of bash's echo, not dash's
one.
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/15938>
This should help highlight the actual test results, as opposed to the setup
and teardown.
Also tuned the "set -x"es a little bit so we get less surrounding noise in
the echo process.
Acked-by: Daniel Stone <daniels@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/15332>
The test suite is full of flakes around transform feedback, atomics, and
tess. But, I hope it can be useful for regression testing core Mesa
reworks.
This required updating the kernel to 5.16.12 to get a more stable boot
process. That kernel rebuild caused an update of the container with
piglit which that was missed in a previous MR, so we got new xfails in x86
swrast.
Acked-by: Ilia Mirkin <imirkin@alum.mit.edu> (nouveau)
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/15201>
Having the Mesa Shader Cache stored on a tmpfs mount point reduces the
tests execution duration by 2-3 %, while preventing several hundreds of
megabytes to be written on the storage media.
Signed-off-by: Cristian Ciocaltea <cristian.ciocaltea@collabora.com>
Reviewed-by: Daniel Stone <daniels@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/14995>
So we can use well-known absolute paths in configuration files.
Otherwise, the install dir is within $CI_PROJECT_DIR, which changes
between jobs.
Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Reviewed-by: Corentin Noël <corentin.noel@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/12828>
We have two testsuites with the same format for fails/flakes/skips files,
and test names that are definitely unique. As I'm about to add a third
testsuite (gtest for libva-utils), so let's have just one file each for
fails/flakes/skips instead of one per type of testsuite. This starts the
move with just the bulk rename of deqp.
Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Acked-By: Mike Blumenkrantz <michael.blumenkrantz@gmail.com>
Acked-by: Daniel Stone <daniels@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/13517>
Changing these variables won't do anything for you otherwise.
Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>.
Reviewed-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/13372>
deqp hasn't been linking against that in quite some time.
Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Reviewed-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/13372>
This should improve their reliability and speed a little by getting
deqp-runner off of asan. This removes the last jobs setting
TEST_LD_PRELOAD, so remove passing that variable around from other
scripts.
Acked-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Reviewed-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/13372>
This is prettier in the log files, less shell code, and for non-suite mode
adds checking that the driver has the right git sha1. Also, no need for
suites to have a DEQP_VER to say which dEQP we should run for the renderer
check.
The version checks can help us make sure that GL version exposed doesn't
accidentally regress, and the ".*git" checks that we're using a git
version of Mesa rather than something that snuck in through distro
packages.
Acked-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Reviewed-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/13372>
We can use the general "how parallel should we go on this runner?" env var
and save a bunch of massaging env var names. Fixes how PIGLIT_PARALLEL
looked like it was useful but actually wasn't passed through to HW
runners.
Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Reviewed-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/13372>
Nobody uses it any more, and you could just put it in DEQP_OPTIONS.
Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Reviewed-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/13372>
In addition to a global skip list introduced in
https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11333
(enabled by
https://gitlab.freedesktop.org/anholt/deqp-runner/-/merge_requests/4)
it is also useful to have a per-driver skip list in addition to a
per-gpu list. Now, there are multiple levels at which skips can be
specified, from least to most specific,
- (deqp|piglit)-all-skips.txt :: affecting all tests
- (deqp|piglit)-$(DRIVER_NAME|VK_DRIVER|GALLIUM_DRIVER)-skips.txt ::
affecting the specified driver
- (deqp|piglit)-$GPU_VERSION-skips.txt :: affecting a specific GPU
This idea could be useful for -fails.txt as well.
Reviewed-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Martin Peres <martin.peres@mupuf.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11426>
Everyone can get artifacts now. Let's make those artifacts useful.
Acked-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Reviewed-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/12110>
The deqp-runner "suite" support lets us specify all the deqp variants at
once and shard them across gitlab jobs, so we reduce docker overhead and
get fewer jobs to review the output of.
Acked-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Reviewed-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/12110>
Now that we've disabled brilinear and have per-pixel cube derivatives, we
can use the same rendering paths that end users will see.
In a few cases, we switch to the no_quad_lod option instead, because
otherwise we get a piglit failure.
Reviewed-by: Roland Scheidegger <sroland@vmware.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/12125>
They're too slow to run in CI even on non-tiled renderers, they don't
block conformance (unless you crash), and provide unreliable warning
results unless you isolate them from other activity on the system.
This means that the following jobs now skip these tests:
- deqp-iris-*
- deqp-llvmpipe (you know, the one mentioned in the comment!)
- deqp-virgl-gl
- deqp-zink-lvp
Acked-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Reviewed-by: Christian Gmeiner <christian.gmeiner@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11333>
hang-detection is a vulkan-based lightweight wrapper from
parallel-deqp-runner that periodically submits empty command buffers
and waits for their completions. If the completion never happens, the
GPU is considered hung, the wrapped script is killed, and the job
should get aborted.
This should have no negative impact on the runtime of dEQP/traces/...,
but will allow saving time when the GPU gets hung as we can abort the
job immediately rather than waiting for the timeout.
In the case of B2C, we are using this tool's error message as a way to
trigger the reboot of the test machine and start again.
v2:
- Use hang-detection already with some jobs (Martin).
Signed-off-by: Andres Gomez <agomez@igalia.com>
Reviewed-by: Martin Peres <martin.peres@mupuf.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11087>
Following the rest of our channels, move CI reporting over. Seems to
still work fine. This affects freedreno and iris.
Reviewed-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11020>
Now, flakes that aren't in the *-flakes.txt get a "NEW" in their report so
I can watch for them.
The bash was unwieldy and made debugging hard, so I switched to python.
Reviewed-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11020>
This will be used to test llvmpipe against Xvfb and freedreno against
Xorg. We keep the core deqp testing on surfaceless because
--deqp-surface-type=pbuffer fails on x11_egl_glx, =fbo has never worked in
VK-GL-CTS, and =window would increase test runtime for all the
swapbuffers.
Reviewed-by: Eric Engestrom <eric@engestrom.ch>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10240>
When loading Vulkan ICD file, it uses the CPU machine identifier to
load the correct one, in case multiple versions are installed.
This is fine if the machine where Mesa has been built and the machine
where the test is run are exactly the same. But this is not always the
case. As example, for armhf architecture, the machine where Mesa is
built is identified as `arm7hlf`, but the Raspberry Pi 4 is identified
as `armv7l`, so it will fail to load the ICD file, though both are
totally compatible.
This allow to define the architecture instead.
Signed-off-by: Juan A. Suarez Romero <jasuarez@igalia.com>
Reviewed-by: Eric Anholt <eric@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8745>
3 commits in 0.5.0:
- 20-40s savings on many of our CI runs by dropping the clever test size
scaling code.
- Even bigger savings (especially on deqp-vk runs) by increasing maximuim
test group size (~1/4 of runtime was spawning deqp on cheza, that cost
is cut by ~75%)
- No more needing to manually set MESA_DEBUG=silent
2 commits in 0.5.1:
- Fixed automatic thread pool sizing to keep all CPUs busy (thanks for
catching that Bas!).
- Automatically size down test groups on short test lists and many CPUs,
so split the list evenly between CPUs (such as on freedreno -options
jobs).
Acked-by: Daniel Stone <daniel@fooishbar.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8787>
This should help us avoid landing memory leaks (and some buffer overflows)
throughout the GL stack. I put the asan lib in x86_test-base because
we'll want asan for lavapipe, too.
This requires keeping debug symbols for the asan drivers in the artifacts,
as otherwise you can't do much with the backtraces it produces.
Closes: #3726
Reviewed-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8530>
When set, we want to summary-limit 0, but the meaning was inverted. Sigh,
coding in bash.
Fixes: bf29daa1b5 ("ci/deqp: Switch to a new dEQP runner written in Rust.")
Acked-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8164>
No need for debug printing of how we computed it any more, brings the deqp
failures more into highlight in the job log.
Reviewed-by: Christian Gmeiner <christian.gmeiner@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8206>
I moved QPA-to-XML conversion to the runner, so Mesa CI (and developers!)
don't need to do quite so much in bash. I also made it clean up caselist
.qpa files since nobody ever wants them and we deleted them anyway. This
cleans up a ton of the job log output.
Additionally, I added a subcommend to turn the .csv into a junit output
that we can expose to gitlab. Now, the pipeline's status page will report
the failed testcases, and the "detail" button will give you a link to the
.XML to view for the failure. (We don't report all testcases because it's
too much load for the gitlab server). Note that this will 404 for the
LAVA runners for now, as they don't retain artifacts in gitlab (the plan
is to eventually have them minio upload the artifacts).
This uprev also includes a deqp output parsing fix, resulting in us
catching a couple more failures in some drivers.
Reviewed-by: Christian Gmeiner <christian.gmeiner@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8206>
When introducing/removing these files, it's easy to forget to update the
yml to point to them. Instead of requiring the separate update, just have
the runner script pick the right one from a single per-gpu variable.
As a result, we now pick up the new deqp-lvp-skips.txt that was added but
not conected. This also required moving some bypass flakes from the
shared a630 flakes list to a separate list, which is a feature because now
we'd notice the introduction of flakes to the gmem path.
Fixes: ab79e6b8e3 ("ci: skip failing test on lavapipe")
Acked-by: Daniel Stone <daniels@collabora.com>
Reviewed-by: Juan A. Suarez <jasuarez@igalia.com>
Reviewed-by: Christian Gmeiner <christian.gmeiner@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8147>
To increase our VK coverage on a630, we want to have two jobs in parallel,
but we still can't hit full coverage so we need the fractional setting to
be separate from gitlab CI's flags for setting up parallel jobs.
Reviewed-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6971>
I found the C++ runner hard to develop on, and we had stability issues and
outstanding feature needs that made me want something I felt good about
hacking on. Thus, Rewrite It In Rust of the deqp runner.
The new runner includes:
- Skip lists don't reshuffle the test list.
- Known-flake handling without resorting to skip lists (fixing our main CI
reliability issue on a3xx right now).
- Per-thread Vulkan shader caches should speed up VK CI runtime.
- Tracking of crashes separate from fails (so we can see progress on that
front).
- Logging of deqp stderr spam (particularly assertion failures!) in the CI
log.
- Integrated QPA filtering so we don't have bash perf issues for it.
- Logging of what caselist to go look at for a given error report (in red,
so it's easier to find in your CI log).
- The code is 1/3 unit tests, and easy to extend for more coverage.
- Non-LAVA CI runs create a failures.csv in artifacts that you can check
in as your deqp-*-fails.txt file.
- Test runtime is included in results.csv so you can debug how to speed up
your CI job.
- Pretty summary at the end of the run of slow/flaky/failed tests.
Since this is a new runner with a different RNG, the test groups are
shuffled one more time. This seems to result in some panfrost T720
stability issues (See its new deqp-panfrost-t720-flakes.txt), and one new
flake in freedreno a630.
Reviewed-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7434>
If your driver is totally broken and can't even report its name, let's
stop here instead of doing a CTS run full of failure to start tests and
reporting them all missing at the end.
Reviewed-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6324>
So far, we've been putting our known flakes that intermittently fail CI
into the skips list. This has two downsides:
1) You don't know when the flakes stop happening and when to delist them
from skips, unless you go do a bunch of manual runs with the skips list
cleared.
2) If the flake was because the previous test left some broken state in
the HW, you may just move your intermittent to a new test.
With this new path, you can list your flakes in the flakes file to keep
them from erroring out people's pipelines. They still get run and
reported as is.
Reviewed-by: Christian Gmeiner <christian.gmeiner@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6392>
So we can get an idea if what are the bottlenecks when looking for
opportunities to speed things up.
Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Reviewed-by: Daniel Stone <daniels@collabora.com>
Reviewed-by: Rohan Garg <rohan.garg@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6101>
fd.o has retuned the x86 runners on packet for -j8. Rather than having to
tweak our CI every time fd.o decides to rebalance job concurrency, respect
what the runner admin has chosen for their builds (this will also be
convenient for people with large local runners).
Reviewed-by: Michel Dänzer <michel@daenzer.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5669>
Guilherme Gallo
Emma Anholt
Juan A. Suarez Romero
Cristian Ciocaltea
Tomeu Vizoso
Charlie Turner
Andres Gomez
Eric Anholt
Rob Clark
Matt Turner