- Add the scripts to the prepared Mesa artifacts for use in later
runner stages.
- Add a template generator (generate_b2c.py) which reads and
validates (very lightly for now) the Gitlab job environment and then
spits out a YAML file describing the necessary test workload to be
sent to a Valve CI gateway.
Acked-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Martin Roukala <martin.roukala@mupuf.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/14660>
For every CI job, put JWT content into a file and unset CI_JOB_JWT
environment var
=======
* virgl jobs:
- Share JWT token file to crosvm instance
- Keep using `export -p` due to high complexity in the scripts
of these jobs. At least, the CI_JOB_JWT will not be leaked,
since it is being unset at the `before_script` phase of each
Mesa CI job.
* iris jobs: Update lava_job_submitter to take token file as argument
- generate-env with CI_JOB_JWT_TOKEN_FILE
- create token file during baremetal init stage
* baremetal jobs: Copy token file to bare-metal NFS
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Reviewed-by: Cristian Ciocaltea <cristian.ciocaltea@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/14004>
We've noticed issues with these tests when uprevving Mesa in Chrome OS.
This CI catches some existing failures, and some debug-build assertion
failures as well.
To do this, uprev deqp-runner for its new gtest-runner command. This
runner is not as efficient as I would hope, due to some expensive code in
gtest. I've reported the issue to gtest and it should be easily fixable,
but for now it at least means we get to use the same baseline/skip/flake
handling we have from deqp and piglit runners.
I also fixed build-libdrm for our rootfses to not throw away libdrm's
share directory, which was causing a bunch of test-time spam from radeon's
libdrm when trying to look up its marketing name tables (not that big of a
deal for deqp-runner, but really noisy for piglit and libva-utils which
make gallium screens approximatly per-test).
Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/13419>
We have two testsuites with the same format for fails/flakes/skips files,
and test names that are definitely unique. As I'm about to add a third
testsuite (gtest for libva-utils), so let's have just one file each for
fails/flakes/skips instead of one per type of testsuite. This starts the
move with just the bulk rename of deqp.
Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Acked-By: Mike Blumenkrantz <michael.blumenkrantz@gmail.com>
Acked-by: Daniel Stone <daniels@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/13517>
The deqp-runner "suite" support lets us specify all the deqp variants at
once and shard them across gitlab jobs, so we reduce docker overhead and
get fewer jobs to review the output of.
Acked-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Reviewed-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/12110>
Trying to get arbitrary strings suitably quoted for shell, embedded in a
YAML file, processed by Python templating, is like seven bad ideas all
embedded into one big can of bees.
Reuse the same script we use for bare-metal to generate the environment,
tar that up into a per-job overlay which is added to the
inter-pipeline-reusable rootfs built by the container jobs and the
intra-pipeline-reusable overlay built by the build jobs.
@anholt wrote a chunk of this - replacing the $ENV_VARS GitLab CI
variable with a Python loop across the POSIX job environment - in
!11192, but this still had YAML quoting nightmares, and was more
needless duplication between LAVA and bare-metal.
The diff is large and annoying, but is mostly a sed job to get
ENV_VARS="FOO=bar BAZ=quux" into FOO: bar\nBAZ: quux.
Signed-off-by: Daniel Stone <daniels@collabora.com>
Co-authored-by: Emma Anholt <emma@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11309>
Now, flakes that aren't in the *-flakes.txt get a "NEW" in their report so
I can watch for them.
The bash was unwieldy and made debugging hard, so I switched to python.
Reviewed-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11020>
Covert the job submission process to a python script for more
robustness and control. allowing easier manipulation of job data.
As a result, it adds retry logic to deal with Infrastructure Errors in LAVA.
_call_proxy() is equipped with a robust retry logic, which I have been
using already in the past few weeks in stress testing to run hundreds
of jobs.
Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11079>
Only the AMD video drivers for xorg are added since there are no other
expected users by now.
v2:
- Remove the start/stop logic from the x.sh script. We don't care
about stopping since that's already managed by gitlab-ci (Emma).
v3:
- Remove mistakenly added ".gitlab-ci/common/start-x.sh"
script (Martin).
Signed-off-by: Andres Gomez <agomez@igalia.com>
Reviewed-by: Martin Peres <martin.peres@mupuf.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10856>
Since we don't manually enumerate the drivers using it, we have to retest
all drivers when changing it (which basically never happens, anyway).
Reviewed-by: Michel Dänzer <mdaenzer@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9353>
This means less custom test-source-dep stuff for these drivers, though it
means that touching the CI expects files will cause a bit more retesting:
- broadcom drivers retest as a group (but Igalia requested that
organization of CI files)
- radv+radeonsi retest as a group
- lvp+llvmpipe retest as a group
Acked-By: Mike Blumenkrantz <michael.blumenkrantz@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9161>
This way, when such a file is modified only the affected driver gets
tested.
It also helps to declutter the .gitlab-ci directory.
Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Reviewed-by: Rohan Garg <rohan.garg@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8757>
This should help us avoid landing memory leaks (and some buffer overflows)
throughout the GL stack. I put the asan lib in x86_test-base because
we'll want asan for lavapipe, too.
This requires keeping debug symbols for the asan drivers in the artifacts,
as otherwise you can't do much with the backtraces it produces.
Closes: #3726
Reviewed-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8530>
v2:
- Squashed the commit to remove tracie jobs (Eric).
v3:
- Rename *-piglit-traces jobs with *-traces.
Signed-off-by: Andres Gomez <agomez@igalia.com>
Reviewed-by: Tomeu Vizoso <tomeu.vizoso@collabora.com> [v2]
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6388>
So far, we've been putting our known flakes that intermittently fail CI
into the skips list. This has two downsides:
1) You don't know when the flakes stop happening and when to delist them
from skips, unless you go do a bunch of manual runs with the skips list
cleared.
2) If the flake was because the previous test left some broken state in
the HW, you may just move your intermittent to a new test.
With this new path, you can list your flakes in the flakes file to keep
them from erroring out people's pipelines. They still get run and
reported as is.
Reviewed-by: Christian Gmeiner <christian.gmeiner@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6392>
As drivers have been tested with more and more traces, the yml file is
becoming a bit unwieldy. As more drivers are going to be tested with
traces, and more traces will be used, split them in per-driver files so
the size stays manageable.
Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Reviewed-by: Reviewed-By: Rohan Garg <rohan.garg@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6185>
As these credentials are valid only for 15 minutes, generate them closer
to when they are going to be used.
Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Reviewed-by: Daniel Stone <daniels@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6124>
When the lava files were moved out of the container, this stopped
working which caused the traces job for Freedreno to not run any traces
at all.
Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Fixes: dcd171f5e9 ("gitlab-ci: More stable URL for kernel and ramdisk artifacts, for LAVA")
Acked-by: Andres Gomez <agomez@igalia.com>
Acked-by: Eric Anholt <eric@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6021>
Upload failed images and the results.yml file to MinIO, to facilitate
debugging.
Also, fix version checking when git is installed as Mesa is going to
output a different renderer string if git is installed.
Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Reviewed-by: Daniel Stone <daniels@collabora.com>
Reviewed-By: Rohan Garg <rohan.garg@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5472>
Place the kernel and ramdisk into a place in the file server so the URL
will only change when the contents also change.
Also put the Mesa build into a separate tarball so the ramdisk's
contents don't change every build.
With proper caching in place, all devices in the same farm need only to
download the mesa tarball once, saving time.
As we switch to MinIO for making kernels and rootfs available to LAVA
devices, we can stop using Docker to distribute them.
Instead, build when needed in separate jobs that push directly to MinIO,
from where LAVA devices can download them.
Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Acked-by: Daniel Stone <daniels@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5515>
This job runs in about one minute on the current set of traces, and has
successfully revealed some bugs in our current rendering. Takes about 7
minutes currently.
Reviewed-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5433>
Like for LAVA, make the tradeoff of moving the test scripts and data (55k)
into the artifacts in order to make the per-build jobs not have to pull
down the git tree (hundreds of MB when you don't hit a cached container
for your specific user, which I see happen multiple times a day in my CI
runs).
To do this, we have to be a bit more careful in some places about our
working directory potentially being dirty.
Reviewed-by: Christian Gmeiner <christian.gmeiner@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5393>
Currently, we store the kernel and ramdisk for each LAVA job in the
artifacts of the job that built them. Because artifacts are stored in
GCE and LAVA labs aren't, this causes a lot of egress with is expensive.
To avoid this, have runners download most of the data via the (cached)
container images once, and for each job upload the kernel and ramdisk to
a server outside GCE.
Right now we only have Collabora's runner with a local web server, so
jobs that go to Baylibre's lab have been disabled.
Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Reviewed-by: Eric Anholt <eric@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4295>
There's some files from the .gitlab-ci directory that are needed in the
test stage and that, because the Mesa repository isn't checked out in
that stage, need to be made available through other means.
Because those files are going to be needed in LAVA devices, place them
ino the tarball containing the built files so it's available to both
gitlab-ci runners and LAVA devices.
Before those files were passed in the artifacts of the Gitlab CI job,
but this commit places them into the built tarball so scripts later in
the pipeline don't need to account for this discrepancy.
Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Reviewed-by: Eric Anholt <eric@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4295>
This supports powering up the device (using an external tool you
provide based on your particular lab), talking over serial to wait for
the fastboot prompt, and then booting a fastboot image on a target
device.
I was previously relying on LAVA for this, but that ran afoul of
corporate policies related to the AGPL. However, LAVA wasn't doing
too much for us, given that gitlab already has a job scheduler and
tagging and runners. We were spending a lot of engineering on making
the two systems match up, when we can just have gitlab do it directly.
Lightly-reviewed-by: Kristian H. Kristensen <hoegsberg@google.com>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4076>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4076>
RADV_FORCE_FAMILY forces creating a null device that allows RADV
to be instanced without AMDGPU.
The Fossilize database only contains pipelines from the Sascha
Vulkan triangle demos at the moment. I will add more once this
is merged.
Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Michel Dänzer <mdaenzer@redhat.com>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3960>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3960>
Fossilize is equivalent to vkpipeline-db but it's definitely more
robust. This is based on the CI traces system.
Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Alexandros Frantzis <alexandros.frantzis@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3960>
Also, adds an example job for radv.
Signed-off-by: Andres Gomez <agomez@igalia.com>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Alexandros Frantzis <alexandros.frantzis@collabora.com>
This should get us better stability of the db410c boards by having a
smaller per-board software stack, with no disks involved (just initramfs).
Additionally, the new cluster is 7 (soon 8) db410cs, while currently the
docker cluster only has 1/4 of its db410cs still running.
Unfortunately, we have to prepare the fastboot boot image during the ARM
drivers build stage, because LAVA relies on publicly available URLs for
the images to load into the bootloaders of the boards, and the only thing
we have for that is gitlab's artifacts.
Note that this testing relies on the boards being freshly flashed with the
linaro v136 firmware to pick up the initramfs size fixes and to stop the
boot at fastboot.
Reviewed-by: Daniel Stone <daniels@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3661>
Introduce automated testing of Mesa by replaying traces with Renderdoc
or Apitrace.
For now only LLVMPipe is tested, but other drivers can be tested if
there's runners with the necessary hardware.
Signed-off-by: Alexandros Frantzis <alexandros.frantzis@collabora.com>
Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Reviewed-by: Eric Anholt <eric@anholt.net>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/2935>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/2935>
Use the normal build job to also prepare the artifacts for LAVA jobs.
For that, the build container needs to also build the test suites,
kernel, ramdisk, etc.
Then the build job will place the just-built Mesa in the ramdisk and the
test job can generate a LAVA job and point to those artifacts.
Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Acked-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Reviewed-by: Daniel Stone <daniels@collabora.com>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3295>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3295>
One job for the quick_gl profile, one for the glslparser & quick_shader
profiles (doing these together takes hardly any more time than
quick_shader alone).
v2:
* Don't break lava tests
v3:
* Remove piglit test artifacts paths:
* Exclude some quick_shader tests again:
- Test whose result flips between pass/fail/skip
- *@vs_in tests, as not the same one of these gets picked every time
v4:
* Do not list passing tests in .gitlab-ci/piglit/*.txt (Eric Anholt)
* Include the test number summary in .gitlab-ci/piglit/*.txt
* Completely disable generating any vs_in tests in the piglit build.
* Remove some more unneded files from the piglit build tree.
* Exclude quick_gl arb_gpu_shader5 tests; they were all skipped anyway,
as llvmpipe doesn't support this extension yet, but occasionally they
would spuriously fail instead.
v5:
* Set LD_LIBRARY_PATH, so we actually test the Mesa build from the
pipeline...
* Verify that wflinfo reports the expected Mesa version
* Pass -noreset to Xvfb
v6:
* Don't use autoscale runners, run piglit with -j4 (Eric Anholt)
Reviewed-by: Eric Anholt <eric@anholt.net>
It's currently only needed for the meson-main and meson-arm64 jobs, not
the other meson build jobs.
Also remove MESON_SHADERDB, just run .gitlab-ci/run-shader-db.sh
directly from the meson-main job.
v2:
* Also run prepare-artifacts.sh in meson-arm64 script
v3:
* Move tarball creation into the new script as well, as it prevented
ccache --show-stats from running in after_script
Reviewed-by: Eric Engestrom <eric.engestrom@intel.com> # v1
Reviewed-by: Eric Anholt <eric@anholt.net>
Andres Gomez
Charlie Turner
Guilherme Gallo
Emma Anholt
Rohan Garg
Daniel Stone
Tomeu Vizoso
Gustavo Padovan
Eric Anholt
Samuel Pitoiset
Alexandros Frantzis
Michel Dänzer