This supports powering up the device (using an external tool you
provide based on your particular lab), talking over serial to wait for
the fastboot prompt, and then booting a fastboot image on a target
device.
I was previously relying on LAVA for this, but that ran afoul of
corporate policies related to the AGPL. However, LAVA wasn't doing
too much for us, given that gitlab already has a job scheduler and
tagging and runners. We were spending a lot of engineering on making
the two systems match up, when we can just have gitlab do it directly.
Lightly-reviewed-by: Kristian H. Kristensen <hoegsberg@google.com>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4076>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4076>
It was added with tracie, but it doesn't depend on it.
Signed-off-by: Andres Gomez <agomez@igalia.com>
Reviewed-by: Eric Engestrom <eric@engestrom.ch>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
For now tests only use these drivers:
* llvmpipe
* softpipe
* freedreno
* lima
* etnaviv
* panfrost
So using rules:changes gitlab feature to run the tests when the changes
made are potentially affecting these drivers.
A few notes:
* the following code:
.piglit-test:
extends:
- .test-gl
- .llvmpipe-rules
makes gitlab replace .test-gl "rules:changes" values by the one from
".llvmpipe-rules".
* rules:changes always matches for non-MR new branches so jobs will always be
created (and they'll be run if their dependencies are run). For pushes to
existing branches the files changed by the push are used to match the
rules:changes path.
* the same gitlab feature could be used for some build jobs
Acked-by: Eric Engestrom <eric@engestrom.ch>
Reviewed-by: Michel Dänzer <mdaenzer@redhat.com>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/2569>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/2569>
RADV_FORCE_FAMILY forces creating a null device that allows RADV
to be instanced without AMDGPU.
The Fossilize database only contains pipelines from the Sascha
Vulkan triangle demos at the moment. I will add more once this
is merged.
Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Michel Dänzer <mdaenzer@redhat.com>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3960>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3960>
It was diabled because RADV is the only driver that tests Vulkan
and running CTS on my personal machine and without recovery is
not safe enough for CI (too long and too unstable).
Now that we are going to test Fossilize with RADV, it's needed to
build the test image for VK unconditionally. As RADV now supports
creating NULL devices, the fossilize jobs can run everywhere.
Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Michel Dänzer <mdaenzer@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3960>
Fossilize is equivalent to vkpipeline-db but it's definitely more
robust. This is based on the CI traces system.
Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Alexandros Frantzis <alexandros.frantzis@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3960>
The stages and mapping of jobs to them are somewhat arbitrary; the goal
is to avoid having to scroll through large numbers of jobs.
v2: (Pierre-Eric Pelloux-Prayer)
* Use even more stages for test jobs
* Give somewhat meaningful names to stages
Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Reviewed-by: Eric Engestrom <eric@engestrom.ch>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3995>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3995>
Also, adds an example job for radv.
Signed-off-by: Andres Gomez <agomez@igalia.com>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Alexandros Frantzis <alexandros.frantzis@collabora.com>
In preparation to having "vk" (Vulkan) along "gl" (OpenGL/ES).
This is so it is clearer which traces belong to which API and also for
the build jobs.
Signed-off-by: Andres Gomez <agomez@igalia.com>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Alexandros Frantzis <alexandros.frantzis@collabora.com>
In preparation for having automated testing with Vulkan traces.
Signed-off-by: Andres Gomez <agomez@igalia.com>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Alexandros Frantzis <alexandros.frantzis@collabora.com>
In preparation for having automated testing with Vulkan traces.
Signed-off-by: Andres Gomez <agomez@igalia.com>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Alexandros Frantzis <alexandros.frantzis@collabora.com>
Since we are at it, replace "cd" with pushd / popd and homogenize how
VK-GL-CTS is built in comparison with other build scripts.
Signed-off-by: Andres Gomez <agomez@igalia.com>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Alexandros Frantzis <alexandros.frantzis@collabora.com>
This should get us better stability of the db410c boards by having a
smaller per-board software stack, with no disks involved (just initramfs).
Additionally, the new cluster is 7 (soon 8) db410cs, while currently the
docker cluster only has 1/4 of its db410cs still running.
Unfortunately, we have to prepare the fastboot boot image during the ARM
drivers build stage, because LAVA relies on publicly available URLs for
the images to load into the bootloaders of the boards, and the only thing
we have for that is gitlab's artifacts.
Note that this testing relies on the boards being freshly flashed with the
linaro v136 firmware to pick up the initramfs size fixes and to stop the
boot at fastboot.
Reviewed-by: Daniel Stone <daniels@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3661>
I want to make sure that I don't introduce warnings in turnip where we
have active work going on, and I also want to make sure that the drivers
we care about testing are warnings-clean.
As with the previous -Werror change, this is for CI only and doesn't
affect end-user builds.
Reviewed-by: Michel Dänzer <mdaenzer@redhat.com>
Reviewed-by: Eric Engestrom <eric@engestrom.ch>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3607>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3607>
I'm working on moving the db410c CI from docker to LAVA, which means we
get to boot a custom kernel. To do that, we need to enable ARCH_QCOM in
the kernel, save the dtb around, and include abootimg in our container so
that we can generate combined kernel/dtb/ramdisk images for fastboot.
LAVA's fastboot support is unable to pack the overlay into an abootimg
image, just a cpio rootfs. We could flash the cpio rootfs after overlay
addition, but that takes 2 minutes to do, and causes wear on the devices.
Instead, we'll bring up the network at boot and use wget to fetch the
overlay. We'll want network support anyway, so that we can transfer the
failure xmls back to the gitlab job's artifacts at some point.
Since the msm GPU and realtek network firmware increase our payload by
3MB, add in firmware compression so that it doesn't waste as much RAM on
devices not using it.
Reviewed-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3928>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3928>
The LLVM libraries were a significant fraction of the entire payload
(55M/250M uncompressed) into the initramfs of the test boards, but
LLVM is only used for the draw module used in select/feedback (which
isn't even tested in CI on ARM yet).
Assume that llvmpipe draw is safe enough for ARM given the coverage on
x86, and disable LLVM for these jobs.
Reviewed-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3928>
radv needs libllvm which increases our ramdisk size
significantly. Since this driver is only build tested,
we can split it out into a separate job.
Signed-off-by: Rohan Garg <rohan.garg@collabora.com>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Eric Anholt <eric@anholt.net>
Reviewed-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3928>
LAVA finds a '#' early in boot and races to emit its shell commands.
Apparently for the current boards those serial commands end up getting
buffered such that things work out, but for db410c and db820c, the buffer
is lost and LAVA gets stuck waiting for the prompt. By setting a prompt,
we can delay our commands until we're actually supposed to emit them (and
suppress a complaint from the lava dispatcher that we're using a risky
prompt!)
Reviewed-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3883>
Introduce automated testing of Mesa by replaying traces with Renderdoc
or Apitrace.
For now only LLVMPipe is tested, but other drivers can be tested if
there's runners with the necessary hardware.
Signed-off-by: Alexandros Frantzis <alexandros.frantzis@collabora.com>
Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Reviewed-by: Eric Anholt <eric@anholt.net>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/2935>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/2935>
Using LLVM 8 for ppc64el and 7 for s390x (which hits some coroutine
related issues with LLVM 8).
There are some test failures we need to ignore for now. Also, the
timeout needs to be bumped from the default 30s for some tests, because
they can take longer under emulation.
Reviewed-by: Adam Jackson <ajax@redhat.com>
Reviewed-by: Eric Anholt <eric@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3643>
Among other things, this increases robustness when copying a docker
image from the main Mesa project to a forked project, avoiding spurious
image rebuilds from scratch.
Also drop the comment about .gitlab-ci/lava-gitlab-ci.yml, it doesn't
include the templates anymore.
Reviewed-by: Adam Jackson <ajax@redhat.com>
Reviewed-by: Eric Anholt <eric@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3643>
I find warnings to be very disruptive to my workflow (using emacs's "go to
next error" feature), and I periodically have to go clean up other
people's drivers to get back to finding my own warnings in the noise. I
know I'm not the only one doing something like this.
We don't want to enable -Werror by default in builds, since it means that
end users will have builds spuriously fail based on what compiler version
and opt flags they have compared to what the devs are using. However, it
is quite easy to have CI ensure that we at least don't introduce warnings
on the compiler version that it uses.
For now I've just enabled it on meson-i386 to cover a bunch of Mesa core
and get us started on ratcheting up warnings-cleanliness in the tree,
without me having to fix up all the drivers at once.
Reviewed-by: Daniel Stone <daniels@collabora.com>
Reviewed-by: Kristian H. Kristensen <hoegsberg@google.com>
Reviewed-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Reviewed-by: Eric Engestrom <eric@engestrom.ch>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3539>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3539>
They were causing trouble with Marge Bot: The project settings require
that the pipeline succeeds before a merge request (MR) can be merged,
otherwise Marge doesn't wait for the pipeline to succeed before merging
an MR assigned to her. But Marge can't start manual jobs, so she would
always time out waiting for pipelines with manual jobs.
To avoid this, use these rules:
* Run the pipeline by default for MRs and main project branches changing
any files affecting it.
* For other MRs, run a single dummy job which always succeeds.
* Don't run any jobs for main project branch changes (e.g. from an MR
having been merged) not affecting the pipeline.
* Allow jobs to be started manually on branches of forked projects, as
before.
Acked-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Acked-by: Daniel Stone <daniels@collabora.com>
Reviewed-by: Eric Engestrom <eric@engestrom.ch>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3361>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3361>
Use the normal build job to also prepare the artifacts for LAVA jobs.
For that, the build container needs to also build the test suites,
kernel, ramdisk, etc.
Then the build job will place the just-built Mesa in the ramdisk and the
test job can generate a LAVA job and point to those artifacts.
Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Acked-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Reviewed-by: Daniel Stone <daniels@collabora.com>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3295>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3295>
They're not available for Debian buster yet, so we have to use upstream
snapshot packages again.
In contrast to earlier, we now store the LLVM APT repository key in Git
instead of re-downloading it every time.
Take one step towards sharing code between the LAVA and non-LAVA jobs,
with the goals of reducing maintenance burden and use of computational
resources.
The env var DEQP_NO_SAVE_RESULTS allows us to skip the procesing of the
XML result files, which can take a long time and is not useful in the
LAVA case as we are not uploading artifacts anywhere at the moment.
Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Reviewed-by: Eric Anholt <eric@anholt.net>
This is used to validate if the driver emits correct LLVM IR.
Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Only Polaris10 is tested at the moment, and I disabled a TON of
tests to keep a CTS run within 5 minutes because my local runner
is a bit slow. A full CTS run takes more than 1h, which means it
will hit the timeout.
RADV CI can only be triggered manually on personal branches to
avoid breaking the world because one runner is definitely not
enough. This will allow us to test it until it's stable enough
to be enabled by default.
Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Acked-by: Michel Dänzer <mdaenzer@redhat.com>
This requires to bump LLVM to 8 because it's the minimum supported
version by RADV.
Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Michel Dänzer <mdaenzer@redhat.com>
Use new rules: instead of only:
For container stage jobs:
* In the main Mesa project, run them by default.
* In merge requests, run them by default if any files affecting pipeline
results are changed.
* In all other cases (in particular branches in personal projects),
don't run them by default but allow triggering them manually.
build & test stage jobs are left at the default (when: on_success), so
they will run automatically once all their dependencies are satisified.
(Using the same rules as above would require these jobs to be manually
triggered as well, which is only possible once all dependency jobs have
passed) Please be considerate of CI runner resources and cancel unneeded
jobs on personal branches with no corresponding merge requests (this can
be done before the jobs start running).
In summary: No more special branch names. Unnecessary job runs are
avoided by default, but jobs which don't run by default can be triggered
manually.
v2:
* Split out LAVA changes to separate commit
* Clarify commit log a little, in particular WRT build/test stage jobs
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com> # v1
Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com> # v1
Reviewed-by: Tomeu Vizoso <tomeu.vizoso@collabora.com> # v1
Acked-by: Eric Engestrom <eric.engestrom@intel.com>
If the runner has a HW device that would be supported, even without
/dev/dri forwarded into the container, it will be enumerated and the tests
on llvmpipe fail with (for example):
libEGL warning: Not allowed to force software rendering when API explicitly selects a hardware device.
libEGL warning: MESA-LOADER: failed to open i965 (search paths /builds/anholt/mesa/install/lib/dri)
Given that we can't necessarily control the DRI devices present on the
runners (particularly for developers bringing their own runners to reduce
the demands on fd.o's shared resources), just skip these tests in CI.
Reviewed-by: Michel Dänzer <mdaenzer@redhat.com>
Commit also updates the Piglit quick_gl.txt, list modifications happened
due to following Piglit commits: c248bf201,c acff58ca, 5603e2e60.
Signed-off-by: Tapani Pälli <tapani.palli@intel.com>
Reviewed-by: Timothy Arceri <tarceri@itsqueeze.com>
And only use --process-isolation false for the quick_gl tests.
This will hopefully avoid variance in the test results that we've been
seeing lately. But even if it doesn't, it should at least help narrow
down the cause of the variance.
Tested-by: Vasily Khoruzhick <anarsoul@gmail.com>
Acked-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
This will make it easier to look at details of failed / skipped tests.
Acked-by: Daniel Stone <daniels@collabora.com>
Reviewed-by: Eric Anholt <eric@anholt.net>
It was basically useless in this form, and processing the JUnit data in
the GitLab backend was pretty expensive.
Acked-by: Daniel Stone <daniels@collabora.com>
Reviewed-by: Eric Anholt <eric@anholt.net>
To pick up updated cts_runner and netcat for the flake reporting.
Signed-off-by: Rob Clark <robdclark@chromium.org>
Acked-by: Eric Engestrom <eric.engestrom@intel.com>
If there are a small number of fails, re-run to determine if they are
flakes, and optionally (if `$FLAKES_CHANNEL` configured) report the
flakes.
This way flakes don't interfere with developers working on other
drivers, but get logged so that the developers working on the flaking
driver can monitor the situation.
Signed-off-by: Rob Clark <robdclark@chromium.org>
Acked-by: Eric Engestrom <eric.engestrom@intel.com>
It seems overkill to me to build scons 7x for every pipeline.
Scons is now build with the oldest llvm version in scons-old-llvm
and with the newest llvm version in scons.
Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Michel Dänzer <mdaenzer@redhat.com>
Reviewed-by: Eric Anholt <eric@anholt.net>
I set the runners to concurrency=1, so they serve only one gitlab-ci job
at at time. Swap over to using the parallel runner now to keep the
runners busy, more efficiently than spawning many docker containers and
downloading artifacts multiple times, and producing easier-to-understand
results for browsing on the web.
This bumps the a306 runners to 4x parallel instead of 2x like before, but
cheza gles3 drops from 6 to 4. Current rough timings of the jobs (if no
container download):
db410c-gles2: 5:00
a630-gles2: 1:30
a630-gles3: 6:00
a630-gles31: 5:30
a630-gles3 is a bit longer than I like, but it should come back down once
I can sort out the NIR algebraic rewinding.
Use hardcoded /cache/mesa/ccache for the cache, so it will be shared by
all jobs of all Mesa projects running on the same runner host. This
should increase the hit rate and decrease the worst case storage used.
Further benefits of directly using a host-mapped directory:
* Saves up to ~1 minute per job for restoring and saving the cache
contents via the GitLab CI cache mechanism
* Cache contents generated by failed jobs are no longer lost
* Jobs running in parallel on the same runner host can get hits from
each other
Also enable compression, so the default maximum cache size of 5G might
be sufficient.
v2:
* Move CCACHE_DIR variable to the .build-linux template
Suggested-by: Eric Anholt <eric@anholt.net>
Reviewed-by: Eric Anholt <eric@anholt.net> # v1
Building GLVND in meson-main doesn't work because this disables
libEGL and it's needed for running shader-db.
Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Michel Dänzer <mdaenzer@redhat.com>
Now that debugoptimized isn't set and that all test jobs depend on
meson-testing, enabling swr shouldn't slowdown the CI.
Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Michel Dänzer <mdaenzer@redhat.com>
This should reduce compile time because optimizations are costly.
Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Michel Dänzer <mdaenzer@redhat.com>
For turnip and RADV testing, we will need a debugoptimized build
without UBSAN. This introduces meson-testing which builds only the
things that are needed by the test stage.
Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Michel Dänzer <mdaenzer@redhat.com>
The 'dri' directory isn't created when building Vulkan drivers.
Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Michel Dänzer <mdaenzer@redhat.com>
Two benefits:
Most docker image related environment variables can now be defined in
the jobs where they're used instead of globally. The DEBIAN_TAG values
are propagated to other jobs via YAML anchors.
Images on https://gitlab.freedesktop.org/mesa/mesa/container_registry
are now organized in separate repositories with a suffix matching the
name of the job which makes sure the image is there.
Acked-by: Eric Anholt <eric@anholt.net>
Reviewed-by: Eric Engestrom <eric@engestrom.ch>
Cleans up .gitlab-ci/ a little, and allows using a single DEBIAN_EXEC
line for all container jobs.
v2:
* Use lava_arm.sh instead of arm_lava.sh for consistency with v2 of the
previous change
Reviewed-by: Eric Anholt <eric@anholt.net> # v1
Reviewed-by: Eric Engestrom <eric@engestrom.ch>
This makes it easier to tell which job is which in a pipeline.
v2:
* Use lava_arm{64,hf} instead of arm{64,hf}_lava to keep these jobs
together in pipeline overviews
Reviewed-by: Eric Anholt <eric@anholt.net> # v1
Reviewed-by: Eric Engestrom <eric@engestrom.ch>
Otherwise there can be weird breakage.
(Removing the include from .gitlab-ci/lava-gitlab-ci.yml doesn't seem
possible unfortunately:
https://gitlab.freedesktop.org/daenzer/mesa/pipelines/79458)
Reviewed-by: Eric Anholt <eric@anholt.net>
Reviewed-by: Eric Engestrom <eric@engestrom.ch>
The ARMHF LLVM package is LLVM 7 but RADV requires LLVM 8.
Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Eric Engestrom <eric.engestrom@intel.com>
RADV requires libdrm-2.4.100 but the distrib package is too old.
Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Eric Engestrom <eric.engestrom@intel.com>
Now that we're not using so many job slots, it's easy to get these
jobs run in a reasonable amount of time (gles3 took 10 minutes for 4
cores, and gles31 was 15 minutes for 4 cores).
Acked-by: Michel Dänzer <mdaenzer@redhat.com>
This runner is a little project by Bas, written in C++, that spawns
threads that then loop grabbing chunks of the (randomly shuffled but
consistently so) test list and hand it to a dEQP instance. As the
remaining list gets shorter, so do the chunks, so hopefully the
threads all complete effectively at once. It also handles restarting
after crashes automatically. I've extended the runner a bit to do
what I was doing in the bash scripts before, like the skip list and
expected failures handling. This project should also be a good
baseline for extending to handle retesting of intermittent failures.
By switching to it, we can have the swrast tests just take up one job
slot on the shared runners and keep their allotment of CPUs busy,
instead of taking up job slots with single-threaded dEQP jobs. It
will also let us (eventually, once I reprovision) switch the freedreno
runners over to threading within the job instead of running concurrent
jobs, so that memory scribbles in one pipeline don't affect unrelated
pipelines, and I can experiment with their parallelism (particularly
on a306 where we are frequently backed up) without trashing other
people's jobs.
What we lose in this process is per-test output in the log (not a big
loss, I think, since we summarize fails at the end and reducing log
length keeps chrome from choking on our logs so badly). We also drop
the renderer sanity checking, since it's not saving qpa files for us
to go poke through. Given that all the drivers involved have fail
lists, if we got the wrong renderer somehow, we'd get a job failure
anyway.
v2: Rebase on droppong of the autoscale cluster and the arm64
build/test split. Use a script to deduplicate the cts-runner
build.
v3: Rebase on the amd64 build/test container split.
Acked-by: Daniel Stone <daniels@collabora.com> (v1)
Reviewed-by: Tomeu Vizoso <tomeu.vizoso@collabora.com> (v2)
This helps cut down our container build time. I've left a few that
we're likely to rev more frequently or I was less confident in
dropping.
v2: Rebase on the build/test container split, now bumps the build
container tag in this commit.
Acked-by: Eric Engestrom <eric.engestrom@intel.com> (v1)
Acked-by: Daniel Stone <daniels@collabora.com> (v1)
Autotools was deprecated for a while and has now been removed, so let's
start using meson here so that we won't have any issues next time we
update libdrm.
Signed-off-by: Eric Engestrom <eric.engestrom@intel.com>
Reviewed-by: Michel Dänzer <mdaenzer@redhat.com>
Same as was done for the ARM images before.
This should make it less painful to update to newer dEQP / piglit as
well as to make changes to the build/test environment.
Reviewed-by: Eric Anholt <eric@anholt.net>
One job for the quick_gl profile, one for the glslparser & quick_shader
profiles (doing these together takes hardly any more time than
quick_shader alone).
v2:
* Don't break lava tests
v3:
* Remove piglit test artifacts paths:
* Exclude some quick_shader tests again:
- Test whose result flips between pass/fail/skip
- *@vs_in tests, as not the same one of these gets picked every time
v4:
* Do not list passing tests in .gitlab-ci/piglit/*.txt (Eric Anholt)
* Include the test number summary in .gitlab-ci/piglit/*.txt
* Completely disable generating any vs_in tests in the piglit build.
* Remove some more unneded files from the piglit build tree.
* Exclude quick_gl arb_gpu_shader5 tests; they were all skipped anyway,
as llvmpipe doesn't support this extension yet, but occasionally they
would spuriously fail instead.
v5:
* Set LD_LIBRARY_PATH, so we actually test the Mesa build from the
pipeline...
* Verify that wflinfo reports the expected Mesa version
* Pass -noreset to Xvfb
v6:
* Don't use autoscale runners, run piglit with -j4 (Eric Anholt)
Reviewed-by: Eric Anholt <eric@anholt.net>
It's currently only needed for the meson-main and meson-arm64 jobs, not
the other meson build jobs.
Also remove MESON_SHADERDB, just run .gitlab-ci/run-shader-db.sh
directly from the meson-main job.
v2:
* Also run prepare-artifacts.sh in meson-arm64 script
v3:
* Move tarball creation into the new script as well, as it prevented
ccache --show-stats from running in after_script
Reviewed-by: Eric Engestrom <eric.engestrom@intel.com> # v1
Reviewed-by: Eric Anholt <eric@anholt.net>
This reverts commit c9df92bf79.
It turns out that gitlab-runner uses kubernetes all wrong, spawning Pods
and sshing into them to run the script instead of Jobs containing the
script to run. This means that when anything goes wrong with the pod
(autoscale, preemption, VM maintenance, cluster reconfiguration), the job
fails and only sometimes gets handled as a runner system failure. Even
worse, due to bugs in either the runner or k8s itself, some classes of
timeout-related failure end up not being reported as failures, and the job
will incorrectly report success!
Disable using the "autoscale" cluster until we can do something else
(docker-machine instead of k8s, or the custom third-party k8s-native
runner).
Reviewed-by: Michel Dänzer <mdaenzer@redhat.com>
Acked-by: Daniel Stone <daniels@collabora.com>
The image used for test jobs is only about 1/6 as big as before, which
may help avoid some issues with some of the test boards.
Inspired by https://gitlab.freedesktop.org/mesa/mesa/issues/2046 .
v2:
* Leave LIBDRM_VERSION at 2.4.99 (Daniel Stone)
* Delete more build artifacts from dEQP tree (Daniel Stone)
v3:
* Set LD_LIBRARY_PATH for ldd
Acked-by: Daniel Stone <daniels@collabora.com> # v2
Reviewed-by: Eric Anholt <eric@anholt.net> # Except for the ldd line
$PWD doesn't work for variables:, it ended up as "/ccache", always
starting with an empty cache.
v2:
* Use relative path and realpath
v3:
* Use $CI_PROJECT_DIR (Eric Anholt)
* Clear ccache stats in before_script if the cache is in $CI_PROJECT_DIR
Fixes: c9df92bf79 "ci: Switch over to an autoscaling GKE cluster for
builds."
Reviewed-by: Eric Anholt <eric@anholt.net>
Reviewed-by: Eric Engestrom <eric.engestrom@intel.com>
The GKE pool we're using is 1-3 32-core VMs, preemptible (to keep
costs down), with 8 jobs concurrent per system. We have plenty of
memory (4G/core), so we run make -j8 to try to keep the cores busy even
when one job is in a single-threaded step (docker image download, git
clone, artifacts processing, etc.) When all jobs are generating work
for all the cores, they'll be scheduled fairly.
The nodes in the pool have 300GB boot disks (over-provisioned in space
to provide enough iops and throughput) mounted to /ccache, and
CACHE_DIR set pointing to them. This means that once a new
autoscaled-up node has run some jobs, it should have a hot ccache from
then on (instead of having to rely on the docker container cache
having our ccache laying around and not getting wiped out by some
other fd.o job). Local SSDs would provide higher performance, but
unfortunately are not supported with the cluster autoscaler.
For now, the softpipe/llvmpipe test runs are still on the shared
runners, until I can get them ported onto Bas's runner so they can be
parallelized in a single job.
Reviewed-by: Michel Dänzer <mdaenzer@redhat.com>
E.g. documentation-only changes cannot affect the outcome of the
pipeline, so don't waste resources on running it.
The thing we need to be careful about here is that the container stage
jobs must always run if any later stage jobs using the corresponding
docker images run. We're currently using the same .ci-run-policy
template for all jobs, so this is trivially true.
v2:
* Add bin/ and common.py (Eric Engestrom)
Acked-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com> # v1
Reviewed-by: Eric Engestrom <eric.engestrom@intel.com>
Commit 9edcce2a32 bumped the required libdrm-amdgpu version to
2.4.100. Update the version we use in our CI scripts to avoid CI
build failures.
Also bump the debian image name for this change to take effect.
Note that amdgpu is only built with the debian-buster image,
so only this image requires an update.
Fixes: 9edcce2a ("ac: get tcc_harvested from the kernel")
Signed-off-by: Alexandros Frantzis <alexandros.frantzis@collabora.com>
Reviewed-by: Erik Faye-Lund <erik.faye-lund@collabora.com>
When resolving a merge-conflict, I accidentally only updated the
ARM64-tag tag. Let's correct this.
Fixes: 3d529c1739 ("gitlab-ci: also build Zink on CI")
This adds a new CI job that runs on windows with MSVC. It currently
builds softpipe and osmesa, and runs the related unit tests. It does
rely on meson's wraps for zlib, but I've set up caching of the wrap
dependencies so hopefully that wont be a problem.
I really wanted to user powershell for this, but there just isn't an
easy way to do that, it's much easier to use batch scripts, so thats
what I used.
The leading `/` for .gitlab-ci/lava... must be removed because windows
doesn't understand it, and when it reads the file the job ends in error.
Reviewed-by: Eric Engestrom <eric@engestrom.ch>
At this point meson should be able to handle all of the non-windows
platforms just fine; we'd like to be able to stop maintaining scons for
those platforms sooner than later.
Reviewed-by: Eric Anholt <eric@anholt.net>
v2:
* Use LLVM 8 from buster-backports
v3:
* Use LLVM 7 again for armhf, llvmpipe is still broken there with LLVM 8
Acked-by: Eric Engestrom <eric.engestrom@intel.com>
This allows running the regression tests.
One downside is that we can't easily build the Vulkan overlay layer,
because only x86 binaries of the glslang validator are available. If
that's important, we could either use those binaries via qemu, or build
it from source.
v2:
* Add :amd64 suffix to existing debian-9/10 job names (Eric Engestrom)
Acked-by: Eric Engestrom <eric.engestrom@intel.com> # v1
Apparently needs: in a definition overwrites inherited ones. So
.deqp-test effectively didn't declare needs: for debian-10, which means
any jobs based on .deqp-test could spuriously run after the debian-10
job failed or was cancelled.
The one debian provides is broken in buster+, so I've just written my
own. This allows meson to find the installed zlib and prevents it from
falling back to wraps.
Reviewed-by: Eric Engestrom <eric.engestrom@intel.com>
While at it, rename to singular "container" for consistency.
Signed-off-by: Eric Engestrom <eric.engestrom@intel.com>
Reviewed-by: Michel Dänzer <mdaenzer@redhat.com>
It simplifies the definitions of jobs using the Debian 10 image.
The needs: was previously missing from the llvmpipe/softpipe test jobs,
so they could spuriously run if the debian-10 job failed or was
cancelled.
Reviewed-by: Eric Engestrom <eric.engestrom@intel.com>
In preparation for testing drivers other than Panfrost in LAVA labs.
Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Reviewed-by: Eric Anholt <eric@anholt.net>
Include Panfrost's gitlab.ci.yml file from Mesa's main .gitlab-ci.yml so
we test on devices with Panfrost.
This uses LAVA to schedule jobs in the devices and will be the base for
testing Etnaviv, Lima, etc.
Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Reviewed-by: Eric Anholt <eric@anholt.net>
Instead of a single cache shared between all jobs, but reduce the
maximum cache size to 1.5G (from 5G).
Rationale for smaller cache:
Pulling & pushing a 5G cache could take a long time. Consider
https://gitlab.freedesktop.org/mesa/mesa/-/jobs/684010 (click the "Show
complete raw" button to see timestamps): Pulling the cache took
1569927241-1569927194 = 47 seconds, pushing it 1569927671-1569927519
= 152, for a total of 199 seconds. The actual build took comparable
1569927518-1569927243 = 275 seconds, despite no cache hits from ccache.
In other words, the cache transfers almost doubled the job duration,
and they would have negated any build time benefits from ccache even
with a high cache hit rate.
Also, the smaller caches avoid blowing up storage requirements for them
too much.
Rationale for per-job caches:
Making a single cache significantly smaller might result in cached
build products from one job getting evicted by another job, reducing
the likelihood of cache hits from previous pipelines.
v2:
* Move up "ccache --max-size=1500M" call (Eric Engestrom)
Reviewed-by: Eric Engestrom <eric.engestrom@intel.com>
Without this, it was theoretically possible for the jobs to run before
the docker image was ready.
v2:
* Use - list syntax instead of [] (Eric Engestrom)
Reviewed-by: Eric Engestrom <eric.engestrom@intel.com>
Reviewed-by: Eric Anholt <eric@anholt.net>
This allows most build jobs to run before the stretch or arm64 docker
images are ready.
v2:
* Use - list syntax instead of [] (Eric Engestrom)
Reviewed-by: Eric Engestrom <eric.engestrom@intel.com>
Reviewed-by: Eric Anholt <eric@anholt.net>
This allows the *-old-llvm jobs to run before the buster docker images
are ready.
v2:
* Use - list syntax instead of [] (Eric Engestrom)
Reviewed-by: Eric Engestrom <eric.engestrom@intel.com>
Reviewed-by: Eric Anholt <eric@anholt.net>
Pros:
* Less fragile due to not mixing packages from stretch and buster
* No longer need to use third-party LLVM packages
* The buster image now uses GCC 8 for C++ as well (previously 6 for C++,
8 for C), allowing to drop some hacks
Con:
* The stretch image now only uses GCC 6 for C as well as C++
* Need separate jobs for testing old LLVM versions
Reviewed-by: Eric Engestrom <eric.engestrom@intel.com>
Yes, some tests fail, but we can turn those into XFAILs at meson time.
Better to keep the things that work working than not cover them at all.
Unfortunately XPASS results will not cause the build to fail until we
update CI to meson 0.51 or newer.
Reviewed-by: Daniel Stone <daniels@collabora.com>
This might allow the arm64 tests to start running earlier.
Reviewed-by: Eric Engestrom <eric.engestrom@intel.com>
Reviewed-by: Eric Anholt <eric@anholt.net>
v2:
* Preserve setting NIR_VALIDATE=0 for all arm64_* jobs
* Preserve setting DEQP_SKIPS=deqp-default-skips.txt for
arm64_a306_gles2 jobs
Reviewed-by: Eric Engestrom <eric.engestrom@intel.com> # v1
Reviewed-by: Eric Anholt <eric@anholt.net>
Support for multiple inheritance was added to GitLab recently.
Reviewed-by: Eric Engestrom <eric.engestrom@intel.com>
Reviewed-by: Eric Anholt <eric@anholt.net>
This allows the arm64_a306_gles2 jobs to run as soon as the meson-arm64
job has finished.
Fixes: 6f0dc087b7 "freedreno: Introduce gitlab-based CI."
Reviewed-by: Eric Engestrom <eric.engestrom@intel.com>
Reviewed-by: Eric Anholt <eric@anholt.net>
Since freedreno's kernel and GPU reset seem to be totally solid, we
don't need to have the complexity of the LAVA setup that panfrost has.
Instead, we can register some boards as shared gitlab runners and have
the jobs run out of a docker container just like we do for llvmpipe.
Just make sure that the DRI device node is passed through to the
containers in the gitlab config ('devices = ["/dev/dri"]' under
runners.docker).
If a runner fails (networking dies, kernel panic, etc.) it'll take out
one build but the rest can keep going since gitlab-runner is what
pulls jobs. Since the runner pulls jobs, it also means that they can
live behind firewalls instead of needing some public address to be
accessed by gitlab.fd.o.
For now, enable it just on db410c (A307) and cheza (A630) as those are
the hardware that I have plenty of. A307 is only testing GLES2 since
running all of GLES3 takes too long for the number of boards I've
brought up.
Acked-by: Rob Clark <robdclark@chromium.org>
Acked-by: Kenneth Graunke <kenneth@whitecape.org>
Reviewed-by: Timothy Arceri <tarceri@itsqueeze.com>
Reviewed-by: Jose Fonseca <jfonseca@vmware.com>
[ Michel Dänzer: Dropped jessie line from debian-install.sh again ]
This way, the test jobs can start running before all build+test jobs
have finished, once the meson-main job has.
Idea suggested by Daniel Stone on IRC.
See https://docs.gitlab.com/ce/ci/directed_acyclic_graph/ and
https://docs.gitlab.com/ce/ci/yaml/README.html#needs for details.
v2:
* Improve commit log (Daniel Stone, Eric Engestrom)
Reviewed-by: Eric Engestrom <eric.engestrom@intel.com>
Reviewed-by: Eric Anholt <eric@anholt.net>
The GLES2 CTS takes about 8 minutes of total runtime (at parallel 4 is
~2 minutes in the test stage if runners are free), while GLES3 takes
about 25. Since the GLES3 run is pretty expensive, just do a cheap
touch test of 1 out of every 10 tests in the test list on MRs, until
we can get the runtime down.
v2: Drop the full run for now until we can bring runtime down or bring
up a dedicated mesa runner.
Reviewed-by: Eric Engestrom <eric@engestrom.ch> (v1)
Reviewed-By: Gert Wollny <gert.wollny@collabora.com> (v1)
This is the start of doing CTS tests on merges to Mesa master. We use
the surfaceless platform so that we don't need to bother bringing up
weston or X11. The surface size is kept low to reduce runtime, but
this comes at the cost of many rendering tests skipping due to
too-small render targets (as we see the impact of Mesa on the shared
runner pool, we can reevaluate this and what set of CTS tests we want
to run).
We split the job up across 4 runners (each at 4 llvmpipe threads), so
that the job can load-balance across our shared runners and finish
sooner (since dEQP is very single-thread-performance bound).
Reviewed-by: Eric Engestrom <eric.engestrom@intel.com>
Now that we're running the drivers we build, building with
optimization is important for keeping our runtime down. Shaves about
4 minutes of runtime off of GLES2 CTS of llvmpipe at 64x64.
v2: Only switch meson-main until we enable CTS for other builds
on request by Michel.
Reviewed-by: Eric Engestrom <eric.engestrom@intel.com>
I want to enable CI of llvmpipe out of the meson-main build. So, kick
classic swrast/osmesa to meson-i386, then promote llvmpipe to
meson-main (along with nine, now that classic osmesa isn't keeping it
out of there).
Reviewed-by: Michel Dänzer <michel.daenzer@amd.com>
Reviewed-by: Eric Engestrom <eric.engestrom@intel.com>