So that if the former fails, build & test jobs won't run, wasting less
time & CI resources.
This requires slight tweaks to the rules: of the former job, to make
sure it always exists when the build jobs do.
Acked-by: Erik Faye-Lund <erik.faye-lund@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7278>
These can catch various common issues in MR settings and Git commit
logs.
The "check mr" job only exists in pre-merge pipelines for MRs, and runs
automatically.
The "check commits" job only exists in pre-merge pipelines for MRs and
in pipelines for forked branches. It runs automatically in the former
case and can be manually triggered in the latter.
v2:
* Use git_archive docker image (Daniel Stone)
* Use a single sanity stage for both jobs (Tomeu Vizoso)
Reviewed-by: Peter Hutterer <peter.hutterer@who-t.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6209>
We put it in the first container stage to prevent it from waiting for
jobs in previous stages, but empty needs: works for that as well.
Reviewed-by: Peter Hutterer <peter.hutterer@who-t.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6209>
I wasted a bunch of time today tracking down a spurious test results
change due to a driver invoking UB by running tests where NIR validation
had failed (instruction reading from components beyond vector size). If
we need to shrink our coverage to get runtimes down, it will still be
better to be catching validation errors in CI.
To keep the test jobs runtime under 10 minutes, I've split a530's gles2 to
two different jobs.
Acked-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7203>
In SDK 25 (Android 7, 2016), the kernel only went up to around 4.4, so
the upstreamed sync API didn't exist, and libsync didn't expose
sync_merge().
In SDK 26 (Android 8, 2017), the kernel is sometimes bumped to 4.9 or
4.14, but libsync has a sync_merge() that operates on either the
downstream or the upstream API.
Since our android-targeting drivers in general use sync objects (requiring
a kernel newer than 7 got) and sync_merge() (suggesting interaction with
external fencing that started in Android 8), make CI build against an SDK
version with the sync API. I think really doing SDK 25 right at this
point would involve backporting mesa with some #ifdefs to not expose sync
APIs that wouldn't work.
Reviewed-by: Kristian H. Kristensen <hoegsberg@google.com>
Acked-by: Tapani Pälli <tapani.palli@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6821>
To support Android drivers, we're going to want to be tracking that Mesa's
build succeeds on a real android toolchain. This still uses the android
stubs since these libs aren't in the NDK.
Note that I had to drop the Intel and AMD drivers currently: we don't have
LLVM cross-compiled for Android in this container, and I'm honestly hoping
ACO saves us from that. Intel has dependencies on libexpat, which AOSP
really doesn't want to bring in, and it looks to me like those dependencies
could be optional.
Reviewed-by: Kristian H. Kristensen <hoegsberg@google.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6700>
This causes a lot of hiccups on the CL tests, but I've got most of
them fixed in another MR in pieces.
This should at least give a much more realistic baseline.
v2: use script in both places
Reviewed-by: Michel Dänzer <mdaenzer@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7073>
This adds support for building clover/llvmpipe and running the
piglit CL tests on it.
It uses the gl testing container, and add builds the libclc
spirv libraries as part of that which requires the llvm spirv
translator in the build container.
It also builds the llvm spirv translator as part of the build
root and creates a mesa build that builds clover for testing
against it. It uses llvm 10 as the baseline.
This drops bswap as it has an oob memory access with llvmpipe
which cause flaky test results. phatk also seems flaky
Reviewed-by: Michel Dänzer <mdaenzer@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6901>
The gles3 and gles31 multisample and 565-no-depth-no-stencil caselists
are also mustpass. And they don't add a significant number of extra
test cases.
The remaining mustpass caselists all involve rotation, which is not
currently supported in the surfaceless deqp build.
Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6553>
Some compile-time regressions go unnoticed because not all gallium
drivers are compiled as part of the meson-clang job. Let's pass an
explicit list of drivers to compile instead of setting GALLIUM_DRIVERS
to "auto" to increase CI coverage.
Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com>
Acked-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Reviewed-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6811>
Since pre-merge pipelines run in the "mesa" namespace now, the rule
would have created the job if the source branch was named "master" (and
the job might have updated the public website).
Acked-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6566>
They are using a in source tree version of the needed libdrm
functionality or are shipping all needed headers in the source tree.
Signed-off-by: Christian Gmeiner <christian.gmeiner@gmail.com>
Reviewed-by: Eric Anholt <eric@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6672>
I'd like to see this new non-UAPI feature bake in CI. More importantly,
it may prevent some classes of flakes on cheza by isolating the processes
on the GPU so that a fault in one doesn't stomp over memory in another.
I've also pulled in a fix that etnaviv needed for their upcoming CI.
We add a few more kernel options while uprevving:
- More interconnect drivers for getting good GPU perf
- PRNG so that we don't get late-in-boot complaints about randomness.
- db820c's power domains and ethernet so hopefully we can switch to this
upstream kernel
This seems to slightly change the flakes happening in bypass mode, so add
them to the list.
Reviewed-by: Kristian H. Kristensen <hoegsberg@google.com>
Reviewed-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6592>
Inform gitlab about the JUnit XML file that tracie produces, so that the
pipeline page can present more detailed information about tracie test
results.
Signed-off-by: Alexandros Frantzis <alexandros.frantzis@collabora.com>
Reviewed-by: Eric Anholt <eric@anholt.net>
Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6598>
Inform gitlab about the JUnit XML file that tracie produces, so that the
pipeline page can present more detailed information about tracie test
results.
Signed-off-by: Alexandros Frantzis <alexandros.frantzis@collabora.com>
Reviewed-by: Eric Anholt <eric@anholt.net>
Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6598>
Only in pre-merge pipelines for MRs, or in pipelines for forked project
branches.
Having the manual job in post-merge pipelines prevented the pages job
from running automatically as well, which could prevent the public
website from getting updated.
Acked-by: Daniel Stone <daniels@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6534>
There's no need to have separate build scripts here, just choose what the
DEQP_TARGET is for the particular container being built. This brings in a
tremendous number of GLES test fixes that haven't made it into a tagged
gles CTS release.
Closes: #2056
Reviewed-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6526>
The version bump gets us various testcase fixes, mostly to test
requirements). While we're rebuilding the container, copy GL CTS stuff
from build-deqp-gl.sh -- we had already included the glcts binary in our
image, but we had unnecessary other binaries and were missing the mustpass
files (container size stays the same overall). Also pull in all the GLES
mustpass lists, not just the main ones -- Rob wants them to increase our
coverage to match what Android CTS covers.
Reviewed-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6526>
Since the last GitLab update, pre-merge pipelines for MRs run in the
mesa/mesa namespace, so this job didn't get created anymore, causing
trouble.
Reviewed-by: Daniel Stone <daniels@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6522>
Since the last GitLab update, pre-merge pipelines for MRs run in the
mesa/mesa namespace, so this job didn't get created anymore, causing
trouble.
Reviewed-by: Daniel Stone <daniels@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6522>
Debian defaults to bfd, which is comically slow. We can't use lld because
the old version we have in the debian stable we use has various bugs.
This required bumping libwayland, which had multiply-defined symbols
issues in the previous release.
Closes: #3236
Reviewed-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6324>
This will keep us from needing new MRs to fix up release build warnings
after the fact. I've mostly derived from meson-gallium, dropping
components that aren't clean yet.
Reviewed-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6462>
Now that we have a more powerful runner and a more stable Docker
configuration, try re-enabling the Windows build.
Signed-off-by: Daniel Stone <daniels@collabora.com>
Reviewed-by: Erik Faye-Lund <erik.faye-lund@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6446>
I've been hacking on softpipe in the process of trying to delete a bunch
of core Mesa code, and want to make sure I don't regress desktop GL
either. The run takes under a minute and a half.
Reviewed-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Reviewed-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6466>
This makes sure that we keep executing the tests so that we can get our
alerts in IRC and know whether the tests are still flaking. It also keeps
us from having adjustments to the skip list causing failures/flakes to
move to different tests (as seen with a530 having to move some xfails
around after changing the skip list)
Reviewed-by: Christian Gmeiner <christian.gmeiner@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6392>
And remove a redundant call to ccache --show-stats
Signed-off-by: Andres Gomez <agomez@igalia.com>
Reviewed-by: Daniel Stone <daniels@collabora.com>
Reviewed-by: Christian Gmeiner <christian.gmeiner@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6343>
As drivers have been tested with more and more traces, the yml file is
becoming a bit unwieldy. As more drivers are going to be tested with
traces, and more traces will be used, split them in per-driver files so
the size stays manageable.
Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Reviewed-by: Reviewed-By: Rohan Garg <rohan.garg@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6185>
We've seen this crash a few times over the past couple of weeks, but
haven't yet managed to reproduce locally and fix. In the meantime, just
skip.
Signed-off-by: Daniel Stone <daniels@collabora.com>
Reviewed-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6352>
This is the initial import of the vallium frontend for gallium.
This is only good enough to run the triangle and the gears demo
(wrongly) from Sascha demos.
Improvements are mostly on the llvmpipe side after this.
It contains an implementation of the Vulkan API which is mapped
onto the gallium API, and is suitable only for SOFTWARE drivers.
Command buffers are recordred into malloced memory, then later
they are played back against the gallium API. The command buffers
are mostly just Vulkan API marshalling but in some places the information is
processed before being put into the command buffer (renderpass stuff).
Execution happens on a separate "graphics" thread, againt the gallium API.
There is only a single queue which wraps a single gallium context.
Resources are allocated via the new resource/memory APIs.
Shaders are created via the context and bound/unbound in the
second thread.
(No HW for reasons - memory management, sw paths for lots of paths,
pointless CPU side queue)
v2: drop mesa_icd, drop cpp_args, drop extra flags, change meson config (Eric)
v2.1: use meson-gallium job
meson pieces:
Reviewed-by: Eric Engestrom <eric@engestrom.ch>
overall:
Acked-by: Roland Scheidegger <sroland@vmware.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6082>
To make sure we don't completely break ACO GFX1030 support.
Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6257>
https://github.com/KhronosGroup/EGL-Registry/pull/95 has moved
a couple of extensions defines and functions to the upstream `eglext.h`,
but when 9a74746bd1 sync'ed these files we broke compilation
of apps that require these symbols on systems that don't have the
updated Khronos headers.
On non-GLVND builds, we still provide these headers, so everything's
fine, but on GLVND builds the Khronos headers are external so we need to
make sure we have a libglvnd version that's recent enough.
Fixes: 9a74746bd1 ("EGL: sync headers with Khronos")
Signed-off-by: Eric Engestrom <eric@engestrom.ch>
Acked-by: Daniel Stone <daniels@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6069>
We anyway depend already on robust network support in the DUTs, and we
can save quite some time this way.
It will also allow us to grow further as we expand coverage.
Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Reviewed-By: Rohan Garg <rohan.garg@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6114>
Some changes unintendedly slipped into an unrelated commit before it was
merged.
This caused kernel modules to be built and installed in the ramdisk,
which caused some devices to fail to boot due to the ramdisk size limit
being surpassed.
These changes weren't in effect until a subsequent commit triggered a
rebuild of the ramdisks.
Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Acked-by: Daniel Stone <daniels@collabora.com>
Fixes: a9560939e0 ("ci: Build-test Panfrost tools")
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6167>
Now that we have MinIO, we can distribute traces better than by direct
downloads from git.
With a caching MinIO instance local to the DUT, total run times should
be noticeably impacted.
Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Reviewed-by: Eric Anholt <eric@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6136>
The MinIO server is sometimes complaining about the submitted date being
too off.
Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Reviewed-by: Daniel Stone <daniels@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6135>
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Acked-by: Daniel Schürmann <daniel@schuermann.dev>
Acked-by: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3521>
For testing Mesa on LAVA devices with the amd64 architecture, build
kernels and rootfs in the same way as we do for arm64 and armhf.
Also add a few trivial jobs for a specific AMD Chromebook.
Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Reviewed-by: Rohan Garg <rohan.garg@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5903>
As we are doing that in several places already and we'll need to build
in others as well.
Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Reviewed-by: Rohan Garg <rohan.garg@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5903>
This also tunes `.freedreno-rules` a bit so that it isn't triggered by
various tools that don't effect the driver build.
The .gitlab-ci directory is kept separate from the toplevel one so that
updates to (for example) reference decode output do not trigger all the
other-driver jobs to run.
Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6070>
We're now using at least twice as many CPU cores per job (on shared
runners), so they only take about half as long, and should still be
under 10 minutes.
Reviewed-by: Eric Anholt <eric@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6011>
If another MR was merged while these were still running for the main
project, the result could be no updated images in the main project
registry (forcing a rebuild of the new images in all forked projects) or
an outdated Mesa website.
Reviewed-by: Eric Anholt <eric@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6011>
Upload failed images and the results.yml file to MinIO, to facilitate
debugging.
Also, fix version checking when git is installed as Mesa is going to
output a different renderer string if git is installed.
Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Reviewed-by: Daniel Stone <daniels@collabora.com>
Reviewed-By: Rohan Garg <rohan.garg@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5472>
fd.o has retuned the x86 runners on packet for -j8. Rather than having to
tweak our CI every time fd.o decides to rebalance job concurrency, respect
what the runner admin has chosen for their builds (this will also be
convenient for people with large local runners).
Reviewed-by: Michel Dänzer <michel@daenzer.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5669>
I hadn't realised these were disabled, llvmpipe now exposes this extension.
One additional failure is fine to get the added testing coverage.
Reviewed-by: Michel Dänzer <mdaenzer@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5973>
I haven't noticed tftp boot issues in last few days, not sure if they
where just a fluke on Mon or if it is somehow related to # of jobs we
run (ie. having more of the c630 runners powered up and running more
of the time).
Let's turn them back on and see what happens.
Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5952>
I mistakenly removed what I thought were remnants of when Freedreno used
LAVA for their DUTs. lava_arm.sh is used for baremetal, so re-add that
code.
Fixes: dcd171f5e9 ("gitlab-ci: More stable URL for kernel and ramdisk artifacts, for LAVA")
Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Reviewed-by: Eric Anholt <eric@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5839>
The commit below changed the rule such that it accidentally also applied
to the non-MR pipelines created by Marge, resulting in Marge triggering
twice as many jobs as necessary.
Fixes: 549b4a3dd4 "gitlab-ci: Automatically run pipelines for Marge
Bot pre-merge only"
Acked-by: Daniel Stone <daniels@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5898>
Since version 2.4.101 there are only xz archives hence the bz2 to xz
change.
Signed-off-by: Karol Herbst <kherbst@redhat.com>
Reviewed-by: Michel Dänzer <mdaenzer@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5906>
Since version 2.4.101 there are only xz archives hence the bz2 to xz
change.
Signed-off-by: Karol Herbst <kherbst@redhat.com>
Reviewed-by: Michel Dänzer <mdaenzer@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4580>
Tomeu discovered that GitLab 12.8 fixed the bug where jobs would
spuriously run even though some of their dependency jobs were skipped.
So we don't need to list indirect dependencies anymore.
Reviewed-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5845>
They seem to be sometimes taking a while to boot, which is triggering CI
timeouts. (Possibly tftp server in bad shape?) Cut out non-essential
a630 CI jobs, and reduce the gles3/gles31 jobs to compensate.
Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5893>
It has no dependencies and costs virtually nothing to build. There is
no downside to enabling it unconditionally, so let's do just that.
Signed-off-by: Eric Engestrom <eric.engestrom@intel.com>
Reviewed-by: Emil Velikov <emil.velikov@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3161>
Currently, scheduled pipelines are only used to rebuild
the git-cache archive daily. There is no point in rebuilding
eveything, so ensure that any normal jobs are removed from
the scheduled pipelines.
Reviewed-by: Michel Dänzer <mdaenzer@redhat.com>
Acked-by: Daniel Stone <daniels@collabora.com>
Signed-off-by: Benjamin Tissoires <benjamin.tissoires@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5804>
the new ci-fairy minio on ci-templates can copy
data to/from the MinIO server with much less permissions.
Upgrading mesa to this commit will allow us to restrict the
git-cache bucket permission to only "fetch" objects, i.e.
not allow anybody to walk through the tree of any repo.
Reviewed-by: Michel Dänzer <mdaenzer@redhat.com>
Signed-off-by: Benjamin Tissoires <benjamin.tissoires@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5804>
Place the kernel and ramdisk into a place in the file server so the URL
will only change when the contents also change.
Also put the Mesa build into a separate tarball so the ramdisk's
contents don't change every build.
With proper caching in place, all devices in the same farm need only to
download the mesa tarball once, saving time.
As we switch to MinIO for making kernels and rootfs available to LAVA
devices, we can stop using Docker to distribute them.
Instead, build when needed in separate jobs that push directly to MinIO,
from where LAVA devices can download them.
Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Acked-by: Daniel Stone <daniels@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5515>
Over the last 7 days, git pulls represented a total of 1.7 TB.
On those 1.7 TB, we can see:
- ~300 GB for the CI farm on hetzner
- ~730 GB for the CI farm on packet.net
- ~680 GB for the rest of the world
We can not really change the rest of the world*, but we can
certainly reduce the egress costs towards our CI farms.
Right now, the gitlab runners are not doing a good job at
caching the git trees for the various jobs we make, and
we end up with a lot of cache-misses. A typical pipeline
ends up with a good 2.8GB of git pull data. (a compressed
archive of the mesa folder accounts for 280MB)
In this patch, we implemented what was suggested in
https://gitlab.com/gitlab-org/gitlab/-/issues/215591#note_334642576
- we host a brand new MinIO server on packet
- jobs can upload files on 2 locations:
* git-cache/<namespace>/<project>/<branch-name>.tar.gz
* artifacts/<namespace>/<project>/<pipeline-id>/
- the authorization is handled by gitlab with short tokens
valid only for the time of the job is running
- whenever a job runs, the runner are configured to execute
(eval) $CI_PRE_CLONE_SCRIPT
- this variable is set globally to download the current cache
from the MinIO packet server, unpack it and replace the
possibly out of date cache found on the runner
- then git fetch is run by the runner, and only the delta
between the upstream tree and the local tree gets pulled.
We can rebuild the git cache in a schedule job (once a day
seems sufficient), and then we can stop the cache miss
entirely.
First results showed that instead of pulling 280MB of data
in my fork, I got a pull of only 250KB. That should help us.
* arguably, there are other farms in the rest of the world, so
hopefully we can change those too.
Reviewed-by: Michel Dänzer <mdaenzer@redhat.com>
Reviewed-by: Peter Hutterer <peter.hutterer@who-t.net>
Signed-off-by: Benjamin Tissoires <benjamin.tissoires@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5428>
Dave Airlie
Daniel Stone
Michel Dänzer
Rhys Perry
Eric Anholt
Alejandro Piñeiro
Rob Clark
Serge Martin
Boris Brezillon
Christian Gmeiner
Alexandros Frantzis
Tapani Pälli
Andres Gomez
Tomeu Vizoso
Adam Jackson
Samuel Pitoiset
Eric Engestrom
Kristian H. Kristensen
Rohan Garg
Karol Herbst
Jonathan Marek
Eric Engestrom
Timothy Arceri
Benjamin Tissoires
Erik Faye-Lund