Commit Graph

17 Commits

Author SHA1 Message Date
Eric Anholt cb655d2554 Revert "ci: Switch over to an autoscaling GKE cluster for builds."
This reverts commit c9df92bf79.

It turns out that gitlab-runner uses kubernetes all wrong, spawning Pods
and sshing into them to run the script instead of Jobs containing the
script to run.  This means that when anything goes wrong with the pod
(autoscale, preemption, VM maintenance, cluster reconfiguration), the job
fails and only sometimes gets handled as a runner system failure.  Even
worse, due to bugs in either the runner or k8s itself, some classes of
timeout-related failure end up not being reported as failures, and the job
will incorrectly report success!

Disable using the "autoscale" cluster until we can do something else
(docker-machine instead of k8s, or the custom third-party k8s-native
runner).

Reviewed-by: Michel Dänzer <mdaenzer@redhat.com>
Acked-by: Daniel Stone <daniels@collabora.com>
2019-11-06 11:38:07 -08:00
Tomeu Vizoso 427d0c4b6a gitlab-ci: Run only LAVA jobs in special-named branches
Run only jobs needed for testing on LAVA devices if a branch starts with
lava-ci-.

This allows developers to have faster test cycles as these pipelines
take only a bit above 8 minutes. Also has the advantage of conserving
resources.

Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Reviewed-by: Eric Engestrom <eric.engestrom@intel.com>
2019-11-05 16:09:47 +01:00
Eric Anholt c9df92bf79 ci: Switch over to an autoscaling GKE cluster for builds.
The GKE pool we're using is 1-3 32-core VMs, preemptible (to keep
costs down), with 8 jobs concurrent per system.  We have plenty of
memory (4G/core), so we run make -j8 to try to keep the cores busy even
when one job is in a single-threaded step (docker image download, git
clone, artifacts processing, etc.)  When all jobs are generating work
for all the cores, they'll be scheduled fairly.

The nodes in the pool have 300GB boot disks (over-provisioned in space
to provide enough iops and throughput) mounted to /ccache, and
CACHE_DIR set pointing to them.  This means that once a new
autoscaled-up node has run some jobs, it should have a hot ccache from
then on (instead of having to rely on the docker container cache
having our ccache laying around and not getting wiped out by some
other fd.o job).  Local SSDs would provide higher performance, but
unfortunately are not supported with the cluster autoscaler.

For now, the softpipe/llvmpipe test runs are still on the shared
runners, until I can get them ported onto Bas's runner so they can be
parallelized in a single job.

Reviewed-by: Michel Dänzer <mdaenzer@redhat.com>
2019-10-31 11:19:43 -07:00
Eric Anholt da6cc72237 ci: Make lava inherit the ccache setup of the .build script.
It was just duplicating the code.

Reviewed-by: Michel Dänzer <mdaenzer@redhat.com>
2019-10-31 11:19:43 -07:00
Tomeu Vizoso 01af59b2d9 gitlab-ci: Disable lima jobs
The runner that submits jobs there is down and will turn some time to
get fixed. Disable them for now to keep the CI green.

Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
2019-10-31 11:08:11 +00:00
Dylan Baker 06e4647cb0 gitlab-ci: refactor out some common stuff for Windows and Linux
Reviewed-by: Eric Engestrom <eric@engestrom.ch>
2019-10-25 22:47:32 +00:00
Neil Armstrong e919c44c3b Revert "ci: Disable lima until its farm can get fixed."
This reverts commit fb9362c6fb.

Signed-off-by: Neil Armstrong <narmstrong@baylibre.com>
Acked-by: Eric Engestrom <eric.engestrom@intel.com>
Reviewed-by: Vasily Khoruzhick <anarsoul@gmail.com>
2019-10-25 20:52:03 +02:00
Tomeu Vizoso 3168b8defa gitlab-ci: Update kernel for LAVA jobs to 5.4-rc4
Update to 5.4-rc4 so we can test Panfrost on devices with Mali T720 and
T820.

A bug was found that prevented things working at all on RK3288 devices,
so we carry a patch for now in my personal fork.

Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Acked-by: Daniel Stone <daniels@collabora.com>
2019-10-24 08:47:37 +02:00
Eric Anholt fb9362c6fb ci: Disable lima until its farm can get fixed.
It's been throwing the following error today:

"<Fault -32603: 'Internal Server Error (contact server administrator
for details): could not extend file "base/17952/18226": No space left
on device\nHINT: Check free disk space.\n'>"

Reviewed-by: Daniel Stone <daniels@collabora.com>
2019-10-21 20:31:34 -07:00
Eric Engestrom 3bcd54f3fc gitlab-ci: set a common job parent for test stage
Signed-off-by: Eric Engestrom <eric.engestrom@intel.com>
Reviewed-by: Michel Dänzer <mdaenzer@redhat.com>
2019-10-15 17:42:39 +01:00
Eric Engestrom aba78c2d38 gitlab-ci: set a common job parent for build stage
Signed-off-by: Eric Engestrom <eric.engestrom@intel.com>
Reviewed-by: Michel Dänzer <mdaenzer@redhat.com>
2019-10-15 17:42:39 +01:00
Eric Engestrom 81b98e99cd gitlab-ci: set a common job parent for container stage
While at it, rename to singular "container" for consistency.

Signed-off-by: Eric Engestrom <eric.engestrom@intel.com>
Reviewed-by: Michel Dänzer <mdaenzer@redhat.com>
2019-10-15 17:42:39 +01:00
Tomeu Vizoso 6397dff6d7 gitlab-ci/lava: Test Lima driver with dEQP
Run dEQP on boards with Mali 400 and 450 in Baylibre's lab.

There's lots of skipped tests because of crashes and undetermined
behavior. May be a good idea to run the tests with valgrind and fix any
issues found.

Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Reviewed-by: Vasily Khoruzhick <anarsoul@gmail.com>
Reviewed-by: Neil Armstrong <narmstrong@baylibre.com>
2019-10-10 14:50:14 +00:00
Tomeu Vizoso 8a168683d0 gitlab-ci/lava: Use files to list tests to skip
As the non-LAVA runner script does, have per-GPU version files listing
the tests that are to be skipped, due to being very slow, unstable, etc.

Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Reviewed-by: Vasily Khoruzhick <anarsoul@gmail.com>
Reviewed-by: Neil Armstrong <narmstrong@baylibre.com>
2019-10-10 14:50:14 +00:00
Michel Dänzer 94cfe59070 gitlab-ci/lava: Add needs: for container image to test jobs
Without this, the test jobs could spuriously run after the container
job failed or was cancelled, even if the build job didn't run at all.

Reviewed-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
2019-10-09 16:19:56 +02:00
Tomeu Vizoso c00f017e65 gitlab-ci/lava: Fix image to use in test jobs
In the test stage, we can use any of the two container images as we
arent going to do anything architecture-dependent when submitting the
jobs to LAVA.

But if we are in a pipeline in which the images need to be rebuilt and
one finishes much earlier than the other, it could happen that the test
job that executes first fails to find the container image.

To avoid that, have each job in the test stage to use the image that has
been already implicitly built by depending on the build job for the
given arch.

Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Reviewed-by: Michel Dänzer <mdaenzer@redhat.com>
2019-10-07 07:31:55 -07:00
Tomeu Vizoso 555c0de8c6 gitlab-ci: Move LAVA-related files into top-level ci dir
In preparation for testing drivers other than Panfrost in LAVA labs.

Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Reviewed-by: Eric Anholt <eric@anholt.net>
2019-10-06 07:47:41 -07:00
Renamed from src/gallium/drivers/panfrost/ci/gitlab-ci.yml (Browse further)