In order to avoid having to dig through artifacts to see what failed.
Signed-off-by: Eric Engestrom <eric@igalia.com>
Acked-by: David Heidelberg <david.heidelberg@collabora.com>
Acked-by: Daniel Stone <daniels@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17804>
Empirically, a successful LAVA boot time should take less than 3
minutes.
LAVA itself is configured to attempt thrice to boot the device,
summing up to 9 minutes.
It is better to retry the boot than cancel the job and re-submit to
avoid the enqueue delay.
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17646>
In the future, bump will be needed to update non-VK tags as wine traces get
used elsewhere.
Reviewed-by: Andres Gomez <agomez@igalia.com>
Signed-off-by: David Heidelberg <david.heidelberg@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17623>
It seems that we sometimes stall out executing "fastboot boot", and if
that happens we want to reboot the board and try again.
Fixes: #6682
Acked-by: David Heidelberg <david.heidelberg@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17607>
It'll be re-used by piglit traces.
Reviewed-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Reviewed-by: Andres Gomez <agomez@igalia.com>
Signed-off-by: David Heidelberg <david.heidelberg@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17573>
Add a global-level variable that allows disabling all jobs that would
have gone to the Igalia lab, to be used in case of outages or failures.
Reviewed-by: Eric Engestrom <eric@igalia.com>
Reviewed-by: Andres Gomez <agomez@igalia.com>
Reviewed-by: Daniel Stone <daniels@collabora.com>
Signed-off-by: David Heidelberg <david.heidelberg@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17178>
Sleep a bit before executing `lava-test-case` to give time for bash to dump shell xtrace messages into
the console, which may cause interleaving with `LAVA_SIGNAL_STARTTC` in some
devices like a618.
The same approach worked for `LAVA_SIGNAL_STARTRUN` since
3b8d10d270 (deafdd86b8d9d0108bc692f479c3b31c4c7d5635_161_164)
was merged.
Closes: #6867
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Reviewed-by: David Heidelberg <david.heidelberg@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17533>
This contains some Device Generated Command tests that will be
useful for an experimental RADV implementation.
Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17463>
I mistakenly applied .gl-rules to the non-freedreno perf jobs, which
caused them to be incorrectly run pre-merge when core GL files changed.
Pull the freedreno core GL performance job rules out, explain a bit more
what is going on, and use it from iris and virgl performance testing.
This also drops running freedreno performance when core vulkan files
change -- freedreno perf testing doesn't have any turnip usage, nor does
it watch for turnip file changes.
Acked-by: Michel Dänzer <mdaenzer@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17386>
If you accidentally re-included your test job core definition after your
driver-specific ruleset, you'd end up running the driver job on every
source code change. This had happened with a630_gles_asan: it included
.baremetal-test-arm64-asan (and thus .baremetal-test) after including
.a630-test, to override .baremetal-test-arm64's depednencies to use asan
artifacts instead.
Acked-by: Michel Dänzer <mdaenzer@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17386>
... and explain what they're doing, compared to the test rules in
test-source-dep.yml.
Unfortunately, we can't really pull them into test-source-dep.yml with
other source deps, because of various '&'-'*' references.
Acked-by: Michel Dänzer <mdaenzer@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17386>
We should be explicit that we are cancelling jobs once the script finds
some log messages that are linked with known issues. That means the
script preemptively retried the job without giving chances to recover.
Adds magenta color to cancelled jobs.
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17389>
Mark test_full_yaml_log with this new marker to be easily run by the
developers.
Make `debian-testing` skip this test with `not slow` marker hint.
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17389>
Implement a log-based retry hint for R8152 issue described in #6681,
which is based on detecting these two consecutive lines:
```
r8152 <USB> eth0: Tx status -71
nfs: server <IP> not responding, still trying
```
Where <IP> and <USB> could be any IP and USB addresses, respectfully.
This commit is a temporary fix since it requires a section-aware log
follower, implemented in !16323. When the cited MR is merged, one will
make a proper fix on top of that.
Closes: #6681
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17389>
In some jobs, such as
https://gitlab.freedesktop.org/gallo/mesa/-/jobs/24904100, the kmsg is
interleaved with stderr/stdout in serial console, making it difficult to
confidently find the log messages to detect when the DUT is booting,
when the DUT is running etc.
Luckily, LAVA sends redundant messages about their signals. We can use
them to mitigate the chance of missing an interleaved message by being
more open to different messages, using the regex on both `debug` and
`target` LAVA log levels.
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16323>
There are some leftovers in the jobs logs after the result log line.
Only print until the init-stage2.sh output, to raise the chance to check
for the test script results at the first glance in the Gitlab logs.
Extra changes:
- Add `hung` status for jobs considered hanging in the Gitlab
- print them after the retry loop
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16323>
This will serve to warn the user that those messages are processed
differently, e.g. the kmsgs does not trigger heartbeats and maybe
eventual targets of hint to retry the job immediately.
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16323>
test_full_yaml_log is a test that will look for a LAVA log YAML file at
`/tmp/log.yaml` and consume it as it was a realtime CI job.
It is useful for debugging issues related with LAVA.
Let's keep it skipped by default, to avoid introducing entire logs into
the codebase.
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16323>
Currently, the submitter consider that every new log that comes from the
DUT console is a signal that the device is healthy, but maybe that is
not the case, since in some kernel hangs/failures, no output is
presented except from some kernel messages.
This commit bypass the heartbeat when the LogFollower detect a kernel
message. Any log line that does follow the kmsg pattern will make the
job labeled as healthy again.
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16323>
Now LogFollower is used to deal with the LAVA logs.
Moreover, this commit adds timeouts per Gitlab section, if a section
takes longer than expected, cancel the job and retry again.
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16323>
- Create LogFollower to capture LAVA log and process it adding some
- GitlabSection and color treatment to it
- Break logs further, make new gitlab sections between testcases
- Implement LogFollower as ContextManager to deal with incomplete LAVA
jobs.
- Use template method to simplify gitlab log sections management
- Fix sections timestamps
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16323>
Gitlab has support for collapsible sections, so it would be good to
create collapsed log sections for the LAVA setup logs. This way, the
Mesa developers to see only the execution of the scripts, instead of
LAVA messages clutter.
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16323>
New tests, dEQP line rasterization test fix that lets Intel pass.
Clears out bogus xfails from 1.3.2.0 uprev on a630, which I suspect were
"we lost the device twice on a full run once, and those fails got pasted
in without checking if it happened a full run again" (since we haven't
seen them in other full run attempts).
Also clears out the a630 vk asan xfails (essentially all tests run) by
turning off leak detection which was just catching leaks in vkcts.
Acked-by: Jason Ekstrand <jason@jlekstrand.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17304>
The job will still fail, but it will at least go through until the end.
Signed-off-by: Martin Roukala (né Peres) <martin.roukala@mupuf.org>
Acked-by: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16841>
This will be used to detect soft resets without aborting the run, but
still make the job fail.
Signed-off-by: Martin Roukala (né Peres) <martin.roukala@mupuf.org>
Acked-by: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16841>
The LLVM backend is not officially supported by the RADV developers,
but it has been useful early during bring-up, or later when users are
experiencing what looks like a compiler bug. It is thus beneficial to
keep it working.
However, maintaining the vkcts expectations for every platform requires
more work and machine time than what we would like to commit to. This
is why we agreed that we would only keep LLVM tested on the latest
family of Radeon GPUs.
Signed-off-by: Martin Roukala (né Peres) <martin.roukala@mupuf.org>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Acked-by: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16841>
Most CI tests are currently running on LLVM 11 (released over 2 years
ago), which predates some of the GPUs we have in CI and prevents
testing RADV's LLVM backend.
LLVM 13 is known to work for RADV, released almost 8 months ago, and
is already available in most distributions. Fedora 36 is even already
on LLVM 14.
So this commit updates x86 testing on llvm 13.
v2:
- store the llvm apt repo key locally (Michel Dänzer)
Signed-off-by: Martin Roukala (né Peres) <martin.roukala@mupuf.org>
Acked-by: Timur Kristóf <timur.kristof@gmail.com>
Acked-by: Daniel Stone <daniels@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17188>
This avoids the risks of concatenating our rulesets missing out on some
file list because of a "never" in the middle.
Reviewed-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Acked-by: Eric Engestrom <eric@engestrom.ch>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17287>
Not only runs less testing when only one driver is impacted, but also
makes sure zink+turnip is turned off when the farm is.
Reviewed-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Acked-by: Eric Engestrom <eric@engestrom.ch>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17287>
This drops the mesa/gallium lists from some build rules, since zink common
rules brings them in already. If we do more driver common rules, we might
end up with those core lists appearing in the yaml multiple times, but
that seems like a small price to pay for not being able to forget some.
Reviewed-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Acked-by: Eric Engestrom <eric@engestrom.ch>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17287>
Eric Engestrom
Guilherme Gallo
David Heidelberg
Emma Anholt
Jesse Natalie
Samuel Pitoiset
Corentin Noël
Martin Roukala (né Peres)