Commit Graph

8 Commits

Author SHA1 Message Date
Guilherme Gallo 2c51b7a9c9 ci/lava: Detect R8152 issues preemptively and retry
Implement a log-based retry hint for R8152 issue described in #6681,
which is based on detecting these two consecutive lines:

```
r8152 <USB> eth0: Tx status -71
nfs: server <IP> not responding, still trying
```

Where <IP> and <USB> could be any IP and USB addresses, respectfully.

This commit is a temporary fix since it requires a section-aware log
follower, implemented in !16323. When the cited MR is merged, one will
make a proper fix on top of that.

Closes: #6681

Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17389>
2022-07-08 12:26:05 +00:00
Guilherme Gallo 45a4b01427 ci/lava: Split lava_log into modules
This script is getting too big, it been hard to extend it.

Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17389>
2022-07-08 12:26:05 +00:00
Guilherme Gallo 20827dfa9b ci/lava: Update license header
Use SPDX to indicate the license.
Update authors of lava_job_submitter.py

Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16323>
2022-07-07 00:28:53 +00:00
Guilherme Gallo 6ba2b33a8c ci/lava: Flexibilize section marker regexes
In some jobs, such as
https://gitlab.freedesktop.org/gallo/mesa/-/jobs/24904100, the kmsg is
interleaved with stderr/stdout in serial console, making it difficult to
confidently find the log messages to detect when the DUT is booting,
when the DUT is running etc.

Luckily, LAVA sends redundant messages about their signals. We can use
them to mitigate the chance of missing an interleaved message by being
more open to different messages, using the regex on both `debug` and
`target` LAVA log levels.

Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16323>
2022-07-07 00:28:53 +00:00
Guilherme Gallo 29af421272 ci/lava: Don't print LAVA debug messages
Remove debug messages from the output in order to unclutter the log a
little more for the developers.

Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16323>
2022-07-07 00:28:53 +00:00
Guilherme Gallo aa26a6ab72 ci/lava: Follow job execution via LogFollower
Now LogFollower is used to deal with the LAVA logs.

Moreover, this commit adds timeouts per Gitlab section, if a section
takes longer than expected, cancel the job and retry again.

Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16323>
2022-07-07 00:28:53 +00:00
Guilherme Gallo 2569d7d7df ci/lava: Create LogFollower and move logging methods
- Create LogFollower to capture LAVA log and process it adding some
- GitlabSection and color treatment to it
- Break logs further, make new gitlab sections between testcases
- Implement LogFollower as ContextManager to deal with incomplete LAVA
  jobs.
- Use template method to simplify gitlab log sections management
- Fix sections timestamps

Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16323>
2022-07-07 00:28:53 +00:00
Guilherme Gallo c86ba3640f ci/lava: Create Gitlab log sections handler
Gitlab has support for collapsible sections, so it would be good to
create collapsed log sections for the LAVA setup logs. This way, the
Mesa developers to see only the execution of the scripts, instead of
LAVA messages clutter.

Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16323>
2022-07-07 00:28:53 +00:00