mesa/.gitlab-ci/container/create-rootfs.sh

294 lines
6.9 KiB
Bash
Raw Permalink Normal View History

#!/bin/bash
set -ex
if [ $DEBIAN_ARCH = arm64 ]; then
ci: Build skqp on ARM64 images This commit makes `kernel+rootfs_arm64` job build and install skqp on ARM64 devices rootfs. Skia repository has a tool to prepare skqp models located at `tools/skqp/cut-release`, which get files from [Skia Gold](https://skia.org/docs/dev/testing/skiagold/), generate files.checksum, rendertests.txt and unittests.txt. One gives a range of commits to let `cut-release` find the right resources to prepare skqp for the user. However, it is failing, since it fails when trying to get image packages from a range of commits via HTTPS from the host https://public-gold.skia.org but it responds with error 404 every time. I tried a range a thousand of commits, yet it still does not give results. The workaround employed was to recover the most recent `files.checksum` and `rendertests.txt` files from the git history and generate `unittests.txt` from `list_gpu_unit_tests` binary. `skqp` runs two lists of tests, `rendertests.txt` and `unittests.txt`. Both must be located inside the `skqp` assets folder. The first list uses GL and GLES to test rendering scenarios. The second runs some unit tests that do not render an image per se. In order to make the first `a630_skqp` to be green, the crashing tests were removed from the test lists and the expectations of the failing ones were updated. It is worth noting that `rendertests.txt` can bring some detail about each test expectation, so each test can have a max pixel error count, to tell `skqp` that it is OK to have at most that number of errors for that test. See also: https://github.com/google/skia/blob/main/tools/skqp/README_ALGORITHM.md As each render backend has a different error count, two different `rendertests.txt` files were created, `src/freedreno/ci/freedreno-a630-skqp-gl_rendertests.txt`, `src/freedreno/ci/freedreno-a630-skqp-gles_rendertests.txt` and , which one refers to GL and GLES tests respectfully. The unit tests file for a630 is located at `src/freedreno/ci/freedreno-a630-skqp_unittests.txt` ``` aaclip domain formats highcontrastfilter rectangle_texture yuv_make_color_space ``` ``` ProcessorOptimizationValidationTest VkProtectedContext_CreateNonprotectedContext VkYCbcrSampler_DrawImageWithYcbcrSampler VkYCbcrSampler_NoYcbcrSurface ``` Each test was updated with the max_error count equal to the first run result. ``` analytic_antialias_inverse async_rescale_and_read_dog_down async_rescale_and_read_dog_up async_rescale_and_read_rose async_rescale_and_read_text_down async_rescale_and_read_text_up async_rescale_and_read_text_up_large async_rescale_and_read_yuv420_rose complexclip2_path_bw encode-platform imageblur_large lcdtextsize onebadarc onefailarc scale-pixels surfaceprops textfilter_color textfilter_image ``` Considering all the following tests results as wrong. ``` async_rescale_and_read_no_bleed backdrop_imagefilter_croprect_persp complexclip2 imageblurrepeatmode mixerCF overdrawcolorfilter patch_alpha patch_primitive rrect_clip_bw scaledemoji_rendering yuv_splitter ``` v2: a) add link to HTML report on job log b) remove extraneous spaces diff c) remove unnecessary conditions from build-skqp.sh d) use fixed skqp source commit SHA v3: a) Use only main skia repository to fetch models and build skqp b) Use list_gpu_unit_tests binary to create a base unittests.txt file c) Remove crashing tests d) Set failing tests expectations for the first skqp run v4: a) Remove clang dependency b) Separate each skqp backend result into its folder c) Regroup a630_skqp in one job v5: a) Separate tests files per driver Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/5580 Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com> Reviewed-by: Emma Anholt <emma@anholt.net> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/14146>
2021-12-07 14:13:00 +00:00
ARCH_PACKAGES="firmware-qcom-media
firmware-linux-nonfree
ci: Build skqp on ARM64 images This commit makes `kernel+rootfs_arm64` job build and install skqp on ARM64 devices rootfs. Skia repository has a tool to prepare skqp models located at `tools/skqp/cut-release`, which get files from [Skia Gold](https://skia.org/docs/dev/testing/skiagold/), generate files.checksum, rendertests.txt and unittests.txt. One gives a range of commits to let `cut-release` find the right resources to prepare skqp for the user. However, it is failing, since it fails when trying to get image packages from a range of commits via HTTPS from the host https://public-gold.skia.org but it responds with error 404 every time. I tried a range a thousand of commits, yet it still does not give results. The workaround employed was to recover the most recent `files.checksum` and `rendertests.txt` files from the git history and generate `unittests.txt` from `list_gpu_unit_tests` binary. `skqp` runs two lists of tests, `rendertests.txt` and `unittests.txt`. Both must be located inside the `skqp` assets folder. The first list uses GL and GLES to test rendering scenarios. The second runs some unit tests that do not render an image per se. In order to make the first `a630_skqp` to be green, the crashing tests were removed from the test lists and the expectations of the failing ones were updated. It is worth noting that `rendertests.txt` can bring some detail about each test expectation, so each test can have a max pixel error count, to tell `skqp` that it is OK to have at most that number of errors for that test. See also: https://github.com/google/skia/blob/main/tools/skqp/README_ALGORITHM.md As each render backend has a different error count, two different `rendertests.txt` files were created, `src/freedreno/ci/freedreno-a630-skqp-gl_rendertests.txt`, `src/freedreno/ci/freedreno-a630-skqp-gles_rendertests.txt` and , which one refers to GL and GLES tests respectfully. The unit tests file for a630 is located at `src/freedreno/ci/freedreno-a630-skqp_unittests.txt` ``` aaclip domain formats highcontrastfilter rectangle_texture yuv_make_color_space ``` ``` ProcessorOptimizationValidationTest VkProtectedContext_CreateNonprotectedContext VkYCbcrSampler_DrawImageWithYcbcrSampler VkYCbcrSampler_NoYcbcrSurface ``` Each test was updated with the max_error count equal to the first run result. ``` analytic_antialias_inverse async_rescale_and_read_dog_down async_rescale_and_read_dog_up async_rescale_and_read_rose async_rescale_and_read_text_down async_rescale_and_read_text_up async_rescale_and_read_text_up_large async_rescale_and_read_yuv420_rose complexclip2_path_bw encode-platform imageblur_large lcdtextsize onebadarc onefailarc scale-pixels surfaceprops textfilter_color textfilter_image ``` Considering all the following tests results as wrong. ``` async_rescale_and_read_no_bleed backdrop_imagefilter_croprect_persp complexclip2 imageblurrepeatmode mixerCF overdrawcolorfilter patch_alpha patch_primitive rrect_clip_bw scaledemoji_rendering yuv_splitter ``` v2: a) add link to HTML report on job log b) remove extraneous spaces diff c) remove unnecessary conditions from build-skqp.sh d) use fixed skqp source commit SHA v3: a) Use only main skia repository to fetch models and build skqp b) Use list_gpu_unit_tests binary to create a base unittests.txt file c) Remove crashing tests d) Set failing tests expectations for the first skqp run v4: a) Remove clang dependency b) Separate each skqp backend result into its folder c) Regroup a630_skqp in one job v5: a) Separate tests files per driver Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/5580 Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com> Reviewed-by: Emma Anholt <emma@anholt.net> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/14146>
2021-12-07 14:13:00 +00:00
libfontconfig1
libgl1
libglu1-mesa
libvulkan-dev
ci: Build skqp on ARM64 images This commit makes `kernel+rootfs_arm64` job build and install skqp on ARM64 devices rootfs. Skia repository has a tool to prepare skqp models located at `tools/skqp/cut-release`, which get files from [Skia Gold](https://skia.org/docs/dev/testing/skiagold/), generate files.checksum, rendertests.txt and unittests.txt. One gives a range of commits to let `cut-release` find the right resources to prepare skqp for the user. However, it is failing, since it fails when trying to get image packages from a range of commits via HTTPS from the host https://public-gold.skia.org but it responds with error 404 every time. I tried a range a thousand of commits, yet it still does not give results. The workaround employed was to recover the most recent `files.checksum` and `rendertests.txt` files from the git history and generate `unittests.txt` from `list_gpu_unit_tests` binary. `skqp` runs two lists of tests, `rendertests.txt` and `unittests.txt`. Both must be located inside the `skqp` assets folder. The first list uses GL and GLES to test rendering scenarios. The second runs some unit tests that do not render an image per se. In order to make the first `a630_skqp` to be green, the crashing tests were removed from the test lists and the expectations of the failing ones were updated. It is worth noting that `rendertests.txt` can bring some detail about each test expectation, so each test can have a max pixel error count, to tell `skqp` that it is OK to have at most that number of errors for that test. See also: https://github.com/google/skia/blob/main/tools/skqp/README_ALGORITHM.md As each render backend has a different error count, two different `rendertests.txt` files were created, `src/freedreno/ci/freedreno-a630-skqp-gl_rendertests.txt`, `src/freedreno/ci/freedreno-a630-skqp-gles_rendertests.txt` and , which one refers to GL and GLES tests respectfully. The unit tests file for a630 is located at `src/freedreno/ci/freedreno-a630-skqp_unittests.txt` ``` aaclip domain formats highcontrastfilter rectangle_texture yuv_make_color_space ``` ``` ProcessorOptimizationValidationTest VkProtectedContext_CreateNonprotectedContext VkYCbcrSampler_DrawImageWithYcbcrSampler VkYCbcrSampler_NoYcbcrSurface ``` Each test was updated with the max_error count equal to the first run result. ``` analytic_antialias_inverse async_rescale_and_read_dog_down async_rescale_and_read_dog_up async_rescale_and_read_rose async_rescale_and_read_text_down async_rescale_and_read_text_up async_rescale_and_read_text_up_large async_rescale_and_read_yuv420_rose complexclip2_path_bw encode-platform imageblur_large lcdtextsize onebadarc onefailarc scale-pixels surfaceprops textfilter_color textfilter_image ``` Considering all the following tests results as wrong. ``` async_rescale_and_read_no_bleed backdrop_imagefilter_croprect_persp complexclip2 imageblurrepeatmode mixerCF overdrawcolorfilter patch_alpha patch_primitive rrect_clip_bw scaledemoji_rendering yuv_splitter ``` v2: a) add link to HTML report on job log b) remove extraneous spaces diff c) remove unnecessary conditions from build-skqp.sh d) use fixed skqp source commit SHA v3: a) Use only main skia repository to fetch models and build skqp b) Use list_gpu_unit_tests binary to create a base unittests.txt file c) Remove crashing tests d) Set failing tests expectations for the first skqp run v4: a) Remove clang dependency b) Separate each skqp backend result into its folder c) Regroup a630_skqp in one job v5: a) Separate tests files per driver Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/5580 Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com> Reviewed-by: Emma Anholt <emma@anholt.net> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/14146>
2021-12-07 14:13:00 +00:00
"
elif [ $DEBIAN_ARCH = amd64 ]; then
# Add llvm 13 to the build image
apt-get -y install --no-install-recommends wget gnupg2 software-properties-common
apt-key add /llvm-snapshot.gpg.key
add-apt-repository "deb https://apt.llvm.org/bullseye/ llvm-toolchain-bullseye-13 main"
ARCH_PACKAGES="firmware-amd-graphics
inetutils-syslogd
iptables
libcap2
libfontconfig1
libelf1
libfdt1
libgl1
libglu1-mesa
libllvm13
libllvm11
libva2
libva-drm2
libvulkan-dev
socat
spirv-tools
sysvinit-core
"
fi
INSTALL_CI_FAIRY_PACKAGES="git
python3-dev
python3-pip
python3-setuptools
python3-wheel
"
apt-get update
apt-get -y install --no-install-recommends \
$ARCH_PACKAGES \
$INSTALL_CI_FAIRY_PACKAGES \
$EXTRA_LOCAL_PACKAGES \
bash \
ca-certificates \
firmware-realtek \
initramfs-tools \
libasan6 \
libexpat1 \
libpng16-16 \
libpython3.9 \
libsensors5 \
libvulkan1 \
libwaffle-1-0 \
libx11-6 \
libx11-xcb1 \
libxcb-dri2-0 \
libxcb-dri3-0 \
libxcb-glx0 \
libxcb-present0 \
libxcb-randr0 \
libxcb-shm0 \
libxcb-sync1 \
libxcb-xfixes0 \
libxdamage1 \
libxext6 \
libxfixes3 \
libxkbcommon0 \
libxrender1 \
libxshmfence1 \
libxxf86vm1 \
netcat-openbsd \
python3 \
python3-lxml \
python3-mako \
python3-numpy \
python3-packaging \
python3-pil \
python3-renderdoc \
python3-requests \
python3-simplejson \
python3-yaml \
sntp \
strace \
waffle-utils \
wget \
xinit \
xserver-xorg-core
# Needed for ci-fairy, this revision is able to upload files to
# MinIO and doesn't depend on git
pip3 install git+http://gitlab.freedesktop.org/freedesktop/ci-templates@34f4ade99434043f88e164933f570301fd18b125
apt-get purge -y \
$INSTALL_CI_FAIRY_PACKAGES
passwd root -d
chsh -s /bin/sh
cat > /init <<EOF
#!/bin/sh
export PS1=lava-shell:
exec sh
EOF
chmod +x /init
#######################################################################
# Strip the image to a small minimal system without removing the debian
# toolchain.
# Copy timezone file and remove tzdata package
rm -rf /etc/localtime
cp /usr/share/zoneinfo/Etc/UTC /etc/localtime
UNNEEDED_PACKAGES="
libfdisk1
"
export DEBIAN_FRONTEND=noninteractive
# Removing unused packages
for PACKAGE in ${UNNEEDED_PACKAGES}
do
echo ${PACKAGE}
if ! apt-get remove --purge --yes "${PACKAGE}"
then
echo "WARNING: ${PACKAGE} isn't installed"
fi
done
apt-get autoremove --yes || true
# Dropping logs
rm -rf /var/log/*
# Dropping documentation, localization, i18n files, etc
rm -rf /usr/share/doc/*
rm -rf /usr/share/locale/*
rm -rf /usr/share/X11/locale/*
rm -rf /usr/share/man
rm -rf /usr/share/i18n/*
rm -rf /usr/share/info/*
rm -rf /usr/share/lintian/*
rm -rf /usr/share/common-licenses/*
rm -rf /usr/share/mime/*
# Dropping reportbug scripts
rm -rf /usr/share/bug
# Drop udev hwdb not required on a stripped system
rm -rf /lib/udev/hwdb.bin /lib/udev/hwdb.d/*
# Drop all gconv conversions && binaries
rm -rf usr/bin/iconv
rm -rf usr/sbin/iconvconfig
rm -rf usr/lib/*/gconv/
# Remove libusb database
rm -rf usr/sbin/update-usbids
rm -rf var/lib/usbutils/usb.ids
rm -rf usr/share/misc/usb.ids
rm -rf /root/.pip
#######################################################################
# Crush into a minimal production image to be deployed via some type of image
# updating system.
# IMPORTANT: The Debian system is not longer functional at this point,
# for example, apt and dpkg will stop working
UNNEEDED_PACKAGES="apt libapt-pkg6.0 "\
"ncurses-bin ncurses-base libncursesw6 libncurses6 "\
"perl-base "\
"debconf libdebconfclient0 "\
"e2fsprogs e2fslibs libfdisk1 "\
"insserv "\
"udev "\
"init-system-helpers "\
"cpio "\
"passwd "\
"libsemanage1 libsemanage-common "\
"libsepol1 "\
"gpgv "\
"hostname "\
"adduser "\
"debian-archive-keyring "\
"libegl1-mesa-dev "\
"libegl-mesa0 "\
"libgl1-mesa-dev "\
"libgl1-mesa-dri "\
"libglapi-mesa "\
"libgles2-mesa-dev "\
"libglx-mesa0 "\
"mesa-common-dev "\
"gnupg2 "\
"software-properties-common " \
# Removing unneeded packages
for PACKAGE in ${UNNEEDED_PACKAGES}
do
echo "Forcing removal of ${PACKAGE}"
if ! dpkg --purge --force-remove-essential --force-depends "${PACKAGE}"
then
echo "WARNING: ${PACKAGE} isn't installed"
fi
done
# Show what's left package-wise before dropping dpkg itself
COLUMNS=300 dpkg-query -W --showformat='${Installed-Size;10}\t${Package}\n' | sort -k1,1n
# Drop dpkg
dpkg --purge --force-remove-essential --force-depends dpkg
# No apt or dpkg, no need for its configuration archives
rm -rf etc/apt
rm -rf etc/dpkg
# Drop directories not part of ostree
# Note that /var needs to exist as ostree bind mounts the deployment /var over
# it
rm -rf var/* opt srv share
# ca-certificates are in /etc drop the source
rm -rf usr/share/ca-certificates
# No need for completions
rm -rf usr/share/bash-completion
# No zsh, no need for comletions
rm -rf usr/share/zsh/vendor-completions
# drop gcc python helpers
rm -rf usr/share/gcc
# Drop sysvinit leftovers
rm -rf etc/init.d
rm -rf etc/rc[0-6S].d
# Drop upstart helpers
rm -rf etc/init
# Various xtables helpers
rm -rf usr/lib/xtables
# Drop all locales
# TODO: only remaining locale is actually "C". Should we really remove it?
rm -rf usr/lib/locale/*
# partition helpers
rm -rf usr/sbin/*fdisk
# local compiler
rm -rf usr/bin/localedef
# Systemd dns resolver
find usr etc -name '*systemd-resolve*' -prune -exec rm -r {} \;
# Systemd network configuration
find usr etc -name '*networkd*' -prune -exec rm -r {} \;
# systemd ntp client
find usr etc -name '*timesyncd*' -prune -exec rm -r {} \;
# systemd hw database manager
find usr etc -name '*systemd-hwdb*' -prune -exec rm -r {} \;
# No need for fuse
find usr etc -name '*fuse*' -prune -exec rm -r {} \;
# lsb init function leftovers
rm -rf usr/lib/lsb
# Only needed when adding libraries
rm -rf usr/sbin/ldconfig*
# Games, unused
rmdir usr/games
# Remove pam module to authenticate against a DB
# plus libdb-5.3.so that is only used by this pam module
rm -rf usr/lib/*/security/pam_userdb.so
rm -rf usr/lib/*/libdb-5.3.so
# remove NSS support for nis, nisplus and hesiod
rm -rf usr/lib/*/libnss_hesiod*
rm -rf usr/lib/*/libnss_nis*