215 lines
8.1 KiB
ReStructuredText
215 lines
8.1 KiB
ReStructuredText
Virtio-GPU Venus
|
|
================
|
|
|
|
Venus is a Virtio-GPU protocol for Vulkan command serialization. The protocol
|
|
definition and codegen are hosted at `venus-protocol
|
|
<https://gitlab.freedesktop.org/olv/venus-protocol>`__. The renderer is
|
|
hosted at `virglrenderer
|
|
<https://gitlab.freedesktop.org/virgl/virglrenderer>`__.
|
|
|
|
The protocol is still under development. This driver and the renderer are
|
|
both considered experimental.
|
|
|
|
Requirements
|
|
------------
|
|
|
|
The Venus renderer requires
|
|
|
|
- Vulkan 1.1
|
|
- ``VK_EXT_external_memory_dma_buf``
|
|
- ``VK_EXT_image_drm_format_modifier``
|
|
- ``VK_EXT_queue_family_foreign``
|
|
|
|
from the host driver. However, it violates the spec in some places currently
|
|
and also relies on implementation-defined behaviors in others. It is not
|
|
expected to work on all drivers meeting the requirements. It has only been
|
|
tested with
|
|
|
|
- ANV 21.1 or later
|
|
- RADV 21.1 or later (the host kernel must have
|
|
``CONFIG_TRANSPARENT_HUGEPAGE`` disabled because of this `KVM issue
|
|
<https://github.com/google/security-research/security/advisories/GHSA-7wq5-phmq-m584>`__)
|
|
|
|
The Venus driver requires supports for
|
|
|
|
- ``VIRTGPU_PARAM_RESOURCE_BLOB``
|
|
- ``VIRTGPU_PARAM_HOST_VISIBLE``
|
|
- ``VIRTGPU_PARAM_CROSS_DEVICE``
|
|
- ``VIRTGPU_PARAM_CONTEXT_INIT``
|
|
|
|
from the virtio-gpu kernel driver, unless vtest is used. Currently, this
|
|
means the `context-init
|
|
<https://gitlab.freedesktop.org/virgl/drm-misc-next/-/tree/context-init>`__
|
|
kernel branch paired with `crosvm
|
|
<https://chromium.googlesource.com/chromiumos/platform/crosvm>`__.
|
|
|
|
vtest
|
|
-----
|
|
|
|
The simplest way to test Venus is to use virglrenderer's vtest server. To
|
|
build virglrenderer with Venus support and to start the vtest server,
|
|
|
|
.. code-block:: console
|
|
|
|
$ git clone https://gitlab.freedesktop.org/virgl/virglrenderer.git
|
|
$ cd virglrenderer
|
|
$ meson out -Dvenus-experimental=true
|
|
$ ninja -C out
|
|
$ ./out/vtest/virgl_test_server --venus
|
|
|
|
In another shell,
|
|
|
|
.. code-block:: console
|
|
|
|
$ export VK_ICD_FILENAMES=<path-to-virtio_icd.x86_64.json>
|
|
$ export VN_DEBUG=vtest
|
|
$ vulkaninfo
|
|
$ vkcube
|
|
|
|
If the host driver of the system is not new enough, it is a good idea to build
|
|
the host driver as well when building the Venus driver. Just remember to set
|
|
:envvar:`VK_ICD_FILENAMES` when starting the vtest server so that the vtest
|
|
server finds the locally built host driver.
|
|
|
|
Virtio-GPU
|
|
----------
|
|
|
|
Because the driver requires ``VIRTGPU_PARAM_CONTEXT_INIT`` from the virtio-gpu
|
|
kernel driver, one must make sure the guest kernel includes the changes from
|
|
the `context-init
|
|
<https://gitlab.freedesktop.org/virgl/drm-misc-next/-/tree/context-init>`__
|
|
branch.
|
|
|
|
To build crosvm,
|
|
|
|
.. code-block:: console
|
|
|
|
$ mkdir crosvm
|
|
$ cd crosvm
|
|
$ wget https://storage.googleapis.com/git-repo-downloads/repo
|
|
$ chmod +x repo
|
|
$ ./repo init -g crosvm -u https://chromium.googlesource.com/chromiumos/manifest.git
|
|
$ ./repo sync
|
|
$ cd src/platform/crosvm
|
|
$ RUSTFLAGS="-L<path-to-virglrenderer>/out/src" cargo build \
|
|
--features "x virgl_renderer virgl_renderer_next default-no-sandbox"
|
|
|
|
Note that crosvm must be built with ``default-no-sandbox`` or started with
|
|
``--disable-sandbox`` in this setup.
|
|
|
|
This is how one might want to start crosvm
|
|
|
|
.. code-block:: console
|
|
|
|
$ sudo LD_LIBRARY_PATH=<...> VK_ICD_FILENAMES=<...> ./target/debug/crosvm run \
|
|
--gpu vulkan=true \
|
|
--display-window-keyboard \
|
|
--display-window-mouse \
|
|
--host_ip 192.168.0.1 \
|
|
--netmask 255.255.255.0 \
|
|
--mac 12:34:56:78:9a:bc \
|
|
--rwdisk disk.qcow2 \
|
|
-p root=/dev/vda1 \
|
|
<path-to-bzImage>
|
|
|
|
assuming a working system is installed to partition 1 of ``disk.qcow2``.
|
|
``sudo`` or ``CAP_NET_ADMIN`` is needed to set up the TAP network device.
|
|
|
|
Virtio-GPU and Virtio-WL
|
|
------------------------
|
|
|
|
In this setup, the guest userspace uses Xwayland and a special Wayland
|
|
compositor to connect guest X11/Wayland clients to the host Wayland
|
|
compositor, using Virtio-WL as the transport. This setup is more tedious, but
|
|
that should hopefully change over time.
|
|
|
|
For now, the guest kernel must be built from the ``chromeos-5.10`` branch of
|
|
the `Chrome OS kernel
|
|
<https://chromium.googlesource.com/chromiumos/third_party/kernel>`__. crosvm
|
|
should also be built with ``wl-dmabuf`` feature rather than ``x`` feature.
|
|
|
|
To build minigbm and to enable minigbm support in virglrenderer,
|
|
|
|
.. code-block:: console
|
|
|
|
$ git clone https://chromium.googlesource.com/chromiumos/platform/minigbm
|
|
$ cd minigbm
|
|
$ CFLAGS=-DDRV_<I915-or-your-driver> OUT=out DESTDIR=out/install make install
|
|
$ cd ../virglrenderer
|
|
$ meson configure out -Dminigbm_allocation=true
|
|
$ ninja -C out
|
|
|
|
Make sure a host Wayland compositor is running. Replace
|
|
``--display-window-keyboard --display-window-mouse`` by
|
|
``--wayland-sock=<path-to-wayland-socket>`` when starting crosvm.
|
|
|
|
In the guest, build and start sommelier, the special Wayland compositor,
|
|
|
|
.. code-block:: console
|
|
|
|
$ git clone https://chromium.googlesource.com/chromiumos/platform2
|
|
$ cd platform2/vm_tools/sommelier
|
|
$ meson out -Dxwayland_path=/usr/bin/Xwayland -Dxwayland_gl_driver_path=/usr/lib/dri
|
|
$ ninja -C out
|
|
$ sudo chmod 777 /dev/wl0
|
|
$ ./out/sommelier -X --glamor
|
|
--xwayland-gl-driver-path=<path-to-locally-built-gl-driver> \
|
|
sleep infinity
|
|
|
|
sommelier requires ``xdg-shell-unstable-v6`` rather than the stable
|
|
``xdg-shell`` from the host compositor. One must make sure the host
|
|
compositor still supports the older extension.
|
|
|
|
Optional Requirements
|
|
---------------------
|
|
|
|
When virglrenderer is built with ``-Dminigbm_allocation=true``, the Venus
|
|
renderer might need to import GBM BOs. The imports will fail unless the host
|
|
driver supports the formats, especially multi-planar ones, and the DRM format
|
|
modifiers of the GBM BOs.
|
|
|
|
In the future, if virglrenderer's ``virgl_renderer_export_fence`` is
|
|
supported, the Venus renderer will require ``VK_KHR_external_fence_fd`` with
|
|
``VK_EXTERNAL_FENCE_HANDLE_TYPE_SYNC_FD_BIT`` from the host driver.
|
|
|
|
A WSI image of the Venus driver is an external image to the host driver. When
|
|
the WSI image is transitioned from ``VK_IMAGE_LAYOUT_UNDEFINED`` after image
|
|
acquisition, the Venus driver does not request the Venus renderer to perform
|
|
an ownership transfer on the external image. It is unclear if the ownership
|
|
transfer is required or not. A specification issue has been filed for
|
|
clarifications. See the comment before ``vn_cmd_fix_image_memory_barrier``
|
|
for more details.
|
|
|
|
VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT
|
|
-----------------------------------
|
|
|
|
The Venus renderer makes assumptions about ``VkDeviceMemory`` that has
|
|
``VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT``. The assumptions are illegal and rely
|
|
on the current behaviors of the host drivers. It should be possible to remove
|
|
some of the assumptions and incrementally improve compatibilities with more
|
|
host drivers by imposing platform-specific requirements. But the long-term
|
|
plan is to create a new Vulkan extension for the host drivers to address this
|
|
specific use case.
|
|
|
|
The Venus renderer assumes a device memory that has
|
|
``VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT`` can be exported as a mmapable dma-buf
|
|
(in the future, the plan is to export the device memory as an opaque fd). It
|
|
chains ``VkExportMemoryAllocateInfo`` to ``VkMemoryAllocateInfo`` without
|
|
checking if the host driver can export the device memory.
|
|
|
|
The dma-buf is mapped (in the future, the plan is to import the opaque fd and
|
|
call ``vkMapMemory``) but the mapping is not accessed. Instead, the mapping
|
|
is passed to ``KVM_SET_USER_MEMORY_REGION``. The hypervisor, host KVM, and
|
|
the guest kernel work together to set up a write-back or write-combined guest
|
|
mapping (see ``virtio_gpu_vram_mmap`` of the virtio-gpu kernel driver). CPU
|
|
accesses to the device memory are via the guest mapping, and are assumed to be
|
|
coherent when the device memory also has
|
|
``VK_MEMORY_PROPERTY_HOST_COHERENT_BIT``.
|
|
|
|
When a ``VkImage`` or a ``VkBuffer`` is created, the Venus renderer does not
|
|
know if the image or the buffer will be bound to such a device memory or not.
|
|
As a result, the Venus renderer unconditionally chains
|
|
``VkExternalMemoryImageCreateInfo`` to ``VkImageCreateInfo`` and chains
|
|
``VkExternalMemoryBufferCreateInfo`` to ``VkBufferCreateInfo`` without
|
|
checking for the host driver support.
|