2021-03-22 15:37:27 +00:00
Perfetto Tracing
================
Mesa has experimental support for `Perfetto <https://perfetto.dev> `__ for
GPU performance monitoring. Perfetto supports multiple
`producers <https://perfetto.dev/docs/concepts/service-model> `__ each with
one or more data-sources. Perfetto already provides various producers and
data-sources for things like:
- CPU scheduling events (`` linux.ftrace `` )
- CPU frequency scaling (`` linux.ftrace `` )
- System calls (`` linux.ftrace `` )
- Process memory utilization (`` linux.process_stats `` )
As well as various domain specific producers.
2022-10-19 11:00:58 +01:00
The mesa Perfetto support adds additional producers, to allow for visualizing
2021-03-22 15:37:27 +00:00
GPU performance (frequency, utilization, performance counters, etc) on the
same timeline, to better understand and tune/debug system level performance:
- pps-producer: A systemwide daemon that can collect global performance
counters.
- mesa: Per-process producer within mesa to capture render-stage traces
2022-08-26 19:14:47 +01:00
on the GPU timeline, track events on the CPU timeline, etc.
2021-03-22 15:37:27 +00:00
2021-05-05 22:58:32 +01:00
The exact supported features vary per driver:
.. list-table :: Supported data-sources
:header-rows: 1
* - Driver
- PPS Counters
- Render Stages
2021-07-13 13:13:26 +01:00
* - Freedreno
- `` gpu.counters.msm ``
- `` gpu.renderstages.msm ``
* - Turnip
- `` gpu.counters.msm ``
2022-08-26 19:14:47 +01:00
- `` gpu.renderstages.msm ``
2021-07-13 13:13:26 +01:00
* - Intel
- `` gpu.counters.i915 ``
2023-01-10 18:36:46 +00:00
- `` gpu.renderstages.intel ``
2021-05-06 12:20:01 +01:00
* - Panfrost
2021-07-13 13:13:26 +01:00
- `` gpu.counters.panfrost ``
2021-05-06 12:27:42 +01:00
-
2021-05-05 22:58:32 +01:00
2021-03-22 15:37:27 +00:00
Run
---
2022-10-19 11:00:58 +01:00
To capture a trace with Perfetto you need to take the following steps:
2021-03-22 15:37:27 +00:00
2022-10-19 11:00:58 +01:00
1. Build Perfetto from sources available at `` subprojects/perfetto `` following
2021-03-22 15:37:27 +00:00
`this guide <https://perfetto.dev/docs/quickstart/linux-tracing> `__ .
2022-11-07 14:05:17 +00:00
2. Create a `trace config <https://perfetto.dev/docs/concepts/config> `__ , which is
2021-03-22 15:37:27 +00:00
a json formatted text file with extension `` .cfg `` , or use one of the config
files under the `` src/tool/pps/cfg `` directory. More examples of config files
can be found in `` subprojects/perfetto/test/configs `` .
3. Change directory to `` subprojects/perfetto `` and run a
2022-11-07 14:05:17 +00:00
`convenience script <https://perfetto.dev/docs/quickstart/linux-tracing#capturing-a-trace> `__
to start the tracing service:
2021-03-22 15:37:27 +00:00
.. code-block :: console
cd subprojects/perfetto
CONFIG=<path/to/gpu.cfg> OUT=out/linux_clang_release ./tools/tmux -n
4. Start other producers you may need, e.g. `` pps-producer `` .
5. Start `` perfetto `` under the tmux session initiated in step 3.
6. Once tracing has finished, you can detach from tmux with :kbd: `Ctrl+b` ,
:kbd: `d` , and the convenience script should automatically copy the trace
files into `` $HOME/Downloads `` .
7. Go to `ui.perfetto.dev <https://ui.perfetto.dev> `__ and upload
`` $HOME/Downloads/trace.protobuf `` by clicking on **Open trace file** .
8. Alternatively you can open the trace in `AGI <https://gpuinspector.dev/> `__
(which despite the name can be used to view non-android traces).
2021-10-27 22:15:33 +01:00
To be a bit more explicit, here is a listing of commands reproducing
the steps above :
.. code-block :: console
# Configure Mesa with perfetto
mesa $ meson . build -Dperfetto=true -Dvulkan-drivers=intel,broadcom -Dgallium-drivers=
# Build mesa
mesa $ ninja -C build
# Within the Mesa repo, build perfetto
mesa $ cd subprojects/perfetto
perfetto $ ./tools/install-build-deps
perfetto $ ./tools/gn gen --args='is_debug=false' out/linux
perfetto $ ./tools/ninja -C out/linux
# Start perfetto
perfetto $ CONFIG=../../src/tool/pps/cfg/gpu.cfg OUT=out/linux/ ./tools/tmux -n
# In parallel from the Mesa repo, start the PPS producer
mesa $ ./build/src/tool/pps/pps-producer
# Back in the perfetto tmux, press enter to start the capture
2022-08-26 19:14:47 +01:00
CPU Tracing
~~~~~~~~~~~
Mesa's CPU tracepoints (`` MESA_TRACE_* `` ) use Perfetto track events when
Perfetto is enabled. They use `` mesa.default `` and `` mesa.slow `` categories.
2022-10-20 10:28:39 +01:00
Currently, only EGL and Freedreno have CPU tracepoints.
2022-08-26 19:14:47 +01:00
2021-11-22 10:56:20 +00:00
Vulkan data sources
~~~~~~~~~~~~~~~~~~~
The Vulkan API gives the application control over recording of command
buffers as well as when they are submitted to the hardware. As a
consequence, we need to ensure command buffers are properly
2022-10-19 11:00:58 +01:00
instrumented for the Perfetto driver data sources prior to Perfetto
2021-11-22 10:56:20 +00:00
actually collecting traces.
2022-11-24 08:55:20 +00:00
This can be achieved by setting the :envvar: `MESA_GPU_TRACES`
2021-11-22 10:56:20 +00:00
environment variable before starting a Vulkan application :
.. code-block :: console
2022-11-24 08:30:50 +00:00
MESA_GPU_TRACES=perfetto ./build/my_vulkan_app
2021-11-22 10:56:20 +00:00
2021-05-05 22:58:32 +01:00
Driver Specifics
~~~~~~~~~~~~~~~~
2021-05-06 12:20:01 +01:00
Below is driver specific information/instructions for the PPS producer.
2021-05-05 22:58:32 +01:00
2021-07-13 13:13:26 +01:00
Freedreno / Turnip
^^^^^^^^^^^^^^^^^^
The Freedreno PPS driver needs root access to read system-wide
performance counters, so you can simply run it with sudo:
.. code-block :: console
sudo ./build/src/tool/pps/pps-producer
Intel
^^^^^
The Intel PPS driver needs root access to read system-wide
2022-11-07 12:30:03 +00:00
`RenderBasic <https://www.intel.com/content/www/us/en/develop/documentation/vtune-help/top/reference/gpu-metrics-reference.html> `__
2021-07-13 13:13:26 +01:00
performance counters, so you can simply run it with sudo:
.. code-block :: console
sudo ./build/src/tool/pps/pps-producer
Another option to enable access wide data without root permissions would be running the following:
.. code-block :: console
sudo sysctl dev.i915.perf_stream_paranoid=0
Alternatively using the `` CAP_PERFMON `` permission on the binary should work too.
2021-11-22 14:24:43 +00:00
A particular metric set can also be selected to capture a different
set of HW counters :
.. code-block :: console
INTEL_PERFETTO_METRIC_SET=RasterizerAndPixelBackend ./build/src/tool/pps/pps-producer
2021-11-21 16:23:57 +00:00
Vulkan applications can also be instrumented to be Perfetto producers.
To enable this for given application, set the environment variable as
follow :
.. code-block :: console
PERFETTO_TRACE=1 my_vulkan_app
2021-05-06 12:20:01 +01:00
Panfrost
^^^^^^^^
2021-05-05 22:58:32 +01:00
2021-05-06 12:20:01 +01:00
The Panfrost PPS driver uses unstable ioctls that behave correctly on
kernel version `5.4.23+ <https://lwn.net/Articles/813601/> `__ and
`5.5.7+ <https://lwn.net/Articles/813600/> `__ .
2021-05-06 12:27:42 +01:00
2021-05-06 12:20:01 +01:00
To run the producer, follow these two simple steps:
2021-05-06 12:27:42 +01:00
2021-05-06 12:20:01 +01:00
1. Enable Panfrost unstable ioctls via kernel parameter:
2021-05-06 12:27:42 +01:00
2021-05-06 12:20:01 +01:00
.. code-block :: console
2021-05-06 12:27:42 +01:00
2021-05-06 12:20:01 +01:00
modprobe panfrost unstable_ioctls=1
2021-05-06 12:27:42 +01:00
2021-05-06 12:20:01 +01:00
Alternatively you could add `` panfrost.unstable_ioctls=1 `` to your kernel command line, or `` echo 1 > /sys/module/panfrost/parameters/unstable_ioctls `` .
2021-05-06 12:27:42 +01:00
2021-05-06 12:20:01 +01:00
2. Run the producer:
2021-05-06 12:27:42 +01:00
2021-05-06 12:20:01 +01:00
.. code-block :: console
2021-05-06 12:27:42 +01:00
2021-05-06 12:20:01 +01:00
./build/pps-producer
2021-05-05 22:58:32 +01:00
2021-03-22 15:37:27 +00:00
Troubleshooting
---------------
Tmux
~~~~
If the convenience script `` tools/tmux `` keeps copying artifacts to your
`` SSH_TARGET `` without starting the tmux session, make sure you have `` tmux ``
installed in your system.
.. code-block :: console
apt install tmux
Missing counter names
~~~~~~~~~~~~~~~~~~~~~
If the trace viewer shows a list of counters with a description like
`` gpu_counter(#) `` instead of their proper names, maybe you had a data loss due
to the trace buffer being full and wrapped.
In order to prevent this loss of data you can tweak the trace config file in
two different ways:
- Increase the size of the buffer in use:
.. code-block :: javascript
buffers {
size_kb: 2048,
fill_policy: RING_BUFFER,
}
- Periodically flush the trace buffer into the output file:
.. code-block :: javascript
write_into_file: true
file_write_period_ms: 250
- Discard new traces when the buffer fills:
.. code-block :: javascript
buffers {
size_kb: 2048,
fill_policy: DISCARD,
}