Commit Graph

5616 Commits

Author SHA1 Message Date
Eric Engestrom 2c67457e5e util/list: rename LIST_ENTRY() to list_entry()
This follows the Linux kernel convention, and avoids collision with
macOS header macro.

Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/6751
Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/6840
Cc: mesa-stable
Signed-off-by: Eric Engestrom <eric@igalia.com>
Acked-by: David Heidelberg <david.heidelberg@collabora.com>
Reviewed-by: Yonggang Luo <luoyonggang@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17772>
2022-07-28 10:10:44 +00:00
Georg Lehmann df4b5914cd nir/fold_16bit_tex_image: Default to only_fold_all.
No driver doesn't use this option.

Signed-off-by: Georg Lehmann <dadschoorse@gmail.com>
Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Reviewed-by: Emma Anholt <emma@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17757>
2022-07-27 18:57:12 +00:00
Timur Kristóf 11e51f7211 radv: Use NIR-based NGG lowering with LLVM.
Also cleanup some unused code.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/12792>
2022-07-27 09:06:24 +00:00
Timur Kristóf 45ed22c87e radv: Properly set LDS size for LLVM NGG shaders.
Update the esgs_ring_size after ac_nir_lower_ngg.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/12792>
2022-07-27 09:06:24 +00:00
Timur Kristóf d3dcd294c0 radv/llvm: Remove incorrect hardcoded workgroup size from NGG GS.
Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/12792>
2022-07-27 09:06:24 +00:00
Marek Olšák f3f00f77ad ac/gpu_info: remove amdgpu_gpu_info parameter from ac_query_gpu_info
Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17411>
2022-07-27 05:01:38 +00:00
Marek Olšák 12c5d64fae ac/gpu_info: remove vram_size and gtt_size in favor of *_kb variants
Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17411>
2022-07-27 05:01:38 +00:00
Marek Olšák f218c3d795 ac/gpu_info: rename info fields to num_cu, memory_bus_width, memory_freq_mhz
Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17411>
2022-07-27 05:01:38 +00:00
Benjamin Cheng 1c06565026 radv: expose disjoint image support
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16510>
2022-07-26 19:50:58 +00:00
Benjamin Cheng 27a24cb382 radv: implement disjoint memory for multiplane images
For descriptor binding, we need to allow up to three buffers to be
referenced by any image.

Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16510>
2022-07-26 19:50:58 +00:00
Benjamin Cheng 6a77ecbe6f radv: consolidate descriptor set buffer counts
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16510>
2022-07-26 19:50:57 +00:00
Benjamin Cheng 72625bea5f radv: refactor image binding into struct
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16510>
2022-07-26 19:50:57 +00:00
Georg Lehmann 7fa9b25c2f radv: Fold 16bit tex sources.
Signed-off-by: Georg Lehmann <dadschoorse@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16979>
2022-07-26 16:54:08 +00:00
Konstantin Seurer 2861d1ff8a radv: Fix out of bounds access in RTPSO creation
Signed-off-by: Konstantin Seurer <konstantin.seurer@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17723>
2022-07-24 14:59:44 +00:00
Daniel Schürmann 5b196716cf radv/rt: simplify lower_rt_instructions()
Reviewed-by: Konstantin Seurer <konstantin.seurer@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17301>
2022-07-22 15:31:23 +00:00
Daniel Schürmann f7c318901d radv/rt: fix nir_builder cursor in lower_rt_instructions()
Fixes: 207ce6d658 ('radv: Add helper to inline shaders into the main shader.')
Reviewed-by: Konstantin Seurer <konstantin.seurer@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17301>
2022-07-22 15:31:23 +00:00
Daniel Schürmann c0945f70df radv/rt: Don't load ClosestHit SBT on every hit, but only once after traversal
Quake II RTX:
Totals from 7 (0.01% of 134913) affected shaders:
CodeSize: 217592 -> 215956 (-0.75%)
Instrs: 39468 -> 39341 (-0.32%)
Latency: 761581 -> 746802 (-1.94%)
InvThroughput: 507721 -> 497870 (-1.94%)
Copies: 4621 -> 4585 (-0.78%)
Branches: 1598 -> 1584 (-0.88%)

Reviewed-by: Konstantin Seurer <konstantin.seurer@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17301>
2022-07-22 15:31:23 +00:00
Daniel Schürmann c39cccec9c radv/rt: use stage ID as handle for general and closestHit shaders
This avoids some code duplication and divergence.

Quake II RTX:
Totals from 7 (0.01% of 134913) affected shaders:
CodeSize: 218880 -> 217592 (-0.59%)
Instrs: 39692 -> 39468 (-0.56%)
Latency: 789091 -> 761581 (-3.49%)
InvThroughput: 526061 -> 507721 (-3.49%)
VClause: 1202 -> 1188 (-1.16%)
Copies: 4649 -> 4621 (-0.60%)
Branches: 1605 -> 1598 (-0.44%)

Reviewed-by: Konstantin Seurer <konstantin.seurer@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17301>
2022-07-22 15:31:23 +00:00
Daniel Schürmann 3750663c72 radv/rt: use derefs for the traversal stack
Reviewed-by: Konstantin Seurer <konstantin.seurer@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17301>
2022-07-22 15:31:23 +00:00
Daniel Schürmann 076ea8b35a radv: create RT traversal as separate shader
This will help in future to keep the main shader slim
when we have actual function calls.

Reviewed-by: Konstantin Seurer <konstantin.seurer@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17301>
2022-07-22 15:31:23 +00:00
Daniel Schürmann 8e056af399 radv: Only create noop-fs for graphics pipelines
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Konstantin Seurer <konstantin.seurer@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17301>
2022-07-22 15:31:23 +00:00
Konstantin Seurer 1a95d43e55 radv: Simplify the meta init fail path
Move most of the the cleanup into radv_device_init_meta.

Signed-off-by: Konstantin Seurer <konstantin.seurer@gmail.com>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17652>
2022-07-22 07:51:14 +00:00
Konstantin Seurer b06e5535c0 radv: Use RADV_META_SUSPEND_PREDICATING
Signed-off-by: Konstantin Seurer <konstantin.seurer@gmail.com>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17625>
2022-07-22 07:25:09 +00:00
Konstantin Seurer fdeca2ca0b radv: Add meta saving/restoring for predicating
There are a bunch of places, where this is done manually.

Signed-off-by: Konstantin Seurer <konstantin.seurer@gmail.com>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17625>
2022-07-22 07:25:09 +00:00
Georg Lehmann 87e3277b82 nir: Rewrite and merge 16bit tex folding pass with 16bit image folding pass.
Allow folding constants/undef sources by sharing more code with the image_store
16bit folding pass.

Allow more than one set of sources because RADV wants two, one for
G16 (ddx/ddy) and one for A16 (all other sources).

Allow folding cube sampling destination conversions on radeonsi/radv because
I think the limitation only applies to sources.

Signed-off-by: Georg Lehmann <dadschoorse@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16978>
2022-07-21 19:15:03 +00:00
Konstantin Seurer 7e7e01fe0c radv: Use desc->format
Signed-off-by: Konstantin Seurer <konstantin.seurer@gmail.com>
Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17490>
2022-07-21 12:48:01 +00:00
Konstantin Seurer ae67294479 radv: Remove format desc null checks
Signed-off-by: Konstantin Seurer <konstantin.seurer@gmail.com>
Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17490>
2022-07-21 12:48:01 +00:00
Samuel Pitoiset 85e859ac5f radv: only force 1x sample for Bresenham lines when pipeline draws lines
Otherwise, this would affect non-line draws. While we are at it,
adjust a comment.

Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/6303
Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-By: Mike Blumenkrantz <michael.blumenkrantz@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17657>
2022-07-21 08:05:35 +02:00
Timur Kristóf 4eb7e53963 radv: Enable task shader feature for NV_mesh_shader.
Still hidden behind RADV_PERFTEST=nv_ms but now advertises
task shader support too.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16531>
2022-07-20 19:00:31 +00:00
Timur Kristóf 28db493cdc radv: Copy BO list to ACE internal CS.
This is necessary to make sure the ACE internal cmdbuf
can access the same memory as the GFX cmdbuf.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16531>
2022-07-20 19:00:30 +00:00
Timur Kristóf 5a1cbafd9d radv: Submit internal compute cmdbuf.
Use scheduled dependencies to create two submissions:
first we submit ACE then GFX.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16531>
2022-07-20 19:00:30 +00:00
Timur Kristóf eeb8366bbd radv: Workaround MEC taskmesh dispatch hang when count buffer has zero.
The DISPATCH_TASKMESH_INDIRECT_MULTI_ACE packet has a firmware bug,
it hangs the GPU when the draw count is zero.

This commit adds a workaround sequence using COND_EXEC packets
which make sure that this indirect packet is never executed when
the draw count is zero.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16531>
2022-07-20 19:00:30 +00:00
Timur Kristóf 7b8f087f6b radv: Support task shaders in secondary cmd buffers.
Special consideration is needed to keep ACE and GFX in sync.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16531>
2022-07-20 19:00:30 +00:00
Timur Kristóf 4c6f83006d radv: Synchronization for task shaders.
Add a separate flush_bits field for tracking cache
flushes in the ACE internal cmdbuf.
In barriers and image transitions we add these flush bits to ACE.

Create a semaphore in the upload BO which makes it possible
for ACE to wait for GFX for the purpose of synchronization.
This is necessary when a barrier needs to block task shaders.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16531>
2022-07-20 19:00:30 +00:00
Timur Kristóf 2479b62869 radv: Implement mesh shading draw calls with task shaders.
This implements NV_mesh_shader draw calls with task shaders.

- On the GFX side:
  DISPATCH_TASKMESH_GFX for all draws
- On the ACE side:
  DISPATCH_TASKMESH_DIRECT_ACE for direct draws
  DISPATCH_TASKMESH_INDIRECT_MULTI_ACE for indirect draws

Additionally, the NV_mesh_shader indirect BO layout is
incompatible with AMD HW, so we add a function that copies
that into a suitable layout.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16531>
2022-07-20 19:00:30 +00:00
Timur Kristóf 99179f8490 radv: Introduce radv_before_taskmesh_draw.
This includes additional code that takes care of
handling the internal ACE cmdbuf.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16531>
2022-07-20 19:00:30 +00:00
Timur Kristóf 79287f5ae6 radv: Flush descriptors and push constants for task shaders.
Task shaders are executed on the internal compute cmdbuf, so they
need special consideration.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16531>
2022-07-20 19:00:30 +00:00
Timur Kristóf ef07c3272a radv: Create internal cmdbuf when a graphics pipeline needs compute.
This is mainly going to be used by task shaders, because
the HW implementation mismatches the API:

- In the API, task shaders are considered graphics shaders which
  are part of a graphics pipeline and the draws are submitted to
  a graphics queue.
- The HW requires the driver to dispatch task shaders on
  an async compute queue.

When a pipeline is bound that has a task shader, create a
driver-internal ACE (async compute engine) cmdbuf which
we are going to submit to an ACE queue.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16531>
2022-07-20 19:00:30 +00:00
Timur Kristóf 44a71594c8 radv: Add dispatch_initiator_task field to radv_device.
This is going to be used with task shader dispatches.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16531>
2022-07-20 19:00:30 +00:00
Timur Kristóf 822e370390 radv: Allow reusing pipeline compute state emit functions.
We are going to reuse them outside of radv_pipeline.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16531>
2022-07-20 19:00:30 +00:00
Timur Kristóf bc4012d7ed radv: Minor cleanup of radv_queue_submit_normal.
- Move the uses_perf_counters ternary expression out of
  the loop into a variable called cs_offset.

- Constify cmd_buffer_count.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16531>
2022-07-20 19:00:30 +00:00
Timur Kristóf d718bea92d radv: Move inline push constants to a new function.
This cleans up radv_flush_constants and also
the new function will be reused later.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16531>
2022-07-20 19:00:30 +00:00
Timur Kristóf c4d60df380 radv: Refactor view index emit to use a per-stage function.
Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16531>
2022-07-20 19:00:30 +00:00
Timur Kristóf 99b7521504 radv: Refactor radv_emit_inline_push_consts to work with radeon_cmdbuf.
Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16531>
2022-07-20 19:00:30 +00:00
Timur Kristóf 19fcf5a4cb radv: Refactor radv_emit_descriptor_pointers to work with radeon_cmdbuf.
Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16531>
2022-07-20 19:00:30 +00:00
Timur Kristóf 21e6d596ed radv: Refactor radv_emit_userdata_address to work with radeon_cmdbuf.
Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16531>
2022-07-20 19:00:30 +00:00
Timur Kristóf aadb3d69d1 radv: Refactor some CP DMA functions to work with radeon_cmdbuf.
Allow emitting these packets without a radv_cmd_buffer object.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16531>
2022-07-20 19:00:30 +00:00
Timur Kristóf e10cbb5d98 radv: Refactor predication for compute queues.
Initialize the inverted predication VA only when it is used
for the first time.

This is needed to get conditional rendering work correctly with
task shaders because the internal compute cmdbuf may not exist
yet when conditional rendering starts.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16531>
2022-07-20 19:00:30 +00:00
Georg Lehmann 333f056edf radv, aco: Don't lower 16bit isign.
Signed-off-by: Georg Lehmann <dadschoorse@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17440>
2022-07-20 14:31:15 +00:00
Georg Lehmann 4be6164ac9 radv: Remove ineg from lower_bit_size_callback callback.
It's always lowered to isub.

Signed-off-by: Georg Lehmann <dadschoorse@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17440>
2022-07-20 14:31:15 +00:00