outputs_written is uint64_t, should count max reg number
by util_last_bit64(). Otherwise the following access will
overflow the allocated array with a smaller size.
cc: mesa-stable
Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Signed-off-by: Qiang Yu <yuq825@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/15765>
Like other optimizations, breaking this pass may not affect functional
correctness. It's also dead simple to unit test the pass, so we have no excuse
not to. Add unit tests for the functionality we currently support, since we just
extended it and want to make sure everything still works.
This includes tests for use of modifiers to get more small constants. There are
lots of subtle gotchas there, so let's add lots of unit tests to make sure we
got it right.
Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16862>
We need to distinguish signed integer instructions from unsigned integer
instructions, to distinguish sign-extension and zero-extension of sources.
Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16862>
Some blitter operations, like clear, doesn't require to save all the
states.
This is particular important because, besides saving time, the blitter
operation restores the state required for the operation, and if we saved
more states than those, these ones won't be restored and will be leak.
So this also fixes some leaks when running CTS tests.
CC: mesa-stable
Signed-off-by: Juan A. Suarez Romero <jasuarez@igalia.com>
Reviewed-by: Iago Toral Quiroga <itoral@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16837>
Call proper pipe reference function to initialize the reference
counting.
Signed-off-by: Juan A. Suarez Romero <jasuarez@igalia.com>
Reviewed-by: Iago Toral Quiroga <itoral@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16837>
CSEL executes on the conversion unit (CVT), while MUX executes on the special
function unit (SFU). Throughput on CVT is 4x higher than SFU, so this is
(almost) always an optimization.
The "real" MUX is still used for unusual cases, like 8-bit and bitselect.
Note that it's easier for us to use MUX everywhere for the IR. This is an easy
fixup to get better codegen on Valhall without touching the core Bifrost code.
shader-db is a bit of a toss up: register pressure and instruction count are
hurt in some cases due to restrictions on FAU access. In particular, a shader
that muxes between two uniforms needs an extra move due to extra constant
(zero). However, in terms of throughput this is still a win: 2 CVT instructions
(MOV + CSEL) have 2x throughput to 1 SFU instruction (MUX). The MOV has
opportunities for CSE, but that can hurt pressure in turn. Overall, cycles are
helped substantially.
total instructions in shared programs: 2728438 -> 2731597 (0.12%)
instructions in affected programs: 414391 -> 417550 (0.76%)
helped: 87
HURT: 1063
helped stats (abs) min: 1.0 max: 6.0 x̄: 5.17 x̃: 6
helped stats (rel) min: 0.19% max: 15.79% x̄: 4.12% x̃: 4.11%
HURT stats (abs) min: 1.0 max: 56.0 x̄: 3.40 x̃: 2
HURT stats (rel) min: 0.11% max: 23.43% x̄: 1.15% x̃: 0.63%
95% mean confidence interval for instructions value: 2.47 3.03
95% mean confidence interval for instructions %-change: 0.61% 0.90%
Instructions are HURT.
total cycles in shared programs: 142103 -> 142015.75 (-0.06%)
cycles in affected programs: 1263.45 -> 1176.20 (-6.91%)
helped: 281
HURT: 176
helped stats (abs) min: 0.015625 max: 2.234375 x̄: 0.50 x̃: 0
helped stats (rel) min: 0.71% max: 54.17% x̄: 16.93% x̃: 15.31%
HURT stats (abs) min: 0.015625 max: 30.0 x̄: 0.30 x̃: 0
HURT stats (rel) min: 0.84% max: 120.00% x̄: 7.16% x̃: 5.00%
95% mean confidence interval for cycles value: -0.33 -0.05
95% mean confidence interval for cycles %-change: -9.08% -6.22%
Cycles are helped.
total cvt in shared programs: 13983.34 -> 14891.70 (6.50%)
cvt in affected programs: 7498.36 -> 8406.72 (12.11%)
helped: 71
HURT: 4711
helped stats (abs) min: 0.0625 max: 0.0625 x̄: 0.06 x̃: 0
helped stats (rel) min: 5.41% max: 40.00% x̄: 10.23% x̃: 9.30%
HURT stats (abs) min: 0.015625 max: 2.640625 x̄: 0.19 x̃: 0
HURT stats (rel) min: 0.18% max: 141.18% x̄: 16.21% x̃: 9.52%
95% mean confidence interval for cvt value: 0.18 0.20
95% mean confidence interval for cvt %-change: 15.21% 16.42%
Cvt are HURT.
total sfu in shared programs: 11320.44 -> 7882.56 (-30.37%)
sfu in affected programs: 7618.50 -> 4180.62 (-45.13%)
helped: 4782
HURT: 0
helped stats (abs) min: 0.0625 max: 10.5625 x̄: 0.72 x̃: 0
helped stats (rel) min: 1.34% max: 100.00% x̄: 41.91% x̃: 37.50%
95% mean confidence interval for sfu value: -0.75 -0.68
95% mean confidence interval for sfu %-change: -42.68% -41.14%
Sfu are helped.
total ls in shared programs: 129660 -> 129690 (0.02%)
ls in affected programs: 25 -> 55 (120.00%)
helped: 0
HURT: 1
total quadwords in shared programs: 1482728 -> 1484128 (0.09%)
quadwords in affected programs: 58624 -> 60024 (2.39%)
helped: 24
HURT: 195
helped stats (abs) min: 8.0 max: 8.0 x̄: 8.00 x̃: 8
helped stats (rel) min: 3.70% max: 20.00% x̄: 10.34% x̃: 10.00%
HURT stats (abs) min: 8.0 max: 24.0 x̄: 8.16 x̃: 8
HURT stats (rel) min: 1.41% max: 50.00% x̄: 4.84% x̃: 2.56%
95% mean confidence interval for quadwords value: 5.70 7.09
95% mean confidence interval for quadwords %-change: 2.22% 4.14%
Quadwords are HURT.
total spills in shared programs: 125 -> 127 (1.60%)
spills in affected programs: 0 -> 2
helped: 0
HURT: 1
total fills in shared programs: 800 -> 828 (3.50%)
fills in affected programs: 0 -> 28
helped: 0
HURT: 1
Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16857>
It's portable, and useful to both Bifrost and Valhall, in the clause scheduler
and in an instruction selection respectively. Move it from the Bifrost clause
scheduler to common code so we can share the benefits.
Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16857>
On Valhall, we always* use memory-allocated IDVS, which does not require min/max
indices. As such, we do not want to calculate min/max indices, as this is quite
slow. Skip this step.
* except for blit shaders, which don't use an index buffer anyway.
Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16867>
Memory-allocated IDVS does not require min/max indices to be calculated, but it
of course requires an index buffer. Extract a helper to upload the index buffer
without calculating bounds.
Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16867>
It's unnecessary and breaks the empty shader optimizations. Noticed while
inspecting a trace from dEQP-GLES3.functional.color_clear.masked_scissored_rgb,
which does not produce any varyings other than gl_Position in its vertex shader
and hence should omit the varying shader.
Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16868>
Top level acceleration structures need the bottom
6 bits to store the root ids of instances. If we
don't require that alignment, more "advanced"
allocators like VMA may sub allocate a buffer
which can lead to the 6 getting lost.
Fixes the Khronos ray tracing Vulkan samples.
Closes: #6598
Signed-off-by: Konstantin Seurer <konstantin.seurer@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16870>
according to the spec, atomic counters can be bound at any offset divisible by 4,
which means that any driver that uses the ssbo lowering pass and doesn't have
a min offset align of 4 is potentially broken
to handle this, use a statevar to inject the misaligned remainder of the offset
into the shader as a uniform. for well-aligned counter binds, the uniform offset
will be 0
Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16749>
this function should be called late to allow for other passes potentially
making changes which affect the states in use by shaders
Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16749>
some hardware can't do a ssbo offset=4, as required by the atomic->ssbo
lowering pass, so for these cases an offset can be passed for the counter
as a uniform, and the shaders can be adjusted accordingly
Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16749>
This pass will merge instructions like these
MOV output[0].x, temp[5].x___;
MOV output[0].yzw, none._001;
into
MOV output[0].xyzw, temp[5].x001;
It is currently very careful with control flow and dependency
tracking, so there is still room for improvements.
Shader-db stats with RV530:
total instructions in shared programs: 132486 -> 132256 (-0.17%)
instructions in affected programs: 6186 -> 5956 (-3.72%)
helped: 65
HURT: 0
total temps in shared programs: 18035 -> 18014 (-0.12%)
temps in affected programs: 295 -> 274 (-7.12%)
helped: 22
HURT: 1
Signed-off-by: Pavel Ondračka <pavel.ondracka@gmail.com>
Reviewed-by: Filip Gawin <filip@gawin.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16657>
This fixes the "Rewrite of inst X failed Can't allocate source
for Inst X src_type=X new_index=X new_mask=X" errors.
The compiler is quite strict when rewriting registers during
the pair allocation and checks that all of the reads of it are
initialized. However the spec doesn't enfore that, and
specifically with control flow depending on user input we can't
really know...
In the following example temp[4].x is written only in one branch,
that might or might not be taken, but this is enough to keep the
compiler happy:
IF aluresult.x___;
MAD temp[4].x, src0.1__, src0.111, src0.000
ENDIF;
src0.xyz = temp[4], src0.w = temp[4]
MAD color[0].xyz, src0.xyz, src0.111, src0.000
MAD color[0].w, src0.w, src0.1, src0.0
After switch to ntt, more IFs are converted to CMP, and the color
write looks like this. Please note that the CMP here is not TGSI
opcode but rather our US_OP_RGB_CMP: src2 >= 0 ? src0 : src1
src0.xyz = temp[4], src0.w = temp[4], src1.xyz = temp[3], src1.w = temp[12], src2.xyz = temp[2]
CMP color[0].xyz, src0.xyz, src1.xyz, -src2.xxx
CMP color[0].w, src0.w, src1.w, -src2.x
At this point temp[4].x is undefined. Now when compiler tries to
allocate register for temp[4] at some previous instruction, it will
find out that it is used as a source in the final CMP and bail out.
Instead of increasing the complexitty even more trying to account for
this, just get rid of the check completelly.
Fixes:
dEQP-GLES2.functional.shaders.indexing.vector_subscript.vec2_dynamic_subscript_write_component_read_fragment,Fail
dEQP-GLES2.functional.shaders.indexing.vector_subscript.vec2_dynamic_subscript_write_direct_read_fragment,Fail
dEQP-GLES2.functional.shaders.indexing.vector_subscript.vec2_dynamic_subscript_write_dynamic_subscript_read_fragment,Fail
dEQP-GLES2.functional.shaders.indexing.vector_subscript.vec2_dynamic_subscript_write_static_loop_subscript_read_fragment,Fail
dEQP-GLES2.functional.shaders.indexing.vector_subscript.vec2_dynamic_subscript_write_static_subscript_read_fragment,Fail
dEQP-GLES2.functional.shaders.indexing.vector_subscript.vec3_dynamic_subscript_write_component_read_fragment,Fail
dEQP-GLES2.functional.shaders.indexing.vector_subscript.vec3_dynamic_subscript_write_direct_read_fragment,Fail
dEQP-GLES2.functional.shaders.indexing.vector_subscript.vec3_dynamic_subscript_write_dynamic_subscript_read_fragment,Fail
dEQP-GLES2.functional.shaders.indexing.vector_subscript.vec3_dynamic_subscript_write_static_loop_subscript_read_fragment,Fail
dEQP-GLES2.functional.shaders.indexing.vector_subscript.vec3_dynamic_subscript_write_static_subscript_read_fragment,Fail
dEQP-GLES2.functional.shaders.indexing.vector_subscript.vec4_dynamic_subscript_write_component_read_fragment,Fail
dEQP-GLES2.functional.shaders.indexing.vector_subscript.vec4_dynamic_subscript_write_direct_read_fragment,Fail
dEQP-GLES2.functional.shaders.indexing.vector_subscript.vec4_dynamic_subscript_write_dynamic_subscript_read_fragment,Fail
dEQP-GLES2.functional.shaders.indexing.vector_subscript.vec4_dynamic_subscript_write_static_loop_subscript_read_fragment,Fail
dEQP-GLES2.functional.shaders.indexing.vector_subscript.vec4_dynamic_subscript_write_static_subscript_read_fragment,Fail
Reviewed-by: Pavel Ondračka <pavel.ondracka@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16657>
The fixes are mostly from 23dfae4c81
dEQP-GLES2.functional.fragment_ops.depth_stencil tests show random
flakes. The ones in failures are showing unexpected pass, however other
random test failures from the same group keep showing so just mark it
all as flakes.
Signed-off-by: Pavel Ondračka <pavel.ondracka@gmail.com>
Reviewed-by: Filip Gawin <filip@gawin.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16657>
It doesn't make sense and was not working anyway. This was spotted
by Filip Gawin in https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/13978
however the fix there was IMO just papering over the problem.
I don't believe that this could manifest as a real issues, because
when all of the swizzles were constant the file would be set to
RC_FILE_NONE already. So in theory this could lead to an issue only
in the close to impossible circumstance that the out of bounds memory
read by constant->u.Immediate[swz] would end with the same exact value
as another inlineable constant in different channel. However in some
circumstances it would lead to following valgrind warnings:
Conditional jump or move depends on uninitialised value(s)
at 0x5D4E690: ieee_754_to_r300_float (radeon_inline_literals.c:61)
by 0x5D4E690: rc_inline_literals (radeon_inline_literals.c:133)
by 0x5D3877A: rc_run_compiler_passes (radeon_compiler.c:436)
by 0x5D38821: rc_run_compiler (radeon_compiler.c:458)
by 0x5D4AF63: r3xx_compile_fragment_program (r3xx_fragprog.c:139)
by 0x5D48377: r300_translate_fragment_shader (r300_fs.c:499)
by 0x5D491B0: r300_pick_fragment_shader (r300_fs.c:601)
by 0x5D2BFEE: r300_create_fs_state (r300_state.c:1072)
by 0x57DDC36: st_create_nir_shader (st_program.c:538)
by 0x57DF10E: st_create_fp_variant (st_program.c:1056)
by 0x57E057C: st_get_fp_variant (st_program.c:1102)
by 0x57E0AB1: st_precompile_shader_variant (st_program.c:1287)
by 0x57E0AB1: st_finalize_program (st_program.c:1333)
by 0x57CB6F3: st_link_nir (st_glsl_to_nir.cpp:958)
Signed-off-by: Pavel Ondračka <pavel.ondracka@gmail.com>
Reviewed-by: Filip Gawin <filip@gawin.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16657>
When there are multiple MOVs with the same destination in loop
in different branches and some readers after the loop, we would
now errorneously copy propagate the last MOV, like in the following
snippet:
BGNLOOP;
...
IF temp[3].x___;
MOV temp[2], const[1].yxxy;
BRK;
ENDIF;
IF temp[4].x___;
MOV temp[2], const[1].xyxy;
BRK;
ENDIF;
...
MOV temp[2], const[1].xyxy;
ENDLOOP;
ADD_SAT temp[0], temp[2], temp[1];
into:
BGNLOOP;
...
IF temp[3].x___;
MOV temp[2], const[1].yxxy;
BRK;
ENDIF;
IF temp[3].y___;
MOV temp[2], const[1].xyxy;
BRK;
ENDIF;
...
ENDLOOP;
ADD_SAT temp[0], const[1].xyxy, temp[1];
We need the copy propagate just for simple cleanups after ttn,
anything more complex should have been handled already in NIR.
So just bail out if any of the readers is after the loop.
No changes in shader-db.
Fixes few piglit tests when loop unrolling is disabled:
spec@glsl-1.10@execution@vs-loop-complex-unroll
spec@glsl-1.10@execution@vs-loop-complex-unroll-nested-break
spec@glsl-1.10@execution@vs-loop-complex-unroll-with-else-break
Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/6467
Signed-off-by: Pavel Ondračka <pavel.ondracka@gmail.com>
Reviewed-by: Filip Gawin <filip@gawin.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16657>
opencl_c_h is defined only for llvm < 15.
Fixes: bcc2df4890 ("clc: speed up compilation by not relying on opencl-c.h")
Signed-off-by: Vinson Lee <vlee@freedesktop.org>
Reviewed-by: Jesse Natalie <jenatali@microsoft.com>
Reviewed-by: Karol Herbst <kherbst@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16808>
This is used for the old, buggy and slow GLSL IR loop unrolling
code. All drivers have now switched to the NIR unrolling code so
here we remove the CAP.
Reviewed-by: Alyssa Rosenzweig <alyssa@collabora.com>
Reviewed-by: Emma Anholt <emma@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16366>
NIR loop unrolling is already enabled so just let it do its job.
Shader-db results (nv120):
total gpr in shared programs: 893490 -> 893898 (0.05%)
gpr in affected programs: 15338 -> 15746 (2.66%)
total instructions in shared programs: 6243205 -> 6237068 (-0.10%)
instructions in affected programs: 71160 -> 65023 (-8.62%)
total bytes in shared programs: 66729616 -> 66664760 (-0.10%)
bytes in affected programs: 759328 -> 694472 (-8.54%)
Reviewed-by: Emma Anholt <emma@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16366>
NIR loop unrolling is already enabled so just let it do its job.
Shader-db results (nv92):
total gpr in shared programs: 734638 -> 735037 (0.05%)
gpr in affected programs: 11058 -> 11457 (3.61%)
total instructions in shared programs: 6073415 -> 6073398 (<.01%)
instructions in affected programs: 10079 -> 10062 (-0.17%)
total bytes in shared programs: 41837432 -> 41838872 (<.01%)
bytes in affected programs: 252504 -> 253944 (0.57%)
Reviewed-by: Emma Anholt <emma@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16366>
Qiang Yu
Alyssa Rosenzweig
Rhys Perry
Juan A. Suarez Romero
Frank Binns
Konstantin Seurer
David Heidelberg
Mike Blumenkrantz
Pavel Ondračka
Filip Gawin
Vinson Lee
Timothy Arceri