- tcinv - Likely Texture Cache Invalidate (unverified)
- icinv - Mostly sure that it is Instruction Cache Invalidate
- dccln - Data Cache Clean
- dcinv - Data Cache Invalidate
- dcflu - Data Cache Flush
The emission of these instructions were not observed in the wild.
TODO: find out the difference between .shr and .all modes of
dccln, dcinv, dcflu.
Signed-off-by: Danylo Piliaiev <dpiliaiev@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/14419>
`kill` changes which fibers are active, thus reodering instructions
which depend on which fibers are active - is wrong.
The issue was hidden because only `ballot(true)` is translated to movmsk
immidiately, while others are passed as MACRO and don't properly
take part in ir3_sched (which does the reordering).
Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/7162
Fixes CTS test (on gen3+):
dEQP-VK.spirv_assembly.instruction.terminate_invocation.terminate.subgroup_ballot
Fixes: b1b80c06a7
("ir3: Implement nir subgroup intrinsics")
Signed-off-by: Danylo Piliaiev <dpiliaiev@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18413>
Before, we needed CP post-sched to copy-propagate references to NIR
registers produced by out-of-ssa. Now that we're in SSA, this pass ends
up not doing anything useful, and actually gets in the way by occasionally
creating a cycle in the DAG.
The entire shader-db impact is:
instructions HURT: shaders/closed/steam/tropico-5/78.shader_test FRAG: 238 -> 242 (1.68%)
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17320>
These will be used to implement the ir3-specific shader preamble
lowering in NIR. shps is conceptually similar to getone (although it
technically can't be duplicated) and shpe is similar to other barriers,
since it has to happen after any stores to the constant file in the
preamble. Add NIR intrinsics and plumbs them through ir3.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/13148>
Normally this wouldn't matter, but it will matter for the upcoming scan
macro because the running tally is communicated through a shared
register across a physical edge. It may also matter if a live-range
split occurs.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/14107>
* shrm - (src2 >> src1) & src3
* shlm - (src2 << src1) & src3
* shrg - (src2 >> src1) | src3
* shlg - (src2 << src1) | src3
* andg - (src2 & src1) | src3
* dp2acc - dot product of two {i,u}8vec2 packed into
SRC1 and SRC2, added to 32b SRC3
* dp4acc - dot product of two {i,u}8vec4 packed into
SRC1 and SRC2, added to 32b SRC3
* wmm - vec4(x_1, x_2, x_3, x_4) * (y_1 + y_2 + y_3 + y_4), which is
duplicated (1 << (SRC3 / 32)) times starting from DST register
* wmm.accu - same as wmm but result is added to DST registers, however
the first reg in each vec4 result is overwritten instead of
accumulating.
Signed-off-by: Danylo Piliaiev <dpiliaiev@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/13986>
The blob uses *both* nops and (ss). It turns out that in some rare cases
the hardware does take more than 6 cycles, at least for movmsk, but
adding nops is unnecessary. I believe the extra nops are only there due
to the immaturity of the blob's implementation of subgroup ops, so we
don't have to copy them - just handle shared reg producers the same as
SFU instructions.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/14246>
Add new centralized functions which will replace the various places we
hardcode 10 for the number of (ss) nops, add numbers for soft (sy) nops
based on similar computerator experiments with ldc, sam, and ldib (the
most common (sy) producers), and add a "systall" metric which is
analogous to sstall. This also fixes some cases where we'd erroniously
count ldl* as (sy) producers instead of (ss) producers when calculating
sstall.
This only switches over the metric reporting to the new functions, so
there is no behavior change. The following commit will switch over
the rest of the compiler.
While we're at it, remove max_sun as it's never set.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/14246>
These macros expand to a mov in an if statement which breaks address
assumption that instruction which produces address and consumes it
are in the same block.
Fixes test:
dEQP-VK.subgroups.ballot_broadcast.framebuffer.subgroupbroadcast_intvertex
Signed-off-by: Danylo Piliaiev <dpiliaiev@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/13931>
Separating atomic opcodes makes possible to express a6xx global
atomics which take iova in SRC1. They would be needed by
VK_KHR_buffer_device_address.
The change also makes easier to distiguish atomics in conditions.
Signed-off-by: Danylo Piliaiev <dpiliaiev@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8717>
Analogous to the pre-RA scheduler. Unfortunately this time it's a bit
more involved because we have to correctly handle (rptN), which is
already relevant for swz. This means we need the index of the
destination register that conflicts with the source register, to handle
swz, and we need to expose that part of ir3_delay. But once that's done,
we can delete ir3_delay_calc_postra.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/13722>
We have a situation in some skia shaders like:
add.f r0.x, ...
(rpt2)nop
mul.f ..., r0.x
sam (xyzw) r0.x, ...
rcp ..., r0.x
Notice that rcp uses the result of the sam instruction, not the add.f,
but we didn't keep track of which instructions kill the sources in
ir3_delay, so we'd add an extra nop, resulting in a disagreement betwen
ir3_delay and the scheduling graph. Since postsched is correct, fix
ir3_delay. This only results in some very slight shader-db changes but
keeps the next commit from changing things.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/13722>
The old code walked the instructions between each ready instruction and
each of its parents for every instruction, which can quickly become
accidentally quadratic. Instead we keep track of the current
"instruction pointer" of the to-be-scheduled instruction, and for each
ready instruction calculate an "earliest possible IP" which is the IP
that needs to be reached before we can schedule it. Because this stays
constant as soon as an instruction becomes ready, we never have to
recompute it and each call to ir3_delay_calc_prera() becomes a simple
comparison and subtract. We only need to iterate over the children and
update their earliest_ip when scheduling an instruction, and we already
do that in util_day_prune_head() so it should be cheap.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/13722>
We were treating them the same as regular cat2/cat3/cat4 immediates, but
that's not right because cat6 sources are only 8 bits.
Our bindless code was handling this before for bindless resources, and
it was disabled for most other things, so this was mostly harmless, but
fixing it will be necessary for handling ldc offsets.
In addition enable tests for this that were just commented out, and add
a custom test making sure that the immediate source is treated as
unsigned.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/13142>
If the coherent flag is present, then we need to not have an incoherent
cache between us and previous stores to the image that were also decorated
as coherent. isam apparently (unsurprisingly) goes through a texture
cache. Use ldib instead, so that we don't get the wrong result.
We would need a similar fix for a4xx, but that uses ldgb and I don't
have hardware to test on.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/12704>
a6xx supports shaders with more than 64k dwords, or at least the shader
size register has increased in size, and the matching name is gone so
there's no reason to be clever here. This doesn't fix anything at the
moment.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/12487>
Rob Clark
Danylo Piliaiev
Amber
Eric Engestrom
Yonggang Luo
Chia-I Wu
Marek Olšák
Emma Anholt
Connor Abbott
Ilia Mirkin
Matt Turner