This needs to be accurate so that when we split and then schedule a new
a0.x/a1.x/p0.x write we will eventually make progress. It wasn't taking
the kill_path into account which could create an infinite loop as we
keep scheduling writes whose uses are blocked because they are memory
instructions not on the kill_path.
Closes: #6413
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16635>
Note: this is a behavior change for arrays, because it will count the
entire array instead of just the components written in the register
pressure calculation. However this is more accurate since this matches
how RA works.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/14107>
This now covers e.g. cat6 instructions as well, and ss will cover
instructions writing shared regs as well. This is split out from the
previous change to avoid too much churn and shouldn't cause any
functional changes.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/14246>
The old code walked the instructions between each ready instruction and
each of its parents for every instruction, which can quickly become
accidentally quadratic. Instead we keep track of the current
"instruction pointer" of the to-be-scheduled instruction, and for each
ready instruction calculate an "earliest possible IP" which is the IP
that needs to be reached before we can schedule it. Because this stays
constant as soon as an instruction becomes ready, we never have to
recompute it and each call to ir3_delay_calc_prera() becomes a simple
comparison and subtract. We only need to iterate over the children and
update their earliest_ip when scheduling an instruction, and we already
do that in util_day_prune_head() so it should be cheap.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/13722>
If you didn't have a freed+ready instruction, you'd redo the live_effect
and check_instr() logic multiple times per instr. Replace the multiple
loops in each function with a ranking that I think is more readable,
reducing the overhead in the process.
debugoptimized dEQP-GLES31.functional.ubo.random.all_per_block_buffers.20
runtime goes from ~3.5s -> ~3.0s on my lazor. No shader-db change.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11855>
When we go to split e.g. a p0.x producer, the only other instructions
ready to schedule are often only p0.x producers. It could happen that
they all have a lower priority than the split instruction. Then we would
immediately schedule the split instruction again, then again try to
schedule one of the other producers, be blocked, and split it, around
and around again, leading to an infinite loop. The following commit
triggered this with
dEQP-GLES3.functional.shaders.discard.dynamic_loop_always on a3xx.
Fixes: d2f4d33 ("freedreno/ir3: new pre-RA scheduler")
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6752>
If we've identified another use that isn't scheduled yet, we can break
right away rather than iterating through all the other uses. While this
could be optimized further, this simple change makes
dEQP-VK.subgroups.ballot_broadcast.compute.subgroupbroadcast_ivec4 go
from 40 seconds to 1.9 seconds on a release build according to my
unscientific testing.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11613>
Also change the indexing in ir3_delayslots, so it's finally sane! To do
this we also have to change foreach_ssa_src_n to index srcs instead of
regs, so that the indexing stays in sync.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11469>
If an instruction's destination is unused, then we shouldn't penalize
it. For example, this helps us schedule atomic operations whose results
aren't read. This works around RA failures when CSE is enabled in some
robustness2 tests.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9842>
In a scenario where there are a lot of texture fetches with constant
coordinates, this prevents the scheduler from scheduling all the setup
instructions after the first group of textures has been scheduled
because they are the only non-syncing thing and scheduling them didn't
decrease tex_delay. Collects with immed/const sources will turn into
moves of those sources, so we should treat them the same.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9842>
The old delay calculation relied on the SSA information staying around,
and wouldn't work once we start introducing phi nodes and making
"normal" values defined in multiple blocks not array regs anymore.
What's worse is that properly inserting phi nodes when splitting live
ranges would make that code even more complicated, and this was the last
place post-RA that actually needed that information.
The new version only compares the physical registers of sources and
destinations. It works by going backwards up to a maximum number of
cycles, so it might be slightly slower when the definition is closer but
should be faster when it is farther away.
To avoid complicating the new method, the old method is kept around, but
only for pre-RA scheduling and it can therefore be drastically
simplified as the array case can be dropped.
ir3_delay_calc() is split into a few variants to avoid an explosion of
boolean arguments in users, especially now that merged_regs now has to
be passed to it.
The new method is a little more complicated when it comes to handling
(rptN), because both the assigner and consumer may be (rptN). This adds
some unit tests for those cases, in addition to dropping the to-SSA code
in the test harness since it's no longer needed.
Finally, ir3_legalize has to be switched to using physical registers for
the branch condition. This was the one place where IR3_REG_SSA remained
after RA.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9842>
If the tex/sfu ssa src is from a different block than the one currently
being scheduled, we do not have a valid sched-node. So fallback to
previous behavior rather than dereference an invalid ptr.
Fixes: 7821e5a3f8 ("ir3/sched: Don't penalize uses of already-waited tex/SFU")
Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10306>
Instead of using a separate outputs array, make the "end" instruction
(or chmask) take the outputs as sources. This works better for the new
RA, because it better models the fact that outputs are consumed all at
the same time. With the old model, each output collect would be assumed
dead after it was processed and subsequent collects could use it when
inserting shuffle code, which wouldn't work, and the new RA also deletes
collect instructions after lowering them to moves so the information
would be gone after RA.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10591>
The "demote" intrinsic has the semantics of D3D discard, which means
it doesn't change the control flow, allowing derivatives to work.
On A6xx there is no known way to check whether invocation was demoted,
thus we use nir_lower_is_helper_invocation.
Add "logical" OPC_DEMOTE which is later translated to "kill".
Such separation is necessary to run "kill" specific optimizations
which are invalid for "demote".
Signed-off-by: Danylo Piliaiev <dpiliaiev@igalia.com>
Reviewed-by: Samuel Iglesias Gonsálvez <siglesias@igalia.com>
Reviewed-by: Eric Anholt <eric@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9460>
Consider a simple loop that does a series of texture instructions and
then reduces the results:
vec4 sum = vec4(0);
for (int i = 0; i < N; i++) {
sum += texture(...);
}
Assume that the loop is unrolled and we schedule the resulting basic
block. Right now, after we schedule the first texture instruction, the
only instructions available to schedule that don't incur a sync are the
instructions to setup the second texture instruction. So we keep picking
the texture instructions, no matter how large N is, resulting in a
pathological schedule for register pressure when N is very large:
sum1 = texture(...);
sum2 = texture(...);
sum3 = texture(...);
...
sum = sum1 + sum2 + sum3 + ...;
In particular this happens with some CTS tests for VK_EXT_robustness2,
where a loop like that with many iterations is marked as [[unroll]],
forcing NIR to unroll it.
This solution is a balance between the current approach and always
scheduling for register pressure (and ignoring sync's). We only allow a
certain number of texture fetches to be in flight before considering
textures to "sync", even though they don't really, both because they
likely *will* sync in reality (overflowing the internal queue of waiting
texture instructions) and because at some point we need the normal
algorithm to kick in and start lowering register pressure.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7571>
Once we insert a use of a given tex or SFU instruction, then we must
wait for that tex/SFU instruction (as well as all earlier ones) to
complete, so we shouldn't penalize further uses, even if a subsequent
tex/SFU instruction gets scheduled after the first use. This especially
matters after the next commit when we start forcibly breaking up long
sequences of texture instructions, since if we schedule a group of 8
texture instructions then we want to schedule the uses of those
instructions in parallel with the next 8 texture instructions to reduce
register pressure.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7571>
Similar to what we do in postsched. It is useful for pre-RA sched to be
a bit aware of things that would cause syncs. In particular for the tex
fetches, since the vecN src/dst tends to limit postsched's ability to
re-order them.
Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4923>
If an instruction's only use is as an output, and it increases register
pressure, then try to avoid scheduling it until there are no other
options.
A semi-common pattern is `fragcolN.a = 1.0`, this pushes all these
immed loads to the end of the shader.
Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4923>
Realize that certain instructions make a vecN live, and account for
this, in hopes of scheduling the remaining components of the vecN
sooner.
Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4440>
This replaces the depth-first search scheduler with a more traditional
ready-list scheduler. It primarily tries to reduce register pressure
(number of live values), with the exception of trying to schedule kills
as early as possible. (Earlier iterations of this scheduler had a
tendency to push kills later, and in particular moving texture fetches
which may not be necessary ahead of kills.)
Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4440>
They were inserting a nop between back to back SFU instrucions. But
that doesn't actually appear to be required. And they get stripped out
later anyways before legalize.
Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4071>
In schedule live value tracking, differentiate between half vs full
precision. Half-precision live values are less costly than full
precision.
Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3989>
Current scheduler thresholds try to ensure there are warps available to
switch to when hiding texture fetch latency. But if there is none to
hide, we should allow scheduler to use more registers to reduce nops.
Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3989>
We want to do this only once. If we have post-RA sched pass, then we
don't want to do it pre-RA. Since legalize is where we resolve the
branch/jumps, we might as well move this into legalize.
Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3569>
This way we can deal with it in one place, *after* all the blocks have
been scheduled. Which will simplify life for a post-RA sched pass.
This has the benefit of already taking into account nop's that legalize
has to insert for non-delay related reasons.
Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3569>
Add some infrastructure to trace scheduler decisions. The next patch
will add some more traces, just splitting this out to reduce clutter.
Signed-off-by: Rob Clark <robdclark@chromium.org>
Eric Engestrom
Marek Olšák
Connor Abbott
Emma Anholt
Rob Clark
Danylo Piliaiev