This fixes a pre-existing bug in ir3, but it showed up even more due to
other changes in this series and it interacts with the logical/physical
CFG split. When both sides of an if end with a jump, a block may become
unreachable via the logical CFG, which can cause problems because it has
no predecessors to figure out the location of live-in non-shared
values. In this case we assume that nir_opt_if has removed any code in
these blocks and just skip processing live-ins for these blocks,
pretending that they aren't live.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6752>
These were a holdover from before the src/dst split and are no longer
necessary. Just don't create any dest registers for instructions that
never have a destination.
This has the side-effect that it becomes easier to replace uses of
dest_regs() with a per-register thing, once we start adding support for
multiple destinations.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11565>
RA was manually fiddling with regs to copy over the parallel copy code,
which has to be done in a different way, but if we switch this all over
at once it shouldn't be a problem.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11469>
Instructions that operate on an array read the previous state of the
array, modify it, and write a new array, at least conceptually before
RA. Previously the same register specified the previous state and acted
as the new state, but this meant that it was both a source and
destination which meant that it was getting in the way of splitting up
sources and destinations. Break out the source into a separate register,
and use the new tied-src infrastructure to share code with a6xx atomics.
With this, there are basically no more special cases for arrays in RA.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11469>
Previously this was hard-coded for a6xx atomic instructions. However
we'll need a way for array destinations to point to the source with the
previous value of the array when we split them up. This is conceptually
the same as tied source/destinations for a6xx atomics, except that array
writes sometimes won't have a previous value to point to. So move this
into the IR so that it can be more dynamic. As a bonus we can move the
knowledge of a6xx atomics out of RA, where it's out-of-place, and into
the a6xx-specific code that creates them.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11469>
With array registers, there are two num's we care about:
1. The base num that the whole array starts at (->array.base)
2. The num that the instruction uses, plus possibly an indirect offset
(->num or ->array.offset)
For parallel copies we always copy the whole array, so (2) is irrelevant
here. For phis and parallel copies inserted for phis, we used
assign_reg() which assigned ->array.base, but we forgot about this when
constructing our own parallel copies for live range splitting, just
setting ->num instead. The parallel copy lowering was also inconsistent
here, using ra_reg_get_num() (which looks at ->array.base for arrays)
for sources but looking at ->num directly for destinations. This makes
everything use ->array.base consistently.
While we're here, make sure to remove IR3_REG_SSA from liveout copies to
make sure printing works correctly.
Fixes: 0ffcb19 ("ir3: Rewrite register allocation")
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11422>
There were two different approaches I saw in the post-RA code for
figuring out what regiser range a relative access touched:
1. Use reg->array.offset and reg->array.size. This is wrong in case
reg->array.offset was non-zero before RA, because array.size is
the size of the whole array and array.offset has the const offset
within the array baked in.
2. Lookup the array from the array ID and use the base + range there.
This is correct, but won't work with the new RA, where an array might
not always be assigned to the same register.
This replaces both methods with a new ir3_register::array.base field,
and switches all the users I could find to it.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9842>
To simplify the pre-RA merge set code and express the result live-range
splitting in RA, we need to add support for parallel copy instructions,
and for the merge set code these parallel copies need to be in SSA form.
Parallel copies have multiple destinations by necessity, but there was
no way to express this in the existing IR. In particular there was no
support for marking a register as being a destination, and no support
for indicating which destination register out of several an SSA source
refers to. This replaces ir3_register::instr with ir3_register::def and
re-purposes ir3_register::instr. I haven't propagated this into common
helpers, like ssa(), because that would vastly increase the amount of
churn and the number of places that produce such instructions should be
limited -- only RA will create parallel copies and they will be
destroyed right after RA. In the future swz will have multiple
destinations too, but it will only be created after RA via parallel copy
lowering.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9842>
Instead of using a separate outputs array, make the "end" instruction
(or chmask) take the outputs as sources. This works better for the new
RA, because it better models the fact that outputs are consumed all at
the same time. With the old model, each output collect would be assumed
dead after it was processed and subsequent collects could use it when
inserting shuffle code, which wouldn't work, and the new RA also deletes
collect instructions after lowering them to moves so the information
would be gone after RA.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10591>
Assigning an array reg removes IR3_REG_ARRAY, which means that
definitions and uses can't be tracked back to the array register's name
and liveness for the components of the array aren't correctly
calculated. To fix this we delay assigning array registers until the
scalar pass.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7711>
Properly handle the difference between split and merged register file
when determining where arrays can fit without conflicting with other
arrays or pre-colored instructions.
1) if not mergedregs, only consider other things with same precision
as potentially conflicting
2) if mergedregs, calculate everything in therms of half-regs and
convert back to fullregs in the end
Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5957>
We shouldn't divide-by-two for half-reg arrays. We set the proper node
interference class, based on `arr->half`.
Fixes a RA fail with 16b arrays:
src/freedreno/ir3/ir3_ra.c:633: name_to_array: Assertion `!"invalid array name"' failed.
Caused by use/def iterators returning `arr->length` vreg namess, but
only assigning the array half that many names.
Also, since we are assigning unique vreg names to each array element,
there is no need to try and convert from half-reg to it's conflicting
full reg when pre-coloring the array elements. Getting us closer to
having half-arrays work sanely with split-register-file (a5xx and
earlier).
Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5957>
It doesn't happen much. But it's annoying when we hit an impossible
condition deep in RA 90% thru a long test run. Add some ra_assert()/
ra_unreachable() helper macros so we can bail cleanly and fail RA.
Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5907>
Fixes a case where you have something like:
aVecOutput.z = aScalarInput;
In particular, skipping over things that are not the first component is
wrong.. in the above case the input we need to precolor is the 3rd
component. But we need to adjust the target register according to the
offset.
Fixes android.hardware.nativehardware.cts.AHardwareBufferNativeTests
Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5601>
Rather than assuming a6xx+ means mergedregs. We can actually (mostly?)
do splitregs on a6xx as well. And GS/DS/HS currently require it, which
might be papering over a bug, or might be something to do with how
chaining shaders works. At any rate, we should at least be consistent,
and not have the compiler thinking we are doing mergedregs when we are
actually doing splitregs.
Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5458>
Teach RA to setup additional interference to prevent textures fetched
before the FS starts from ending up in a register that is too high to
encode.
Fixes mis-rendering in multiple playcanv.as webgl apps.
Note that the regression was not actually 733bee57eb8's fault, but
that was the commit that exposed the problem.
Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/3108
Fixes: 733bee57eb ("glsl: lower samplers with highp coordinates correctly")
Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5431>
For cases where instructions have a src and/or dst type, validate that
it matches the src/dst register types. And for cases where there are
different opcodes for half vs full, validate that the opcode matches.
Now that we maintain this properly throughout the stages of the ir, we
can drop the fixups from the RA pass.
Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5048>
In a later patch, this will get folded into an IR3_PASS() macro, at
least for most passes. But to do that, it is better to standardize
on printing the ir3 after the pass.
Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5048>
We were failing to tell the allocator about the restriction that scalar
texture instructions (allocated as scalar regs) couldn't be allocated such
that the start of the full unwritemasked vector started before r0. There
was a patch in select_reg_callback on a6xx that tried to work around that,
but you could still end up backed into a corner you shouldn't be because
we didn't tell the RA what it needed.
Fixes compiler assertion failures on a300-a400's blit_z shader, used for
Z32F gmem blits.
Looks like as a result we get tighter register allocation but more nops:
instructions in affected programs: 757945 -> 760356 (0.32%)
nops in affected programs: 317983 -> 320468 (0.78%)
non-nops in affected programs: 27525 -> 27451 (-0.27%)
mov in affected programs: 3098 -> 3023 (-2.42%)
dwords in affected programs: 109664 -> 110656 (0.90%)
last-baryf in affected programs: 112701 -> 112847 (0.13%)
full in affected programs: 4326 -> 4011 (-7.28%)
sstall in affected programs: 120550 -> 120836 (0.24%)
(ss) in affected programs: 13939 -> 13918 (-0.15%)
(sy) in affected programs: 3006 -> 2786 (-7.32%)
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4562>
The intersects() function assumes that inside each instruction values
always die before they are defined, so that if the end of one range is
the same instruction as the beginning of the next then they don't
intersect. However, this isn't the case for values that become live at
the beginning of a basic block, which become live *before* the first
instruction, or instructions that die at the end of a basic block which
die after the last instruction.
For example, imagine that we have two values, A which is defined earlier
in the block and B which is defined in the last instruction of the block
and both die at the end of the basic block (e.g. are used in the next
iteration of a loop). We would compute a range for A of, say, (10, 20)
and for B of (20, 20) since each block's end_ip is the same as the ip of
the last instruction, and RA would consider them to not interfere.
There's a similar problem with values that become live at the beginning.
The fix is to offset the block's start_ip and end_ip by one so that they
don't correspond to any actual instruction. One way to think about this
is that we're adding fake instructions at the beginning and end of a
block where values become live & die. We could invert the order, so that
values consumed by each instruction are considered dead at the end of
the previous instruction, but then values that become dead at the
beginning of the basic block would incorrectly have an empty live range,
with a similar problem at the end of the basic block if we try to say
that values are defined at the beginning of the next instruction. So
the extra padding instructions are unavoidable.
This fixes an accidental infinite loop in the shader for
dEQP-VK.spirv_assembly.type.scalar.u32.switch_vert.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4614>
The src of the SFU instruction could also be array/reg (non-SSA).
Handle this case too.
The postsched cp pass makes this scenario more likely.
Fixes: cc82521de4 ("freedreno/ir3: round-robin RA")
Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4440>
Since we are re-assigning the scalars anyways in the second pass, assign
them to the highest free reg in the first pass (rather than lowest) to
allow packing vecN regs as low as possible.
Note this required some changes specifically for tex instructions with a
single component writemask that is not necessarily .x, as previously
these would get assigned in the first RA pass, and since they are still
scalar, we'd end up w/ some r47.* and other similarly way-to-high
assignments after the 2nd pass.
Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4272>
Using the output of the first pass isn't ideal, as it can bake in the
losses from fragmentation which the scalar pass is intended to fill in.
This gets worse when we start using "vectorish" instructions, due to
higher use of vecN values.
Instead, we can just use the outputs of the liveness analysis to get a
more accurate # of maximum live values at any point.
Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4272>
Decouple the messy logic of figuring out vreg names defined/used by an
instruction from the logic of what to do about it by introducing
iterators. There is still *some* array vs ssa special casing in
ra_block_compute_live_ranges(), but less than before. And this will
avoid introducing a second copy of the def/use logic in a following
patch which uses the liveranges to calculate the maximum # of live
values (which is the optimal target for max physical register window
to round-robin within).
Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4272>
Account for the # of regs an instruction writes, and fix an off-by-one.
(We are about to replace this with calculating the register target using
the live-ranges, but in debugging that it was useful to assert() if it
chose a higher target.)
Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4272>
Connor Abbott
Eric Anholt
Rob Clark
Kristian H. Kristensen