The shader won't be available for deserialized variants, so we need to
include all the info we need for compiling variants to be in the
variant. Most of the things we dug out of the shader were various bits
from nir_shader_info which we move into ir3_shader_variant.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16147>
We weren't considering the other destinations when allocating a
destination, so we could allocate overlapping destinations. This wasn't
done before because we never had a need for it, but the subgroup
reduction macros will need it.
The trickiest part of this is that we have to rewrite the
compress_regs_left fallback, because we may have to move around the
other already-allocated destinations. We now have a list of destinations
to (re)allocate in addition to the popped live intervals. For the rest
of the destination handling, we can just bail out if the proposed spot
for something overlaps another destination, but for the fallback we have
to handle all the cases gracefully. I also added support for odd
combinations of multiple destinations where some of them are tied, which
we'll use in the next commit to handle early-clobber destinations and
which will actually be used because one of the destinations of the
subgroup reduction macro will be early-clobber. The result is that the
order of intervals to allocate is now a lot more complicated.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/14107>
For pcopies we only care about the register's type, i.e. whether its a
half-register and whether it's an array (plus its size). Copying over
other flags like IR3_REG_RELATIV just leads to sadness and validator
assertions.
Fixes: 0ffcb19b9d ("ir3: Rewrite register allocation")
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/14107>
Before, we were careful to
1. Get the source physreg.
2. Allocate the destination.
3. Insert a copy with the source being the physreg from step 1.
and this guaranteed that if the tied source were moved in step 2 we'd
still insert a copy from the correct place. However this won't work with
multiple destinations because an earlier destination could've already
moved the tied source around. Instead flip steps 2 and 3 (we'll insert
the copy before we allocate the interval, but that's ok) and run the
first two steps in a separate loop before any destinations are
allocated.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/14107>
This allows the wavesize to be controlled per-shader. This will be used
by VK_EXT_subgroup_size_control, and freedreno will also need it if
legacy ARB_shader_ballot is to be supported (since it forces a wavesize
of 64 or less).
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/13960>
If barriers are used, it must be possible for all waves in the workgroup
to execute concurrently. Thus we may have to reduce the registers limit.
Fixes a hang in "Digital Combat Simulator".
Signed-off-by: Danylo Piliaiev <dpiliaiev@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/14110>
If we're allocating a source then we force is_killed to false, not to
true. Fixes a regression in
dEQP-GLES31.functional.synchronization.in_invocation.image_atomic_write_read
later.
Fixes: 0ffcb19b9d ("ir3: Rewrite register allocation")
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/14246>
This would've caught the previous issue earlier. We checked that the
physreg made sense when inserting via ra_file_insert() but not
ra_push_interval() which is used for live-range splitting.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/13818>
Let's assume we have a vec2 collect instruction with killed sources that
are non-contiguous and the entire rest of the register file is blocked,
which can happen when our register target is very tight. It's impossible
to just insert move instructions to resolve this, but we can make space
by swapping one of the killed sources with the value next to the other,
assuming it's also scalar.
This commit implements that idea, preventing us from falling back to the
terrible shuffle-everything approach in this case.
total instructions in shared programs: 1566648 -> 1565117 (-0.10%)
instructions in affected programs: 13332 -> 11801 (-11.48%)
helped: 30
HURT: 5
helped stats (abs) min: 6 max: 535 x̄: 51.77 x̃: 25
helped stats (rel) min: 2.67% max: 33.63% x̄: 12.28% x̃: 9.58%
HURT stats (abs) min: 1 max: 6 x̄: 4.40 x̃: 6
HURT stats (rel) min: 0.18% max: 5.13% x̄: 2.41% x̃: 2.13%
95% mean confidence interval for instructions value: -75.05 -12.43
95% mean confidence interval for instructions %-change: -13.18% -7.18%
Instructions are helped.
total mov in shared programs: 77336 -> 76683 (-0.84%)
mov in affected programs: 2135 -> 1482 (-30.59%)
helped: 29
HURT: 5
helped stats (abs) min: 2 max: 227 x̄: 23.31 x̃: 10
helped stats (rel) min: 6.06% max: 72.73% x̄: 31.83% x̃: 30.00%
HURT stats (abs) min: 2 max: 9 x̄: 4.60 x̃: 4
HURT stats (rel) min: 14.29% max: 69.23% x̄: 34.00% x̃: 27.78%
95% mean confidence interval for mov value: -33.21 -5.20
95% mean confidence interval for mov %-change: -32.94% -11.35%
Mov are helped.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/13143>
I'm going to want to use this in other passes, so lets let the
allocation hang off the pass's context. Also, while we're at it,
fix the error path leak in ir3_ra().
Fixes: 0ffcb19b9d ("ir3: Rewrite register allocation")
Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/12923>
Support for spilling shared registers to normal registers is still TODO.
There are also several improvements to be made, like rematerialization.
Note, there is one behavior change to register pressure accounting: we
now include half registers in the current full pressure directly in
mergedregs mode, rather than adding the max half pressure to the max
full pressure afterwards, which might result in lower calculated max
pressure in some cases with half registers. This is needed for spilling,
since we need to make sure the total pressure including half registers
is below the maximum at each instruction. Because the entire pass is
rewritten, including the register pressure calculating parts, it didn't
seem worth it to separate out this change.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/12033>
This was just wrong - we need to check against the entire register file,
and we need to include removed full regs even if the register we're
trying to insert is a half-reg, or else we could run out of space when
reinserting full regs after it. There does need to be an additional
check so that we don't try to insert a half-reg beyond the half-reg
limit, but that has to happen in addition to the normal check.
This fixes KHR-GLES31.core.arrays_of_arrays.InteractionArgumentAliasing6
once spilling is added.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/12033>
RA uses this to pop and then reinsert intervals when shuffling around
registers. For spilling, we want to remove the interval and also mark
all its descendants as removed. Since "remove_all" sounds more like the
latter, rename the old "remove_all" to "remove_temp". "remove_all" was
already exposed in ir3_ra.h, so there's no need to add it.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/12033>
It can happen that we create an enormous merge set, even larger than the
entire register file, in which case find_best_gap() would loop
infinitely. This seems to be triggered more often with
IR3_SHADER_DEBUG=spillall, since it actually happened with a CTS test.
Just bail out in that case.
Fixes: 0ffcb19b9d ("ir3: Rewrite register allocation")
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/12033>
When we mark live-through sources that are merged with the destination
as killed, we kept the bitsets in sync, but we forgot to keep them in
sync when unmarking them after allocating the destination. The result
was that "available" wasn't correct for any instruction afterwards. This
resulted in a bad register allocation with IR3_SHADER_DEBUG=spillall for
a dEQP-VK test.
While we're changing this, use ra_foreach_src().
Fixes: 0ffcb19b9d ("ir3: Rewrite register allocation")
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/12033>
Otherwise when an interval is removed and then re-inserted it could
have an invalid/corrupted parent link and child tree. I think RA
happened to never do this, but spilling will.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/12033>
This fixes a pre-existing bug in ir3, but it showed up even more due to
other changes in this series and it interacts with the logical/physical
CFG split. When both sides of an if end with a jump, a block may become
unreachable via the logical CFG, which can cause problems because it has
no predecessors to figure out the location of live-in non-shared
values. In this case we assume that nir_opt_if has removed any code in
these blocks and just skip processing live-ins for these blocks,
pretending that they aren't live.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6752>
These were a holdover from before the src/dst split and are no longer
necessary. Just don't create any dest registers for instructions that
never have a destination.
This has the side-effect that it becomes easier to replace uses of
dest_regs() with a per-register thing, once we start adding support for
multiple destinations.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11565>
RA was manually fiddling with regs to copy over the parallel copy code,
which has to be done in a different way, but if we switch this all over
at once it shouldn't be a problem.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11469>
Instructions that operate on an array read the previous state of the
array, modify it, and write a new array, at least conceptually before
RA. Previously the same register specified the previous state and acted
as the new state, but this meant that it was both a source and
destination which meant that it was getting in the way of splitting up
sources and destinations. Break out the source into a separate register,
and use the new tied-src infrastructure to share code with a6xx atomics.
With this, there are basically no more special cases for arrays in RA.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11469>
Previously this was hard-coded for a6xx atomic instructions. However
we'll need a way for array destinations to point to the source with the
previous value of the array when we split them up. This is conceptually
the same as tied source/destinations for a6xx atomics, except that array
writes sometimes won't have a previous value to point to. So move this
into the IR so that it can be more dynamic. As a bonus we can move the
knowledge of a6xx atomics out of RA, where it's out-of-place, and into
the a6xx-specific code that creates them.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11469>
With array registers, there are two num's we care about:
1. The base num that the whole array starts at (->array.base)
2. The num that the instruction uses, plus possibly an indirect offset
(->num or ->array.offset)
For parallel copies we always copy the whole array, so (2) is irrelevant
here. For phis and parallel copies inserted for phis, we used
assign_reg() which assigned ->array.base, but we forgot about this when
constructing our own parallel copies for live range splitting, just
setting ->num instead. The parallel copy lowering was also inconsistent
here, using ra_reg_get_num() (which looks at ->array.base for arrays)
for sources but looking at ->num directly for destinations. This makes
everything use ->array.base consistently.
While we're here, make sure to remove IR3_REG_SSA from liveout copies to
make sure printing works correctly.
Fixes: 0ffcb19 ("ir3: Rewrite register allocation")
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11422>
There were two different approaches I saw in the post-RA code for
figuring out what regiser range a relative access touched:
1. Use reg->array.offset and reg->array.size. This is wrong in case
reg->array.offset was non-zero before RA, because array.size is
the size of the whole array and array.offset has the const offset
within the array baked in.
2. Lookup the array from the array ID and use the base + range there.
This is correct, but won't work with the new RA, where an array might
not always be assigned to the same register.
This replaces both methods with a new ir3_register::array.base field,
and switches all the users I could find to it.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9842>
To simplify the pre-RA merge set code and express the result live-range
splitting in RA, we need to add support for parallel copy instructions,
and for the merge set code these parallel copies need to be in SSA form.
Parallel copies have multiple destinations by necessity, but there was
no way to express this in the existing IR. In particular there was no
support for marking a register as being a destination, and no support
for indicating which destination register out of several an SSA source
refers to. This replaces ir3_register::instr with ir3_register::def and
re-purposes ir3_register::instr. I haven't propagated this into common
helpers, like ssa(), because that would vastly increase the amount of
churn and the number of places that produce such instructions should be
limited -- only RA will create parallel copies and they will be
destroyed right after RA. In the future swz will have multiple
destinations too, but it will only be created after RA via parallel copy
lowering.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9842>
Instead of using a separate outputs array, make the "end" instruction
(or chmask) take the outputs as sources. This works better for the new
RA, because it better models the fact that outputs are consumed all at
the same time. With the old model, each output collect would be assumed
dead after it was processed and subsequent collects could use it when
inserting shuffle code, which wouldn't work, and the new RA also deletes
collect instructions after lowering them to moves so the information
would be gone after RA.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10591>
Assigning an array reg removes IR3_REG_ARRAY, which means that
definitions and uses can't be tracked back to the array register's name
and liveness for the components of the array aren't correctly
calculated. To fix this we delay assigning array registers until the
scalar pass.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7711>
Properly handle the difference between split and merged register file
when determining where arrays can fit without conflicting with other
arrays or pre-colored instructions.
1) if not mergedregs, only consider other things with same precision
as potentially conflicting
2) if mergedregs, calculate everything in therms of half-regs and
convert back to fullregs in the end
Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5957>
We shouldn't divide-by-two for half-reg arrays. We set the proper node
interference class, based on `arr->half`.
Fixes a RA fail with 16b arrays:
src/freedreno/ir3/ir3_ra.c:633: name_to_array: Assertion `!"invalid array name"' failed.
Caused by use/def iterators returning `arr->length` vreg namess, but
only assigning the array half that many names.
Also, since we are assigning unique vreg names to each array element,
there is no need to try and convert from half-reg to it's conflicting
full reg when pre-coloring the array elements. Getting us closer to
having half-arrays work sanely with split-register-file (a5xx and
earlier).
Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5957>
It doesn't happen much. But it's annoying when we hit an impossible
condition deep in RA 90% thru a long test run. Add some ra_assert()/
ra_unreachable() helper macros so we can bail cleanly and fail RA.
Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5907>
Eric Engestrom
Connor Abbott
Danylo Piliaiev
Rob Clark
Eric Anholt