This duplicates the lowering from nir_lower_packing. However, nir_lower_packing
also lowers a pile of other instructions that we do implement natively, and this
is easier than adding a bunch of knobs to nir_lower_packing to get just what we
need.
Fixes test-printf address_space_4.
Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18656>
Implement like other workgroup barriers. No subgroup barriers yet, but that
doesn't seem needed yet.
Fixes test_basic.async_copy_global_to_local and a pile of other OpenCL tests.
Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18656>
Array index always goes in the fourth 16-bit component on Valhall. I'm unsure
whether that should also apply to Bifrost. f256ec2a88 ("pan/bi: Fix 1DArray
image coordinate retrieval") says that it should be in the third component on
Bifrost, but I can't remember why that would be the case.
Fixes OpenCL test image_streams.write.1darray on Valhall.
Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18656>
Works around LLVM/SPIR-V stupidity. In effect this means we always use typeless
image stores, which is good enough for both CL and GL.
Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18656>
Scalarizing phis results in vector constructions (nir_op_vec) of the same size
as the phi, so a wide phi (>128-bit) will result in a wide vector op that the
backend can't handle. These wide vector ops can always be copypropped away, but
that relies on running NIR copy/prop after scalarizing phis, which was not
always happening before. By scalarizing phis before the opt loop instead of
after, we guarantee that copyprop and DCE run to completion and we get
appropriately lowered code in the backend.
Fixes parts of integer_ops.integer_divideAssign with longs.
Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18656>
Fixes test_bruteforce.isnormal. We don't implement fisnormal in the backend, but
actually lower_bool_to_bitsize was failing earlier since there's no fisnormal32
to lower to either.
Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18656>
While we have a POPCOUNT.i32 instruction, we do not have v2i16/v4i8 variants.
The code generated by lower_to_bitsize doesn't seem any better than what we
could do ourselves, so let's use that.
While we're at it, give bitfield_reverse the same treatment as we have only
BITREV.i32. I don't think we can get <32-bit bitfield_reverse in either GL or
CL, but that seems likely to change in the future. (It looks to be valid SPIR-V,
at least.)
Fixes integer_ops.popcount.
Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18656>
The word offset is already handled by the above code, there's no need to
restrict the further restrict the swizzle. This pattern can come up with OpenCL.
Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18656>
The following IR is valid NIR:
vec1 16 ssa_0 = ...
vec1 32 ssa_1 = pack_32_2x16 ssa_0.xx
In this case, pack_32_2x16 takes in a two component vector, but the source
itself ssa_0 has only a single component. This is fine due to the shuffle, but
will fail the assert. Remove the assert and all is well.
Fixes test_relational.shuffle_copy.
Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18656>
These need a simple two-instruction lowering regardless of the size of float
involved. Fixes integer_ops.integer_divideAssign
Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18656>
We don't have an 8-bit CSEL, so this is the best we can do. It's easier to write
the lowering as an algebraic rule since we don't need to do anything clever.
Fixes integer_ops.integer_clamp.
Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18656>
When we lower swizzles, we move source modifiers (except for the swizzle) after
the swizzle operation. In particular, we change the order of composition for
negates and abs. However, copying the source will copy the modifiers unless we
specifically strip the extra modifiers. That's harmless in practice on Bifrost,
which doesn't check for extraneous modifiers, but is incorrect IR and trips an
assertion in the Valhall packing code.
Fixes test_relations.relational_bitselect.
Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18656>
We're about to extend this pass to support 8-bit swizzles. That will be a
nontrivial change, so let's get some testing for what's already in the pass.
Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18656>
Fixes math_bruteforce.atan2 and contractions tests.
For OpenCL, we want to flush fp32 and preserve fp16, applying to both inputs and
outputs so F16_TO_F32 acts as preserve, which implements CL spec text:
> Denormalized numbers for the half data type which may be generated when
converting a float to a half using vstore_half and converting a half to a float
using vload_half cannot be flushed to zero
Note that our libclc builds flush denorms and rusticl does not advertise denorms
so we're expected to flush to zero. rusticl correctly sets the desired float
controls, we just have to match to the hardware requirements.
Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18656>
In NIR, txf does not take a sampler. However, in the hardware it does take a
sampler. If there is no sampler bound and we use txf, the hardware will read
back all-0's due to bounds checking. As a workaround, bind a trivial sampler and
use that.
As-is this workaround is Valhall specific, making use of an extra resource
table. I'm punting on generalizing back to Bifrost until I can discuss the issue
in more depth with Jason and Karol and figure out the right fix.
Fixes api.image_properties_query.
Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18656>
Indirect dispatch does not actually require any dynamic memory allocation, even
with shared memory. We just need to set wls_instances to some (mostly arbitrary)
value, statically allocate memory based on that, and let the hardware throttle
workgroups to fit if needed.
Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Reviewed-by: Boris Brezillon <boris.brezillon@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18661>
If memory allocation fails, we look for a suitable sized BO in the BO cache and
wait until we can use its memory. That usually works, but there's a case when it
can fail despite sufficient memory in the system: BOs in the BO cache
contributing to memory pressure but none of them being of sufficient size. This
case is not just theoretical: it's seen in the OpenCL
test_non_uniform_work_group, which puts the system under considerable memory
pressure with an unusual allocation pattern.
To handle this case, try evicting *everything* from the BO cache and stalling
in order to allocate, if the above attempts failed. Fixes the following error:
DRM_IOCTL_PANFROST_CREATE_BO failed: No space left on device
on the aforementioned OpenCL test.
Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Acked-by: Boris Brezillon <boris.brezillon@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18579>
Fix defects reported by Coverity Scan.
Resource leak (RESOURCE_LEAK)
leaked_storage: Variable used going out of scope leaks the storage it points to.
leaked_storage: Variable multiple_uses going out of scope leaks the storage it points to.
Fixes: 8fb415fee2 ("pan/bi: Reduce some moves when going out-of-SSA")
Signed-off-by: Vinson Lee <vlee@freedesktop.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18653>
This test times out occasionally. Let's disable it for now.
Reported-by: David Heidelberg <david.heidelberg@collabora.com>
Acked-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Acked-by: David Heidelberg <david.heidelberg@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18578>
We need to look at the job header pointers themselves, not the memory objects
that contain them, because there can be (and usually is) multiple jobs per BO.
Fixes: 3da8c9193c ("panfrost: Handle Job VA cycles when decoding a dump file")
Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Reviewed-by: Boris Brezillon <boris.brezillon@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18539>
This is a common pattern. Make it more ergonomic. With the help of Coccinelle:
@@
expression E;
expression R;
expression instruction;
@@
-instruction->src[E] = bi_replace_index(instruction->src[E], R);
+bi_replace_src(instruction, E, R);
Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17794>
Prior to register allocation, destinations must be in SSA form, except for the
"fake SSA" briefly used in the current RA (for which bi_is_ssa returns true
anyway). These asserts duplicate the asserts in the validator. If there's any
coverage lost here, that's just a sign the validator needs to run more often.
Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17794>
Frequently, we want to iterate over the SSA variables read by an instruction,
while skipping over constants and uniforms. Add and use a macro for this
pattern. A few redundant asserts are removed.
Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17794>
Now that variables have flat naming (no more SSA/reg distinction), we can just
use .value directly. This cleans up the RA a bit.
Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17794>
Now that non-SSA nodes are an internal implementation detail of RA, the
entrypoints should be made private to make sure no other passes use the
RA-specific liveness analysis.
Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17794>
Mike Blumenkrantz
Alyssa Rosenzweig
David Heidelberg
Vinson Lee
Thomas H.P. Andersen
Jason Ekstrand
Boris Brezillon