r300: Keep rc_rename_regs() from overflowing RC_REGISTER_MAX_INDEX

This pass tries to move register usage closer to SSA, and for large
shaders this means we can overflow the register index, which only has
RC_REGISTER_INDEX_BITS size. This creates invalid code and leads to
crash at a later stage. Limit the pool of available registers to
RC_REGISTER_MAX_INDEX, currently is was two times the number of
shader instructions.

This means we'll fail the compile right away if we wanted more than
RC_REGISTER_MAX_INDEX temps, but when we've got that many we're
already well past how many instructions we can support anyway.

CC: mesa-stable
Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/6017
Signed-off-by: Pavel Ondračka <pavel.ondracka@gmail.com>
Reviewed-by: Emma Anholt <emma@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17393>

src/gallium/drivers/r300/compiler/radeon_rename_regs.c

@@ -29,6 +29,8 @@
  * \file
  */
 
+#include "util/u_math.h"
+
 #include "radeon_rename_regs.h"
 
 #include "radeon_compiler.h"
@@ -60,7 +62,7 @@ void rc_rename_regs(struct radeon_compiler *c, void *user)
 			return;
 	}
 
-	used_length = 2 * rc_recompute_ips(c);
+	used_length = MIN2(2 * rc_recompute_ips(c), RC_REGISTER_MAX_INDEX);
 	used = memory_pool_malloc(&c->Pool, sizeof(unsigned char) * used_length);
 	memset(used, 0, sizeof(unsigned char) * used_length);