radv: cleanup creating the decompress/resummarize pipelines

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4389>

radv: rename extra graphics pipeline decompress/resummarize fields

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4389>

radv: rename decompress/resummarize depth/stencil functions

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4389>

turnip: fix compute shaders crashing after geometry shader change

Fixes: 1af71bee734da7d8 ("turnip: Set has_gs in ir3_shader_key")
Signed-off-by: Jonathan Marek <jonathan@marek.ca>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4483>

nir: make opt_if_loop_terminator() less strict

nir_cf_{extract,reinsert}() can't stitch a block together
if the block we are extracting ends in a jump but other jumps
nested in further ifs should be fine to move.

Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4477>

radeonsi: don't lower constant arrays to uniforms in GLSL IR

This re-enables the change made in 2f5783bc2b82 which was
incorrectly disabled by 3e1dd99adca5.

For radeonsi, we will prefer the NIR pass as it'll generate better code
(some index calculation and a single load vs. a load, then index
calculation, then another load) and oftentimes NIR optimization can kick
in and make all the access indices constant.

Fixes: 3e1dd99adca5 ("radeonsi: Remove a bunch of default handling of pipe caps.")
Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4474>

meson: fix debug build on Android

debug_stack functions are implemented in another file for Android.
Also add backtrace library dependency.

Reviewed-by: Chia-I Wu <olvaffe@gmail.com>
Reviewed-bu: Kristian H. Kristensen <hoegsber@google.com>
Signed-off-by: Dominik Behr <dbehr@chromium.org>
Acked-by: Dylan Baker <dylan@pnwbakers.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/2435>

radv: Store 64-bit availability bools if requested.

Fixes dEQP-VK.query_pool.*.reset_before_copy.* on RAVEN.

CC: <mesa-stable@lists.freedesktop.org>
Closes: https://gitlab.freedesktop.org/mesa/mesa/issues/2296
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4334>

gallivm: Add missing header for powf.

Fix build error after llvm-11 commit 3a29393b4709 ("Remove
math.h/cmath include from DataTypes.h").

src/gallium/auxiliary/gallivm/lp_bld_format_srgb.c: In function ‘lp_build_linear_to_srgb’:
src/gallium/auxiliary/gallivm/lp_bld_format_srgb.c:194:44: error: implicit declaration of function ‘powf’ [-Werror=implicit-function-declaration]
  194 |                                  exp2f_c * powf(coeff_f, 1.0f / exp_f));
      |                                            ^~~~
src/gallium/auxiliary/gallivm/lp_bld_format_srgb.c:194:44: warning: incompatible implicit declaration of built-in function ‘powf’
src/gallium/auxiliary/gallivm/lp_bld_format_srgb.c:78:1: note: include ‘<math.h>’ or provide a declaration of ‘powf’
   77 | #include "lp_bld_format.h"
  +++ |+#include <math.h>
   78 |

Signed-off-by: Vinson Lee <vlee@freedesktop.org>
Reviewed-by: Roland Scheidegger <sroland@vmware.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4473>

turnip: Drop dep_llvm from dependencies

Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4478>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4478>

turnip: Make Android platform build

We still don't have a way to keep this from breaking, but I don't
think this ever built.  Let's call it progress.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4478>

turnip: Stub out VK_KHR_external_{fence,semaphore}_fd

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4478>

turnip: Add missing VKAPI_ATTR annotations

Make sure the types match.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4478>

tracie: Reformat code to fix indentation

Signed-off-by: Rohan Garg <rohan.garg@collabora.com>
Reviewed-by: Alexandros Frantzis <alexandros.frantzis@collabora.com>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4435>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4435>

tracie: Print results in a machine readable format

Signed-off-by: Rohan Garg <rohan.garg@collabora.com>
Reviewed-by: Alexandros Frantzis <alexandros.frantzis@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4435>

freedreno/a6xx: Set a level's pitch based on minified level0 pitch, not width0.

Found from piglit fbo-generatemipmaps failures, then tracked down with the
texturator test.  The piece that really revealed things was finding that
1024x1 linear RGBA8 on the older blob drivers would have a pitch of 5120
instead of 4096, and the following levels minified that pitch.

Fixes ~124 piglit tests (~8.5% of piglit failures) on cheza.

Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3987>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3987>

freedreno: Add the outline of a test for a6xx texture layout.

Trying to work out texture layout by remembering what things looked like
in texturator is hard.  Instead, let's use texture layouts from tracing
the blob as a source of truth to make sure that we pick the same layouts
they do (and don't break known-good ones).  More testcases will be added
as I fix layout bugs.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3987>

freedreno/a6xx: Drop the "alignment" layout temporary.

It's just 1 for !3d, which means that the align we're doing in that case
is pointless.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3987>

freedreno/a6xx: Remove the "aligned_height" temporary.

Now that we're not incrementally minifying height, we can just modify it.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3987>

freedreno/a6xx: Sink the per-level size temps inside the loop.

u_minify(n, 1) is no cheaper than u_minify(n, level), and this makes the
logic a lot simpler to follow.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3987>

gitlab-ci: Run merge request pipelines automatically only for Marge Bot

MR pipelines not triggered by Marge Bot can still be triggered manually.

Motivation: The main & forked Mesa project CI pipelines combined are
currently generating over 1 TB of egress traffic per week. ~80% of this
is from pre-merge pipelines. Assuming this corresponds to 4 pre-merge
and one post-merge pipeline per MR on average, this change could
potentially eliminate up to ~60% of the overall traffic (by preventing
3 of the 4 pre-merge pipelines from running automatically).

(Of course, this could be subverted if all jobs of the other pipelines
were triggered manually anyway... In most cases, manually triggering
just a few jobs should suffice)

v2:
* $GITLAB_USER_NAME was the wrong variable, $GITLAB_USER_LOGIN should
  do the trick.

Suggested-by: Marek Olšák <maraeo@gmail.com>
Reviewed-by: Adam Jackson <ajax@redhat.com>
Acked-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4432>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4432>

gitlab-ci: Don't require triggering build/test jobs manually

Let them run automatically once all their dependencies have passed.

Reviewed-by: Adam Jackson <ajax@redhat.com>
Acked-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4432>

gitlab-ci/lava: Add needs: for container image to test jobs (again)

Without this, the test jobs could spuriously run after the container
job failed or was cancelled, even if the build job didn't run at all.

(I already did this in 94cfe590703018cf3d34a0c1f8667064919bf843, but it
got dropped accidentally in 22d976454f4e50142116f4544c0bbf11134ce991)

Reviewed-by: Adam Jackson <ajax@redhat.com>
Acked-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4432>

gitlab-ci: Rename "paths" YAML anchor to "all_paths"

To avoid confusion with `paths:` elements.

Reviewed-by: Adam Jackson <ajax@redhat.com>
Acked-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4432>

anv/gen12: Lower VK_KHR_multiview using Primitive Replication

Identify if view_index is used only for position calculation, and use
Primitive Replication to implement Multiview in Gen12.  This feature
allows storing per-view position information in a single execution of
the shader, treating position as an array.

The shader is transformed by adding a for-loop around it, that have an
iteration per active view (in the view_mask).  Stores to the position
now store into the position array for the current index in the loop,
and load_view_index() will return the view index corresponding to the
current index in the loop.

The feature is controlled by setting the environment variable
ANV_PRIMITIVE_REPLICATION_MAX_VIEWS, which defaults to 2 if unset.
For pipelines with view counts larger than that, the regular
instancing will be used instead of Primitive Replication.  To disable
it completely set the variable to 0.

v2: Don't assume position is set in vertex shader; remove only stores
    for position; don't apply optimizations since other passes will
    do; clone shader body without extract/reinsert; don't use
    last_block (potentially stale). (Jason)

    Fix view_index immediate to contain the view index, not its order.
    Check for maximum number of views supported.
    Add guard for gen12.

v3: Clone the entire shader function and change it before reinsert;
    disable optimization when shader has memory writes. (Jason)

    Use a single environment variable with _DEBUG on the name.

v4: Change to use new nir_deref_instr.
    When removing stores, look for mode nir_var_shader_out instead
    of the walking the list of outputs.
    Ensure unused derefs are removed in the non-position part of the
    shader.
    Remove dead control flow when identifying if can use or not
    primitive replication.

v5: Consider all the active shaders (including fragment) when deciding
    that Primitive Replication can be used.
    Change environment variable to ANV_PRIMITIVE_REPLICATION.
    Squash the emission of 3DSTATE_PRIMITIVE_REPLICATION into this patch.
    Disable Prim Rep in blorp_exec_3d.

v6: Use a loop around the shader, instead of manually unrolling, since
    the regular unroll pass will kick in.
    Document that we don't expect to see copy_deref or load_deref
    involving the position variable.
    Recover use_primitive_replication value when loading pipeline from
    the cache.
    Set VARYING_SLOT_LAYER to 0 in the shader.  Earlier versions were
    relying on ForceZeroRTAIndexEnable but that might not be
    sufficient.
    Disable Prim Rep in cmd_buffer_so_memcpy.

v7: Don't use Primitive Replication if position is not set, fallback
    to instancing; change environment variable to be
    ANV_PRIMITVE_REPLICATION_MAX_VIEWS and default it to 2 based on
    experiments.

Reviewed-by: Rafael Antognolli <rafael.antognolli@intel.com>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/2313>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/2313>

intel/fs: Allow multiple slots for position

Change brw_compute_vue_map() to also take the number of pos slots.  If
more than one slot is used, the VARYING_SLOT_POS is treated as an
array.

When using Primitive Replication, instead of a single position, the
VUE must contain an array of positions.  Padding might be
necessary (after clip distance) to ensure rest of attributes start
aligned.

v2: Add note about array in the commit message and assert that
    pos_slots >= 1 to make clear 0 is invalid. (Jason)
    Move padding to be after the clip distance.

v3: Apply the correct offset when gathering the sources from outputs.

Reviewed-by: Jason Ekstrand <jason@jlekstrand.net> [v2]
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Reviewed-by: Rafael Antognolli <rafael.antognolli@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/2313>

intel/gen12: Add XML description for 3DSTATE_PRIMITIVE_REPLICATION

v2: Use groups for the 16-element arrays "Viewport Offset"
    and "RTAI Offset". (Ken)

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Reviewed-by: Rafael Antognolli <rafael.antognolli@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/2313>

nir: Add per_view attribute to nir_variable

If a nir_variable is tagged with per_view, it must be an array with
size corresponding to the number of views.  For slot-tracking, it is
considered to take just the slot for a single element -- drivers will
take care of expanding this appropriately.

This will be used to implement the ability of having per-view position
in a vertex shader in Intel platforms.

Acked-by: Rafael Antognolli <rafael.antognolli@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/2313>

mesa: add support for NV_pixel_buffer_object

Signed-off-by: Simon Ser <contact@emersion.fr>
Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Reviewed-by: Ian Romanick <ian.d.romanick@intel.com>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4422>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4422>

turnip: implement timestamp query

Passes tests in:
dEQP-VK.pipeline.timestamp.*

Signed-off-by: Jonathan Marek <jonathan@marek.ca>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4027>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4027>

turnip: Enable geometryShader device feature

Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4436>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4436>

turnip: Enable geometry shaders for CP_DRAWs

Enable geometry shading on draw if the pipeline has a geometry
stage.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4436>

turnip: Populate tu_pipeline.active_stages

This can be used to determine if the pipeline has a specific shader
stage (e.g. geometry shader).

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4436>

turnip: Update maxGeometryShaderInvocations to match blob

Geometry shaders support an invocations parameter up to a limit
defined by maxGeometryShaderInvocations. This was set to 127, but
executing with invocations > 32 causes a crash. As it turns out, the
blob only advertises a max of 32 invocations, so we set that in
turnip as well.

Fixes dEQP-VK.geometry.instanced.draw_*_instances_{127, 64}_geometry_invocations

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4436>

turnip: Selectively configure GRAS_LAYER_CNTL

One of the features of geometry shaders is the ability to render to
different layers by assigning to the gl_Layer (Layer in SPIR-V)
builtin.

While have already plumbed the layer regid to the geometry shader,
we also need to GRAS_LAYER_CNTL to actually use layered rendering.
In addition, gmem does not support layered rendering, so we need to
force sysmem.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4436>

turnip: Set up REG_A6XX_SP_GS_CONFIG

Updates GS_CONFIG and HLSQ_GS_CNTL registers to match those emitted
by the blob and fd.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4436>

turnip: Configure VFD_CONTROL with gsheader and primitiveid

This commit updates VFD_CONTROL to use the GS header and primitive
ID sysvals if a geometry shader stage is present in the pipeline.
Like in the case of VPC, the code here is adapted from fd6_program.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4436>

turnip: Configure VPC for geometry shaders

This commit updates tu6_emit_vpc to selectively emit GS-specifc
configuration. Most of this is repurposed from fd6_program.c.

This also refactors `link_geometry_stages` to ir3_nir_lower_tess.c
so it can be shared between fd and tu.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4436>

turnip: Emit geometry shader obj and related consts

Like with other shader types, we need to emit the geometry shader
object and the consts it uses. In addition, we need to emit
additional geometry-specific consts that link primitive/vertex stride
between the vs and gs. In conjunction with the gsheader, these are
used by the vs to determine where to stlw outputs and used by the gs
to determine where to ldlw those outputs from.

FD emits these consts in the draw call because in GL, you can mix
and match shaders in different programs. In Vulkan, however, we
compile and link the shaders at pipeline creation, so we can emit
these in the pipeline IB instead.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4436>

turnip: Set has_gs in ir3_shader_key

The ir3 compiler only lowers the VS and GS for geometry shading if
the corresponding has_gs key is set in the shader key. Without it,
GS-specific intrinsics like load_per_vertex_input won't get lowered
and the GS header will be initialized with invalid values.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4436>

radv: Print shader stage before disassembly.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3576>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3576>

aco: Print shader stage in aco_print_program.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3576>

radv: Enable ACO for NGG VS/TES, but disable NGG for ACO GS.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3576>

aco/ngg: Run GS_ALLOC_REQ on priority 3 for NGG VS and TES.

It is recommended to do this as quickly as possible.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3576>

aco/ngg: Schedule position exports of NGG VS/TES.

Similarly to the HW VS stage, the HW NGG GS stage also
benefits from executing these exports as early as possible.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3576>

aco/ngg: Implement NGG VS and TES.

When NGG is used, vertex and tess eval shaders are executed on the
hardware NGG geometry stage. There is a series of steps they
must perform:

* Request GS space using GS_ALLOC_REQ
* Export the primitive
* Finally, export the normal VS outputs

In this commit, two modes are implemented:

* "late" which matches what the RADV LLVM backend currently does
* "early" which is an optimized version as seen in radeonsi

Vulkan doesn't allow the shader to write the edge flags, so we can
currently always use the "early" mode.

Exporting the primitive ID is also supported by having the GS threads
write that into LDS and reading them from LDS in the ES threads.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3576>

aco/ngg: Setup NGG VS and TES stages.

ngg_vertex_gs and ngg_tess_eval_gs work very similarly to
vertex_vs and tess_eval_vs, but they run on the HW NGG GS stage.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3576>

aco/ngg: Fix exports for NGG VS and TES.

The exports in NGG VS and TES work just like VS exports,
so the assembler needs to fix these too in the same manner.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3576>

aco/ngg: Initialize exec mask for NGG VS and TES.

They behave like merged ESGS shaders, so the exec mask needs
to be manually initialized for these NGG shaders too.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3576>

aco/ngg: Add new stage for hw_ngg_gs.

This is needed to distinguish between NGG and legacy.
Otherwise, vertex_geometry_gs and ngg_vertex_geometry_gs
have the same value, which we want to avoid.

Also, there is no such thing as ngg_vertex_tess_control_hs.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3576>

aco: Treat s_setprio as a scheduling barrier.

We want to execute instructions after s_setprio in the given
priority, so we must prevent the scheduler from scheduling beyond
s_setprio, otherwise some instructions could be executed in a
different priority.

Rename hazard_fail_memtime to hazard_fail_unreorderable and include
s_setprio in the list of unreorderable opcodes.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3576>

aco: Extract merged_wave_info_to_mask to its own function.

Currently we only use this at the beginning of merged shader parts,
but we are going to need to use it with some NGG code as well.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3576>

aco: Print block_kind_export_end.

Useful when debugging issues with exports.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3576>

aco: Extract uniform if handling to separate functions.

Currently we only use this for uniform ifs that come from NIR,
but we are going to need to use it with some NGG parts as well.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3576>

aco: Fix crash in insert_wait_states.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4465>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4465>

pan/bit: Wire up add/add op+test

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4470>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4470>

pan/bit: Add fmin/max16 tests

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4470>

pan/bit: Enable more debug for `run`

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4470>

pan/bit: Add min/max support to interpreter

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4470>

pan/bit: Unify test frontends

Random.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4470>

pan/bi: Force ADD scheduling for MINMAX

Might be GPU version specific.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4470>

pan/bi: Fix incorrect abs flip in fma/fadd16

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4470>

pan/bi: Set BI_MODS for MINMAX

We support it.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4470>

pan/bi: Add ADD add/min/max fp32 packing

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4470>

pan/bi: Structify ADD unit add/min/max

..since it's missing for FMA

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4470>

pan/bi: Implement min/max on FMA

Unfortunately, while this looks fine to the disasm, it's raising
INSTR_INVALID_ENC on my g31 board here. Looks like it might be ADD only
on newer Bifrost.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4470>

pan/bit: Add special unit test

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4470>

pan/bit: Add special op interpreting

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4470>

pan/bi: Add fp16 support for frcp/frsq

More ops.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4470>

pan/bi: Add 32-bit _FAST packing

For frcp/frsq on newer Bifrost.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4470>

pan/bi: Remove nontrivial SPECIAL ops

These require a lot more handholding in the IR than we can deal with at
this stage; we need to restrict ourselves to frcp/sqrt.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4470>

aco: zero-initialize Temp

Fixes dEQP-VK.transform_feedback.* crashes from accesses garbage
temporaries in emit_extract_vector().

Fixes: 85521061 ("aco: prepare helper functions for subdword handling")
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4463>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4463>

aco: remove divergence check in sanitize_if()

We also need to do this if a side ends in a divergent break.

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4461>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4461>

nir/lower_amul: fix slot calculation

Fixes incorrect indexing in
dEQP-GLES31.functional.ssbo.layout.instance_array_basic_type.packed.mat2x3

Signed-off-by: Rob Clark <robdclark@chromium.org>
Reviewed-by: Eric Anholt <eric@anholt.net>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4455>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4455>

nir: add some swizzle helpers

Signed-off-by: Rob Clark <robdclark@chromium.org>
Reviewed-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Reviewed-by: Eric Anholt <eric@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4455>

nir: Assert memory loads are aligned

We've had alignment parameters on these operations for a while but a
bunch of places weren't setting them.  That should be resolved now so we
can start validating that they're always set.

Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4441>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4441>

util: move and adjust the vertex upload heuristic equation from u_vbuf

This will also be used by glthread.

The new equation is optimized for glthread.

Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4466>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4466>

vbo: expose helper function vbo_get_minmax_index_mapped for glthread

Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4466>

mesa: split _mesa_primitive_restart_index into a function without gl_context

Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4466>

mesa: precompute _mesa_primitive_restart_index during state changes

Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4466>

mesa: remove no longer needed _mesa_is_bufferobj function

All buffers have Name != 0.

Note that there is no longer the pointer dereference to get Name,
so it's faster.

Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4466>

mesa: remove NullBufferObj

Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4466>

mesa: don't ever bind NullBufferObj to glBindBuffer(Base,Range) slots

Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4466>

mesa: don't ever bind NullBufferObj for glBindBuffer targets

Since VAOs don't use NullBufferObj for vertex attribs anymore, let's remove
more uses of NullBufferObj.

Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4466>

mesa: don't ever set NullBufferObj in gl_vertex_array_binding

This improves performance by 5% in the game "torcs", FPS: 98.83 -> 103.73

It does a lot of glPush/PopClientAttrib, which exacerbates the overhead
of setting NullBufferObj.

Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4466>

mesa: optimize initialization of new VAOs

Precompute the default state in gl_context, and just copy it when we create
a VAO.

This also helps glPushClientAttrib function, which always creates a VAO,
which has a substantial CPU overhead in profiles.

Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4466>

android: aco: add various compiler statistics

Fixes a building error due to compiler/aco_statistics.cpp
missing in src/amd/Makefile.sources

Fixes: b1544352 ("aco: add various compiler statistics")
Signed-off-by: Mauro Rossi <issor.oruam@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>

nir: fix wrong assignment to buffer in xfb_varyings_info

Tested with dEQP-VK.transform_feedback.fuzz.various_buffers.buffers100_instance_array_vertex

Signed-off-by: Hyunjun Ko <zzoon@igalia.com>
Reviewed-by: Tapani Pälli <tapani.palli@intel.com>
Cc: mesa-stable@lists.freedesktop.org
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4459>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4459>

mesa/st: release variants for active programs before unref

Programs can be shared among many contexts and each program holds a
variant list which has context specific variants. When context gets
destroyed it must make sure it relases all variants, otherwise remaining
context that utilizes same program will attempt to save a zombie shader
for already deleted context when releasing program and its variants.

Fixes:
   dEQP-EGL.functional.sharing.gles2.program.render

and other flaky multihread dEQP-EGL failures.

v2: pass program pointer via & (Marek)

Signed-off-by: Tapani Pälli <tapani.palli@intel.com>
Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
Cc: mesa-stable@lists.freedesktop.org
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4386>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4386>

mesa/st: unbind shader state before deleting it

Signed-off-by: Tapani Pälli <tapani.palli@intel.com>
Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
Cc: mesa-stable@lists.freedesktop.org
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4386>

pan/bit: Add mode to run unit tests

Probably the most useful of the bunch going forward.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4458>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4458>

pan/bit: Make run more useful

..by printing some output.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4458>

pan/bit: Add csel tests

..and pull out common instruction generation to reduce duplication in
tests a bit.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4458>

pan/bit: Add CSEL to interpreter

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4458>

pan/bit: Add FMA tests

Now that the earlier reg ctrl issue is fixed these should pass.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4458>

pan/bit: Add 16-bit fmod tests

These raise another set of issues -- indeed, not all of these tests are
passing, since it turns out I have an actual bug in the packing code. So
after all this work, test bringup has identified an actual issue :)

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4458>

pan/bit: Add verbose printing for tests

We'd like to dump both the generated IR (so we know exactly what's being
tested) as well as the compiled program (so we know what's running for
comparison).

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4458>

pan/bit: Add helper for generating floating mod tests

We can iterate them, isn't that adorable!

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4458>

pan/bit: Add packing test framework

Given an instruction, we'd like to wrap it in a clause with some I/O on
each end so we can pack it up and send it to the hardware to compare
against the simulator.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4458>

pan/bit: Implement floating source mods

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4458>