glx: use anonymous namespace to avoid -Wodr issues when building with LTO enabled

Signed-off-by: Eric Engestrom <eric@engestrom.ch>
Reviewed-by: Adam Jackson <ajax@redhat.com>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/2597>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/2597>

glx: fix 630 times -Wlto-type-mismatch when building with LTO enabled

The prototypes are simply copied from include/GL/gl*.h

Signed-off-by: Eric Engestrom <eric@engestrom.ch>
Reviewed-by: Adam Jackson <ajax@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/2597>

Revert "spirv: Rewrite CFG construction"

This reverts commit fa5a36dbd474fb3c755da51553c6ca18dab76a06.

Revert "gallivm: disable rgtc/latc SNORM accellerated fetches"

This reverts commit 4897e70ccd3987d470ec8622d473ee3405f6e96f.

Fixed in previous commits.

Reviewed-by: Roland Scheidegger <sroland@vmware.com>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4425>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4425>

gallivm/rgtc: enable fast path for snorm types.

As per Roland's suggestions it should be easy to enable the
fast path fetch for rgtc snorm as well here.

Reviewed-by: Roland Scheidegger <sroland@vmware.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4425>

gallivm/rgtc: fix the truncation to 8-bit

The 8 bit type wasn't 8-bit so when doing signed work we lost
the sign bit. This fixes it to use a proper vector type,
even if we just end up in here with the 1-wide path for now.

Reviewed-by: Roland Scheidegger <sroland@vmware.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4425>

glsl: don't limit fp16 lowering to frag

This restriction doesn't belong in core code.

Signed-off-by: Rob Clark <robdclark@chromium.org>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4423>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4423>

freedreno: limit fp16 to frag and compute

To avoid dealing w/ some backend varying linking issues when mixing
precision vs geom/tess.

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4423>

freedreno/ir3: also precompile compute shaders for shaderdb

Similar as with draw shaders, nothing will trigger the final variant in
shader-db.

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4423>

freedreno: fix missing locking

Fixes: d0b3ccb0607 ("freedreno: Fix detection of being in a blit for acc queries.")
Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4423>

freedreno/a6xx: add some compute logging

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4423>

freedreno/ir3/cf: use ssa-uses

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4423>

freedreno/ir3: add a pass to collect SSA uses

We don't really track these as the ir is transformed, but it would be a
useful thing for some passes to have.  So add a pass to collect this
information.  It uses instr->data (generic per-pass ptr), with the
hashsets hanging under a mem_ctx for easy disposal at the end of the
pass.

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4423>

freedreno/ir3/cf: skip array load/store

Don't fold conversions into array (incl phi lowered to regs/array).
These aren't SSA.  Avoids crashes in particular in frag shaders with
flow control, which would leave a dangling array write disconnect from
the original cov src.

Possibly this could be slightly relaxed, if there is no other consumer
of the src, and it were in the same block.  But it would require
updating block->keeps, and taking care of barrier state.  Which isn't a
thing the cf pass does currently.

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4423>

freedreno/ir3: fixup cat3 32b vs 16b

These should be keyed on src arg type.

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4423>

freedreno/ir3/cf: handle widening too

We can also fold f16->f32 conversions.

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4423>

nir: fix definition of imadsh_mix16 for vectors

Fixes: c27b3758fa0 ("nir/opcodes: Add new 'umul_low' and 'imadsh_mix16' opcodes")
Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4423>

aco: use MUBUF to load subdword SSBO

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-By: Timur Kristóf <timur.kristof@gmail.com>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4002>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4002>

aco: implement 8bit/16bit store_ssbo

Currently without alignment check, so that
we can only use the _byte and _short versions
and multi-component stores are split.

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-By: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4002>

aco: implement 8bit/16bit load_buffer

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-By: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4002>

aco: implement storagePushConstant8 & storagePushConstant16

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-By: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4002>

aco: implement vec2/3/4 with subdword operands

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-By: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4002>

aco: prepare helper functions for subdword handling

- get_alu_src()
- emit_extract_vector()
- emit_split_vector()

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-By: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4002>

aco: add byte_align_scalar() & trim_subdword_vector() helper functions

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-By: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4002>

aco: add missing conversion operations for small bitsizes

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-By: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4002>

aco: don't vectorize 8/16bit load/store_ssbo

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-By: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4002>

aco: don't assume split_vector(create_vector) has the same number of elements when optimizing

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-By: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4002>

aco: don't propagate SGPRs into subdword PSEUDO instructions

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-By: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4002>

aco: lower subdword shuffles correctly.

Note that subdword swaps are not yet implemented

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-By: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4002>

aco: add builder function for subdword copy()

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-By: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4002>

aco: small refactoring of shuffle code lowering

Uses now bytes instead of 32bit size

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-By: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4002>

aco: align subdword registers during RA when necessary

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-By: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4002>

aco: adapt register allocation for subdword registers

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-By: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4002>

aco: create helper function to collect variables from register area

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-By: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4002>

aco: add notion of subdword registers to register allocator

To not having to split the register file into single bytes,
we maintain a map with registers which contain subdword variables.

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-By: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4002>

aco: remove unnecessary reg_file.fill() operation in get_reg_create_vector()

No pipelinedb changes

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-By: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4002>

aco: fix Temp and assignment of renamed operands during RA

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-By: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4002>

aco: print subdword registers

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-By: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4002>

aco: validate RA of subdword assignments

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-By: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4002>

aco: validate uninitialized operands

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-By: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4002>

aco: validate register alignment of subdword operands and definitions

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-By: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4002>

aco: validate p_create_vector with subdword elements properly

Also allows for undef operands

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-By: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4002>

aco: refactor regClass setup for subdword VGPRs

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-By: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4002>

aco: add emission support for register-allocated sdwa sels

Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Reviewed-By: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4002>

aco: add sub-dword regclasses

Co-authored-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-By: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4002>

aco: print and validate opsel

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Reviewed-By: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4002>

aco: add SDWA_instruction

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Reviewed-By: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4002>

aco: add comparison operators for PhysReg

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-By: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4002>

aco: make PhysReg in units of bytes

Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Reviewed-By: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4002>

nir: fix unpack_64_4x16 in lower_alu_to_scalar()

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-By: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4002>

drm-shim: stub libdrm's use of realpath()

libdrm started using realpath to get the type of bus associated with a
given device. This stubs the very specific usage that prevents
drm-shim's device from being listed.

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Eric Anholt <eric@anholt.net>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4429>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4429>

drm-shim: return device platform as specified

v2: Embed the libdrm dependency inside the drm-shim dependency

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Acked-by: Eric Anholt <eric@anholt.net> (v1)
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4429>

spirv: Rewrite CFG construction

This commit completely rewrites the way we extract a structured CFG from
SPIR-V.  The new approach is different in a few ways:

 1. It does a breadth-first search instead of depth-first.  This means
    that we've visited the merge node for a construct before we visit
    any of the nodes inside the construct.  This makes it easier to
    validate things like loop and switch nesting.

 2. We record more information in the CFG.  Earlier commits added a
    parent pointer to vtn_cf_node but we now record all of the merge and
    other special blocks for each CFG node.  This lets us validate
    things more precisely.

 3. It makes heavy use of merge blocks for walking the CFG.  Previously,
    we sort of used them as hints for trying to guess the CFG structure
    but things got dicey whenever a merge was missing.  We had some
    heuristics for how to handle short-circuiting if statements but it
    was a bunch of special cases.

    Now, we make them a fundamental part of walking the CFG.  When we
    encounter a control-flow construct, we add the body components of
    the construct to the BFS work list and then jump to the merge block
    if one exists to continue scanning the current CFG nesting level.
    If no merge block exists, we assume that means that control-flow
    never re-converges in a normal way and that the only way to get back
    to normality is with a direct jump such as a loop break or continue.
    This should make things far more robust when trying to deal with the
    more creative placement (or lack thereof) of merge instructions.

Reviewed-by: Alan Baker <alanbaker@google.com>
Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3820>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3820>

spirv: Add a parent field to vtn_cf_node

This makes it easier to crawl up the CF tree when trying to validate the
incoming SPIR-V control-flow.

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3820>

spirv: Make vtn_function a vtn_cf_node

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3820>

spirv: Make vtn_case a vtn_cf_node

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3820>

spirv: Add cast and loop helpers for vtn_cf_node

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3820>

spirv: Add a vtn_block() helper

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3820>

intel/nir: Enable load/store vectorization

This commit enables the I/O vectorization pass that was originally
written for ACO for Intel drivers.  We enable it for UBOs, SSBOs, global
memory, and SLM.  We only enable vectorization for the scalar back-end
because it vec4 makes certain alignment assumptions.

Shader-db results with iris on ICL:

    total instructions in shared programs: 16077927 -> 16068236 (-0.06%)
    instructions in affected programs: 199839 -> 190148 (-4.85%)
    helped: 324
    HURT: 0
    helped stats (abs) min: 2 max: 458 x̄: 29.91 x̃: 4
    helped stats (rel) min: 0.11% max: 38.94% x̄: 4.32% x̃: 1.64%
    95% mean confidence interval for instructions value: -37.02 -22.80
    95% mean confidence interval for instructions %-change: -5.07% -3.58%
    Instructions are helped.

    total cycles in shared programs: 336806135 -> 336151501 (-0.19%)
    cycles in affected programs: 16009735 -> 15355101 (-4.09%)
    helped: 458
    HURT: 154
    helped stats (abs) min: 1 max: 77812 x̄: 1542.50 x̃: 75
    helped stats (rel) min: <.01% max: 34.46% x̄: 5.16% x̃: 2.01%
    HURT stats (abs)   min: 1 max: 22800 x̄: 336.55 x̃: 20
    HURT stats (rel)   min: <.01% max: 17.11% x̄: 2.12% x̃: 1.00%
    95% mean confidence interval for cycles value: -1596.83 -542.49
    95% mean confidence interval for cycles %-change: -3.83% -2.82%
    Cycles are helped.

    total sends in shared programs: 814177 -> 809049 (-0.63%)
    sends in affected programs: 15422 -> 10294 (-33.25%)
    helped: 324
    HURT: 0
    helped stats (abs) min: 1 max: 256 x̄: 15.83 x̃: 2
    helped stats (rel) min: 1.33% max: 67.90% x̄: 21.21% x̃: 15.38%
    95% mean confidence interval for sends value: -19.67 -11.98
    95% mean confidence interval for sends %-change: -23.03% -19.39%
    Sends are helped.

    LOST:   7
    GAINED: 2

Most of the helped shaders were in the following titles:

 - Doom
 - Deus Ex: Mankind Divided
 - Aztec Ruins
 - Shadow of Mordor
 - DiRT Showdown
 - Tomb Raider (Rise, I think)

Five of the lost programs are SIMD16 shaders we lost from dirt showdown.
The other two are compute shaders in Aztec Ruins which switched from
SIMD8 to SIMD16.

Vulkan pipeline-db stats on ICL:

    Instructions in all programs: 296780486 -> 293493363 (-1.1%)
    Loops in all programs: 149669 -> 149669 (+0.0%)
    Cycles in all programs: 90999206722 -> 88513844563 (-2.7%)
    Spills in all programs: 1710217 -> 1730691 (+1.2%)
    Fills in all programs: 1931235 -> 1958138 (+1.4%)

By far the most help was in the Tomb Raider games.  A couple of Batman
games with DXVK were also helped.  In Shadow of the Tomb Raider:

    Instructions in all programs: 41614336 -> 39408023 (-5.3%)
    Loops in all programs: 32200 -> 32200 (+0.0%)
    Cycles in all programs: 1875498485 -> 1667034831 (-11.1%)
    Spills in all programs: 196307 -> 214945 (+9.5%)
    Fills in all programs: 282736 -> 307113 (+8.6%)

Benchmarks of real games I've done on this patch:

 - Rise of the Tomb Raider: +3%
 - Shadow of the Tomb Raider: +10%

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Reviewed-by: Ian Romanick <ian.d.romanick@intel.com>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4367>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4367>

nir/load_store_vectorize: Add support for nir_var_mem_global

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4367>

nir/load_store_vectorize: Use nir_iadd_imm for offsets

This makes it capable of handling 64-bit offsets

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4367>

nir/load_store_vectorize: Fix shared atomic info

These were clearly copied and pasted from SSBOs.  The shared atomics
don't have an SSBO index so their offset is src0 and data is src1.

Fixes: ce9205c03bd20 "nir: add a load/store vectorization pass"
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4367>

intel/nir: Lower memory access bit sizes later

We're about to do load/store vectorization right before this but we need
that to happen after we've done a round of optimization.  Otherwise,
we'll be getting unoptimized NIR in from ANV and the vectorizer won't be
able to do anything with it.

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Reviewed-by: Ian Romanick <ian.d.romanick@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4367>

iris: Set alignments on cbuf0 and constant reads

Reviewed-by: Ian Romanick <ian.d.romanick@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4367>

anv: Improve brw_nir_lower_mem_access_bit_sizes

This commit makes us take both bit size and alignment into account so
that we can properly handle cases such as when we have a 32-bit store
to an 8-bit-aligned address.

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Reviewed-by: Ian Romanick <ian.d.romanick@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4367>

intel/fs: Choose memory message type based on bit size

Thanks to the NIR vectorizing pass, we're about to see alignments that
are higher than the bit size.  Previously, we could use either and we
just happened to choose alignment (probably the wrong choice) so it's
harmless to switch to detecting based on bit size.  This commit changes
things to take both into account which is more accurate to what the
messages we're using do.  We also beef up the asserts and make them more
consistent, more accurate, and more complete.

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Reviewed-by: Ian Romanick <ian.d.romanick@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4367>

ir3: Disable copy prop for immediate ldlw offsets

Immediate offsets are currently collapsed for ldlw, but ldlw does
behave correctly with immediate values. For example,
`ldlw.u32 r0.x, l[4], 1` actually means to use the value of
regid 4 (r1.x) as the offset when we actually want it to use the
imm value of 4 as the offset.

This commit disables copy prop for ldlw offsets so the same
intrinsic gets compiled to:
  mov.u32u32 r0.y, 0x00000004
  ldlw.u32 r0.x, l[r0.y], 1

Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4439>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4439>

radv: fix null winsys gpu_info array

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Fixes: de550805c5d ('radv/winsys: spoof some values for num_render_backends in the null winsys')
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4437>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4437>

pan/midgard: Fix a divide by zero in emit_alu_bundle

util_dynarray_grow_bytes divides by eltsize, but it's possible for
bundle->padding to be zero.

I changed the other call to util_dynarray_grow_bytes for consistency.

Reviewed-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4397>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4397>

turnip: Advertise 8 bit subpixel precision

Previously, turnip advertised 4-bit subpixel precision when in
practice, a6xx seems to render with 8-bit precision. This caused
dEQP-VK.renderpass2.suballocation.subpass_dependencies.late_fragment_tests.*
to fail because they compare images rendered with turnip against
ones rendered via a software reference implementation parameterized
by turnip's VkPhysicalDeviceLimits.subPixelPrecisionBits value.

Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4172>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4172>

mesa: update pipeline when re-linking a program in use

Updating was only done for bound program, so add the
same logic for existing pipelines.

This fixes piglit test arb_shader_storage_buffer_object-issue1258.
It might also help the following issue:
  https://gitlab.freedesktop.org/mesa/mesa/-/issues/1258

Reviewed-by: Timothy Arceri <tarceri@itsqueeze.com>
Acked-by: Marek Olšák <marek.olsak@amd.com>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4404>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4404>

nv50: don't try to upload MSAA settings for BUFFER textures

We need the MSAA scaling parameters to properly fetch samples from MSAA
textures. These are stored in the miptree which wraps all regular
textures. However it does not wrap buffer textures, so make sure to skip
them rather than accessing out-of-bounds or unmapped memory.

Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/2727
Fixes: 3bd40073b98 ("nv50: add support for texelFetch'ing MS textures")
Signed-off-by: Ilia Mirkin <imirkin@alum.mit.edu>
Cc: mesa-stable@lists.freedesktop.org
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4424>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4424>

intel/aub_viewer: fix access to freed memory

Windows closed while we're displaying them might lead to invalid
memory accessed, so use the safe iterators on the list of windows.

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Tapani Pälli <tapani.palli@intel.com>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4430>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4430>

radv, aco: collect statistics if requested but executables are not

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/2965>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/2965>

aco: add vmem/smem score statistic

This isn't perfect (for example, changes might not be too meaningful when
comparing shaders with different control flow) but it should be useful for
evaluating scheduler changes.

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Acked-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/2965>

aco: add various compiler statistics

Adds these statistics:
- hash of code and constant data
- number of instructions
- number of copies from pseudo-instructions
- number of branches
- estimate of cycles spent not waiting in s_waitcnt
- number of vmem/smem "clauses"
- sgpr/vgpr usage before scheduling

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Acked-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/2965>

radv: add code for exposing compiler statistics

Statistics will be added to ACO in later commits.

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/2965>

EGL: Add eglSetDamageRegionKHR to GLVND dispatch list

This was missed in the original conversion, which added support for
eglSetDamageRegionKHR to local EGL exports, but forgot to generate
updated dispatch for GLVND.

Signed-off-by: Daniel Stone <daniels@collabora.com>
Reviewed-by: Emil Velikov <emil.velikov@collabora.com>
Reviewed-by: Eric Engestrom <eric@engestrom.ch>
Fixes: 9827547313c7 ("egl/android: support for EGL_KHR_partial_update")
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4403>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4403>

docs: update calendar, add news item, and link releases notes for 20.0.4

Note that the next 20.0.x releases numbers have been shifted as this was
not one of the planned releases.

Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4428>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4428>

docs/relnotes: add sha256sum for 20.0.4

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4428>

docs: add release notes for 20.0.4

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4428>

util/xmlconfig: fix sha1 comparison code

Fixes: 8f48e7b1e99 ("util/xmlconfig: add new sha1 application attribute")
Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/2730
Reviewed-by: Dave Airlie <airlied@redhat.com>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4426>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4426>

radv/llvm: enable 16-bit storage features on GFX6-GFX7

Should allow to play Doom Eternal on GFX6-GFX7 because the
driver now supports storageBuffer16BitAccess.

It's now supported and all CTS tests pass.

Closes: https://gitlab.freedesktop.org/mesa/mesa/issues/857
Cc: 20.0 <mesa-stable@lists.freedesktop.org>
Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4339>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4339>

ac/nir: split 16-bit SSBO stores on GFX6

Due to possible alignment issues, make sure to split stores of
16-bit vectors.

Doom Eternal requires storageBuffer16BitAccess.

Cc: 20.0 <mesa-stable@lists.freedesktop.org>
Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4339>

ac/nir: split 16-bit load/store to global memory on GFX6

Due to possible alignment issues, make sure to split loads/stores
of 16-bit vectors.

Doom Eternal requires storageBuffer16BitAccess.

Cc: 20.0 <mesa-stable@lists.freedesktop.org>
Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4339>

radv/llvm: enable 8-bit storage features on GFX6-GFX7

It's now supported and all CTS tests pass.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4339>

ac/nir: split 8-bit SSBO stores on GFX6

Due to possible alignment issues, make sure to split stores of
8-bit vectors.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4339>

ac/nir: split 8-bit load/store to global memory on GFX6

Due to possible alignment issues, make sure to split loads/stores
of 8-bit vectors.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4339>

aco: always optimize v_mad to v_madak in presence of literals

v_mad and v_madak are both 64-bit instructions, so it doesn't
increase code size to always apply a 32-bit literal instead of
using v_mad and a sgpr which contains that literal.

Found with some Youngblood shaders but help some other games.

vkpipeline-db (VEGA10):
Totals from affected shaders:
SGPRS: 46168 -> 46016 (-0.33 %)
VGPRS: 45576 -> 45564 (-0.03 %)
Code Size: 5187208 -> 5179584 (-0.15 %) bytes
Max Waves: 3297 -> 3297 (0.00 %)

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4410>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4410>

glsl/lower_precision: Use vector.back() instead of vector.end()[-1]

The use of vector.end()[-1] seems to generate warnings in Coverity about
not allowing a negative argument to a parameter. The intention with the
code snippet is just to access the last element of the vector. The
vector.back() call acheives the same thing, is clearer and will
hopefully fix the Coverity warning.

I’m not exactly sure why Coverity thinks the array index can’t be
negative. cplusplus.com says that vector::end() returns a random access
iterator and that the type of the array index operator argument to that
should be the difference type for the container. It then also says that
difference_type for a vector is "a signed integral type".

Reviewed-by: Eric Anholt <eric@anholt.net>

clover: fix build with single library clang build

Closes: #2560
Signed-off-by: Karol Herbst <kherbst@redhat.com>
Reviewed-by: Michel Dänzer <mdaenzer@redhat.com>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4417>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4417>

radv: Filter extensions not whitelisted for Android

Android enforces through CTS a whitelist of Vulkan extensions that are
allowed in each Android version. When building radv for Android, disable
extensions that are unknown to the version of Android for which
radv is being built.

Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4398>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4398>

st/vdpau: make query test for 2D support

The 3D check has been there since the dawn of time, but I see no reason
for it, most likely a typo. When the surfaces are actually created, they
use the 2D resource type (as expected).

Signed-off-by: Ilia Mirkin <imirkin@alum.mit.edu>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4108>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4108>

st/vdpau: avoid asserting on new VDP_YCBCR_* formats

Depending on user's vdpau headers, not all of those defines may exist.
Eventually we may want a private copy of these, but this is simple
enough for now.

Fixes asserts when running vdpauinfo which supports these recently added
formats.

Signed-off-by: Ilia Mirkin <imirkin@alum.mit.edu>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4108>

nir/from_ssa: Only chain movs when a src is also a dest

The algorithm we use for resolving parallel copy instructions plays this
little shell game with the values.  The reason for this is that it lets
us handle cases where, for instance we have a -> b and b -> a and we
need to use a temporary to do a swap.  One result of this algorithm is
that it tends to emit a lot of mov chains which are typcially really bad
for GPUs where a mov is far from free.  For instance, it's likely to
turn this:

    r16 = ssa_0; r17 = ssa_0; r18 = ssa_0; r15 = ssa_0

into this:

    r15 = mov ssa_0
    r18 = mov r15
    r17 = mov r18
    r16 = mov r17

which, if it's the only thing in a block (this is common for phis) is
impossible for a scheduler to fix because of the dependencies and you
end up with significant stalling.  If, on the other hand, we only do the
chaining in the actual case where we need to free up a so that it can be
used as a destination, we can emit this:

    r15 = mov ssa_0
    r18 = mov ssa_0
    r17 = mov ssa_0
    r16 = mov ssa_0

which is far nicer to the scheduler.  On Intel, our copy propagation
pass will undo the chain for us so this has no shader-db impact.
However, for less intelligent back-ends, it's probably a lot better.

Reviewed-by: Rob Clark <robdclark@chromium.org>
Reviewed-by: Connor Abbott <cwabbott0@gmail.com>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4412>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4412>

freedreno: Rename RB_DONE_TS

This makes the various cache_flush implementations make more sense.

Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4065>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4065>

freedreno: Cleanup event names

It turns out that every *_TS event, i.e. every event which requires a
seqno pointer, also allows generating an interrupt in the kernel, at
least since a3xx. And furthermore these interrupts are named by the kgsl
kernel driver and already in envytools. Therefore it's possible to map
out what the *_TS events are with 100% certainty, given access to the
hardware, by sending a CP_EVENT_WRITE with bit 31 set, unmasking all
interrupts in the kernel, and logging which ones get hit. I've done this
for a6xx, and I've also looked at the a5xx firmware, and the list of TS
interrupts is the same as a6xx, so I have a pretty good idea of what the
a5xx events are. I also fixed a few related things along the way:

- VIZQUERY_END overlaps with WT_DONE_TS, but VIZQUERY_START was also a
mess, with neither VIZQUERY_START nor HLSQ_FLUSH using variants. I added
what seems like reasonable variants, based on the existing comment
and the fact that HLSQ_FLUSH is only used in Mesa with a3xx and a4xx.
- CACHE_FLUSH_AND_INVALIDATE seems to come straight from R600, and I
have no idea if it's actually valid with a2xx, but given that RB_DONE_TS
exists in the interrupt mask since a3xx, I guessed that RB_DONE_TS
hasn't changed position since then and put it down as a3xx+ and limited
CACHE_FLUSH_AND_INVALIDATE to a2xx. Someone with the relevant hardware
should be able to confirm.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4065>

gallivm: fix stream id fetch

Fetching the stream id directly can crash since bld->immediates may not
exist (if there's too many immediates or we use the array due to indirect
accesses). So just call emit_fetch_immediate instead.

v2: fix the swizzle

Reviewed-by: Dave Airlie <airlied@redhat.com>
Reviewed-by: Brian Paul <brianp@vmware.com>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4416>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4416>

gallivm: switch the mask6/mask7 cases for signed rgtc formats

This fixes some regressions where -1.0/1.0 results got flipped, but it's still
broken in some cases.

Reviewed-by: Dave Airlie <airlied@redhat.com>
Reviewed-by: Brian Paul <brianp@vmware.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4416>

gallivm: fix rgtc2 format

In some cases, there can be garbage in the upper bits after the channel
decode - for dxt5 this didn't matter (as the upper bits are shifted out
anyway) but for rgtc2 formats it does.

Reviewed-by: Dave Airlie <airlied@redhat.com>
Reviewed-by: Brian Paul <brianp@vmware.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4416>