intel/fs: Only stall after sending all memory fence messages

In Gen11+, when emitting a fence for both L3 and SLM, the generated
code would look like

    SEND, MOV (for stall), SEND, MOV (for stall)

This commit change that so two SENDs are emitted before the MOVs for
stall.  This is similar to the approach used in Ivy Bridge for the
render fence.

Reviewed-by: Francisco Jerez <currojerez@riseup.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3278>

intel/fs,vec4: Pull stall logic for memory fences up into the IR

Instead of emitting the stall MOV "inside" the
SHADER_OPCODE_MEMORY_FENCE generation, use the scheduling fences when
creating the IR.

For IvyBridge, every (data cache) fence is accompained by a render
cache fence, that now is explicit in the IR, two
SHADER_OPCODE_MEMORY_FENCEs are emitted (with different SFIDs).

Because Begin and End interlock intrinsics are effectively memory
barriers, move its handling alongside the other memory barrier
intrinsics.  The SHADER_OPCODE_INTERLOCK is still used to distinguish
if we are going to use a SENDC (for Begin) or regular SEND (for End).

This change is a preparation to allow emitting both SENDs in Gen11+
before we can stall on them.

Shader-db results for IVB (i965):

    total instructions in shared programs: 11971190 -> 11971200 (<.01%)
    instructions in affected programs: 11482 -> 11492 (0.09%)
    helped: 0
    HURT: 8
    HURT stats (abs)   min: 1 max: 3 x̄: 1.25 x̃: 1
    HURT stats (rel)   min: 0.03% max: 0.50% x̄: 0.14% x̃: 0.10%
    95% mean confidence interval for instructions value: 0.66 1.84
    95% mean confidence interval for instructions %-change: 0.01% 0.27%
    Instructions are HURT.

  Unlike the previous code, that used the `mov g1 g2` trick to force
  both `g1` and `g2` to stall, the scheduling fence will generate `mov
  null g1` and `mov null g2`.  During review it was decided it was not
  worth keeping the special codepath for the small effect will have.

Shader-db results for HSW (i965), BDW and SKL don't have a change
on instruction count, but do report changes in cycles count, showing
SKL results below

    total cycles in shared programs: 341738444 -> 341710570 (<.01%)
    cycles in affected programs: 7240002 -> 7212128 (-0.38%)
    helped: 46
    HURT: 5
    helped stats (abs) min: 14 max: 1940 x̄: 676.22 x̃: 154
    helped stats (rel) min: <.01% max: 2.62% x̄: 1.28% x̃: 0.95%
    HURT stats (abs)   min: 2 max: 1768 x̄: 646.40 x̃: 362
    HURT stats (rel)   min: <.01% max: 0.83% x̄: 0.28% x̃: 0.08%
    95% mean confidence interval for cycles value: -777.71 -315.38
    95% mean confidence interval for cycles %-change: -1.42% -0.83%
    Cycles are helped.

  This seems to be the effect of allocating two registers separatedly
  instead of a single one with size 2, which causes different register
  allocation, affecting the cycle estimates.

while ICL also has not change on instruction count but report changes
negative changes in cycles

    total cycles in shared programs: 352665369 -> 352707484 (0.01%)
    cycles in affected programs: 9608288 -> 9650403 (0.44%)
    helped: 4
    HURT: 104
    helped stats (abs) min: 24 max: 128 x̄: 88.50 x̃: 101
    helped stats (rel) min: <.01% max: 0.85% x̄: 0.46% x̃: 0.49%
    HURT stats (abs)   min: 2 max: 2016 x̄: 408.36 x̃: 48
    HURT stats (rel)   min: <.01% max: 3.31% x̄: 0.88% x̃: 0.45%
    95% mean confidence interval for cycles value: 256.67 523.24
    95% mean confidence interval for cycles %-change: 0.63% 1.03%
    Cycles are HURT.

  AFAICT this is the result of the case above.

Shader-db results for TGL have similar cycles result as ICL, but also
affect instructions

    total instructions in shared programs: 17690586 -> 17690597 (<.01%)
    instructions in affected programs: 64617 -> 64628 (0.02%)
    helped: 55
    HURT: 32
    helped stats (abs) min: 1 max: 16 x̄: 4.13 x̃: 3
    helped stats (rel) min: 0.05% max: 2.78% x̄: 0.86% x̃: 0.74%
    HURT stats (abs)   min: 1 max: 65 x̄: 7.44 x̃: 2
    HURT stats (rel)   min: 0.05% max: 4.58% x̄: 1.13% x̃: 0.69%
    95% mean confidence interval for instructions value: -2.03 2.28
    95% mean confidence interval for instructions %-change: -0.41% 0.15%
    Inconclusive result (value mean confidence interval includes 0).

  Now that more is done in the IR, more dependencies are visible and
  more SWSB annotations are emitted.  Mixed with different register
  allocation decisions like above, some shaders will see more `sync
  nops` while others able to avoid them.

  Most of the new `sync nops` are also redundant and could be dropped,
  which will be fixed in a separate change.

Reviewed-by: Francisco Jerez <currojerez@riseup.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3278>

intel/fs: Allow FS_OPCODE_SCHEDULING_FENCE stall on registers

It will generate the MOVs (or SYNC_NOP in Gen12+) needed for stall.

Reviewed-by: Francisco Jerez <currojerez@riseup.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3278>

radv: Expose 4G element texel buffers.

Old value seems to be copied from anv.

Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4787>

iris: Fix downcast of bound_vertex_buffers from uint64_t to int

This is the wrong data type, the original field - and the values we're
adding in - are both 64-bit unsigned.  Keep the original data type.

Thanks to Dave Airlie for finding this while reading the code.

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4802>

intel/ir: Remove scheduling-based cycle count estimates.

The cycle count estimation logic part of the scheduler is now
redundant with the shader performance modeling pass, and the estimates
can be consolidated into the brw::performance analysis result object
instead of being part of the CFG, which guarantees that the estimates
cannot be accessed without previously calling the
performance_analysis::require() method, which makes sure that the
right analysis pass is executed at the right time if we don't already
have up-to-date cached results.

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>

intel/ir: Pass block cycle count information explicitly to disassembler.

So we can eventually remove the cycle count estimates from the CFG
data structure and consolidate performance information in the
brw::performance object.

It would be cleaner to pass the brw::performance object directly to
the disassembler but that isn't straightforward since the disassembler
is built as a plain C file unlike the rest of the compiler back-end.

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>

intel/ir: Use brw::performance object instead of CFG cycle counts for codegen stats.

These should be more accurate than the current cycle counts, since
among other things they consider the effect of post-scheduling passes
like the software scoreboard on TGL.  In addition it will enable us to
clean up some of the now redundant cycle-count estimation
functionality in the instruction scheduler.

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>

intel/fs: Add INTEL_DEBUG=no32 debugging flag.

This is useful in order to identify codegen issues caused by SIMD32.
It doesn't currently have any effect on compute shaders since SIMD32
dispatch is only enabled for CS when it's strictly necessary to do so
in order to support the workgroup size requested for the shader --
That might change in the future though when we hook up the SIMD32
heuristic to CS compilation.

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>

intel/fs: Implement performance analysis-based SIMD32 heuristic for fragment shaders.

The heuristic enables the SIMD32 fragment shader based on whether the
IR performance modeling pass predicts it to have greater throughput
than the SIMD16 and SIMD8 variants of the same shader.  It would be
straightforward to do the same thing in order to control whether
SIMD16 dispatch is enabled, but it's pending additional performance
evaluation.

The INTEL_DEBUG=do32 option is left around in order to force the
SIMD32 shader to be used regardless of the result of the heuristic,
since it's useful as a debugging aid e.g. in order to identify
SIMD32-specific codegen issues which may be masked by the SIMD32
heuristic, or cases where the heuristic is incorrectly disabling
SIMD32 shaders that offer a performance advantage.

Currently this is only enabled on Gen6+, since SIMD32 codegen support
is incomplete on earlier platforms.

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>

intel/fs: Heap-allocate fs_visitors in brw_compile_fs().

This makes brw_compile_fs() look a bit more similar to
brw_compile_cs().  It saves us three v*_shader_stats local variables,
and will save us additional triplicated declarations as we start
tracking IR performance analysis results.

The triplicated cfg pointers are left around because they're set to
NULL to mark specific dispatch modes as disabled (e.g. in order to
enforce hardware restrictions).  Doing the same thing with the visitor
pointers would cause data leaks.

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>

intel/ir: Import shader performance analysis pass.

This introduces an analysis pass intended to estimate several
performance statistics of the shader, including cycle count latency
and throughput values, based on static modeling.  It has instruction
performance information more comprehensive than the current scheduling
pass for all platforms between Gen4-11, and works on both the FS and
VEC4 back-end.

The most immediate purpose of this pass is to implement a heuristic
meant to determine whether using SIMD32 dispatch for a fragment shader
can be expected to help more than it hurts.  In addition this will
allow the effect of passes run after scheduling (e.g. the TGL software
scoreboard pass and the VEC4 dependency control pass) to be visible in
shader-db statistics.

But that isn't the end of the story, other potential applications of
this pass (not part of this MR) I've been playing around with are:

 - Implement a similar SIMD16 heuristic allowing the identification of
   inefficient SIMD16 fragment shaders.

 - Implement similar SIMD16 and SIMD32 heuristics for the compute
   shader stage -- Currently compute shader builds always use the
   SIMD16 shader if available and never use the SIMD32 shader unless
   strictly necessary, which is suboptimal under certain conditions.

 - Hook up to the instruction scheduler in order to improve the
   accuracy of its timing information.

 - Use as heuristic in order to drive the selection of scheduling
   modes (Matt was experimenting with that).

 - Plug to the TGL software scoreboard pass in order to implement a
   more effective SBID token allocation algorithm, since in general
   the optimal token allocation depends on the timings of all
   instructions in the program.

 - Use its bottleneck detection functionality in order to implement a
   heuristic computing a more optimal bound for the number of fragment
   shader threads executed in parallel (by adjusting the
   MaximumNumberofThreadsPerPSD control of 3DSTATE_PS).

As a follow-up I'm planning to submit updated timing information for
Gen12 platforms -- Everything else required to support Gen12 like SWSB
handling is already included in this patch, but there were some IP
concerns regarding the TGL timing parameters since they cannot
currently be obtained with the documentation and hardware which is
publicly available.  The timing parameters for any previous Gen7-11
platforms can be obtained by anyone by sampling the timestamp register
using e.g. shader_time, though I have some more convenient
instrumentation coming up.

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>

intel/vec4: Fix constness of vec4_instruction::reads_flag() and ::writes_flag().

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>

intel/fs: Replace fs_visitor::bank_conflict_cycles() with stand-alone function.

This will be re-usable by the IR performance analysis pass.

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>

intel/fs: Fix constness of argument of fs_instruction_scheduler::is_compressed().

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>

intel/fs: Rename half() helpers to quarter(), allow index up to 3.

Makes more sense considering SIMD32.  Relaxing the assertion in
brw_ir_fs.h will be required in order to avoid assertion failures on
SNB with SIMD32 fragment shaders.

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>

intel/ir: Add missing initialization of backend_reg::offset during construction.

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>

intel/fs/gen12: Fix Render Target Read header setup for new thread payload layout.

In Gen12 the Poly 0 Info DWORD containing the Viewport Index and
Render Target Index fields were moved from r0.0 to r1.1 in order to
make room for dual-polygon dispatch.  The render target message format
was updated to expect that information in the same location, so we
didn't need to make any changes for framebuffer fetch to work with
SIMD8 and SIMD16 dispatch.  Unfortunately that won't work with SIMD32,
since the render target message header is assembled from r0 and r2
instead of r1, and the r2 thread payload wasn't updated with an
additional copy of the same information.  We need to fix things up
manually instead.  This avoids a handful of
EXT_shader_framebuffer_fetch regressions in combination with SIMD32
fragment shaders.

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>

intel/fs/gen12: Work around dual-source blending hangs in combination with SIMD32.

This applies the same work-around I commited as b84fa0b31e67
"intel/fs/gen11: Work around dual-source blending hangs in combination
with SIMD32." to Gen12, which seems to suffer from the same hardware
bug found empirically.  The failure mode seems to be identical.

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>

intel/fs/gen12: Fix hangs with per-sample SIMD32 fragment shader dispatch.

The Gen12 docs are rather contradictory regarding the dispatch
configurations supported by the fragment shader -- The same table
present in previous generations seems to imply that only one dispatch
mode can be enabled when doing per-sample shading, but a restriction
documented in the 3DSTATE_PS_BODY page implies the opposite: That
SIMD32 can only be used in combination with some other dispatch mode.

The latter seems to match the behavior of real hardware as I could
tell from my testing: A bunch of multisample test-cases that do
per-sample shading hang if we only provide a SIMD32 shader.

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>

mesa: Follow OpenGL conversion rules for values that exceed storage size

Section 2.2.2 (Data Conversions For State Query Commands) of the
OpenGL 4.5 spec says:

  Following these steps, if a value is so large in magnitude that
  it cannot be represented by the returned data type, then the
  nearest value representable using that type is returned.

The current code doesn't do the correct thing, because it truncates a
long (potentially a 64bit values) to an int.

Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/2828
Fixes: 53c36dfcfe3eb3749a53267f054870280afb0d71
       ("replace IROUND with util functions")

Reviewed-by: Matt Turner <mattst88@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4673>

pan/bit: Add BITWISE test

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4790>

pan/bit: Interpret BI_BITWISE

No shifting yet.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4790>

pan/bi: Handle iand/ior/ixor in NIR->BIR

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4790>

pan/bi: Pack BI_BITWISE

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4790>

pan/bi: Add bitwise modifiers

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4790>

freedreno/a6xx: invalidate tex state cache entries on rebind

When a resource's backing bo changes, its seqno will be incremented.
Which would result in a new tex state cache key, and nothing to clean
up the old tex state until the sampler view/state is destroyed.  But
in some games, that may never happen, or at least not happen before
we run out of memory.

Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/2830
Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4744>

freedreno: rebind_resource() *before* bo changes

This will matter in the next patch, where we need the original
rsc->seqno.

It means slight shuffling of where we call rebind_resource() in the
`fd_try_shadow_resource()` path.

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4744>

freedreno: rebind resource in all contexts

If the resource is rebound, we need to invalidate in all contexts.

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4744>

freedreno: optimize rebind_resource()

Track how resources are used, ie. which state they may potentially dirty
if the backing bo is changed/reallocated, to optimize rebind_resource().

This will be more important in a later patch when we hook up eviction of
entries in a6xx tex state cache.

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4744>

freedreno: mark more state dirty when rebinding resources

Plus a bonus typo fix.

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4744>

freedreno: don't realloc idle bo's

The `DISCARD_WHOLE_RESOURCE` is just a hint.  And `rebind_resource()` is
a bunch of faffing about (and going to get worse in a later patch), so
let's not bother when the bo is already idle.

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4744>

freedreno: small whitespace fix

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4744>

gallium/swr: Fix crashes and failures in vertex fetch

This commit fixes two problems:
- In some cases SWR does not correctly report to Gallium
  which formats are supported.
- Incorrect LLVM instructions are used in vertex fetch in some situations

Reviewed-by: Krzysztof Raszkowski <krzysztof.raszkowski@intel.com>
Reviewed-by: Bruce Cherniak <bruce.cherniak@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4788>

freedreno/log-parser: support to read gzip'd logs

~50MB gzip'd log files are nicer than ~300MB uncompressed

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4750>

freedreno/a6xx: pre-calculate expected vsc stream sizes

We should only rely on overflow detection for indirect draws, where we
have no other option.

This doesn't use quite the worst-possible-case sizes, which in practice
seem to be ~20x larger than what is required.  But instead uses roughly
half of that.

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4750>

freedreno: add helper to estimate # of bins per pipe

For vsc size calculation, we need to know the # of bins per pipe.  Or at
least the worst-case # of bins, assuming we don't eliminate an unused depth/
stencil buffer.

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4750>

freedreno/a6xx+tu: rename VSC_DATA/VSC_DATA2

These are the draw-stream and primitive-stream, so lets give them more
descriptive names.

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4750>

aco: fix vgpr nir_op_vecn with sgpr operands

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4772>

aco: improve clamped integer addition disassembly workaround

Make it work with 16-bit and GFX10.

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4772>

aco: add various GFX10 int16 opcodes

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4772>

aco: fix sub-dword overwrite check in RA validator

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4772>

aco: fix sub-dword out-of-bounds check in RA validator

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4772>

aco: add missing adjust_max_used_regs()

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4772>

aco: improve RA for uneven p_split_vector

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4772>

aco: don't recurse in sub-dword get_reg_simple()

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4772>

aco: split self-intersecting copies instead of swapping

Example situation:
v1 = {v0.hi, v1.lo}
v0.hi = v1.hi

The 4-byte copy's definition is completely used, but swapping it makes no
sense. We have to split it to generate correct code:
swap(v0.hi, v1.lo)
swap(v0.hi, v1.hi)

Found in dEQP-VK.spirv_assembly.type.vec3.i16.constant_composite_vert

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4772>

aco: fix neighboring register check in get_reg_simple()

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4772>

aco: check alignment of non-subdword registers in get_reg_specified()

When splitting a v6b vector into v1 and v2b components, we should ensure
the v1 definition doesn't start at the upper half.

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4772>

aco: make RegisterFile::block() take a regclass

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4772>

anv: Claim VK_EXT_robustness2 support

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4767>

anv: Handle null vertex buffer bindings

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4767>

anv: Handle NULL descriptors

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4767>

nir/combine_stores: Handle volatile

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4767>

nir/dead_write_vars: Handle volatile

We can't remove volatile writes and we can't combine them with other
volatile writes so all we can do is clear the unused bits.

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4767>

nir/copy_prop_vars: Report progress when deleting self-copies

Fixes: 62332d139c8f6 "nir: Add a local variable-based copy prop..."
Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4767>

nir/copy_prop_vars: Handle volatile better

For deref_store, we can still delete invalid stores that write to
statically OOB data.  For everything, we need to make sure that we kill
aliases of destinations even if it's volatile.

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4767>

vulkan: Update Vulkan XML and headers to 1.2.139

Acked-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4767>

anv: Allow all clear colors for texturing on Gen11+

Starting with Gen11, we have two indirect clear colors: An unconverted
float/int version which is us used for rendering and a converted pixel
value version which is used for texturing.  Because the one used for
texturing is stored as a single pixel of that color, it works no matter
what format is being used.  Because it's a simple HW indirect and
doesn't involve copying surface states around, we can use it in the
sampler without having to worry about surface states having out-of-date
clear values.  The result is that we can now allow any clear color when
texturing.

This cuts the number of resolves in a RenderDoc trace of Dota2 by 95%
on Gen11+ (you read that right) and improves perf by 3.5%.  It improves
perf in a handful of other workloads by < 1%.

Reviewed-by: Rafael Antognolli <rafael.antognolli@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4393>

anv: Use anv_layout_to_aux_usage for color during render passes

Previously, we tried to treat color image layouts as a special case
during render passes.  This is largely an artifact of history as our
initial understanding of Vulkan placed much more emphasis on render
passes than our current understanding.  The only real practical use for
magic layouts in the middle of a render pass, as far as I can tell, is
to allow more clear colors to get passed through to input attachments.
However, most apps aren't very creative with their clear colors and very
few of them (none coming from DXVK) actually use render passes in any
interesting way.  Therefore, the risk of being able to pass fewer clear
colors through to input attachments should be minimal.

There are, however, three very big advantages to this change:

 1. We are now consistent in our handling of aux usage and layouts
    between color and depth/stencil.

 2. We are now actually following the layout guidelines from the app and
    aren't nearly as likely to see strange behavior due to us overriding
    the image layouts manually.

 3. It's more obviously correct.  While I think our old render pass code
    was probably correct, it was full of corner cases and it's very
    possible that it was behaving badly in weird ways.  This follows the
    Vulkan API much more blindly and, as such, is more likely to be
    correct and behave the same as other implementations.

Reviewed-by: Rafael Antognolli <rafael.antognolli@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4393>

anv: Split color_attachment_compute_aux_usage in two

In particular, we split out an anv_can_fast_clear_color_view helper
which only cares about fast-clear and not aux_usage itself.

Reviewed-by: Rafael Antognolli <rafael.antognolli@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4393>

anv: Rework depth_stencil_attachment_compute_aux_usage

Instead of making it a function that pretends to choose aux usage (which
isn't what it does at all), make it a function which returns whether or
not we want to do a fast clear.  This is far more accurate to the
purpose of the function.

Reviewed-by: Rafael Antognolli <rafael.antognolli@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4393>

anv: Refactor cmd_buffer_setup_attachments

This commit just renames some things so that we use names for temporary
variables which are more consistent with other places in the code-base.

Reviewed-by: Rafael Antognolli <rafael.antognolli@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4393>

anv: Stop allowing non-zero clear colors in input attachments

Previously, we bent over backwards to allow non-zero clear colors input
attachments whenever we could.  However, very few apps use input
attachments and very few use non-zero clear colors.  Getting rid of
support for non-zero clear colors input attachments will allow us to
treat them identically to textures which should help us simplify things
a good bit.

Reviewed-by: Rafael Antognolli <rafael.antognolli@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4393>

anv: Disallow fast-clears which require format-reinterpretation

In order to actually hit this case you have to be using a very odd
color/view combination.  The common cases of clear-to-zero and 0/1 clear
colors with an sRGB view don't require any re-interpretation.  This is
probably better than always resolving whenever we have a format mismatch
like we are today because that hits the sRGB case every time.

Reviewed-by: Rafael Antognolli <rafael.antognolli@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4393>

intel: Move swizzle_color_value from blorp to ISL

Reviewed-by: Rafael Antognolli <rafael.antognolli@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4393>

anv: Allocate surface states per-subpass

Instead of allocating surface states for attachments in BeginRenderPass,
we now allocate them in begin_subpass.  Also, since we're zeroing
things, we can be a bit cleaner about or implementation and just fill
out all those passes for which we have allocated surface states.

Reviewed-by: Rafael Antognolli <rafael.antognolli@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4393>

anv: Split command buffer attachment setup in three

This commit splits genX(cmd_buffer_setup_attachments)() into three
functions: one which sets up cmd_buffer->state.attachments, one which
allocates surface states, and one which fills out the surface states.
While we're here, we make both functions take the framebuffer (if any)
as an argument instead of pulling it from the command buffer so it's
more clear what things are inputs to the functions.  We also make the
render pass and framebuffer parameters const as those are immutable
objects.  The only functional change here should be that we now
vk_zalloc the attachments which should be a bit safer.

Reviewed-by: Rafael Antognolli <rafael.antognolli@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4393>

anv: Mark images written in end_subpass

This makes a lot more sense than marking them written in begin_subpass
since, at that point, we haven't written them yet.  This should reduce
the chances of accidental extra resolves.

Reviewed-by: Rafael Antognolli <rafael.antognolli@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4393>

anv: Use ANV_FROM_HANDLE for pInheritanceInfo fields

Reviewed-by: Rafael Antognolli <rafael.antognolli@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4393>

anv: Assert surface states are valid

Reviewed-by: Rafael Antognolli <rafael.antognolli@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4393>

anv: Stop filling out the clear color in compute_aux_usage

It's a pointless micro-optimization that just makes compute_aux_usage
unnecessarily entangled with setting up surface states.

Reviewed-by: Rafael Antognolli <rafael.antognolli@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4393>

anv: Add TRANSFER_SRC to pass usage not subpass usage

The subpass usage flags are supposed to always be one bit and never
multiple bits.  However, when adding in TRANSFER_SRC usage for resolve
attachments we were adding it to the subpass bits and not the render
pass bits.  This potentially is causing issues where images aren't
getting marked written properly.

Reviewed-by: Rafael Antognolli <rafael.antognolli@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4393>

anv: Return an error if allocating attachment memory fails

Reviewed-by: Rafael Antognolli <rafael.antognolli@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4393>

radv: advertise VK_AMD_memory_overallocation_behavior

Doom Eternal explicitly allows overallocation via this extension
but that shouldn't change anything because it's the default RADV
behavior.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4785>

radv: track memory heaps usage if overallocation is explicitly disallowed

By default, RADV supports overallocation by the sense that it doesn't
reject an allocation if the target heap is full.

With VK_AMD_overallocation_behaviour, apps can disable overallocation
and the driver should account for all allocations explicitly made by
the application, and reject if the heap is full.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4785>

radv: remove unused radv_device_memory::map_size field

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4785>

nir/algebraic: Require operands to iand be 32-bit

With the mask value 0x80000000, the other operand must be 32-bit.  This
fixes failures in
dEQP-VK.subgroups.ballot_mask.ext_shader_subgroup_ballot.*.gl_subgroupgemaskarb_*
tests from Vulkan 1.2.2 CTS.

Checking one of the tests, it appears that the tests are doing 64-bit
iand with 0x0000000080000000, then comparing the result with zero.

Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/2834
Fixes: 88eb8f190bd ("nir/algebraic: Simplify logic to detect sign of an integer")
Reviewed-by: Eric Anholt <eric@anholt.net>
Reviewed-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4770>

freedreno/ir3/ra: only assign array base in first pass

In particular, we specifically don't want to let the base change between
passes, as it could end up conflicting with registers assigned in the
first pass.

Mostly-closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/2838
Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4780>

freedreno/ir3/ra: split out helper for array assignment

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4780>

freedreno/ir3/ra: use ir3_debug_print helper

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4780>

freedreno/ir3/ra: remove unused variable

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4780>

freedreno/computer: add script to test widening/narrowing

Just something I hacked together to help figure out which instructions
can fold in a wideing/narrowing conversion.

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4780>

pan/bi: Add initial fcmp test

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4789>

pan/bit: Interpret CMP

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4789>

pan/bit: Prepare condition evaluation for vectors

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4789>

pan/bi: Relax double-abs condition

Only if both ports (<==> registers) same.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4789>

pan/bi: Pack fma.fcmp16

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4789>

pan/bi: Factor out fp16 abs logic

Also used for fcmp16

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4789>

pan/bi: Pack FMA 32 FCMP

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4789>

pan/bi: Fix source mod testing for CMP

Outputs u32.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4789>

pan/bi: Structify ADD ICMP 32

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4789>

pan/bi: Structify FMA ICMP 16

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4789>

pan/bi: Structify FMA ICMP 32

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4789>

pan/bi: Structify ADD FCMP16

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4789>

pan/bi: Structify FMA FCMP16

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4789>

pan/bi Strucitfy ADD FCMP 32

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4789>

pan/bi: Structify FMA FCMP

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4789>

pan/bi: Remove bi_round_op

No purpose.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4789>

pan/bi: Deduplicate csel/cmp cond

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4789>