iris: Implement buffer-local memory barrier based on cache coherency matrix.

This takes advantage of the previously introduced cache tracking
infrastructure in order to define a multi-purpose barrier operation
that allows the caller to order memory operations with respect to
previous operations performed on the same buffer from any other cache
domain.

v2: Assorted CPU overhead micro-optimizations (Francisco).
v3: Use C99 designated initializers (Ken).

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3875>

iris: Update cache coherency matrix on PIPE_CONTROL.

This introduces a batch synchronization boundary at every PIPE_CONTROL
command, and updates the cache coherency status tracked during batch
construction according to the specified control bits.

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3875>

iris: Introduce cache coherency matrix for batch-local memory ordering.

This introduces a representation of the cache coherency status of the
GPU at any point in the batch.  This is done by defining a matrix C of
synchronization sequence numbers such that at any point of batch
construction, a memory operation from domain i introduced into the
batch is guaranteed to be ordered after any memory operation from
domain j in a previous batch section with seqno n if the following
condition holds:

  C_i_j >= n

This allows us to efficiently determine whether additional flushing
and/or invalidation is required in order to access a buffer object
from some arbitrary domain.

Except for batch buffer reset which requires clearing the whole
matrix, all operations on the matrix are either O(n) or O(1) on the
number of caching domains (which is basically constant).

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3875>

iris: Report use of any in-flight buffers on first draw call after sync boundary.

This is the main performance trade-off of this cache tracking
mechanism: In order for the seqno vector of buffer objects to be
accurate, they need to be marked as used again every time the batch is
split into a new synchronization section if they remain bound to the
pipeline.  This can be achieved easily by re-using
iris_restore_render_saved_bos() and iris_restore_compute_saved_bos(),
which currently serve a similar purpose across batch buffer
boundaries.

The impact on Piglit drawoverhead results seems to be within a
standard deviation of the current results.

XXX - It might be possible to completely remove the current
      iris_batch::contains_draw flag at a small additional performance
      cost.

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3875>

iris: Drop redundant iris_address::write flag.

The write flag is redundant since it can be inferred easily from the
iris_address::access domain.  This allows the iris_address struct to
be laid out more efficiently in memory, leading to a measurable
improvement in several Piglit Drawoverhead test-cases.

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3875>

iris: Annotate all BO uses with domain and sequence number information.

Probably the most annoying patch to review from the whole series --
Mark every buffer object use as accessed through some caching domain
with the sequence number of the current synchronization section of the
batch.  The additional argument of iris_use_pinned_bo() makes sure I'd
have gotten a compile error if I had missed any buffer added to the
batch validation list.

There are only a few exceptions where a buffer is left untracked while
adding it to the validation list, justified below:

 - Batch buffers: These are strictly read-only for the moment.

 - BLORP buffer objects: Their seqnos are bumped manually at the end
   of iris_blorp_exec() instead, in order to avoid plumbing domain
   information through BLORP address combining.

 - Scratch buffers: The contents of these are strictly thread-local.

 - Shader images and SSBOs: Accesses of these buffers are explicitly
   synchronized at the API level.

v2: Opt out of tracking more aggressively (Ken): In addition to the
    above, surface states, binding tables, instructions and most
    dynamic states are now left untracked, which means a *lot* more BO
    uses marked IRIS_DOMAIN_NONE which need to be reviewed extremely
    carefully, since the cache tracker won't be able to provide any
    coherency guarantees for them.

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3875>

iris: Bracket batch operations which access memory within sync regions.

This delimits all batch operations which access memory between
iris_batch_sync_region_start() and iris_batch_sync_region_end() calls.
This makes sure that any buffer objects accessed within the region are
considered in use through the same caching domain until the end of the
region.

Adding any buffer to the batch validation list outside of a sync
region will lead to an assertion failure in a future commit, unless
the caller explicitly opted out of the cache tracking mechanism.

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3875>

iris: Add infrastructure to partition batch into sync boundaries.

This introduces some minimalistic infrastructure which will be used in
order to partition the batch into a series of sections, each one with
a unique, monotonically-increasing sequence number.  Section
boundaries will typically lie at points in the batch where the
execution and memory coherency status of some previous commands are
known, e.g. at batch buffer boundaries or PIPE_CONTROL commands.

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3875>

iris: Add batch-local synchronization book-keeping to iris_bo.

The purpose of this is to represent the cache coherency state of a
buffer as a vector of integers (AKA seqnos), one for each incoherent
caching domain of the GPU.  A seqno will identify a single section of
a batch buffer uniquely across the whole pipe_screen (which means that
there will be no ambiguity about what context a given seqno belongs to
even if there are multiple threads accessing the same buffer in
parallel), and is guaranteed to be allocated in monotonically
increasing order within any given context.  The iris_bo_bump_seqno()
helper is provided for marking the last update of a buffer from a
given caching domain in a lockless manner.

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3875>

panfrost: Mark point sprites as todo on Bifrost

Emulating them will be a rather annoying dance. Let's not worry about
this until further down the line when we have a better sence of how to
do handle them efficiently.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5290>

panfrost: Fix gl_PointSize out of GL_POINTS

In this case, vs->writes_point_size is true as the VS writes
gl_PointSize, but panfrost_writes_points_size() is false as we are not
drawing points so the hardware doesn't process it. Thus the varying
descriptor is emitted but elements is never written. When the VS runs,
it will attempt to write to elements, a NULL pointer.

The behaviour is architecture-independent. On Midgard, the write
silently fails, hence why this bug was never noticed before. On Bifrost,
this raises an MMU fault.

The fix is to set the format to VARYING_DISCARD to ignore the write.

Noticed on Neverball.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5290>

panfrost: Prefer sysval for gl_PointCoord on Bifrost

It's like gl_FragCoord. Still not implemented. This unfortunately makes
point sprites a lot more complicated.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5290>

pan/bi: Disassemble gl_PointCoord reads.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5290>

panfrost: Explicitly convert to 32-bit for logic-ops

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Reported-by: Icecream95 <ixn@keemail.me>
Fixes: 19b4e586f62 ("panfrost: Switch to pan_lower_framebuffer")
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5289>

panfrost: Readd MIDGARD_SHADERLESS quirk to t760

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Reported-by: Icecream95 <ixn@keemail.me>
Fixes: e53d27de61b ("panfrost: Add quirks for blend shader types")
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5289>

iris: Extend iris_context dirty state flags to 128 bits.

We're nearly out of dirty bits, and some patches pending review on
GitLab no longer apply due to that.  Make room for them by splitting
off shader stage-specific bits into a separate stage_dirty mask.

An alternative would be to split compute-related bits into a separate
mask, but that would prevent the '<< stage' indexing done in various
parts of the driver from working.

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5279>

iris: Simplify iris_batch_prepare_noop().

This makes iris_batch_prepare_noop() return a boolean instead of
passing through the relevant set of dirty flags.  It will make it
easier to change the representation of dirty flags.

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5279>

nir/lower_tex: fixes for fp16 yuv lowering

Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/3079
Signed-off-by: Rob Clark <robdclark@chromium.org>
Reviewed-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Reviewed-by: Eric Anholt <eric@anholt.net>
Reviewed-by: Kristian H. Kristensen <hoegsberg@google.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5318>

nir/builder: add bitsize conversion helpers

Signed-off-by: Rob Clark <robdclark@chromium.org>
Reviewed-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Reviewed-by: Eric Anholt <eric@anholt.net>
Reviewed-by: Kristian H. Kristensen <hoegsberg@google.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5318>

nir: extract out convert_to_bitsize() helper

Signed-off-by: Rob Clark <robdclark@chromium.org>
Reviewed-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Reviewed-by: Eric Anholt <eric@anholt.net>
Reviewed-by: Kristian H. Kristensen <hoegsberg@google.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5318>

nir: get_base_type() should return enum type

Needed by the next patch, for c++ code which is more strict about
conversions between integers and enums.

Signed-off-by: Rob Clark <robdclark@chromium.org>
Reviewed-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Reviewed-by: Eric Anholt <eric@anholt.net>
Reviewed-by: Kristian H. Kristensen <hoegsberg@google.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5318>

panfrost: Handle writes_memory correctly

We need to pass it thru to EARLY_Z and WRITES_GLOBAL instead of ignoring
and assuming respectively. Nontrivial performance fix.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5300>

panfrost: Document MALI_WRITES_GLOBAL bit

We've been setting this unconditionally -- oops!

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5300>

panfrost: Update MALI_EARLY_Z description

Via the ES3.1 early-z testing force, I've confirmed this bit is e-z.
I've also confirmed e-z must be disabled for global writes, as expected.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5300>

iris: remove unused iris_bo->swizzle_mode

Signed-off-by: Marcin Ślusarz <marcin.slusarz@intel.com>
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5313>

aco: sign-extend input/identity for 16-bit subgroup ops on GFX6-GFX7

16-bit subgroup ops are implemented with 32-bit instructions
on GFX6-GFX7.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5227>

aco: fix subdword copies on GFX6-GFX7

SDWA is only GFX8+. Use v_mov_b32 since the upper 16 bits don't matter.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5227>

aco: implement 16-bit nir_intrinsic_quad_* on GFX6-GFX7

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5227>

aco: implement 16-bit reduce operations on GFX6-GFX7

No fp16 on GFX6-GFX7.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5227>

pan/bi: Handle vectorized load_const

In preparation for 16-bit vectors.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5307>

pan/bi: Passthrough second argument of F32_TO_F16

At the NIR level this is a second vector source of the first (only)
argument; at the BIR level this is a pair of scalars.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5307>

pan/bi: Pack second argument of F32_TO_F16

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5307>

pan/bi: Fix SEL.16 swizzle

2 scalar arguments, not 1 vector.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5307>

pan/bi: Handle SEL with vec3 16-bit

Otherwise we end up with a missing argument.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5307>

panfrost: Passthrough NATIVE loads/stores

Now that we handle load_output directly, this works for e.g. RGB565 on
Midgard.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5285>

pan/mdg: Handle regular nir_intrinsic_load_output

Instead of the vendored version. Only for blend shaders at the moment,
frag shaders fb_fetch has a lot more going on.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5285>

pan/mdg: Allow f2u8 and friends thru

Now that we can handle destination sizes directly, this keeps us from
needing to chew through so many conversions.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5285>

pan/mdg: Handle f2u8

This is similar to f2u16.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5285>

pan/mdg: Fold roundmode into applicable instructions

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5285>

pan/mdg: Implement *_rtz conversions with roundmode

Use rte as the canonical type.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5285>

pan/mdg: Lower roundmodes

So now we can use the IR field semantically.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5285>

pan/mdg: Add opcode roundmode property

When the output is rounded in a specified direction.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5285>

pan/mdg: Add roundmode enum

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5285>

pan/mdg: Distinguish blend shaders in internal shader-db

Since these shaders are purely internal, the optimization criteria are a
bit different, so it's worth calling attention to this when dumping.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5285>

panfrost: Only use AFBC YTR with RGB and RGBA

The "lossless colorspace transform" is lossy for R and RG formats.

Reviewed-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5293>

panfrost: Decode AFBC flag bits

Reviewed-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5293>

glsl: when NIR linker enable use it to resize uniform arrays

Here we turn on uniform array resizing in the NIR linker and disable
the GLSL IR resizing pass when the NIR linker is enabled.

This will potentially make uniform arrays smaller due to NIR
optimising away more uniform uses.

Shader-db results (SKL):

total instructions in shared programs: 14947192 -> 14944093 (-0.02%)
instructions in affected programs: 138088 -> 134989 (-2.24%)
helped: 822
HURT: 4

total cycles in shared programs: 324868402 -> 324794597 (-0.02%)
cycles in affected programs: 3904170 -> 3830365 (-1.89%)
helped: 2333
HURT: 1485

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4910>

glsl: gather uniform dereference info before main linking loop

We want to gather information for all stages here before the main
linking loop. In the following patch we will use to information
to reduce the size of uniform arrays where possible.

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4910>

glsl: add update_array_sizes() helper to the NIR uniform linker

This will be used to reduce the size of uniform arrays and replace
the current glsl ir pass. Doing this in NIR allows us to better
optimise the size of uniform arrays.

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4910>

glsl: add struct to gather more info about uniform array access

This will be used in the following patches to allow the linker
to resize uniform arrays based on array dereferences.

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4910>

util: add BITSET_LAST_BIT() helper

This is the reverse of BITSET_FFS()

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4910>

i965: call brw_nir_lower_uniforms() after uniform linking is complete

i965 currently uses the NIR uniform linker for spirv support. Until
now the only reason there has been no issue with calling the
lowering pass before the linker is because no garbage collection
is done between the calls.

An upcoming change to the linker will add an optimisation to resize
unform arrays where possible. Because lowering causes the array
defs to no longer be used the new optimisation ends up resizing the
arrays to 0. To fix this we move the lowering call after the
linking calls.

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4910>

gbm: document that gbm_bo_map exposes a linear view

Drivers (Gallium, i965) expose a linear view of the buffer via
gbm_bo_map.

Signed-off-by: Simon Ser <contact@emersion.fr>
Reviewed-by: Daniel Stone <daniel@fooishbar.org>
Reviewed-by: Eric Anholt <eric@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5238>

glsl: Don't replace lrp pattern with lrp if arguments are not floats

We don't have "lrp(int, int, int)" and validation of ir_triop_lrp
fails down the road.

Fixes: 8d37e991
Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/3059
Signed-off-by: Danylo Piliaiev <danylo.piliaiev@globallogic.com>
Tested-by: Witold Baryluk <witold.baryluk@gmail.com>
Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5257>

spirv: Use scoped barriers for SpvOpControlBarrier

If use_scoped_barrier is set to true, we don't have to split the control
and memory barriers.

Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com>
Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4900>

intel/compiler: Extract control barriers from scoped barriers

Add a lowering pass extracting all control barriers embedded in scoped
barriers into proper control barriers so we can get rid of the logic
inserting control barriers when an SpvOpControlBarrier with WorkGroup
scope is parsed in spirv_to_nir().

Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com>
Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4900>

nir: Replace the scoped_memory barrier by a scoped_barrier

SPIRV OpControlBarrier can have both a memory and a control barrier
which some hardware can handle with a single instruction. Let's
turn the scoped_memory_barrier into a scoped barrier which can embed
both barrier types. Note that control-only or memory-only barriers can
be supported through this new intrinsic by passing NIR_SCOPE_NONE to the
unused barrier type.

Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com>
Suggested-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4900>

spirv: Split the vtn_emit_scoped_memory_barrier() logic

We are about to add support for scoped control+memory barriers. Let's
move the convert from SPIRV to NIR enums logic in helpers so we can
easily re-use them.

Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com>
Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4900>

radv: enable zero VRAM for all VKD3D (DX12->VK) games

To fix rendering issues with Metro Exodus, RE2 and 3 and probably
more titles. It seems the default behaviour of DX12 anyways.

Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/3064
Cc: <mesa-stable@lists.freedesktop.org>
Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5262>

radv: enable zero VRAM for Doom Eternal

That fixes some rendering issues. Probably some unitialized data
from the game.

Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/3064
Cc: <mesa-stable@lists.freedesktop.org>
Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5262>

gitlab-ci: bump piglit checkout commit

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4797>

glsl/spirv: remove dead uniforms in spirv nir linker

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Reviewed-by: Eric Anholt <eric@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4797>

glsl: remove dead uniforms in the nir linker

This is now possible as we do uniform linking via a nir based linker.

Shader-db results for IRIS (SKL):

total instructions in shared programs: 14947192 -> 14946397 (<.01%)
instructions in affected programs: 39498 -> 38703 (-2.01%)
helped: 230
HURT: 18

total cycles in shared programs: 324868402 -> 324847058 (<.01%)
cycles in affected programs: 706701 -> 685357 (-3.02%)
helped: 599
HURT: 449

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Reviewed-by: Eric Anholt <eric@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4797>

glsl: add can_remove_uniform() helper to the NIR linker

This helper reflects the rules we follow in the GLSL IR linker when
deciding if we can remove a dead uniform. This check is required to
avoid regressions when turning on NIR dead uniform clean up in the
following patch.

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Reviewed-by: Eric Anholt <eric@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4797>

nir: add callback to nir_remove_dead_variables()

This allows us to do API specific checks before removing variable
without filling nir_remove_dead_variables() with API specific code.

In the following patches we will use this to support the removal
of dead uniforms in GLSL.

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Reviewed-by: Eric Anholt <eric@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4797>

nir: add glsl_get_ifc_packing() helper

This will be used in the following patch.

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Reviewed-by: Eric Anholt <eric@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4797>

pan/mdg: Don't double-replicate blend on T720

We already do this unconditionally in NIR.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5305>

radv: Use common gfx10_format_table.h

Save some python code and build time, as well as some code duplication.

Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5291>

radv: Include gfx10_format_table.h only from a single source file.

The radeonsi variant has everything in the header, so lets not
include it everywhere.

Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5291>

radeonsi: Define gfx10_format in the common header.

So we don't have to have multiple definitions of the struct when
sharing with radv.

While at it put the table properly in a C file so we don't have to
deal with multiple definitions, and the struct definition isn't
in generated source.

Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5291>

amd/common,radeonsi: Move gfx10_format_table to common.

Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5291>

radeonsi: Explicitly map Z16_UNORM_S8_UINT to None for GFX10.

We should always use separate planes for textures with this format.

Fixes: 273ead81f1a "util/format: Add VK_FORMAT_D16_UNORM_S8_UINT."
Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5291>

Revert "CI: Disable Panfrost T720/T760"

Switches have been rewired, VLANs have been reconfigured, network
elements with non-functional remote management have been removed from
racks and thrown on desks in anger.

This reverts commit ae6e1aee7d1bd49ae494b8a25ca33d092a3a145a.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5301>

ci: bare-metal: make it possible to use a script for serial

Makes it possible to use e.g. a ser2net script to talk to the devices.

Signed-off-by: Christian Gmeiner <christian.gmeiner@gmail.com>
Reviewed-by: Eric Anholt <eric@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5268>

zink: Use store_dest_raw instead of storing an uint

I cleaned up the other similar call-sites, but somehow missed this one.
There's nothing different with this, so let's also fix this.

Fixes: 16339646f03 ("zink/spirv: rename functions a bit")
Reviewed-by: Mike Blumenkrantz <michael.blumenkrantz@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5250>

radv: Explicitly cast TIMESTAMP_NOT_READY value to uin32_t where needed.

Fixes a clang warning.

Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5228>

aco: Use correct reference type in for-range-loop.

Fixes a clang warning.

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5228>

aco: Don't std::move temporary object.

Fixes the following clang warning:

mesa/src/amd/compiler/aco_optimizer.cpp:2928:15: warning: moving a temporary object prevents copy elision [-Wpessimizing-move]
   ctx.uses = std::move(dead_code_analysis(program));

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5228>

aco: Don't declare 'Block' as class, but define as struct.

Fixes clang warnings.

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5228>

radv: Don't take absolute value of unsigned type.

Fixes clang warnings.

Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5228>

radv/aco: Always enable subgroup shuffle.

It is now supported by both backends on all hw.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5223>

aco: Implement subgroup shuffle on GFX6-7.

GFX6 and GFX7 don't have the ds_bpermute (or permute) instruction,
but we would like to support subgroup shuffle on these old GPUs.

So we introduce a new pseudio instruction which will be lowered
to an "unrolled loop" that emulates bpermute on GFX6 and GFX7
using readlane instructions, while also respecting the exec mask
thanks to v_cmpx.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5223>

aco/gfx10: Refactor of GFX10 wave64 bpermute.

The emulated GFX10 wave64 bpermute no longer needs a linear_vgpr,
so we don't consider it a reduction anymore. Additionally, the
code is slightly reorganized in preparation for the GFX6 emulated
bpermute.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5223>

radeonsi: add a hack to disable TRUNC_COORD for shadow samplers

This fixes dEQP-GLES3.functional.shaders.texture_functions.textureprojlodoffset.sampler2dshadow_vertex.

This is probably a dEQP bug.

Fixes: d573d1d82524b8a2e5f56938069cabc0f0176a0e
Acked-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5209>

radeonsi: pass at most 3 images and/or shader buffers via user SGPRs for compute

This should slightly decrease shader lifetime.

Acked-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5209>

radeonsi: remove const_buffers_declared hacks

This was a bug that was uncovered by 4553fc66a5f23607c2e872d8ac8755c747bd0bd2.

Piglit: spec@arb_uniform_buffer_object@maxblocks

Fixes: 4553fc66a5f23607c2e872d8ac8755c747bd0bd2
Acked-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5209>

radeonsi: remove unused leftover code for INDIRECT_BUFFER inside IBs

Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5209>

nir: gather which images are MSAA

Reviewed-by: Jason Ekstrand <jason@jlekstrand.net>
Reviewed-by: Eric Anholt <eric@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5209>

nir: gather which images are buffers

Reviewed-by: Jason Ekstrand <jason@jlekstrand.net>
Reviewed-by: Eric Anholt <eric@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5209>

nir: don't count samplers and images in interface blocks

Acked-by: Jason Ekstrand <jason@jlekstrand.net>
Reviewed-by: Eric Anholt <eric@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5209>

ac/nir: support v2f16 derivatives

Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5003>

ac/nir: set the second v_cvt_pkrtz argument to undef if it's unused

Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5003>

ac/nir: select v_cvt_pkrtz for all conversions from f32 to f16 for radeonsi

Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5003>

ac/nir: handle nir_op_[fiu]2[fiu]mp opcodes

Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5003>

ac/nir: support 16-bit data in image opcodes

Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5003>

ac/nir: support 16-bit data in buffer_load_format opcodes

Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5003>

ac/nir: remove type and num_channels args from ac_build_buffer_store_common

They were only used for type overloading where we can just use
the type of data.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5003>

ac/nir: support vector types in the type suffix of overloaded intrinsics

Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5003>

ac/nir: use more types from ac_llvm_context

Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5003>

ac: rename has_double_rate_fp16 -> has_packed_math_16bit

Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5003>