amd/common: Add total alignment calculation.

Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5194>

radv: Allocate values/predicates at the end of the image.

Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5194>

radv: Disable HTILE in ac_surface.

Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5194>

radv: Disable DCC in ac_surface.

Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5194>

radv: Use offsets in surface struct.

Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5194>

radv: Rely on ac_surface for avoiding cmask for linear images.

Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5194>

radv: Enforce the contiguous memory for DCC layers in ac_surface.

Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5194>

radv: Pass no_metadata_planes info in to ac_surface.

Also do not allocate aux surfaces for multi-plane images. I may
have messed up and used plane 1 offsets for the other planes as well.
I cannot imagine that sharing aux surfaces between the planes will
work well.

Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5194>

radv: Use ac_surface to determine fmask enable.

Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5194>

ci: add U-Boot specific fetch strings

U-Boot's fastboot over udp generates the following output:
  Listening for fastboot command on x.y.z.w

Also add a general 'data abort' error string seen with an
too old U-Boot version:
  https://github.com/u-boot/u-boot/commit/95712af

Signed-off-by: Christian Gmeiner <christian.gmeiner@gmail.com>
Reviewed-by: Eric Anholt <eric@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5258>

ci: extend expect-output.sh

We need to support different fastboot fetch strings for different
bootloader solutions. Lets extend expect-output.sh to support
multiple fetch strings (-f) and add support for error catch
strings (-e) to stop the CI run early.

Signed-off-by: Christian Gmeiner <christian.gmeiner@gmail.com>
Reviewed-by: Eric Anholt <eric@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5258>

freedreno/computerator: fix missing dependency on generated header

Fixes:
```
 ../mesa-freedreno-20.2.0_pre/src/freedreno/computerator/ir3_asm.c:25:10: fatal error: 'ir3/ir3_parser.h' file not found
 #include "ir3/ir3_parser.h"
          ^~~~~~~~~~~~~~~~~~
 1 error generated.
```

Fixes: da467817e3e ("freedreno/ir3: Move ir3 assembler to backend compiler")
Signed-off-by: Rob Clark <robdclark@chromium.org>
Reviewed-by: Eric Engestrom <eric@engestrom.ch>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5340>

glapi: remove deprecated .getchildren() that has been replace with an iterator

Cc: mesa-stable
Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/3086
Signed-off-by: Eric Engestrom <eric@engestrom.ch>
Reviewed-by: Vinson Lee <vlee@freedesktop.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5342>

radv/aco: enable 64-bit atomic features if RADV is linked with LLVM 8

Just in case someone links RADV with this old LLVM 8 and wants ACO.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5331>

svga: Performance fixes

This is a squash commit of in house performance fixes and misc bug fixes
for GL4.1 support.

Performance fixes:
* started using system memory for constant buffer to gain 3X performance boost with metro redux

Misc bug fixes:
* fixed usage of vertexid in shader
* added empty control point phase in hull shader for zero ouput control point
* misc shader signature fixes
* fixed clip_distance input declaration
* clearing the dirty bit for the surface while using direct map if surface is already flushed
  and there is no pending primitive

This patch also uses SVGA_RETRY macro for commands retries. Part of it is already
used in previous patch.

Reviewed-by: Brian Paul <brianp@vmware.com>
Reviewed-by: Charmaine Lee <charmainel@vmware.com>
Signed-off-by: Neha Bhende <bhenden@vmware.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5317>

svga: Add GL4.1(compatibility profile) support in svga driver

This patch is a squash commit of a very long in-house patch series.

Reviewed-by: Brian Paul <brianp@vmware.com>
Reviewed-by: Charmaine Lee <charmainel@vmware.com>
Signed-off-by: Neha Bhende <bhenden@vmware.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5317>

svga/include: Headers for GL4.1 support

This brings in the new types, enums and #defines for GL 4.1
features in the virtual device.

Reviewed-by: Brian Paul <brianp@vmware.com>
Reviewed-by: Charmaine Lee <charmainel@vmware.com>
Signed-off-by: Neha Bhende <bhenden@vmware.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5317>

winsys/drm: Add GL4.1 support in drm winsys

This is to check whether virtual hardware has SM5 support

Reviewed-by: Brian Paul <brianp@vmware.com>
Reviewed-by: Charmaine Lee <charmainel@vmware.com>
Signed-off-by: Neha Bhende <bhenden@vmware.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5317>

util: Add util functionality for GL4.1 support

This patch adds the following tgsi utilities

* tgsi_dynamic_indexing: This utility flattens out the dyanamic indexing of constant buffers
* tgsi_vpos: This utility writes zeros to position at index 0 in vertex shader.
  This utility can be used if there is no shader output in vertex shader
* util_make_tess_ctrl_passthrough_shader: This adds passthough tessellation control shader.
  Input of passthrough tess ctrl shader is output of vertex shader
  and output is input of tessellation eval shader.
  If program has tessellation eval shader but no tessellation control shader,
  this utility can be used to create passthrough tessellation control shader.

Reviewed-by: Brian Paul <brianp@vmware.com>
Reviewed-by: Charmaine Lee <charmainel@vmware.com>
Signed-off-by: Neha Bhende <bhenden@vmware.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5317>

freedreno/a6xx: more early-z

Technically we only have to do late-z in the alpha-test or discard case
if depth-write is enabled.  If depth write is disabled, the depth read /
test / conditional-write interlock that we need to emulate is not a
problem, so we can still use early-z test.

There is a slightly weird case when there is no zsbuf attachment (see
dEQP-GLES31.functional.fbo.no_attachments.*) where the hw wants us to
use LATE_Z.. not entirely sure if this is an interaction with occlusion
query or just a pecularity of how the hw works when there is no depth
buffer.

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5336>

ci: bump virglrenderer to latest version

Need this for upcoming GL 4.0 llvmpipe support.

Reviewed-by: Elie Tournier <elie.tournier@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5323>

turnip: Simplify vertex buffer bindings.

We were remapping the bindings so the HW binding points were consecutive,
which there's no need for.  Now that we don't shuffle, we can mostly drop
the dependency on the pipeline for this SDS.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5321>

turnip: Don't bother clamping VB size.

From the VK spec: "All elements of pOffsets must be less than the size of
the corresponding element in pBuffers"

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5321>

turnip: Move vertex buffer bindings to SET_DRAW_STATE.

This means that the HW can skip over the vertex buffer state when it's not
used in a bin.  The blob also has this behavior.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5321>

llvmpipe: move coroutines out of noopt case

the virgl CI code was using the noopt path and crashing with a
wierd can't select llvm.coro.subfn.addr error, turns out we have
to call the cleanup pass no matter what.

This enable a lot more virgl gles31 passes, but we have
to disable tessellation shaders as now they executed, they
crash due to missing OES_gpu_shader5, I should try and reenable
them when llvmpipe is further along

Fixes: d32690b43c91d ("gallivm: add coroutine pass manager support")
Reviewed-by: Roland Scheidegger <sroland@vmware.com>
Acked-by: Elie Tournier <elie.tournier@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5320>

pan/mdg: Ensure ld_vary_16 is aligned

Otherwise packing may fail.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Fixes: 5f8dd413bcc ("pan/mdg: Handle 16-bit ld_vary")
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5339>

freedreno/a6xx: Fix VFD_CONTROL emit

The FETCH_CNT field isn't actually the FETCH count. We don't have a
lot of data where it's different from DECODE_CNT, so there's not much
to go by. It could be number of VFD_DEST_CNTL or maybe DECODE_CNT for
binning.  For now, setting both to number of DEST_CNTL gets Google
Earth working again.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5324>

radv: Always expose non-visible local memory type on dedicated GPUs

DOOM Eternal expects this type, but RADV doesn't expose it when the VRAM
is entirely host-visible, in my case on Fiji. Matches AMDVLK behavior.

Closes: https://gitlab.freedesktop.org/mesa/mesa/issues/3054
Cc: <mesa-stable@lists.freedesktop.org>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5308>

pan/mdg: Legalize inverts with constants

We need to force src_invert to be in the right place even if we flip
when lowering an embedded->inline constant.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Fixes: 449e5ded934 ("pan/mdg: Treat inot as a modifier")
Reported-by: Icecream95 <ixn@keemail.me>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5299>

nir: reuse existing psiz-variable

For shaders where there's already a psiz-variable, we should rather
reuse it than create a second one. This can happen if a shader writes
gl_PointSize, but disables GL_PROGRAM_POINT_SIZE.

Fixes: 878c94288a8 ("nir: add lowering-pass for point-size mov")
Reviewed-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5328>

i965: fix export of GEM handles

We reuse DRM file descriptors internally. Therefore when we export a
GEM handle we must do so in the file descriptor used externally.

v2: Fix dmabuf leak
    Fix GEM handle leaks by tracking exported handles

v3: Check os_same_file_description error (Michel)
    Don't create multiple exports for a given GEM table

v4: Add WARN_ONCE (Ken)

v5: Remove blank line (Ian)
    Remove unused field (Ian)

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/2882
Fixes: 4094558e8643 ("i965: share buffer managers across screens")
Tested-by: Eric Engestrom <eric@engestrom.ch>
Tested-by: Tapani Pälli <tapani.palli@intel.com>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4861>

iris: fix export of GEM handles

We reuse DRM file descriptors internally. Therefore when we export a
GEM handle we must do so in the file descriptor used externally.

This change also fixes a file descriptor leak of the FD given at
screen creation.

v2: Don't bother checking fd equals, they're always different
    Fix dmabuf leak
    Fix GEM handle leaks by tracking exported handles

v3: Check os_same_file_description error (Michel)
    Don't create multiple exports for a given GEM table

v4: Add WARN_ONCE (Ken)
    Rename external_fd to winsys_fd

v5: Remove export lock in favor of bufmgr's

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/2882
Fixes: 7557f1605968 ("iris: share buffer managers accross screens")
Tested-by: Eric Engestrom <eric@engestrom.ch>
Tested-by: Tapani Pälli <tapani.palli@intel.com>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4861>

i965: don't forget to set screen on duped image

We'll start using this field more for querying image properties.
Without it we run into a crash.

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Cc: <mesa-stable@lists.freedesktop.org>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4861>

iris: fix BO destruction in error path

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Cc: <mesa-stable@lists.freedesktop.org>
Tested-by: Tapani Pälli <tapani.palli@intel.com>
Reviewed-by: Ian Romanick <ian.d.romanick@intel.com>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4861>

mesa: Fix NetBSD compiler macro.

Reported-by: Rafał Mikrut <mikrutrafal54@gmail.com>
Fixes: a63b90712aad ("mesa: also check for __NetBSD__")
Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/3015
Signed-off-by: Vinson Lee <vlee@freedesktop.org>
Reviewed-by: Eric Engestrom <eric@engestrom.ch>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5191>

freedreno/a6xx: also consider alpha-test for ztest-mode

Looks like we don't have CI coverage for this (since deqp==GLES) but
alpha test is conceptually the same as frag shaders with discard, and
should be handled as such.

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5298>

freedreno/a6xx: add early-lrz-late-z mode

Now that we are doing a better job of managing LRZ, add support for the
EARLY_LRZ_LATE_Z mode.  Since we properly disable LRZ write in cases
where we don't know a fragment's z value during the binning pass (or
when blend is enabled in a later draw, meaning we will need the earlier
fragment's color), we can enable a mode that keeps the early-lrz test
when the frag shader has kill/discard.  This will only discard geometry
that is definitely not visible.

This is a pretty big win for games/benchmarks that have a lot of frag
shaders with kill/discard.  More than 10% gain for gfxbench trex/mh and
40% gain for mh31.

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5298>

freedreno/a6xx: re-work LRZ state tracking

In particular, properly detect reversal of depth-test direction.
With that we can remove a lot of cases where we were unnecessarily
invalidating LRZ, which was simply papering over the direction-
reversal issue in deqp.

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5298>

freedreno/a6xx: update depth-plane control regs

And document the early-lrz-late-z mode.

Initially I thought this would be two bits to control early-lrz vs
early-z.  But having early-z without early-lrz does not make sense,
and the way the values line up makes an enum fit better.

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5298>

freedreno/a6xx: sync registers from envytools

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5298>

freedreno/ir3: split kill from no_earlyz

Unlike other conditions which prevent early-discard of fragments, kill
does not prevent early LRZ test.  Split `has_kill` from `no_earlyz` so
we can take advantage of this.

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5298>

docs/features.txt: Update for freedreno

We've had GL_OES_texture_cube_map_array for a while for a4xx+ and
support for geometry and tessellation for a6xx+.

Reviewed-by: Eric Anholt <eric@anholt.net>
Reviewed-by: Rob Clark <robdclark@chromium.org>
Acked-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5319>

freedreno/a6xx: Turn on robustness extensions

With UBO access going through LDC, all memory access uses buffer based
io primitives.  We can then advertise
PIPE_CAP_ROBUST_BUFFER_ACCESS_BEHAVIOR and
PIPE_CAP_DEVICE_RESET_STATUS_QUERY, which turn on GL_EXT_robustness,
GL_KHR_robust_buffer_access_behavior and GL_KHR_robustness.

Reviewed-by: Eric Anholt <eric@anholt.net>
Reviewed-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5319>

vdpau: Fix wrong calloc sizeof argument.

Fix warning reported by Coverity Scan.

Wrong sizeof argument (SIZEOF_MISMATCH)
suspicious_sizeof: Passing argument 3544UL (sizeof
(vlVdpPresentationQueue)) to function calloc that returns a pointer of
type vlVdpPresentationQueueTarget * is suspicious because a multiple of
sizeof (vlVdpPresentationQueueTarget) /*16*/ is expected.

Fixes: 65fe0866aec7 ("vl: implemented a few functions and made stubs to get mplayer running")
Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/3026
Signed-off-by: Vinson Lee <vlee@freedesktop.org>
Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5182>

OPTIONAL: iris: Perform BLORP buffer barriers outside of iris_blorp_exec() hook.

The iris_blorp_exec() hook needs to be executed under a single
indivisible sync region, which means that in cases where we need to
emit a PIPE_CONTROL for a buffer barrier we won't be able to track the
subsequent commands separately from the previous commands, which will
prevent us from optimizing out subsequent PIPE_CONTROLs if we
encounter the same buffers again.  In particular I've encountered this
situation in some SynMark test-cases which perform lots of BLORP
operations with the same buffer bound as both source and destination
(in order to generate mipmaps): In such a scenario if the source
requires flushing we'd also end up flushing for the destination
redundantly, even though a single PIPE_CONTROL would have been
sufficient.

This avoids a 4.5% FPS regression in SynMark OglHdrBloom and a 3.5%
FPS regression in SynMark OglMultithread.

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3875>

iris: Remove iris_flush_depth_and_render_caches().

This helper is unused now.

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3875>

iris: Emit single render target flush PIPE_CONTROL on format mismatch.

The big-hammer iris_flush_depth_and_render_caches() is largely
redundant whenever a format mismatch is detected from
iris_cache_flush_for_render().  There is no need to kick the depth,
sampler nor constant caches in that case.

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3875>

iris: Open-code iris_cache_flush_for_read() and iris_cache_flush_for_depth().

These have become one-liners now so they can be easily inlined.

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3875>

iris: Remove render cache hash table-based synchronization.

The render cache hash table is now *mostly* redundant with the more
general seqno matrix-based cache tracking mechanism.  Most hash table
operations are now gone except for the format mismatch checks done in
iris_cache_flush_for_render().  Redundant code removed as a separate
patch for bisectability.

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3875>

iris: Remove depth cache set tracking and synchronization.

The depth cache set is now redundant with the more general seqno
matrix-based cache tracking mechanism.  Removed as a separate patch
for bisectability.

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3875>

iris: Perform compute predraw flushes from compute batch.

Whenever iris_predraw_resolve_inputs() ends up doing a flush or
invalidate, we really want it to be on the same batch which is going
to consume the result.  Any resolves should still be performed from
the render batch thanks to the previous patch.

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3875>

iris: Remove batch argument of iris_resource_prepare_access() and friends.

The resolves performed by this function are only expected to work from
the render batch, so make sure we use it independently of the batch
the caller wants to use.  This function provides no synchronization
guarantees anyway, the caller is expected to insert any cache flushing
and synchronization required for the resolved surface to be visible to
the target batch.

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3875>

iris: Insert buffer barrier in existing cache flush helpers.

As a first step to phasing out the current hashtable-based depth and
render cache tracking mechanisms.

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3875>

iris: Implement buffer-local memory barrier based on cache coherency matrix.

This takes advantage of the previously introduced cache tracking
infrastructure in order to define a multi-purpose barrier operation
that allows the caller to order memory operations with respect to
previous operations performed on the same buffer from any other cache
domain.

v2: Assorted CPU overhead micro-optimizations (Francisco).
v3: Use C99 designated initializers (Ken).

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3875>

iris: Update cache coherency matrix on PIPE_CONTROL.

This introduces a batch synchronization boundary at every PIPE_CONTROL
command, and updates the cache coherency status tracked during batch
construction according to the specified control bits.

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3875>

iris: Introduce cache coherency matrix for batch-local memory ordering.

This introduces a representation of the cache coherency status of the
GPU at any point in the batch.  This is done by defining a matrix C of
synchronization sequence numbers such that at any point of batch
construction, a memory operation from domain i introduced into the
batch is guaranteed to be ordered after any memory operation from
domain j in a previous batch section with seqno n if the following
condition holds:

  C_i_j >= n

This allows us to efficiently determine whether additional flushing
and/or invalidation is required in order to access a buffer object
from some arbitrary domain.

Except for batch buffer reset which requires clearing the whole
matrix, all operations on the matrix are either O(n) or O(1) on the
number of caching domains (which is basically constant).

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3875>

iris: Report use of any in-flight buffers on first draw call after sync boundary.

This is the main performance trade-off of this cache tracking
mechanism: In order for the seqno vector of buffer objects to be
accurate, they need to be marked as used again every time the batch is
split into a new synchronization section if they remain bound to the
pipeline.  This can be achieved easily by re-using
iris_restore_render_saved_bos() and iris_restore_compute_saved_bos(),
which currently serve a similar purpose across batch buffer
boundaries.

The impact on Piglit drawoverhead results seems to be within a
standard deviation of the current results.

XXX - It might be possible to completely remove the current
      iris_batch::contains_draw flag at a small additional performance
      cost.

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3875>

iris: Drop redundant iris_address::write flag.

The write flag is redundant since it can be inferred easily from the
iris_address::access domain.  This allows the iris_address struct to
be laid out more efficiently in memory, leading to a measurable
improvement in several Piglit Drawoverhead test-cases.

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3875>

iris: Annotate all BO uses with domain and sequence number information.

Probably the most annoying patch to review from the whole series --
Mark every buffer object use as accessed through some caching domain
with the sequence number of the current synchronization section of the
batch.  The additional argument of iris_use_pinned_bo() makes sure I'd
have gotten a compile error if I had missed any buffer added to the
batch validation list.

There are only a few exceptions where a buffer is left untracked while
adding it to the validation list, justified below:

 - Batch buffers: These are strictly read-only for the moment.

 - BLORP buffer objects: Their seqnos are bumped manually at the end
   of iris_blorp_exec() instead, in order to avoid plumbing domain
   information through BLORP address combining.

 - Scratch buffers: The contents of these are strictly thread-local.

 - Shader images and SSBOs: Accesses of these buffers are explicitly
   synchronized at the API level.

v2: Opt out of tracking more aggressively (Ken): In addition to the
    above, surface states, binding tables, instructions and most
    dynamic states are now left untracked, which means a *lot* more BO
    uses marked IRIS_DOMAIN_NONE which need to be reviewed extremely
    carefully, since the cache tracker won't be able to provide any
    coherency guarantees for them.

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3875>

iris: Bracket batch operations which access memory within sync regions.

This delimits all batch operations which access memory between
iris_batch_sync_region_start() and iris_batch_sync_region_end() calls.
This makes sure that any buffer objects accessed within the region are
considered in use through the same caching domain until the end of the
region.

Adding any buffer to the batch validation list outside of a sync
region will lead to an assertion failure in a future commit, unless
the caller explicitly opted out of the cache tracking mechanism.

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3875>

iris: Add infrastructure to partition batch into sync boundaries.

This introduces some minimalistic infrastructure which will be used in
order to partition the batch into a series of sections, each one with
a unique, monotonically-increasing sequence number.  Section
boundaries will typically lie at points in the batch where the
execution and memory coherency status of some previous commands are
known, e.g. at batch buffer boundaries or PIPE_CONTROL commands.

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3875>

iris: Add batch-local synchronization book-keeping to iris_bo.

The purpose of this is to represent the cache coherency state of a
buffer as a vector of integers (AKA seqnos), one for each incoherent
caching domain of the GPU.  A seqno will identify a single section of
a batch buffer uniquely across the whole pipe_screen (which means that
there will be no ambiguity about what context a given seqno belongs to
even if there are multiple threads accessing the same buffer in
parallel), and is guaranteed to be allocated in monotonically
increasing order within any given context.  The iris_bo_bump_seqno()
helper is provided for marking the last update of a buffer from a
given caching domain in a lockless manner.

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3875>

panfrost: Mark point sprites as todo on Bifrost

Emulating them will be a rather annoying dance. Let's not worry about
this until further down the line when we have a better sence of how to
do handle them efficiently.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5290>

panfrost: Fix gl_PointSize out of GL_POINTS

In this case, vs->writes_point_size is true as the VS writes
gl_PointSize, but panfrost_writes_points_size() is false as we are not
drawing points so the hardware doesn't process it. Thus the varying
descriptor is emitted but elements is never written. When the VS runs,
it will attempt to write to elements, a NULL pointer.

The behaviour is architecture-independent. On Midgard, the write
silently fails, hence why this bug was never noticed before. On Bifrost,
this raises an MMU fault.

The fix is to set the format to VARYING_DISCARD to ignore the write.

Noticed on Neverball.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5290>

panfrost: Prefer sysval for gl_PointCoord on Bifrost

It's like gl_FragCoord. Still not implemented. This unfortunately makes
point sprites a lot more complicated.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5290>

pan/bi: Disassemble gl_PointCoord reads.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5290>

panfrost: Explicitly convert to 32-bit for logic-ops

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Reported-by: Icecream95 <ixn@keemail.me>
Fixes: 19b4e586f62 ("panfrost: Switch to pan_lower_framebuffer")
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5289>

panfrost: Readd MIDGARD_SHADERLESS quirk to t760

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Reported-by: Icecream95 <ixn@keemail.me>
Fixes: e53d27de61b ("panfrost: Add quirks for blend shader types")
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5289>

iris: Extend iris_context dirty state flags to 128 bits.

We're nearly out of dirty bits, and some patches pending review on
GitLab no longer apply due to that.  Make room for them by splitting
off shader stage-specific bits into a separate stage_dirty mask.

An alternative would be to split compute-related bits into a separate
mask, but that would prevent the '<< stage' indexing done in various
parts of the driver from working.

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5279>

iris: Simplify iris_batch_prepare_noop().

This makes iris_batch_prepare_noop() return a boolean instead of
passing through the relevant set of dirty flags.  It will make it
easier to change the representation of dirty flags.

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5279>

nir/lower_tex: fixes for fp16 yuv lowering

Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/3079
Signed-off-by: Rob Clark <robdclark@chromium.org>
Reviewed-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Reviewed-by: Eric Anholt <eric@anholt.net>
Reviewed-by: Kristian H. Kristensen <hoegsberg@google.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5318>

nir/builder: add bitsize conversion helpers

Signed-off-by: Rob Clark <robdclark@chromium.org>
Reviewed-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Reviewed-by: Eric Anholt <eric@anholt.net>
Reviewed-by: Kristian H. Kristensen <hoegsberg@google.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5318>

nir: extract out convert_to_bitsize() helper

Signed-off-by: Rob Clark <robdclark@chromium.org>
Reviewed-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Reviewed-by: Eric Anholt <eric@anholt.net>
Reviewed-by: Kristian H. Kristensen <hoegsberg@google.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5318>

nir: get_base_type() should return enum type

Needed by the next patch, for c++ code which is more strict about
conversions between integers and enums.

Signed-off-by: Rob Clark <robdclark@chromium.org>
Reviewed-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Reviewed-by: Eric Anholt <eric@anholt.net>
Reviewed-by: Kristian H. Kristensen <hoegsberg@google.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5318>

panfrost: Handle writes_memory correctly

We need to pass it thru to EARLY_Z and WRITES_GLOBAL instead of ignoring
and assuming respectively. Nontrivial performance fix.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5300>

panfrost: Document MALI_WRITES_GLOBAL bit

We've been setting this unconditionally -- oops!

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5300>

panfrost: Update MALI_EARLY_Z description

Via the ES3.1 early-z testing force, I've confirmed this bit is e-z.
I've also confirmed e-z must be disabled for global writes, as expected.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5300>

iris: remove unused iris_bo->swizzle_mode

Signed-off-by: Marcin Ślusarz <marcin.slusarz@intel.com>
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5313>

aco: sign-extend input/identity for 16-bit subgroup ops on GFX6-GFX7

16-bit subgroup ops are implemented with 32-bit instructions
on GFX6-GFX7.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5227>

aco: fix subdword copies on GFX6-GFX7

SDWA is only GFX8+. Use v_mov_b32 since the upper 16 bits don't matter.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5227>

aco: implement 16-bit nir_intrinsic_quad_* on GFX6-GFX7

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5227>

aco: implement 16-bit reduce operations on GFX6-GFX7

No fp16 on GFX6-GFX7.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5227>

pan/bi: Handle vectorized load_const

In preparation for 16-bit vectors.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5307>

pan/bi: Passthrough second argument of F32_TO_F16

At the NIR level this is a second vector source of the first (only)
argument; at the BIR level this is a pair of scalars.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5307>

pan/bi: Pack second argument of F32_TO_F16

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5307>

pan/bi: Fix SEL.16 swizzle

2 scalar arguments, not 1 vector.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5307>

pan/bi: Handle SEL with vec3 16-bit

Otherwise we end up with a missing argument.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5307>

panfrost: Passthrough NATIVE loads/stores

Now that we handle load_output directly, this works for e.g. RGB565 on
Midgard.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5285>

pan/mdg: Handle regular nir_intrinsic_load_output

Instead of the vendored version. Only for blend shaders at the moment,
frag shaders fb_fetch has a lot more going on.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5285>

pan/mdg: Allow f2u8 and friends thru

Now that we can handle destination sizes directly, this keeps us from
needing to chew through so many conversions.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5285>

pan/mdg: Handle f2u8

This is similar to f2u16.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5285>

pan/mdg: Fold roundmode into applicable instructions

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5285>

pan/mdg: Implement *_rtz conversions with roundmode

Use rte as the canonical type.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5285>

pan/mdg: Lower roundmodes

So now we can use the IR field semantically.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5285>

pan/mdg: Add opcode roundmode property

When the output is rounded in a specified direction.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5285>

pan/mdg: Add roundmode enum

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5285>

pan/mdg: Distinguish blend shaders in internal shader-db

Since these shaders are purely internal, the optimization criteria are a
bit different, so it's worth calling attention to this when dumping.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5285>

panfrost: Only use AFBC YTR with RGB and RGBA

The "lossless colorspace transform" is lossy for R and RG formats.

Reviewed-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5293>

panfrost: Decode AFBC flag bits

Reviewed-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5293>

glsl: when NIR linker enable use it to resize uniform arrays

Here we turn on uniform array resizing in the NIR linker and disable
the GLSL IR resizing pass when the NIR linker is enabled.

This will potentially make uniform arrays smaller due to NIR
optimising away more uniform uses.

Shader-db results (SKL):

total instructions in shared programs: 14947192 -> 14944093 (-0.02%)
instructions in affected programs: 138088 -> 134989 (-2.24%)
helped: 822
HURT: 4

total cycles in shared programs: 324868402 -> 324794597 (-0.02%)
cycles in affected programs: 3904170 -> 3830365 (-1.89%)
helped: 2333
HURT: 1485

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4910>