intel/gen12+: Reserve 4KB of URB space per bank for Compute Engine

This patch is required to fix 11K+ vulkan CTS failures we were
getting with way_size_per_bank of 4 (see next patch).

Thanks to Sagar Ghuge and Jordan Justen for all the hard work of
debugging and testing.

Signed-off-by: Anuj Phogat <anuj.phogat@gmail.com>
Reviewed-by: Jason Ekstrand <jason@jlekstrand.net>
Reviewed-by: Sagar Ghuge<sagar.ghuge@intel.com>

virgl: Use align_free for align_malloc allocated buffer

Signed-off-by: Szymon Andrzejuk <s.andrzejuk@samsung.com>
Reviewed-by: Gurchetan Singh <gurchetansingh@chromium.org>

freedreno/drm: readonly cmdstream

Noticed that we weren't consistently making cmdstream buffers
gpu-readonly.  Fix that and drop the need to pass flags to
fd_bo_new_ring().

Signed-off-by: Rob Clark <robdclark@chromium.org>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3663>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3663>

intel/fs: Write the address register with NoMask for MOV_INDIRECT

This fixes a hang in the following Vulkan CTS test on TGL-LP:

    dEQP-VK.descriptor_indexing.storage_buffer_dynamic_in_loop

Cc: mesa-stable@lists.freedesktop.org
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3642>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3642>

intel/tools: Handle strides better when dumping buffers

The old code would only break at stride boundaries if the stride was
less than 32B; otherwise it would just break every 32B.  This commit
makes it break at stride boundaries and 32B boundaries (starting from
the last stride).  This makes reading large vertex buffers in aubinator
much nicer.

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3642>

intel/disasm: SEND has two sources on Gen12+

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3642>

intel/eu/validate: Don't validate regions of sends

Otherwise, the validator tries to read the type of src1 of a SEND/SENDS
which doesn't actually have a type field.  This prevents validation
issues in the next commit.

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3642>

aco: fix image_atomic_cmp_swap

Fixes: 71440ba0f5512fe455be66ca48b253ecc37478a9 ('aco: reorder VMEM operands in ACO IR')
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3652>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3652>

aco: fix MUBUF VS input loads when expanding vec3 to vec4 on GFX6

When some unused channels are skipped and that we expand vec3 loads
to vec4 loads, we have to adjust the fourth component.

While we are at it, add an assertion to make sure we don't use
MUBUF for vec3 loads on GFX6.

Closes: https://gitlab.freedesktop.org/mesa/mesa/issues/2450
Closes: https://gitlab.freedesktop.org/mesa/mesa/issues/2442
Fixes: 6aecc316 ("aco: fix VS input loads with MUBUF on GFX6")
Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3641>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3641>

gallium/swr: Fix gcc 4.8.5 compile error

Stop using C++14 feature so it can be compile on default centos7
gcc compiler.

Reviewed-by: Jan Zielinski <jan.zielinski@intel.com>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3640>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3640>

swr: Fix build with GCC 10.

GCC 10 added _mm256_storeu2_m128i.
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=91341

This patch fixes this build error with GCC 10.

In file included from src/gallium/drivers/swr/rasterizer/codegen/gen_knobs.cpp:39:
../src/gallium/drivers/swr/rasterizer/common/os.h:178:20: error: ‘void _mm256_storeu2_m128i(__m128i*, __m128i*, __m256i)’ redeclared inline without ‘gnu_inline’ attribute
  178 | static INLINE void _mm256_storeu2_m128i(__m128i* hi, __m128i* lo, __m256i a)
      |                    ^~~~~~~~~~~~~~~~~~~~
In file included from /usr/lib/gcc/x86_64-redhat-linux/10/include/immintrin.h:51,
                 from /usr/lib/gcc/x86_64-redhat-linux/10/include/x86intrin.h:32,
                 from ../src/gallium/drivers/swr/rasterizer/common/os.h:107,
                 from src/gallium/drivers/swr/rasterizer/codegen/gen_knobs.cpp:39:
/usr/lib/gcc/x86_64-redhat-linux/10/include/avxintrin.h:1580:1: note: ‘void _mm256_storeu2_m128i(__m128i_u*, __m128i_u*, __m256i)’ previously defined here
 1580 | _mm256_storeu2_m128i (__m128i_u *__PH, __m128i_u *__PL, __m256i __A)
      | ^~~~~~~~~~~~~~~~~~~~

Signed-off-by: Vinson Lee <vlee@freedesktop.org>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3650>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3650>

gallium/swr: fix gcc warnings

Few changes to make gcc happy.

Reviewed-by: Jan Zielinski <jan.zielinski@intel.com>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3629>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3629>

zink: implement support for derivative-control

Reviewed-by: Dave Airlie <airlied@redhat.com>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3645>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3645>

zink: implement load_instance_id

Reviewed-by: Dave Airlie <airlied@redhat.com>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3644>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3644>

zink: enable texture-buffer objects

This seems to work as-is, and just need enabling. There's a few piglit
failures, but those seems to be problems with the tests, where they
don't handle lacking GL3-support.

Reviewed-by: Dave Airlie <airlied@redhat.com>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3647>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3647>

radeonsi: Add support for midstream bitrate change in encoder

BACKPORT: Remove |picture| argument from enc->begin in radeon_vcn_enc.c

Signed-off-by: Satyajit Sahu <satyajit.sahu@amd.com>
Reviewed-by: Boyuan Zhang <Boyuan.Zhang@amd.com>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3426>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3426>

panfrost: Use DBG macro to avoid noise in the console

It pollutes the output of programs that use Panfrost and can confuse its
callers, such as test runners.

Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Reviewed-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3625>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3625>

pan/midgard: Handle nir_intrinsic_load_barycentric_centroid

To avoid hitting the assert in the default case, add a nop for this
intrinsic.

dEQP-GLES3.functional.transform_feedback.random.interleaved.lines.3

Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Reviewed-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3625>

panfrost: Add more info to some assertions

Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Reviewed-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3625>

panfrost: Print intended field when decoding

Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Reviewed-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3625>

anv: Always fill out the AUX table even if CCS is disabled

Cc: "20.0" mesa-stable@lists.freedesktop.org
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3454>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3454>

iris: Plumb deref block size through to 3DSTATE_SF

Cc: "20.0" mesa-stable@lists.freedesktop.org
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3454>

anv: Plumb deref block size through to 3DSTATE_SF

Cc: "20.0" mesa-stable@lists.freedesktop.org
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3454>

intel/blorp: Plumb deref block size through to 3DSTATE_SF

Cc: "20.0" mesa-stable@lists.freedesktop.org
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3454>

intel/common: Return the block size from get_urb_config

Cc: "20.0" mesa-stable@lists.freedesktop.org
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3454>

anv: Emit URB setup earlier

Cc: "20.0" mesa-stable@lists.freedesktop.org
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3454>

iris: Consolodate URB emit

Now that we don't have to carry a URB state emit function for BLORP we
can roll some stuff together and drop a genX helper.

Cc: "20.0" mesa-stable@lists.freedesktop.org
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3454>

intel/blorp: Always emit URB config on Gen7+

Previously, i965/iris tried to reuse the currently programmed URB config
if it was good enough for BLORP, rather than reprogramming it each time.
However, this will make some things harder on Gen12+ and we've not seen
any performance impact from emitting URB more frequently in ANV.

This makes the blorp <-> driver interface a bit simpler on Gen7+ because
now all the driver has to do is to provide the L3$ config rather than
trying to hand off URB re-config to blorp.

Cc: "20.0" mesa-stable@lists.freedesktop.org
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3454>

intel: Take a gen_l3_config in gen_get_urb_config

Instead of making each driver pass in the same push constant size and do
it's own L3$ config URB size calculation, just make them pass in their
L3$ configuration.

Cc: "20.0" mesa-stable@lists.freedesktop.org
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3454>

i965: Re-emit l3 state before BLORP executes

If BLORP is the first thing to execute, we may not have set the L3$
config yet.  That's not normally a problem but we're about to add code
to BLORP which will look at brw_context::l3::config and we'd like that
to be initialized.  It's also just good practice.

Cc: "20.0" mesa-stable@lists.freedesktop.org
Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3454>

iris: Use the URB size from the L3$ config

Cc: "20.0" mesa-stable@lists.freedesktop.org
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3454>

iris: Store the L3$ configs in the screen

We only calculate them based on device info and never change them so
this seems like a reasonable place to put them.  We could also put them
in the context, but that's not accessible from iris_init_*_context.

Cc: "20.0" mesa-stable@lists.freedesktop.org
Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3454>

iris: Set SLMEnable based on the L3$ config

Cc: "20.0" mesa-stable@lists.freedesktop.org
Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3454>

intel/genxml: Drop SLMEnable from L3CNTLREG on Gen11

SML is no longer in the L3$ on Gen11+.  It's not incredibly clear from
the docs but no Gen11 platforms are in the list of platforms on which
this bit exists.  Also, we've been always setting it false on Gen11 in
ANV and i965 thanks to GEN_L3P_SLM being zero with no ill effects.

Cc: "20.0" mesa-stable@lists.freedesktop.org
Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3454>

anv,iris: Set 3DSTATE_SF::DerefBlockSize to per-poly on Gen12+

According to the BSpec, this should prevent hangs when using shaders
with large URB entries.  A more precise fix can be done but it requires
re-arranging URB setup.

Cc: mesa-stable@lists.freedesktop.org
Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3454>

genxml: Add a new 3DSTATE_SF field on gen12

Cc: mesa-stable@lists.freedesktop.org
Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3454>

docs/release-calendar: 20.0.0-rc1 has been released

turnip: Fix vkCmdCopyQueryPoolResults with available flag

Previously, calling vkCmdCopyQueryPoolResults with the
VK_QUERY_RESULT_WITH_AVAILABILITY_BIT flag set the query result
field in the buffer to 0 if unavailable and the query result if
available. This was a misunderstanding of the Vulkan spec, and this
commit corrects the behavior to emitting a separate available
result in addition to the query result.

Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3560>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3560>

turnip: Fix vkGetQueryPoolResults with available flag

Previously, calling vkGetQueryPoolResults with the
VK_QUERY_RESULT_WITH_AVAILABILITY_BIT flag set the query result
field in *pData to 0 if unavailable and the query result if
available. This was a misunderstanding of the Vulkan spec, and this
commit corrects the behavior to eriting a separate available result
in addition to the query result.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3560>

turnip: Free event->bo on vkDestroyEvent

Fixes a leak from freeing event but not event->bo.

Reviewed-by: Jonathan Marek <jonathan@marek.ca>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3639>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3639>

loader: Fix leak of kernel driver name

This is strdup'd, it needs to be freed.

CID: 1458032
Fixes: f93bb2fb102 ("loader: Check if the kernel driver is i915 before loading iris")
Reviewed-by: Eric Engestrom <eric@engestrom.ch>
Reviewed-by: Sagar Ghuge <sagar.ghuge@intel.com>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3630>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3630>

docs: Update SWR tessellation support

Update features.txt to reflect ARB_tessellation_shader
support in SWR

Reviewed-by: Krzysztof Raszkowski <krzysztof.raszkowski@intel.com>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3636>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3636>

i965: Use brw_batch_references in tex_busy check

If the batch references the buffer, we will have to flush the batch
immediately before mapping it, at which point it will be busy.

(This bug has existed for a long time...even going back to BLT-era...)

Fixes: 779923194c6 ("i965/tex_image: Use meta for instead of the blitter PBO TexImage and GetTexImage")
Fixes: d5d4ba9139a ("i965/tex_subimage: use meta instead of the blitter for PBO TexSubImage")
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3616>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3616>

etnaviv: drm-shim: add GC400

These are the ETNAVIV_PARAM's returned from a GC400 found on a
STM32MP157C-DK2 Discovery Board running mainline kernel.

Signed-off-by: Christian Gmeiner <christian.gmeiner@gmail.com>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3195>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3195>

lima: add noheap debug option

Disable using heap buffer when set.

Reviewed-by: Vasily Khoruzhick <anarsoul@gmail.com>
Signed-off-by: Qiang Yu <yuq825@gmail.com>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3264>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3264>

lima: create heap buffer with new interface if available

Newly added heap buffer create interface can create a
large enough buffer whose backup memory can increase
dynamically as needed.

Reviewed-by: Vasily Khoruzhick <anarsoul@gmail.com>
Tested-by: Andreas Baierl <ichgeh@imkreisrum.de>
Signed-off-by: Qiang Yu <yuq825@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3264>

lima: sync lima_drm.h with kernel

Reviewed-by: Vasily Khoruzhick <anarsoul@gmail.com>
Tested-by: Andreas Baierl <ichgeh@imkreisrum.de>
Signed-off-by: Qiang Yu <yuq825@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3264>

lima: fix lima_set_vertex_buffers()

When setting the vertex buffers, lima calls
util_set_vertex_buffers_mask() to reference and copy buffers. That
function
function adds dst with start_slot internally, so lima should not offset
the destination address again.

This is discovered when comparing with other drivers, and fixed by
removing the extra offset in lima_set_vertex_buffers().

This fixes draws that get translated in u_vbuf, because u_vbuf adds
extra vertex buffers when translating.

Signed-off-by: Icenowy Zheng <icenowy@aosc.io>
Reviewed-by: Andreas Baierl <ichgeh@imkreisrum.de>
Reviewed-by: Vasily Khoruzhick <anarsoul@gmail.com>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3620>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3620>

turnip: hook up cmdbuffer event set/wait

Gets some basic tests under "dEQP-VK.synchronization.*event*" passing

Signed-off-by: Jonathan Marek <jonathan@marek.ca>
Reviewed-by: Kristian H. Kristensen <hoegsberg@google.com>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3123>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3123>

etnaviv: drop default state for PE_STENCIL_CONFIG_EXT2

It gets emitted when needed.

Signed-off-by: Christian Gmeiner <christian.gmeiner@gmail.com>
Reviewed-by: Jonathan Marek <jonathan@marek.ca>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3631>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3631>

docs: add new features for RADV/ACO.

Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3627>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3627>

radv: refactor physical device properties

Based on ANV. This removes a bunch of duplicated code for properties.

Fixes: 1b8d99e2885 ("radv: bump conformance version to 1.2.0.0")
Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3626>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3626>

freedreno: remove flush-queue

Signed-off-by: Rob Clark <robdclark@chromium.org>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3503>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3503>

freedreno: add gmem_lock

The gmem state is split out now, so it does not require synchronization.
But gmem rendering still accesses vsc state from the context.

TODO maybe there is a better way?  For gen's that don't do vsc resizing,
this is probably easier.. but for a6xx there isn't really a great
position for more fine grained locking.  Maybe it doesn't matter since
in practice the lock shouldn't be contended.

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3503>

freedreno: add gmem state cache

Which also has the benefit of getting rid of fd_context::gmem.

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3503>

freedreno: get GMEM state from batch

Prep work to reduce churn in next patch.

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3503>

freedreno/a2xx: constify gmem state

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3503>

freedreno/a3xx: constify gmem state

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3503>

freedreno/a4xx: constify gmem state

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3503>

freedreno/a5xx: constify gmem state

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3503>

freedreno/a6xx: constify gmem state

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3503>

freedreno: constify fd_vsc_pipe

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3503>

freedreno: constify fd_tile

In a following patch, when we cache the gmem state, we will want to
treat the gmem state as immuatable.  So start converting things to
const to make this more clear.. fd_tile is a good place to start.

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3503>

freedreno: consolidate GMEM state

The tile and vsc_pipe arrays are really part of the GMEM configuration.
So pull these out of fd_context and into fd_gmem_stateobj.

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3503>

freedreno: extract vsc pipe bo from GMEM state

Prep work for reorganizing GMEM state and extracting out of fd_context.
The vsc pipe bo was the one thing that doesn't change with GMEM/tile
config.

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3503>

turnip: remove unused descriptor state dirty

It was only used to be initialized to zero. Not even updated as
descriptor sets are bind.

As far as I understand, setting the bit TU_CMD_DIRTY_DESCRIPTOR_SET on
tu_cmd_state.dirty is used instead.

Reviewed-by: Jonathan Marek <jonathan@marek.ca>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3624>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3624>

aco: Fix the meaning of is_atomic.

Previously, is_atomic really meant "is not atomic", contrary to its name.
This commit fixes it to mean what one would think it means.

Fixes: 69bed1c9186c3e24ad54089218d58c5f7b83befe
Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3618>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3618>

iris: Support multiple chained batches.

There was never much point in artificially limiting chaining to two
batches - we can trivially support arbitrary length chains.

Currently, we should only ever have 1 or 2, but this may change.

Reviewed-by: Tapani Pälli <tapani.palli@intel.com>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3613>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3613>

iris: Make iris_emit_default_l3_config pull devinfo from the batch

No need to pass it, we can just use batch->screen->devinfo.

Reviewed-by: Tapani Pälli <tapani.palli@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3613>

iris: Drop 'engine' from iris_batch.

For the moment, everything is I915_EXEC_RENDER, so this isn't necessary.
But even should that change, I don't think we want to handle multiple
engines in this manner.

Nowadays, we have batch->name (IRIS_BATCH_RENDER, IRIS_BATCH_COMPUTE,
possibly an IRIS_BATCH_BLIT for blorp batches someday), which describes
the functional usage of the batch.  We can simply check that and select
an engine for that class of work (assuming there ever is more than one).

Reviewed-by: Tapani Pälli <tapani.palli@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3613>

tu: Fix binning address setup after pack macros change.

This fixes a regression in "vkcube -m headless" rendering, but upsettingly
none of my CTS tests I've been using.

Fixes: 59f29fc845ce ("turnip: Convert the rest of tu_cmd_buffer.c over to the new pack macros.")
Caught-by: Jonathan Marek <jonathan@marek.ca>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3609>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3609>

turnip: Enable occlusionQueryPrecise

This commit enables the occlusionQueryPrecise feature. No additonal
work is required as occlusion queries are already implemented to
track exact sample counts.

Also enables a number of extra tests on the Vulkan CTS.

Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3605>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3605>

aco: simplify gathering of MIMG address components

This patch has a slight effect on pipelinedb:
Totals from affected shaders:
SGPRS: 23616 -> 21504 (-8.94 %)
VGPRS: 15088 -> 14444 (-4.27 %)
Spilled SGPRs: 0 -> 0 (0.00 %)
Spilled VGPRs: 0 -> 0 (0.00 %)
Private memory VGPRs: 0 -> 0 (0.00 %)
Scratch size: 0 -> 0 (0.00 %) dwords per thread
Code Size: 662660 -> 664600 (0.29 %) bytes
LDS: 49 -> 49 (0.00 %) blocks
Max Waves: 3079 -> 3204 (4.06 %)

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3602>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3602>

aco: simplify adjust_sample_index_using_fmask() & get_image_coords()

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3602>

aco: fix register allocation with multiple live-range splits

This patch fixes register allocation if multiple live-range splits
occur to the same variable within one instruction.

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3602>

aco: reorder VMEM operands in ACO IR

For all VMEM instructions, the resource constant is now
in operands[0]. For MIMG instructions, the sampler shares
operands[1] with write data in case this instruction writes memory.
Moving the VADDR to be the last operand for MIMG is the first step to
support Navi NSA encoding.

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3602>

nir: Make nir_deref_path_init skip trivial casts

In a NIR generated using SPIR-V initializers to variables, copy
propagation can end up transforming

    vec1 32 ssa_33 = deref_var &@1 (shared mat2x4)
    vec1 32 ssa_35 = mov ssa_33
    vec1 32 ssa_7 = deref_cast (mat2x4 *)ssa_35 (shared mat2x4)  /* ptr_stride=0 */

into

    vec1 32 ssa_33 = deref_var &@1 (shared mat2x4)
    vec1 32 ssa_7 = deref_cast (mat2x4 *)ssa_33 (shared mat2x4)  /* ptr_stride=0 */

Before the optimization, the "head" of a path of deref that uses ssa_7
will be the cast.  After, it will be the variable in ssa_33.  Since
the types are the same, this is a trivial cast that would be picked up
by nir_opt_deref.

If we need to compare such deref-chain after optimization with another
deref-chain for the same variable, the compare function will get
confused by the cast in the middle.

One alternative would be to add nir_opt_deref to places that compare
derefs, but that might not scale well, so skip the trivial casts when
generating the paths instead.

Motivated by the discussion in
https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3047#note_383660.

Reviewed-by: Jason Ekstrand <jason@jlekstrand.net>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3420>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3420>

aco: fix exec mask consistency issues

There seems to be more, these are just the ones found in
Detroit: Become Human shaders.

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3257>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3257>

aco: parallelcopy exec mask before s_wqm

It can be used later and we want any uses to not be fixed to exec, so it's
definition can't be fixed to exec because of how exec masks interact with
register demand calculation.

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3257>

aco: fill reg_demand with sensible information in add_coupling_code()

process_block() will use this to determine the register demand of the
before the current instruction. Previously, it was filled with zeroes
which could result in process_block() only using the register demand
of after the current instruction.

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3257>

aco: improve assertion at the end of spiller

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3257>

aco: set exec_potentially_empty after continues/breaks in nested IFs

Fixes: 93c8ebfa ('aco: Initial commit of independent AMD compiler')
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3257>

aco: error when block has no logical preds but VGPRs are live at the start

This would have caught the liveness error fixed in the previous commit.

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3257>

aco: don't always add logical edges from continue_break blocks to headers

Otherwise, code like this will be broken:
loop {
   if (...) {
      break;
   } else {
      break;
   }
}
The continue_or_break block doesn't have any logical predecessors but it's
a logical predecessor of the header block. This liveness error breaks the
spiller in init_live_in_vars() (under "keep variables spilled on all
incoming paths") and eventually creates garbage reloads.

Fixes: 93c8ebfa ('aco: Initial commit of independent AMD compiler')
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3257>

aco: only create parallelcopy to restore exec at loop exit if needed

The operand isn't fixed to exec, which can mess up the spiller. This also
adds a new situation where a phi is needed.

Fixes dEQP-VK.ssbo.layout.random.descriptor_indexing.2 and an assertion
when compiling a Detroit: Become Human shader.

Fixes: 93c8ebfa ('aco: Initial commit of independent AMD compiler')
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3257>

aco: don't update demand in add_coupling_code() for loop headers

We don't need to update it since it won't be used later.

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3257>

aco: don't consider loop header blocks branch blocks in add_coupling_code

Loops without continues create header blocks with only 1 predecessor.

CC: <mesa-stable@lists.freedesktop.org>
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3257>

aco: fix target calculation when vgpr spilling introduces sgpr spilling

A shader might require vgpr spilling but not require sgpr spilling. In
that case, the spiller lowers the sgpr target by 5 which could mean sgpr
spilling is then required. Then the vgpr target has to be lowered to make
space for the linear vgprs. Previously, space wasn't make for the linear
vgprs.

Found while testing the spiller on the pipeline-db with a lowered limit

Fixes: a7ff1bb5b9a78cf57073b5e2e136daf0c85078d6
   ('aco: simplify calculation of target register pressure when spilling')

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3257>

radv/gfx10: re-enable NGG GS

Now that NGG GS queries are implemented, it should be safe enough
to enable NGG GS by default. It can be disabled with RADV_DEBUG=nongg
if necessary.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3380>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3380>

radv/gfx10: implement NGG GS queries

The number of generated primitives is only counted by the hardware
if GS uses the legacy path. For NGG GS, we need to accumulate that
value in the NGG GS itself. To achieve that, we use a plain GDS
atomic operation.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3380>

radv/gfx10: add a separate flag for creating a GDS OA buffer

For implementing NGG GS queries, we decided to use GDS but GDS OA
is only required for NGG streamout.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3380>

winsys/amdgpu: Close KMS handles for other DRM file descriptions

When a BO or amdgpu_screen_winsys is destroyed.

Should fix leaking such BOs in other DRM file descriptions.

v2:
* Pass the correct file descriptor to drmIoctl (Pierre-Eric
  Pelloux-Prayer)
* Use _mesa_hash_table_remove
v3:
* Close handles in amdgpu_winsys_unref as well
v4:
* Adapt to amdgpu_winsys::sws_list_lock.

Closes: https://gitlab.freedesktop.org/mesa/mesa/issues/2270
Fixes: 11a3679e3aba "winsys/amdgpu: Make KMS handles valid for original
                     DRM file descriptor"

Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3582>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3582>

winsys/amdgpu: Re-use amdgpu_screen_winsys when possible

Namely, if os_same_file_description determined that the DRM file
descriptor references the same file description.

v2:
* Adapt to amdgpu_winsys::sws_list_lock.
v3:
* Fix comparison of amdgpu_screen_winsys file descriptions, see
  https://gitlab.freedesktop.org/mesa/mesa/issues/2413 .
* Lock amdgpu_winsys::sws_list_lock for traversing the sws_list in
  amdgpu_winsys_create.

Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3582>

anv: Rename a variable

The name "desc" shadows another variable.  Name it "desc_data" like all
of the other descriptor data variables in this file.

anv/block_pool: Ensure allocations have contiguous maps

Because softpin block pools are made up of a set of BOs with different
maps, it was possible for a single state to end up straddling blocks.
To fix this, we pass a contiguous size to anv_block_pool_grow and it
ensures that the next allocation in the pool will have at least that
size.

We also add an assert in anv_block_pool_map to ensure we always get
contiguous maps.  Prior to the changes to anv_block_pool_grow, the unit
tests failed with this assert.  With this patch, the tests pass.

This was causing problems on Gen12 where we allocate the pages for the
AUX table from the dynamic state pool.  The first chunk, which gets
allocated very early in the pool's history, is 1MB which was enough that
it was getting multiple BOs.  This caused the gen_aux_map code to write
outside of the map and overwrite the instruction state pool buffer which
lead to GPU hangs.

Fixes: 731c4adcf9b "anv/allocator: Add support for non-userptr"
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>

anv: Re-use one old BT block in reset_batch_bo_chain

We intentionally throw away all but one BT block but then we set
cmd_buffer->bt_block to ANV_STATE_NULL instead of the one we hung on to.
This causes the command buffer to immediately re-emit STATE_BASE_ADDRESS
the first time a BT is needed for no good reason.

Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>

anv: Set actual state pool sizes when we have softpin

Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>

nir/algebraic: add some half packing optimizations

pipeline-db (ACO):
Totals from affected shaders:
SGPRS: 29200 -> 29200 (0.00 %)
VGPRS: 17372 -> 17372 (0.00 %)
Spilled SGPRs: 105 -> 105 (0.00 %)
Spilled VGPRs: 0 -> 0 (0.00 %)
Private memory VGPRs: 0 -> 0 (0.00 %)
Scratch size: 0 -> 0 (0.00 %) dwords per thread
Code Size: 1406576 -> 1389256 (-1.23 %) bytes
LDS: 83 -> 83 (0.00 %) blocks
Max Waves: 3976 -> 3976 (0.00 %)

pipeline-db (LLVM):
Totals from affected shaders:
SGPRS: 21320 -> 21320 (0.00 %)
VGPRS: 17056 -> 17036 (-0.12 %)
Spilled SGPRs: 22 -> 22 (0.00 %)
Spilled VGPRs: 503 -> 487 (-3.18 %)
Private memory VGPRs: 0 -> 0 (0.00 %)
Scratch size: 396 -> 396 (0.00 %) dwords per thread
Code Size: 1441244 -> 1423292 (-1.25 %) bytes
LDS: 463 -> 463 (0.00 %) blocks
Max Waves: 3609 -> 3611 (0.06 %)

v2: add pattern for ishr

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Connor Abbott <cwabbott0@gmail.com>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/2271>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/2271>

nir/algebraic: add patterns for a >> #b << #b

Fixes compilation of a Battlefront 2 shader with ACO by removing VGPR
spilling. The reassociation makes it worse on LLVM though.

pipeline-db (ACO):
Totals from affected shaders:
SGPRS: 10704 -> 10688 (-0.15 %)
VGPRS: 18736 -> 18528 (-1.11 %)
Spilled SGPRs: 70 -> 70 (0.00 %)
Spilled VGPRs: 0 -> 0 (0.00 %)
Private memory VGPRs: 0 -> 0 (0.00 %)
Scratch size: 0 -> 0 (0.00 %) dwords per thread
Code Size: 909696 -> 885796 (-2.63 %) bytes
LDS: 225 -> 225 (0.00 %) blocks
Max Waves: 1115 -> 1129 (1.26 %)

pipeline-db (LLVM):
Totals from affected shaders:
SGPRS: 8472 -> 8424 (-0.57 %)
VGPRS: 14284 -> 14368 (0.59 %)
Spilled SGPRs: 0 -> 0 (0.00 %)
Spilled VGPRs: 442 -> 503 (13.80 %)
Private memory VGPRs: 0 -> 0 (0.00 %)
Scratch size: 268 -> 396 (47.76 %) dwords per thread
Code Size: 862568 -> 853028 (-1.11 %) bytes
LDS: 0 -> 0 (0.00 %) blocks
Max Waves: 971 -> 964 (-0.72 %)

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Connor Abbott <cwabbott0@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/2271>

aco: fix VS input loads with MUBUF on GFX6

Only MTBUF supports vec3.

Fixes: 03a0d39366d ("aco: use MUBUF in some situations instead of splitting vertex fetches")
Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3615>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/merge_requests/3615>