mesa/st: release variants for active programs before unref

Programs can be shared among many contexts and each program holds a
variant list which has context specific variants. When context gets
destroyed it must make sure it relases all variants, otherwise remaining
context that utilizes same program will attempt to save a zombie shader
for already deleted context when releasing program and its variants.

Fixes:
   dEQP-EGL.functional.sharing.gles2.program.render

and other flaky multihread dEQP-EGL failures.

v2: pass program pointer via & (Marek)

Signed-off-by: Tapani Pälli <tapani.palli@intel.com>
Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
Cc: mesa-stable@lists.freedesktop.org
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4386>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4386>

mesa/st: unbind shader state before deleting it

Signed-off-by: Tapani Pälli <tapani.palli@intel.com>
Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
Cc: mesa-stable@lists.freedesktop.org
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4386>

pan/bit: Add mode to run unit tests

Probably the most useful of the bunch going forward.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4458>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4458>

pan/bit: Make run more useful

..by printing some output.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4458>

pan/bit: Add csel tests

..and pull out common instruction generation to reduce duplication in
tests a bit.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4458>

pan/bit: Add CSEL to interpreter

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4458>

pan/bit: Add FMA tests

Now that the earlier reg ctrl issue is fixed these should pass.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4458>

pan/bit: Add 16-bit fmod tests

These raise another set of issues -- indeed, not all of these tests are
passing, since it turns out I have an actual bug in the packing code. So
after all this work, test bringup has identified an actual issue :)

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4458>

pan/bit: Add verbose printing for tests

We'd like to dump both the generated IR (so we know exactly what's being
tested) as well as the compiled program (so we know what's running for
comparison).

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4458>

pan/bit: Add helper for generating floating mod tests

We can iterate them, isn't that adorable!

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4458>

pan/bit: Add packing test framework

Given an instruction, we'd like to wrap it in a clause with some I/O on
each end so we can pack it up and send it to the hardware to compare
against the simulator.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4458>

pan/bit: Implement floating source mods

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4458>

pan/bit: Implement outmods

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4458>

pan/bit: Add preliminary FMA/ADD/MOV implementations

Missing some details about modifiers but the core structure should
look like this for 32 and 16-bit, I think. My sincerest apologies for
the macro magic, I tried to make it the least bad I could but trying to
keep down repitition.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4458>

pan/bit: Handle read/write

We case the various sources and destinations to model register file
access and passthrough in particular.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4458>

pan/bit: Stub out BIR interpreter

We'd like to step through a BIR program to evaluate it for testing.
Let's stub out some infrastructure for modeling Bifrost.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4458>

pan/bi: Match CSEL argument order with hw

It turns out ports need to be in order of the arguments of an
instruction (port 3, that is), which breaks on instructions whose IR
argument order is different from the packed order, like csel. So fix
that.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4458>

pan/bi: Add helper to debug port assignment

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4458>

pan/bi: Handle BIFROST_FIRST_WRITE_FMA_P2_READ_P3

It's a special case for unclear reasons, and if you mess it up you get
INSTR_INVALID_ENC. Isn't hardware fun?

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4458>

pan/bi: Allow BI_FMA to take mods

It doesn't take abs but it can take outmod/neg.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4458>

pan/bi: Don't gobble zero ports

In case we've reading/writing R0.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4458>

pan/bi: Fix negation in ADD.v2f16

When we flip the sources we need to flip the negates as well to stay
consistent.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4458>

pan/bi: Fix duplicated source in ADD.v2f16

Typo.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4458>

pan/bi: Export bi_class_name

For use in naming tests.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4458>

lima: avoid situations when scissor minx > maxx or miny > maxy

Clip scissor to viewport and then to framebuffer to avoid that since
hardware can't handle this case.

Reviewed-by: Qiang Yu <yuq825@gmail.com>
Signed-off-by: Vasily Khoruzhick <anarsoul@gmail.com>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4427>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4427>

etnaviv: convert perfmon queries to acc queries

Signed-off-by: Christian Gmeiner <christian.gmeiner@gmail.com>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/1530>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/1530>

etnaviv: move generic perfmon functionality into own file

This change removes the basic infrastructure to work with perfmon
from the perfmon query impl and puts it into its own place.
Makes the whole series easier to review and ends smaller changes.

Signed-off-by: Christian Gmeiner <christian.gmeiner@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/1530>

etnaviv: extend acc sample provide with an allocate(..)

We might be able to sub-class etna_acc_query.

Signed-off-by: Christian Gmeiner <christian.gmeiner@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/1530>

etnaviv: extend result(..) to return if data is ready

For the upcoming conversion of perfmon queries to the acc query
framework we need a way to tell that the data is not ready.

Signed-off-by: Christian Gmeiner <christian.gmeiner@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/1530>

etnaviv: make use of a fixed size array to track of all acc query provider

Signed-off-by: Christian Gmeiner <christian.gmeiner@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/1530>

etnaviv: extend acc query provider with supports(..) function

Move the logic if a query provider supports a query_type
directly to the provider.

Signed-off-by: Christian Gmeiner <christian.gmeiner@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/1530>

etnaviv: rework wait/flush logic

Saves us from doing an extra flush in !wait case and seems more
logical now. Also evaluate etna_bo_cpu_prep(..) retun value.

Signed-off-by: Christian Gmeiner <christian.gmeiner@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/1530>

etnaviv: reset no_wait_cnt after triggered flush

Signed-off-by: Christian Gmeiner <christian.gmeiner@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/1530>

etnaviv: explicitly call resource_written(..)

We might end in cases where etna_acc_get_query_result(..) gets called
within one draw call (aka before flushing). At this point the status
of the resource was not set but gets used in etna_acc_get_query_result(..)
to handle different wait cases. Fix this issue by calling resource_written(..)
explicitly.

Signed-off-by: Christian Gmeiner <christian.gmeiner@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/1530>

etnaviv: rework etna_acc_sample_provider

Simplify the interface a sampler provider needs to implement. The start(..)
and stop(..) functions got called by resume(..) and suspend(..) so lets
get rid of start(..) and stop(..). Also the way we count and use samples
is much easier to follow now.

Signed-off-by: Christian Gmeiner <christian.gmeiner@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/1530>

etnaviv: rename hw queries to acc queries

The name hw queries was choosen as occlusion queries are 'feeling'
like nothing special. It is possible to interact with them only
via the command stream - unlike perfom queries where some kernel
magic is needed.

Accumulated HW queries is a much better name for this type of queries.
We read some hardware values over some draw calls and need to accumulate
them to get the final result.

This is some prep work for the following perfmon changes.

Signed-off-by: Christian Gmeiner <christian.gmeiner@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/1530>

glx: use anonymous namespace to avoid -Wodr issues when building with LTO enabled

Signed-off-by: Eric Engestrom <eric@engestrom.ch>
Reviewed-by: Adam Jackson <ajax@redhat.com>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/2597>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/2597>

glx: fix 630 times -Wlto-type-mismatch when building with LTO enabled

The prototypes are simply copied from include/GL/gl*.h

Signed-off-by: Eric Engestrom <eric@engestrom.ch>
Reviewed-by: Adam Jackson <ajax@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/2597>

Revert "spirv: Rewrite CFG construction"

This reverts commit fa5a36dbd474fb3c755da51553c6ca18dab76a06.

Revert "gallivm: disable rgtc/latc SNORM accellerated fetches"

This reverts commit 4897e70ccd3987d470ec8622d473ee3405f6e96f.

Fixed in previous commits.

Reviewed-by: Roland Scheidegger <sroland@vmware.com>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4425>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4425>

gallivm/rgtc: enable fast path for snorm types.

As per Roland's suggestions it should be easy to enable the
fast path fetch for rgtc snorm as well here.

Reviewed-by: Roland Scheidegger <sroland@vmware.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4425>

gallivm/rgtc: fix the truncation to 8-bit

The 8 bit type wasn't 8-bit so when doing signed work we lost
the sign bit. This fixes it to use a proper vector type,
even if we just end up in here with the 1-wide path for now.

Reviewed-by: Roland Scheidegger <sroland@vmware.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4425>

glsl: don't limit fp16 lowering to frag

This restriction doesn't belong in core code.

Signed-off-by: Rob Clark <robdclark@chromium.org>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4423>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4423>

freedreno: limit fp16 to frag and compute

To avoid dealing w/ some backend varying linking issues when mixing
precision vs geom/tess.

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4423>

freedreno/ir3: also precompile compute shaders for shaderdb

Similar as with draw shaders, nothing will trigger the final variant in
shader-db.

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4423>

freedreno: fix missing locking

Fixes: d0b3ccb0607 ("freedreno: Fix detection of being in a blit for acc queries.")
Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4423>

freedreno/a6xx: add some compute logging

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4423>

freedreno/ir3/cf: use ssa-uses

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4423>

freedreno/ir3: add a pass to collect SSA uses

We don't really track these as the ir is transformed, but it would be a
useful thing for some passes to have.  So add a pass to collect this
information.  It uses instr->data (generic per-pass ptr), with the
hashsets hanging under a mem_ctx for easy disposal at the end of the
pass.

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4423>

freedreno/ir3/cf: skip array load/store

Don't fold conversions into array (incl phi lowered to regs/array).
These aren't SSA.  Avoids crashes in particular in frag shaders with
flow control, which would leave a dangling array write disconnect from
the original cov src.

Possibly this could be slightly relaxed, if there is no other consumer
of the src, and it were in the same block.  But it would require
updating block->keeps, and taking care of barrier state.  Which isn't a
thing the cf pass does currently.

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4423>

freedreno/ir3: fixup cat3 32b vs 16b

These should be keyed on src arg type.

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4423>

freedreno/ir3/cf: handle widening too

We can also fold f16->f32 conversions.

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4423>

nir: fix definition of imadsh_mix16 for vectors

Fixes: c27b3758fa0 ("nir/opcodes: Add new 'umul_low' and 'imadsh_mix16' opcodes")
Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4423>

aco: use MUBUF to load subdword SSBO

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-By: Timur Kristóf <timur.kristof@gmail.com>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4002>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4002>

aco: implement 8bit/16bit store_ssbo

Currently without alignment check, so that
we can only use the _byte and _short versions
and multi-component stores are split.

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-By: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4002>

aco: implement 8bit/16bit load_buffer

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-By: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4002>

aco: implement storagePushConstant8 & storagePushConstant16

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-By: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4002>

aco: implement vec2/3/4 with subdword operands

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-By: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4002>

aco: prepare helper functions for subdword handling

- get_alu_src()
- emit_extract_vector()
- emit_split_vector()

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-By: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4002>

aco: add byte_align_scalar() & trim_subdword_vector() helper functions

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-By: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4002>

aco: add missing conversion operations for small bitsizes

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-By: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4002>

aco: don't vectorize 8/16bit load/store_ssbo

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-By: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4002>

aco: don't assume split_vector(create_vector) has the same number of elements when optimizing

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-By: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4002>

aco: don't propagate SGPRs into subdword PSEUDO instructions

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-By: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4002>

aco: lower subdword shuffles correctly.

Note that subdword swaps are not yet implemented

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-By: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4002>

aco: add builder function for subdword copy()

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-By: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4002>

aco: small refactoring of shuffle code lowering

Uses now bytes instead of 32bit size

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-By: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4002>

aco: align subdword registers during RA when necessary

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-By: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4002>

aco: adapt register allocation for subdword registers

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-By: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4002>

aco: create helper function to collect variables from register area

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-By: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4002>

aco: add notion of subdword registers to register allocator

To not having to split the register file into single bytes,
we maintain a map with registers which contain subdword variables.

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-By: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4002>

aco: remove unnecessary reg_file.fill() operation in get_reg_create_vector()

No pipelinedb changes

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-By: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4002>

aco: fix Temp and assignment of renamed operands during RA

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-By: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4002>

aco: print subdword registers

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-By: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4002>

aco: validate RA of subdword assignments

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-By: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4002>

aco: validate uninitialized operands

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-By: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4002>

aco: validate register alignment of subdword operands and definitions

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-By: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4002>

aco: validate p_create_vector with subdword elements properly

Also allows for undef operands

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-By: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4002>

aco: refactor regClass setup for subdword VGPRs

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-By: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4002>

aco: add emission support for register-allocated sdwa sels

Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Reviewed-By: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4002>

aco: add sub-dword regclasses

Co-authored-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-By: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4002>

aco: print and validate opsel

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Reviewed-By: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4002>

aco: add SDWA_instruction

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Reviewed-By: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4002>

aco: add comparison operators for PhysReg

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-By: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4002>

aco: make PhysReg in units of bytes

Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Reviewed-By: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4002>

nir: fix unpack_64_4x16 in lower_alu_to_scalar()

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-By: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4002>

drm-shim: stub libdrm's use of realpath()

libdrm started using realpath to get the type of bus associated with a
given device. This stubs the very specific usage that prevents
drm-shim's device from being listed.

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Eric Anholt <eric@anholt.net>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4429>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4429>

drm-shim: return device platform as specified

v2: Embed the libdrm dependency inside the drm-shim dependency

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Acked-by: Eric Anholt <eric@anholt.net> (v1)
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4429>

spirv: Rewrite CFG construction

This commit completely rewrites the way we extract a structured CFG from
SPIR-V.  The new approach is different in a few ways:

 1. It does a breadth-first search instead of depth-first.  This means
    that we've visited the merge node for a construct before we visit
    any of the nodes inside the construct.  This makes it easier to
    validate things like loop and switch nesting.

 2. We record more information in the CFG.  Earlier commits added a
    parent pointer to vtn_cf_node but we now record all of the merge and
    other special blocks for each CFG node.  This lets us validate
    things more precisely.

 3. It makes heavy use of merge blocks for walking the CFG.  Previously,
    we sort of used them as hints for trying to guess the CFG structure
    but things got dicey whenever a merge was missing.  We had some
    heuristics for how to handle short-circuiting if statements but it
    was a bunch of special cases.

    Now, we make them a fundamental part of walking the CFG.  When we
    encounter a control-flow construct, we add the body components of
    the construct to the BFS work list and then jump to the merge block
    if one exists to continue scanning the current CFG nesting level.
    If no merge block exists, we assume that means that control-flow
    never re-converges in a normal way and that the only way to get back
    to normality is with a direct jump such as a loop break or continue.
    This should make things far more robust when trying to deal with the
    more creative placement (or lack thereof) of merge instructions.

Reviewed-by: Alan Baker <alanbaker@google.com>
Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3820>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3820>

spirv: Add a parent field to vtn_cf_node

This makes it easier to crawl up the CF tree when trying to validate the
incoming SPIR-V control-flow.

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3820>

spirv: Make vtn_function a vtn_cf_node

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3820>

spirv: Make vtn_case a vtn_cf_node

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3820>

spirv: Add cast and loop helpers for vtn_cf_node

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3820>

spirv: Add a vtn_block() helper

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3820>

intel/nir: Enable load/store vectorization

This commit enables the I/O vectorization pass that was originally
written for ACO for Intel drivers.  We enable it for UBOs, SSBOs, global
memory, and SLM.  We only enable vectorization for the scalar back-end
because it vec4 makes certain alignment assumptions.

Shader-db results with iris on ICL:

    total instructions in shared programs: 16077927 -> 16068236 (-0.06%)
    instructions in affected programs: 199839 -> 190148 (-4.85%)
    helped: 324
    HURT: 0
    helped stats (abs) min: 2 max: 458 x̄: 29.91 x̃: 4
    helped stats (rel) min: 0.11% max: 38.94% x̄: 4.32% x̃: 1.64%
    95% mean confidence interval for instructions value: -37.02 -22.80
    95% mean confidence interval for instructions %-change: -5.07% -3.58%
    Instructions are helped.

    total cycles in shared programs: 336806135 -> 336151501 (-0.19%)
    cycles in affected programs: 16009735 -> 15355101 (-4.09%)
    helped: 458
    HURT: 154
    helped stats (abs) min: 1 max: 77812 x̄: 1542.50 x̃: 75
    helped stats (rel) min: <.01% max: 34.46% x̄: 5.16% x̃: 2.01%
    HURT stats (abs)   min: 1 max: 22800 x̄: 336.55 x̃: 20
    HURT stats (rel)   min: <.01% max: 17.11% x̄: 2.12% x̃: 1.00%
    95% mean confidence interval for cycles value: -1596.83 -542.49
    95% mean confidence interval for cycles %-change: -3.83% -2.82%
    Cycles are helped.

    total sends in shared programs: 814177 -> 809049 (-0.63%)
    sends in affected programs: 15422 -> 10294 (-33.25%)
    helped: 324
    HURT: 0
    helped stats (abs) min: 1 max: 256 x̄: 15.83 x̃: 2
    helped stats (rel) min: 1.33% max: 67.90% x̄: 21.21% x̃: 15.38%
    95% mean confidence interval for sends value: -19.67 -11.98
    95% mean confidence interval for sends %-change: -23.03% -19.39%
    Sends are helped.

    LOST:   7
    GAINED: 2

Most of the helped shaders were in the following titles:

 - Doom
 - Deus Ex: Mankind Divided
 - Aztec Ruins
 - Shadow of Mordor
 - DiRT Showdown
 - Tomb Raider (Rise, I think)

Five of the lost programs are SIMD16 shaders we lost from dirt showdown.
The other two are compute shaders in Aztec Ruins which switched from
SIMD8 to SIMD16.

Vulkan pipeline-db stats on ICL:

    Instructions in all programs: 296780486 -> 293493363 (-1.1%)
    Loops in all programs: 149669 -> 149669 (+0.0%)
    Cycles in all programs: 90999206722 -> 88513844563 (-2.7%)
    Spills in all programs: 1710217 -> 1730691 (+1.2%)
    Fills in all programs: 1931235 -> 1958138 (+1.4%)

By far the most help was in the Tomb Raider games.  A couple of Batman
games with DXVK were also helped.  In Shadow of the Tomb Raider:

    Instructions in all programs: 41614336 -> 39408023 (-5.3%)
    Loops in all programs: 32200 -> 32200 (+0.0%)
    Cycles in all programs: 1875498485 -> 1667034831 (-11.1%)
    Spills in all programs: 196307 -> 214945 (+9.5%)
    Fills in all programs: 282736 -> 307113 (+8.6%)

Benchmarks of real games I've done on this patch:

 - Rise of the Tomb Raider: +3%
 - Shadow of the Tomb Raider: +10%

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Reviewed-by: Ian Romanick <ian.d.romanick@intel.com>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4367>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4367>

nir/load_store_vectorize: Add support for nir_var_mem_global

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4367>

nir/load_store_vectorize: Use nir_iadd_imm for offsets

This makes it capable of handling 64-bit offsets

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4367>

nir/load_store_vectorize: Fix shared atomic info

These were clearly copied and pasted from SSBOs.  The shared atomics
don't have an SSBO index so their offset is src0 and data is src1.

Fixes: ce9205c03bd20 "nir: add a load/store vectorization pass"
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4367>

intel/nir: Lower memory access bit sizes later

We're about to do load/store vectorization right before this but we need
that to happen after we've done a round of optimization.  Otherwise,
we'll be getting unoptimized NIR in from ANV and the vectorizer won't be
able to do anything with it.

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Reviewed-by: Ian Romanick <ian.d.romanick@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4367>

iris: Set alignments on cbuf0 and constant reads

Reviewed-by: Ian Romanick <ian.d.romanick@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4367>