radv: Enable lowering dynamic quad broadcasts.

This will lower dynamic quad broadcasts into something that both
LLVM and ACO can understand. On hardware which supports shuffles,
they are lowered to shuffle, on older hardware (GFX6-7) they will
get lowered to constant quad broadcasts.

Fixes dEQP-VK.subgroups.quad.*.subgroupquadbroadcast_nonconst_*

Cc: mesa-stable@lists.freedesktop.org
Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4147>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4147>

nir: Add ability to lower non-const quad broadcasts to const ones.

Some hardware doesn't support subgroup shuffle, and on such hardware
it makes no sense to lower quad broadcasts to shuffle. Instead, let's
lower them to four const quad broadcasts, paired with bcsel instructions.

Cc: mesa-stable@lists.freedesktop.org
Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Jason Ekstrand <jason@jlekstrand.net>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4147>

gen_release_notes: resolve ambiguity by renaming `version` to `previous_version` and `next_version` to `this_version`

Signed-off-by: Eric Engestrom <eric@engestrom.ch>
Reviewed-by: Dylan Baker <dylan@pnwbakers.com>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4113>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4113>

gen_release_notes: fix version in "you should wait" message

Fixes: 86079447da1e00d49db0 ("scripts: Add a gen_release_notes.py script")
Signed-off-by: Eric Engestrom <eric@engestrom.ch>
Reviewed-by: Dylan Baker <dylan@pnwbakers.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4113>

pan/bi: Interpret register allocation results

Once LCRA has run, we have a map from IR indices to byte offsets into
the register file, so we need to "install" these results, rewriting the
IR to use native registers and fixing up writemasks/swizzles to
substitute vectorization for adjacent registers (for LCRA, we're
modeling in terms of real vectors).

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4158>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4158>

pan/bi: Add register allocator

We model the machine as vector (with restrictions) to natively handle
mixed types and I/O and other goodies. We use LCRA for the heavylifting.
This commit adds only the modeling to feed into LCRA and spit LCRA
solutions back; next commit will integrate it with the IR.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4158>

pan/bi: Fix missing src_types

We want types to be consistent throughout the IR so we don't have to
make exceptions to parse things out. These cases just got missed.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4158>

pan/bi: Fix vector handling of readmasks

The issue was messing with liveness analysis... with Midgard we look at
the writemask to decide how the instruction behaves. Here, since our ALU
is scalar (except for subdivision which doesn't have proper writemasks
anyway) we just look at the component count directly -- either 4 for
vector instructions (essentially - for smaller loads we can replicate
manually without much burden), or 1 for scalar.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4158>

pan/bi: Minor fixes in iteration macros

Found during RA bringup.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4158>

pan/midgard: Remove incorrect comment in RA

Ironically, this comment was mistakenly added by the commit that fixed
the purported issue in the comment (1bce7fdecd86 - found by `git blame`)

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4158>

panfrost: Move lcra to panfrost/util

We'll want to use it for the Bifrost RA as well.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4158>

glsl/list: use uintptr_t for exec_node_data()'s subtraction

This fixes UBSan warnings when foreach_list_typed_safe() passes NULL:
pointer index expression with base 0x000000000000 overflowed to 0xffffffffffffffa8

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Matt Turner <mattst88@gmail.com>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4157>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4157>

aco: fix uninitialized data error in waitcnt pass

Shouldn't create any incorrect waitcnts but may create suboptimial
waitcnts in rare cases.

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4133>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4133>

ac/llvm: add missing optimization barrier for 64-bit readlanes

Otherwise, LLVM optimizes it but it's actually incorrect.

Fixes: 0f45d4dc2b1 ("ac: add ac_build_readlane without optimization barrier")
Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3585>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3585>

iris: toggle on PIPE_CAP_MIXED_COLOR_DEPTH_BITS

This enables additional EGL configs where we have depth/stencil buffer
with different number of bits per pixel than color buffer has. This
enables some Android games to work that require such config.

Signed-off-by: Tapani Pälli <tapani.palli@intel.com>
Reviewed-by: Ilia Mirkin <imirkin@alum.mit.edu>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4127>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4127>

turnip: Add tu6_control struct.

Follow the way that freedreno is doing so that we could see the whole
layout of the scratch buffer.

Signed-off-by: Hyunjun Ko <zzoon@igalia.com>
Reviewed-by: Jonathan Marek <jonathan@marek.ca>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3942>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3942>

turnip: Enable VK_EXT_transform_feedback

Signed-off-by: Hyunjun Ko <zzoon@igalia.com>
Reviewed-by: Jonathan Marek <jonathan@marek.ca>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3942>

turnip: Implement an empty function vkCmdDrawIndirectByteCountEXT

TODO. We should implement this since indirect draw is enabled.

Signed-off-by: Hyunjun Ko <zzoon@igalia.com>
Reviewed-by: Jonathan Marek <jonathan@marek.ca>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3942>

turnip: Implement stream-out emit and vkApis for transform feedback

1. Implement vkCmdBindTransformFeedbackBuffersEXT,
vkCmdBeginTransformFeedbackEXT and vkCmdEndTransformFeedbackEXT.
 - Not handling counter buffers yet.
2. Implement streamout emit function, mostly taken from fd6_emit.c

v2. Replace emit_pkt4 funcs with emit_regs.

v3. Don't copy the state of stream-output from tu_pipeline.

v4. Set zero to VPC_SO_CNTL/VPC_SO_BUF_CNTL in tu6_init_hw.

Signed-off-by: Hyunjun Ko <zzoon@igalia.com>
Reviewed-by: Jonathan Marek <jonathan@marek.ca>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3942>

turnip: Setup stream-output when linking program

Mostly taken from fd6_program.c.

v2. Note that it forces to use full VS instead of binning pass VS if
there's stream output as the binning pass VS will have outputs on
other than position/psize stripped out, which is the same as freedreno.

v3. fix indentation.

v4. Use register index instead of location when setup streamout.

Signed-off-by: Hyunjun Ko <zzoon@igalia.com>
Reviewed-by: Jonathan Marek <jonathan@marek.ca>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3942>

turnip: Define structs for transform feedback

Define new structures for streamout buffers and state.
Most members of the state struct are taken from freedreno driver.

v2. Use IR3_MAX_SO_* and avoid using magic values.

v3. Remove the state of stream-output in tu_cmd_state and use one in
tu_pipeline and split out reset and enabled fields.

Signed-off-by: Hyunjun Ko <zzoon@igalia.com>
Reviewed-by: Jonathan Marek <jonathan@marek.ca>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3942>

turnip: Gather information for transform feedback

- Add one member to the existed ir3_stream_output so that we could
assign location information from nir_xfb_info, rather than defining
new struct.

- Redefine maximum of so buffers, streams and outputs, which will be
used for turnip.

- Also enable caps for transform feedback for spirv_to_nir.

v2. Remove redefined maximums and use IR3_MAX_SO_* and add
IR3_MAX_SO_STREAMS.

v3. Remove the newly added location field so that we could keep aligned
with 32 bytes. Instead we create an array mapping between the location
and consecutive index, which is GL driver is doing.

Signed-off-by: Hyunjun Ko <zzoon@igalia.com>
Reviewed-by: Jonathan Marek <jonathan@marek.ca>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3942>

egl/android: set window usage flags

When creating an egl surface from an ANativeWindow, the window's usage
flags need to be set so that buffers are allocated properly.

Signed-off-by: David Stevens <stevensd@chromium.org>
Reviewed-by: Tapani Pälli <tapani.palli@intel.com>
Reviewed-by: Lepton Wu <lepton@chromium.org>

ci: Make a simple little bare-metal fastboot mode for db410c.

This supports powering up the device (using an external tool you
provide based on your particular lab), talking over serial to wait for
the fastboot prompt, and then booting a fastboot image on a target
device.

I was previously relying on LAVA for this, but that ran afoul of
corporate policies related to the AGPL.  However, LAVA wasn't doing
too much for us, given that gitlab already has a job scheduler and
tagging and runners.  We were spending a lot of engineering on making
the two systems match up, when we can just have gitlab do it directly.

Lightly-reviewed-by: Kristian H. Kristensen <hoegsberg@google.com>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4076>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4076>

ci: Fix installation of firmware for db410c's nic.

The debian firmware package doesn't actually contain it, costing us a
minute of boot time waiting for it to show up.

Lightly-reviewed-by: Kristian H. Kristensen <hoegsberg@google.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4076>

ci: Print the renderer/version that our dEQP invocation is using.

This is useful for sanity checking how the driver loads.

Lightly-reviewed-by: Kristian H. Kristensen <hoegsberg@google.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4076>

intel/compiler: fix cmod propagation optimisations

Knowing following:
 - CMP writes to flag register the result of
   applying cmod to the `src0 - src1`.
   After that it stores the same value to dst.
   Other instructions first store their result to
   dst, and then store cmod(dst) to the flag
   register.
 - inst is either CMP or MOV
 - inst->dst is null
 - inst->src[0] overlaps with scan_inst->dst
 - inst->src[1] is zero
 - scan_inst wrote to a flag register

There can be three possible paths:

 - scan_inst is CMP:

   Considering that src0 is either 0x0 (false),
   or 0xffffffff (true), and src1 is 0x0:

   - If inst's cmod is NZ, we can always remove
     scan_inst: NZ is invariant for false and true. This
     holds even if src0 is NaN: .nz is the only cmod,
     that returns true for NaN.

   - .g is invariant if src0 has a UD type

   - .l is invariant if src0 has a D type

 - scan_inst and inst have the same cmod:

   If scan_inst is anything than CMP, it already
   wrote the appropriate value to the flag register.

 - else:

   We can change cmod of scan_inst to that of inst,
   and remove inst. It is valid as long as we make
   sure that no instruction uses the flag register
   between scan_inst and inst.

Nine new cmod_propagation unit tests:
 - cmp_cmpnz
 - cmp_cmpg
 - plnnz_cmpnz
 - plnnz_cmpz (*)
 - plnnz_sel_cmpz
 - cmp_cmpg_D
 - cmp_cmpg_UD (*)
 - cmp_cmpl_D (*)
 - cmp_cmpl_UD

(*) this would fail without changes to brw_fs_cmod_propagation.

This fixes optimisation that used to be illegal (see issue #2154)

= Before =
 0: linterp.z.f0.0(8) vgrf0:F, g2:F, attr0<0>:F
 1: cmp.nz.f0.0(8) null:F, vgrf0:F, 0f
= After =
 0: linterp.z.f0.0(8) vgrf0:F, g2:F, attr0<0>:F

Now it is optimised as such (note change of cmod in line 0):

= Before =
 0: linterp.z.f0.0(8) vgrf0:F, g2:F, attr0<0>:F
 1: cmp.nz.f0.0(8) null:F, vgrf0:F, 0f
= After =
 0: linterp.nz.f0.0(8) vgrf0:F, g2:F, attr0<0>:F

No shaderdb changes

Closes: https://gitlab.freedesktop.org/mesa/mesa/issues/2154
Signed-off-by: Yevhenii Kolesnikov <yevhenii.kolesnikov@globallogic.com>
Reviewed-by: Matt Turner <mattst88@gmail.com>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3348>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3348>

pan/bi: Fix swizzle for second argument to ST_VARY

Off-by-one.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4150>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4150>

pan/bi: Implement nir_op_ffma

We have native FMA which works for graphics usage (unlike Midgard where
it's really reserved for compute for various reasons), let's use it.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4150>

pan/bi: Add dead code elimination pass

Now that we have liveness analysis, we can cleanup the IR considerably.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4150>

pan/bi: Add liveness analysis pass

Now that all the guts are shared with Midgard, it's just a matter of
wiring it in.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4150>

pan/bi: Add bi_max_temp helper

Instead of trying to reindex all the times, just be okay with consistent
but sparse indices, then figuring out the max index is easy enough.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4150>

pan/bi: Add bi_next/prev_op helpers

From Midgard. These are surprisingly helpful.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4150>

pan/bi: Add bi_bytemask_of_read_components helpers

Same purpose as the Midgard version, but the implementation is
*dramatically* simpler thanks to our more regular IR.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4150>

pan/bi: Paste over bi_has_arg

While we're at it, cleanup the Midgard one.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4150>

panfrost: Sync Midgard/Bifrost control flow

We can move e v e n more code to be shared and let bi_block inherit from
pan_block, which will allow us to use the shared data flow analysis.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4150>

panfrost: Move liveness analysis to root panfrost/

This way we can share the code with Bifrost.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4150>

pan/midgard: Subclass midgard_block from pan_block

Promote as much as we feasibly can while keeping it Midgard/Bifrost
agnostic.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4150>

pan/midgard: Sync midgard_block field names with Bifrost

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4150>

pan/midgard: Decontextualize liveness analysis core

We mostly just need the temp_count from it.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4150>

pan/midgard: Localize `visited` tracking

Instead of a property on the block, just track it within the function to
minimize IR dependencies.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4150>

pan/bi: Implement sysvals

Now that it's all abstracted nicely with an implementation shared with
Midgard, this is pretty easy to get.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4150>

pan/bi: Switch to panfrost_program

...now that it's shared.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4150>

panfrost: Move Midgard sysval code to common Panfrost

We'll use this all as-is in Bifrost.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4150>

pan/midgard: Remove dest_override sysval argument

Unused, noticed while working on porting over to Bifrost.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4150>

pan/midgard: Decontextualize midgard_nir_assign_sysval_body

Now all sysval code should be fairly generic.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4150>

pan/midgard: Remove indexing dependency of sysvals

Ideally we would sync the compilers to use the same indexing scheme but
that's a lot more Midgard refactoring than I have time for right now.
This is good enough honestly.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4150>

pan/midgard: Adjust sysval-related prototypes

We'd like to share this big chunk of code with Bifrost but that requires
removing the compiler_context parameter... which is totally unused in
fact!

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4150>

pan/midgard: Remove unused iterators

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4150>

panfrost: Promote midgard_program to panfrost/util

We'll want Bifrost to reuse the same linking mechanisms for the most
part.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4150>

gitlab-ci: build RADV in meson-i386 to avoid 32-bit build failures

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Michel Dänzer <mdaenzer@redhat.com>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4044>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4044>

radv: fix 32-bits build (again)

Fixes: dcfc08f5b8a ("radv/sqtt: describe begin/end command buffers with user markers")
Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4044>

mesa: don't unroll glMultiDrawElements with user indices for gallium

Acked-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3591>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3591>

gallium: add PIPE_CAP_DRAW_INFO_START_WITH_USER_INDICES

Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3591>

vbo: fix vbo_copy_vertices for GL_PATCHES and adjacency primitive types

Fixes: 4c6323c49f1 - vbo: handle GS and tess primitive types when splitting Begin/End
Reviewed-by: Mathias Fröhlich <Mathias.Froehlich@web.de>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3591>

vbo: fix transitions from glVertexN to glVertexM where M < N

Fixes: 1f6e53e2 "vbo: don't store glVertex values temporarily into exec"
Reviewed-by: Mathias Fröhlich <Mathias.Froehlich@web.de>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3591>

vbo: use vbo_exec_wrap_upgrade_vertex for glVertex in ATTR_UNION

We can't decrease the size for glVertex before a flush,
so use vbo_exec_wrap_upgrade_vertex directly.

Reviewed-by: Mathias Fröhlich <Mathias.Froehlich@web.de>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3591>

st/mesa: keep serialized NIR instead of nir_shader in st_program

This decreases memory usage, because serialized NIR is more compact.

The first variant is created from nir_shader for uncached shaders.
All other variants are created from serialized NIR.

Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/2909>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/2909>

gitlab-ci: Don't restrict ppc64el/s390x build jobs to gstreamer runners

The packet runners have beefier CPUs now and don't seem to run into
test timeouts anymore.

Reviewed-by: Daniel Stone <daniels@collabora.com>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4128>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4128>

gitlab-ci: Sort packages to install alphabetically

Signed-off-by: Andres Gomez <agomez@igalia.com>
Reviewed-by: Eric Engestrom <eric@engestrom.ch>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>

gitlab-ci: Remove unneeded python3-pilkit dependency

It was added with tracie, but it doesn't depend on it.

Signed-off-by: Andres Gomez <agomez@igalia.com>
Reviewed-by: Eric Engestrom <eric@engestrom.ch>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>

gitlab-ci: Fix indentation and dangerous "\" in the last multiline line

Signed-off-by: Andres Gomez <agomez@igalia.com>
Reviewed-by: Eric Engestrom <eric@engestrom.ch>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>

vc4: Fix query_dmabuf_modifiers mis-reporting external_only property

vc4_screen_query_dmabuf_modifiers doesn't consider that the given format
may only be supported by lowering, which only happens for external
textures.

Reviewed-by: Iago Toral Quiroga <itoral@igalia.com>
Reviewed-by: Eric Anholt <eric@anholt.net>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4063>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4063>

aco: Don't store TCS outputs to LDS when we're sure that none are read.

This allows us not to write an output to LDS, even if it has
an indirect offset.

No pipeline DB changes.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Tested-by: Marge Bot <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3964>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3964>

aco: Only write TCS outputs to LDS when they are read by the TCS.

Note that tess factors are always read at the end of the shader,
so those are still always saved to LDS.

Totals from affected shaders:
VGPRS: 25244 -> 25164 (-0.32 %)
Code Size: 1768268 -> 1690804 (-4.38 %) bytes
Max Waves: 4947 -> 4953 (0.12 %)

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3964>

aco: Store tess factors in VMEM only at the end of the shader.

This optimizes out several superfluous stores of the tess factors,
especially if the shader wrote those outputs multiple times.

Pipeline DB changes on GFX10:
Totals from affected shaders:
SGPRS: 30384 -> 29536 (-2.79 %)
Code Size: 2260720 -> 2214484 (-2.05 %) bytes

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3964>

aco: Don't generate an if when the first part of a merged HS or GS is empty.

In some cases (eg. in a few tessellation CTS tests) the VS part of
a merged HS is completely empty. Let's not generate a divergent if
in these cases. (LLVM also doesn't do it.)

No pipeline DB changes, only affects the CTS.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3964>

radv: Enable ACO on all stages.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3964>

aco: Enable running TES as ES, including merged TES+GS.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3964>

radv: Enable ACO for TES when there is no GS.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3964>

aco: Implement loading TES inputs.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3964>

aco: Enable streamout when TES runs on the HW VS stage.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3964>

aco: Store TES outputs when TES runs on the HW VS stage.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3964>

aco: Use TES output info when TES runs on the VS stage.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3964>

aco: Setup tessellation evaluation shader variables.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3964>

radv: Enable ACO for tessellation control shaders.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3964>

aco: Fix LS VGPR init bug on affected hardware.

Vega 10 and Raven have a HW bug: when the HS thread count is zero,
the LS input arguments are loaded in the wrong registers. This commit
works around this by using the registers where the data actually is,
for the affected arguments.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3964>

aco: Store VS outputs correctly when tessellation is used.

When tessellation is used, the VS runs on the HW LS stage (merged
into HS on GFX9-10). This commit enables such VS to store its
outputs properly in LDS so that the TCS can load them as its
per-vertex inputs.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3964>

aco: Implement tessellation control shader input/output.

Tessellation control shaders can have per-vertex inputs,
and both per-vertex and per-patch outputs. TCS can not only store,
but also load their outputs.

The TCS outputs are stored in RING_HS_TESS_OFFCHIP in VMEM, which
is where the TES reads them from. Additionally, the are also stored
in LDS to make sure they can be loaded fast when read by the TCS.

Tessellation factors are always just stored in LDS.
At the end of the shader, the first shader invocation reads these
from LDS and writes them to RING_HS_TESS_FACTOR in VMEM, and
additionally to RING_HS_TESS_OFFCHIP when they are read by
the Tessellation Evaluation Shader.

This implementation matches the memory layouts used by radv_nir_to_llvm.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3964>

aco: Fix combining DS additions in the optimizer.

Previously, it was calculated incorrectly for 64-bit writes and reads.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3964>

aco: Slight fix to lds_store and lds_load.

This commit fixes lds_store and lds_load so that they can
properly support 32 and 64-bit loads and stores; and makes
them a little more reusable so they can be used by
tessellation control shaders.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3964>

aco: Refactor VS output stores in preparation for tessellation.

This commit takes the new helpers into use by the VS output
store function. This function is also where the VS outputs will
be handled when the VS runs on the HW LS stage.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3964>

aco: Refactor load_per_vertex_input in preparation for tessellation.

This commit carves out the GS per-vertex input load, and takes the
new helper functions into use.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3964>

aco: Introduce new helpers for calculating address offsets.

These helpers are going to make it unnecessary to reimplement the
(almost) same address offset calculation in mulitple places.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3964>

aco: Introduce new VMEM load/store helpers.

These are going to be used for loading and storing inputs
and outputs in various stages, such as GS, TCS and TES.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3964>

aco: Remove esgs_itemsize from LDS alignment calculation.

It was problematic to have it, because some shader stages might
not even know about the esgs_itemsize, for example TCS and
the merged VS+TCS stages.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3964>

aco: Extract LDS alignment calculation to a separate function.

This function is going to be reused in multiple functions when
storing or loading something in the LDS.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3964>

aco: Remove vertex_geometry_gs assertion from merged shaders.

We are going to support more kinds of merged shaders, such
as vertex_tess_control_hs and tess_eval_geometry_gs.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3964>

aco: Use mesa shader stage when loading inputs.

This makes it more clear which stages should load these inputs.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3964>

aco: Setup correct HW stages when tessellation is used.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3964>

aco: Implement load_view_index for TCS and TES.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3964>

aco: Implement memory_barrier_tcs_patch.

TCS outputs are going to be written to LDS, so it
has to use memory_barrier_shared in order to ensure
that it waits for LDS writes.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3964>

aco: Implement control_barrier for tessellation control shaders.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3964>

aco: Implement load_invocation_id for tessellation control shaders.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3964>

aco: Implement load_patch_vertices_in.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3964>

aco: Implement load_primitive_id for tessellation shaders.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3964>

aco: Implement load_tess_coord.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3964>

aco: Setup tessellation control shader variables.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3964>

aco: Extract setup_gs_variables into a separate function.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3964>

radv: Move some helper functions to the radv_shader.h header file.

Move calculate_tess_lds_size and get_tcs_num_patches to radv_shader.h
ACO will need to call these functions too.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3964>