etnaviv: introduce struct etna_compiler

This struct will be used to for state saved across compiler
invocations.

Signed-off-by: Christian Gmeiner <christian.gmeiner@gmail.com>
Reviewed-by: Lucas Stach <l.stach@pengutronix.de>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5996>

tu: Enable vertex & fragment stores & atomics

Note that there are some extra tess fails, but they're probably
unrelated to the actual feature. There were also some xfails that were
created as part of an earlier attempt to enable the feature which were
fixed in the meantime, so remove them.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5738>

tu: Detect invalid-for-binning renderpass dependencies

This is all that was missing for stores & atomics.

Closes: #3196
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5738>

tu: Fix hangs for DS with no output

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5738>

tu: Fix empty blit scissor case

With vertexPipelineStoresAndAtomics enabled, fixes:
dEQP-VK.tessellation.invariance.one_minus_tess_coord_component.quads_fractional_even_spacing_cw_point_mode
dEQP-VK.tessellation.invariance.tess_coord_component_range.triangles_fractional_even_spacing_ccw_point_mode

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5738>

spirv: Also copy over binding information for atomic counters

I missed this if statement so atomic counters weren't getting bindings
and, when you have more than one of them, that meant they were all
getting combined into one.

Fixes: 3584cb09bc15 "spirv: Give atomic counters their own variable mode"
Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6060>

tu: Implement VK_KHR_draw_indirect_count

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6007>

tu: Add missing wfi to tu6_emit_hw()

It needs to be there before changing CCU state. This was accidentally
deleted in f494799a7f09deebacb5696fde7514e3329de246 when it should've
been moved.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6007>

tu: Integrate WFI/WAIT_FOR_ME/WAIT_MEM_WRITES with cache tracking

Track them via pending_flush_bits. Previously WFI was only tracked in
flush_bits and WAIT_FOR_ME was emitted directly. This means that we don't
emit WAIT_FOR_ME or WAIT_FOR_IDLE if there wasn't a cache flush or other
write by the GPU. Also split up host writes from sysmem writes, as only
the former require WFI/WAIT_FOR_ME.

Along the way, I also realized that we were missing proper handling of
transform feedback counter writes which require WAIT_MEM_WRITES. Plumb
that through as well. And CmdDrawIndirectByteCountEXT needs a
WAIT_FOR_ME as it does not wait for WFI internally.

As an example of what this does, a typical barrier for transform
feedback with srcAccess = VK_TRANSFORM_FEEDBACK_WRITE_COUNTER_BIT_EXT
and dstAccess = VK_ACCESS_INDIRECT_COMMAND_READ_BIT used to emit on
A650:

- WAIT_FOR_IDLE

and now we emit:

- WAIT_MEM_WRITES
- WAIT_FOR_ME

So we've eliminated a useless WFI and added some necessary waits.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6007>

freedreno: Add INDIRECT_COUNT CP_DRAW_INDIRECT_MULTI variants

These have an indirect count which is loaded from an iova, and the
minimum is taken between the indirect and direct counts. Note, I also
had to fix gen_header.py to deal with the extra-long names we get.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6007>

freedreno: Clean up CP_DRAW_MULTI_INDIRECT definition

Depends on the envytools changes to make the "addvariant" magic work in
order to decode this correctly, and to be able to print the register
names directly.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6007>

turnip: remove extra gmem alignment

Now that we clear the PITCHALIGN" field when filling GMEM input attachment
descriptors, we can get rid of the extra tile width alignment on a630/a640.

With the "block_align_shift" value change, this brings down the default
gmem_align from 16k to 4k on a630/a640 and down from 24k to 12k on a650,
to match the gallium driver.

Signed-off-by: Jonathan Marek <jonathan@marek.ca>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5528>

etnaviv: explicitly set nir_variable_mode

No functional changes - fixes the following assert:
  nir_lower_io_impl: Assertion `!(modes & ~supported_modes)' failed.

Signed-off-by: Christian Gmeiner <christian.gmeiner@gmail.com>
Reviewed-by: Lucas Stach <l.stach@pengutronix.de>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5997>

radv: clean up remaining pipeline init functions

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5837>

radv: remove useless return value to radv_pipeline_scratch_init()

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5837>

radv: add radv_pipeline_init_shader_stages_state()

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5837>

radv: constify all radv_pipeline_generate_*() helpers

To make clear that the pipeline should be read only.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5837>

radv: assign pipeline gfx fields before PM4 emission

To be able to constify all radv_pipeline_generate_*() helpers.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5837>

radv: clean up binning state initialization

It's no longer emitted directly in the pipeline.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5837>

radv: clean up adjusting MSAA state if conservative rast is enabled

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5837>

radv: add radv_pipeline_generate_vgt_gs_out()

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5837>

radv: add radv_pipeline_init_input_assembly_state()

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5837>

radv: clean up tessellation state emission

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5837>

radv: remove unnecessary radv_tessellation_state::lds_size

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5837>

radv: set LDS TCS size at shaders creation for GFX9+

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5837>

radv: align the LDS size in calculate_tess_lds_size()

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5837>

radv: remove one unnecessary param to radv_generate_graphics_pipeline_key()

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5837>

radv: remove no-op si_multiwave_lds_size_workaround()

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5837>

radv: remove unnecessary radv_tessellation_state::num_patches

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5837>

radv: clean up radv_compute_generate_pm4()

For consistency regarding how the graphics pipeline is built.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5837>

radv: reduce the number of allocated dwords for compute CS

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5837>

radv: clean up PA_SC_CLIPRECT_RULE emission

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5837>

radv: clean up VGT_SHADER_STAGES_EN emission

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5837>

radv: emit PA_SC_LINE_CNTL as part of the rasterization state

While we are at it, remove one useless field in radv_multisample_state.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5837>

radv: emit more invariant registers as part of the initial gfx state

This reduces the number of emitted packets for pipelines.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5837>

radv: remove outdated TODO related to PA_SU_VTX_CNTL.PIX_CENTER

It should be always 1, nothing more to check.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5837>

radv: remove set but unused radv_pipeline::vertex_elements

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5837>

radv: remove declared but unused radv_pipeline::is_dual_src

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5837>

bin/khronos-update: add workaround for python bug 9625

The bug causes `choices` to break `nargs='*'`.

Signed-off-by: Eric Engestrom <eric@engestrom.ch>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6057>

bin/khronos-update: add support for the SPIRV files

Signed-off-by: Eric Engestrom <eric@engestrom.ch>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6004>

bin/khronos-update: having a folder in include/ is not a requirement

Signed-off-by: Eric Engestrom <eric@engestrom.ch>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6004>

v3d: Retry with the fallback scheduler when RA fails

v3d_compile is now split out into a helper function that gets called a
second time if compilation fails the first time with the result
reporting the register allocation failed. The second time it is run with
the fallback scheduler to try and increase the chances of successfully
allocating the registers.

v2: Add a performance debug message when using the fallback scheduler.

Reviewed-by: Iago Toral Quiroga <itoral@igalia.com>
Reviewed-by: Alejandro Piñeiro <apinheiro@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5953>

v3d: Changed v3d_compile:failed to an enum

Instead of just having a bool status for the failure, there is now an
enum so that the compilation can report a more detailed status.
Currently this is only used to report whether the failure was due to
failed register allocation. The “failed” bool doesn’t seem to actually
have been used anywhere so this doesn’t really change a lot.

Reviewed-by: Iago Toral Quiroga <itoral@igalia.com>
Reviewed-by: Alejandro Piñeiro <apinheiro@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5953>

nir/schedule: Add an option for a fallback scheduling algorithm

The current scheduling algorithm favors parallelism a bit too
aggressively and sometimes generates shaders that fail register
allocation. This happens even if the threshold is set to zero to force
it to always use the CSR instruction choosing algorithm.

This patch adds an option to use an even more aggressive fallback that
just always picks the instruction with the shortest maximum delay in the
hope that that will generate the least register pressure. The intention
is to use this as a last resort after register allocation fails in order
to at least have a working shader.

Reviewed-by: Alejandro Piñeiro <apinheiro@igalia.com>
Acked-by: Iago Toral Quiroga <itoral@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5953>

v3d: Mark scheduling dependency for prim id and first output

The input primitive ID is read from the VPM in the same memory segment
as the outputs. This means that writing the GS header to VPM location 0
needs to be done after reading the primitive ID. This patch adds a
dependency between the load_primitive_id intrinsic and the store_output
intrinsic for location 0 to stop the scheduler from reordering them.

v2: Use an enum for the dependency class number.
v3: Add "GS" to the class enum name.

Reviewed-by: Iago Toral Quiroga <itoral@igalia.com>
Reviewed-by: Alejandro Piñeiro <apinheiro@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5953>

nir/schedule: Add a callback for backend-specific dependencies

Adds a callback function to nir_schedule_options to give the backend a
chance to add custom dependencies between certain intrinsics. The
callback can assign a class number to the intrinsic and then set a read
or write dependency on that class.

v2: Use a linked-list of schedule nodes for the dependency classes
    instead of a fixed-sized array.

Reviewed-by: Iago Toral Quiroga <itoral@igalia.com>
Reviewed-by: Alejandro Piñeiro <apinheiro@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5953>

nir/schedule: Store a pointer to the options struct in scoreboard

Instead of copying the individual members of nir_schedule_options into
the scoreboard, it now just keeps a pointer to the options. This avoids
the duplicated comments and makes it easier to add more options later.

Reviewed-by: Iago Toral Quiroga <itoral@igalia.com>
Reviewed-by: Alejandro Piñeiro <apinheiro@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5953>

nir/scheduler: Move nir_scheduler to its own header

nir_schedule already has a struct for options which makes it more than
just a function declaration. Later patches intend to add more structs to
complement these options. In order to make the code easier to manage,
this moves the nir_scheduler-related parts out of nir.h to their own
header.

Reviewed-by: Iago Toral Quiroga <itoral@igalia.com>
Reviewed-by: Alejandro Piñeiro <apinheiro@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5953>

spirv: Rework our handling of images and samplers

Previously, objects of type OpTypeImage or OpTypeSampler were treated as
vtn_pointers and objects of type OpTypeSampledImage were a special-use
vtn_sampled_image struct.  This commit changes that so that all of those
objects are stored in vtn_ssa_values.  Each of images, samplers, and
sampled images, are stored as a scalar or vector nir_ssa_def whose
components are NIR deref values.  We now use vtn_type_get_nir_type to
re-resolve those as-needed into GLSL sampler types for NIR.

This simplification has a number of benefits:

 1. We can git rid of the rest of our special-cases for handling images
    and samplers in function arguments.  Now that they're treated as
    structs at the glsl_type level, the generic paths can handle images
    and samplers.

 2. We can now construct composite values containing images and samplers
    internally.  It's unclear from the SPIR-V spec whether or not this
    is allowed and it's not a pattern that GLSLang currently generates
    thanks to GLSL rules.  However, if we do start seeing SPIR-V that
    contains such composites, we should now be able to handle it.

 3. SPIR-V OpNull and OpUndef instructions can now create samplers,
    images, and sampled images.  The NIR generated won't likely be fully
    valid but, given a NIR pass to do something sensible, it should be a
    thing we can compile.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5278>

anv,turnip,radv,clover,glspirv: Run nir_copy_prop before nir_opt_deref

We're about to make the SPIR-V -> NIR path generate a bit more complex
SSA chains for certain derefs.  This will ensure we don't regress anyone
when we start making vec2's of derefs.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5278>

spirv: More heavily use vtn_ssa_value in function parameter handling

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5278>

spirv: Remove a dead case in function parameter handling

Ever since 31a7476335f911a, we've set something for vtn_type::type for
all pointer types.  For logical pointer types, it's uint32_t.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5278>

spirv: Add a helper for getting the NIR type of a vtn_type

There are a few cases, atomic counters being one example, where the type
used by vtn_ssa_value is not the same as the type we want NIR to use in
derefs and variables.  To solve this, we add a helper which converts
between the types for us.  In the next commit, we'll be adding another
major user of this: images and samplers.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5278>

spirv: Give atomic counters their own variable mode

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5278>

spirv: Drop the sampled boolean from vtn_type

It was set but never used.  We always check the glsl_type instead.

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5278>

spirv: Add better checks for SSA value types

Primarily, we check for two things:

 1. That we only ever add SSA values via vtn_push_ssa_value and
    vtn_copy_value.

 2. That the type of the SSA value matches the SPIR-V destination type.

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5278>

spirv: Hand-roll fewer vtn_ssa_value creations

Previously, we created our vtn_ssa_value in _vtn_variable_load_store
manually as we did the recursive load/store.  Instead, we now create the
SSA value before calling into the recursive function.  This is a tiny
bit less efficient but it removes a case of hand-rolling vtn_ssa_value
creation.  For symmetry, we make _vtn_block_load_store assume the value
is already created.  Finally, we remove a trivial hand-rolled case in
vtn_composite_extract.

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5278>

spirv: Simplify vtn_ssa_value creation

For three different functions which create vtn_ssa_values, we had three
completely different implementations.  This unifies them all to roughly
the same algorithm.  While we're at it, we take advantage of the
nir_build_imm helper to avoid some extra code in vtn_const_ssa_value.

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5278>

spirv/subgroups: Refactor to use vtn_push_ssa

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5278>

spirv/subgroups: Stop incrementing w

The w++ is to handle a differences between the KHR extension and Vulkan
1.1 feature where the Vulkan 1.1 instructions take an scope parameter.
While incrementing w technically works, it's really subtle and very easy
to miss when reading the code.

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5278>

spirv/glsl450: Use vtn_push_ssa_value

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5278>

spirv/alu: Use vtn_push_ssa_value

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5278>

spirv: Refactor vtn_push_ssa

We rename it to vtn_push_ssa_value, move it to spirv_to_nir, and remove
the unnecessary type parameter.

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5278>

spirv: Use the new helpers in OpConvertUToPtr/PtrToU

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5278>

spirv: Add a vtn_get_nir_ssa helper

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5278>

spirv/amd: Use vtn_push_nir_ssa

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5278>

spirv: Add a vtn_push_nir_ssa helper

This makes it easy to write a simple NIR SSA value

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5278>

spirv: Rename push_value_pointer to push_pointer

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5278>

spirv: Add a helpers for getting types of values

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5278>

spirv: Use nir_bany/ball for OpAny/All

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5278>

spirv: Clean up OpSignBitSet

For some reason, we were doing a signed shift vectors and an unsigned
shift for scalars.  We then plug it into i2b so it should make no
difference whatsoever.  The fact that we're doing different things for
vectors vs. scalars is bonkers.  Let's simplify the code a bit.

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5278>

spirv: Fix indentation in vtn_handle_ptr

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5278>

spirv: Drop the void *ptr from vtn_value

It isn't being used for anything.

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5278>

spirv: Simplify our handling of NonUniform

The original implementation of SPV_EXT_descriptor_indexing was extremely
paranoid about the NonUniform qualifier, trying to fetch it from every
possible location and propagate it through access chains etc.  However,
the Vulkan spec is quite nice to us on this and has very strict rules
for where the NonUniform decoration has to be placed.  For image and
texture operations, we can search for the decoration on the spot when we
process the image or texture op.  For pointers, we continue putting it
on the pointer but we don't bother trying to do anything silly like
propagate it through casts.

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5278>

nir/vtn: Add support for 8 and 16 vector ball/bany

Reviewed-by: Boris Brezillon <boris.brezillon@collabora.com>
Reviewed-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Reviewed-by: Jason Ekstrand <jason@jlekstrand.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6030>

nir: Support 8 and 16 component vectors for reduceable intrinsics

Reviewed-by: Boris Brezillon <boris.brezillon@collabora.com>
Reviewed-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Reviewed-by: Jason Ekstrand <jason@jlekstrand.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6030>

nir: Support algebraic opts on vectors larger than 4

Reviewed-by: Boris Brezillon <boris.brezillon@collabora.com>
Reviewed-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Reviewed-by: Jason Ekstrand <jason@jlekstrand.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6030>

nir: Support vec8/vec16 in nir_lower_bit_size

Reviewed-by: Gert Wollny <gert.wollny@collabora.com>
Reviewed-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Reviewed-by: Jason Ekstrand <jason@jlekstrand.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6030>

freedreno: sync registers from envytools

Pull in a bunch of fixes and updates.. mostly using varset correctly,
and fixes for implicit bools.

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6052>

tu: Enable VK_EXT_depth_clip_enable

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6047>

aco: ensure readfirstlane subdword operands are always dword aligned

Cc: 20.1 <mesa-stable@lists.freedesktop.org>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6024>

aco: prevent infinite recursion in RA for subdword variables

Cc: 20.1 <mesa-stable@lists.freedesktop.org>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6024>

aco: don't split store data if it was already split into more elements

Cc: 20.1 <mesa-stable@lists.freedesktop.org>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6024>

aco: ensure to not extract more components than have been fetched

Fixes: 7015d2c249e1f7814bf5681ccd049e49e4d6495c ('aco: fix scratch loads which cross element_size boundaries')
Cc: 20.1 <mesa-stable@lists.freedesktop.org>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6024>

radv: Add timeline syncobj for timeline semaphores.

Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5600>

radv: Add winsys functions for timeline syncobj.

Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5600>

radv: Add winsys support for submitting timeline syncobj.

Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5600>

radv: Add thread for timeline syncobj submission.

For cross-process timelines we have to have a thread to wait
till the requested points become available.

The functions actually dealing with timeline semaphores stubbed out, to
implement in the next patch. As such the thread code shouldn't trigger
yet.

The core idea is that we still use the refcount mechanism that we use with
emulated timelines, though the native timeline syncobj don't participate
in the refcounting. This way we keep the ordering of submission in a queue
as each submission is also blocked by its predecessor.

Where we change behavior is when the number of blockers reaches 0. In the
new code we check if we need to wait for the timeline semaphores to
be available and if so we won't execute the submission immediately but
pass it to the submission thread.

Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5600>

radv/winsys: Add binary syncobj ABI changes for timeline semaphores.

To facilitate cross-process timeline semaphores we have to deal with
the fact that the syncobj signal operation might be submitted a
small finite time after the wait operation.

For that we start using DRM_SYNCOBJ_WAIT_FLAGS_WAIT_FOR_SUBMIT during
the wait operation so we properly wait instead of returning an error.

Furthermore, to make this effective for syncobjs that get reused we
actually have to reset them after the wait. Otherwise the wait before
submit would get the previous fence instead of waiting for the new
thing to submit.

The obvious choice, resetting the syncobj after the CS submission
has 2 issues though:

1) If the same semaphore is used for wait and signal we can't reset it.
   This is solvable by only resetting the semaphores that are not in the
   signal list.
2) The submitted work might be complete before we get to resetting the
   syncobj. If there is a cycle of submissions that signals it again and
   finishes before we get to the reset we are screwed.

Solution:
Copy the fence into a new syncobj and reset the old syncobj before
submission. Yes I know it is more syscalls :( At least I reduced the
alloc/free overhead by keeping a cache of temporary syncobjs.

This also introduces a syncobj_reset_count as we don't want to reset
syncobjs that are part of an emulated timeline semaphore. (yes, if
the kernel supports timeline syncobjs we should use those instead,
but those still need to be implemented and if we depend on them in
this patch ordering dependencies get hard ...)

Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5600>

amd: Add detection of timeline semaphore support.

Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5600>

nir/ lower_int_to_float: Handle umax and umin

8e1b75b3 introduced umax/umin in order to lower iand/ior for (n)eq zero.
That breaks the lower_int_to_float pass, because umax and umin weren't
handled there.

Tested with lima. The other users of nir_lower_int_to_float
(etnaviv, freedreno) should also have that issue.

Reviewed-by: Vasily Khoruzhick <anarsoul@gmail.com>
Reviewed-by: Jason Ekstrand <jason@jlekstrand.net>
Signed-off-by: Andreas Baierl <ichgeh@imkreisrum.de>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6043>

ci: Use half as many parallel softpipe / virgl test jobs

We're now using at least twice as many CPU cores per job (on shared
runners), so they only take about half as long, and should still be
under 10 minutes.

Reviewed-by: Eric Anholt <eric@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6011>

ci: Do not mark container / pages jobs as interruptible

If another MR was merged while these were still running for the main
project, the result could be no updated images in the main project
registry (forcing a rebuild of the new images in all forked projects) or
an outdated Mesa website.

Reviewed-by: Eric Anholt <eric@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6011>

ci: Use FDO_CI_CONCURRENT in run-shader-db.sh as well

Noticed while checking job logs for it being used elsewhere.

Reviewed-by: Eric Anholt <eric@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6011>

ci: Namespace trace artifacts to the job number

Put artifacts in a per-job folder, because if a job is retried then it
will try to upload a file to the same key and fail with the following
error:

403 Client Error: Forbidden for url:
https://minio-packet.freedesktop.org/artifacts/daenzer/mesa/180609/gl-panfrost-t860/results.yml

Also, to prevent in the future similar clashes if several trace files
share the same name, upload the images with their checksums as their
names. This will also make it easier to fetch images for comparison with
the references.

Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Reviewed-by: Eric Anholt <eric@anholt.net>
Reviewed-by: Andres Gomez <agomez@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6029>

nir_ allow nir_lower_clip_halfz to run in tess eval shader

Reviewed-by: Eric Anholt <eric@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6027>

nir: allow lower_psiz_mov to run in tessellation stages

Reviewed-by: Eric Anholt <eric@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6027>

Revert "llvmpipe: Use the default behavior of ALLOW_MAPPED_BUFFERS."

This reverts commit 6ec49066498097c2647d030ebe1b99a00bc38734.

This broke:
GTF-GL45.gtf21.GL3Tests.texture_lod_bias.*

not sure why but revert for now.

Reviewed-by: Eric Anholt <eric@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6042>

llvmpipe/ms: fix sign extension bug in rasterizer.

/glcts --deqp-surface-width=1024 --deqp-surface-height=64  --deqp-case=KHR-GL45.texture_view.view_sampling  --deqp-surface-type=fbo
was failing but only for width 1024.

The test was filling a 4x4 ms texture, but leaving the viewport set to 1024x64.

This was resulting in this code incorrectly sign extending a value, and passing
it into the mask generator and getting the wrong values. Explicit cast
avoids the sign extension and fixes the above test.

Reviewed-by: Roland Scheidegger <sroland@vmware.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6006>

intel/ir/gen12+: Work around FS performance regressions due to SIMD32 discard divergence.

This avoids some performance regressions on Gen12 platforms caused by
SIMD32 fragment shaders reported in titles like Dota2, TF2, Xonotic,
and GFXBench5 Car Chase and Aztec Ruins.

The most obvious pattern in the regressing shaders I identified among
these workloads is that they all had non-uniform discard statements,
which are handled rather optimistically by the current IR analysis
pass: No penalty is currently applied to the SIMD32 variant of the
shader in the form of differing branching weights like we do for other
control flow instructions in order to account for the greater
likelihood of divergence of a SIMD32 shader.

Simply changing that by giving the same treatment to discard
statements as we give to other branching instructions seemed to hurt
more than it helped on platforms earlier than Gen12, since it reversed
most of the improvement obtained from SIMD32 fragment shaders in
Manhattan for no measurable benefit in other workloads (Manhattan has
a handful of shaders with statically non-uniform discard statements
which actually perform better in SIMD32 mode due to their approximate
dynamic uniformity).  For that reason this change is applied to Gen12+
platforms only.

I've been running a number of tests trying to understand the
difference in behavior between Gen12 and earlier platforms, and most
of the evidence I've gathered seems to point at EU fusion being the
culprit: Unlike previous generations, on Gen12 EUs are arranged in
pairs which execute instructions in lockstep, giving an effective warp
size of 64 threads in SIMD32 mode, which seems to increase the
likelihood for control flow divergence in some of the affected shaders
significantly.

Fixes: 188a3659aea6dec9acf1 "intel/ir: Import shader performance analysis pass."
Reported-by: Caleb Callaway <caleb.callaway@intel.com>
Reviewed-by: Matt Turner <mattst88@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5910>