radeonsi: add return value to gfx10_ngg_calculate_subgroup_info

radeonsi: add return value to gfx10_ngg_calculate_subgroup_info

gfx10_ngg_calculate_subgroup_info uses assert to detect invalid configuration,
but if asserts are disabled it will continue its execution.

This commits adds a boolean return value to let the caller know that something
went wrong and that the results mustn't be used.

Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/3103
Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5401>