From e9967aba61a7cc62ce13ab691d14bb5a26a86896 Mon Sep 17 00:00:00 2001 From: Kenneth Graunke Date: Fri, 12 Oct 2012 11:17:39 -0700 Subject: [PATCH] mesa/vbo: Fix scaling issue in 10-bit signed normalized packing. MIME-Version: 1.0 Content-Type: text/plain; charset=utf8 Content-Transfer-Encoding: 8bit For the 10-bit components, the divisor was incorrect. A 10-bit signed integer can represent -2^9 through 2^9 - 1, which leads to the following ranges: (float)value.x -> [ -512, 511] 2.0F * (float)value.x -> [-1024, 1022] 2.0F * (float)value.x + 1.0F -> [-1023, 1023] So dividing by 511 would incorrectly scale it to approximately: [-2.001956947, 2.001956947]. To correctly scale to [-1.0, 1.0], we need to divide by 1023. This correctly implements the desktop GL rules. ES 3.0 has different rules, but those will be implemented in a separate patch. Signed-off-by: Kenneth Graunke Tested-by: Marek Olšák --- src/mesa/vbo/vbo_attrib_tmp.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/mesa/vbo/vbo_attrib_tmp.h b/src/mesa/vbo/vbo_attrib_tmp.h index 6bc53bab3d2..9a4df695d39 100644 --- a/src/mesa/vbo/vbo_attrib_tmp.h +++ b/src/mesa/vbo/vbo_attrib_tmp.h @@ -117,7 +117,7 @@ static inline float conv_i10_to_norm_float(int i10) { struct attr_bits_10 val; val.x = i10; - return (2.0F * (float)val.x + 1.0F) * (1.0F / 511.0F); + return (2.0F * (float)val.x + 1.0F) * (1.0F / 1023.0F); } static inline float conv_i2_to_norm_float(int i2) -- 2.30.2