From: Kenneth Graunke Date: Tue, 16 Aug 2016 05:26:42 +0000 (-0700) Subject: mesa: Fix uf10_to_f32() scale factor in the E == 0 and M != 0 case. X-Git-Url: https://git.libre-soc.org/?a=commitdiff_plain;h=01e99cba043084be7477e3a801029bfee581ab87;p=mesa.git mesa: Fix uf10_to_f32() scale factor in the E == 0 and M != 0 case. GL_EXT_packed_float, 2.1.B Unsigned 10-Bit Floating-Point Numbers: 0.0, if E == 0 and M == 0, 2^-14 * (M / 32), if E == 0 and M != 0, 2^(E-15) * (1 + M/32), if 0 < E < 31, INF, if E == 31 and M == 0, or NaN, if E == 31 and M != 0, In the second case (E == 0 and M != 0), we were multiplying the mantissa by 2^-20, when we should have been multiplying by 2^-19 (which is 2^(-14 + -5), or 2^-14 * 2^-5, or 2^-14 / 32). The previous section defines the formula for 11-bit numbers, which is: 2^-14 * (M / 64), if E == 0 and M != 0, In other words, we had accidentally copy and pasted the 11-bit code to the 10-bit case, and neglected to change the exponent. Fixes dEQP-GLES3.functional.pbo.renderbuffer.r11f_g11f_b10f_triangles when run with surface dimensions of 1536x1152 or 1920x1080. Cc: mesa-stable@lists.freedesktop.org References: https://code.google.com/p/chrome-os-partner/issues/detail?id=56244 Signed-off-by: Kenneth Graunke Reviewed-by: Stephane Marchesin Reviewed-by: Antia Puentes --- diff --git a/src/util/format_r11g11b10f.h b/src/util/format_r11g11b10f.h index c9e967cf31a..f6cd4ac696a 100644 --- a/src/util/format_r11g11b10f.h +++ b/src/util/format_r11g11b10f.h @@ -191,7 +191,7 @@ static inline float uf10_to_f32(uint16_t val) if (exponent == 0) { if (mantissa != 0) { - const float scale = 1.0 / (1 << 20); + const float scale = 1.0 / (1 << 19); f32.f = scale * mantissa; } } else if (exponent == 31) {