mesa/vbo: Fix scaling issue in 10-bit signed normalized packing.
authorKenneth Graunke <kenneth@whitecape.org>
Fri, 12 Oct 2012 18:17:39 +0000 (11:17 -0700)
committerKenneth Graunke <kenneth@whitecape.org>
Thu, 22 Nov 2012 04:29:38 +0000 (20:29 -0800)
commite9967aba61a7cc62ce13ab691d14bb5a26a86896
tree106d600c4ca297d1ad78feb08dc567d7fe5b7763
parente2df37f69a4e513b756fc193a544d48d8fd0ddac
mesa/vbo: Fix scaling issue in 10-bit signed normalized packing.

For the 10-bit components, the divisor was incorrect.  A 10-bit signed
integer can represent -2^9 through 2^9 - 1, which leads to the following
ranges:

       (float)value.x          -> [ -512,  511]
2.0F * (float)value.x          -> [-1024, 1022]
2.0F * (float)value.x + 1.0F   -> [-1023, 1023]

So dividing by 511 would incorrectly scale it to approximately:
[-2.001956947, 2.001956947].  To correctly scale to [-1.0, 1.0], we need
to divide by 1023.

This correctly implements the desktop GL rules.  ES 3.0 has different
rules, but those will be implemented in a separate patch.

Signed-off-by: Kenneth Graunke <kenneth@whitecape.org>
Tested-by: Marek Olšák <maraeo@gmail.com>
src/mesa/vbo/vbo_attrib_tmp.h