mesa: fix possible precision issues in pack/unpack/fetch functions
authorMarek Olšák <maraeo@gmail.com>
Sun, 11 Dec 2011 15:18:36 +0000 (16:18 +0100)
committerMarek Olšák <maraeo@gmail.com>
Mon, 12 Dec 2011 07:04:51 +0000 (08:04 +0100)
commitfc52534f012837a39c03a764eb611d460210514a
tree43ce8892cc63d6cdc010094402e12390d8699a9a
parent4298c88f656c191f3daca0c341850dd8c23f5f92
mesa: fix possible precision issues in pack/unpack/fetch functions

GLfloat doesn't have enough precision to exactly represent 0xffffff
and 0xffffffff. (and a reciprocal of those, if I am not mistaken)

If -ffast-math is enabled, using GLfloat causes assertion failures in:
- fbo-blit-d24s8
- fbo-depth-sample-compare
- fbo-readpixels-depth-formats
- glean/depthStencil

For example:
fbo-depth-sample-compare: main/format_unpack.c:1769:
unpack_float_z_Z24_X8: Assertion `dst[i] <= 1.0F' failed.

Reviewed-by: Brian Paul <brianp@vmware.com>
src/mesa/main/format_pack.c
src/mesa/main/format_unpack.c
src/mesa/swrast/s_texfetch_tmp.h