meta: Mask Stencil.Clear against stencilMax in _mesa_meta_Clear
authorPeter Clifton <pcjc2@cam.ac.uk>
Wed, 24 Nov 2010 15:24:31 +0000 (15:24 +0000)
committerBrian Paul <brianp@vmware.com>
Wed, 24 Nov 2010 19:14:54 +0000 (12:14 -0700)
commitee88727df8b133e99608e5d30cbd50fb6e11628a
tree8a05d306a234af18bb9cf0894185b29107f3e25d
parent03a4f97a68dc0bf7008efcfdb61fd7033357d1ab
meta: Mask Stencil.Clear against stencilMax in _mesa_meta_Clear

This fixes incorrect behaviour when the stencil clear value exceeds
the size of the stencil buffer, for example, when set with:

glClearStencil (~1); /* Set a bit pattern of 111...11111110 */
glClear (GL_STENCIL_BUFFER_BIT);

The clear value needs to be masked by the value 2^m - 1, where m is the
number of bits in the stencil buffer. Previously, we passed the value
masked with 0x7fffffff to _mesa_StencilFuncSeparate which then clamps,
NOT masks the value to the range 0 to 2^m - 1.

The result would be clearing the stencil buffer to 0xff, rather than 0xfe.

Signed-off-by: Peter Clifton <pcjc2@cam.ac.uk>
Signed-off-by: Brian Paul <brianp@vmware.com>
src/mesa/drivers/common/meta.c