mesa: Fix return type of _mesa_get_format_bytes() (#37351)
authorAdam Jackson <ajax@redhat.com>
Fri, 20 May 2011 22:21:15 +0000 (18:21 -0400)
committerAdam Jackson <ajax@redhat.com>
Mon, 23 May 2011 15:07:34 +0000 (11:07 -0400)
commite8b1c6d6f55f5be3bef25084fdd8b6127517e137
tree0bbb1b20dd02eea0f360d037b554b5c8f38db37c
parentd0c6d24a9ebe31b3a805e8f5845fc7e6fa552a98
mesa: Fix return type of  _mesa_get_format_bytes() (#37351)

Despite that negative values aren't sensible here, making this unsigned
is dangerous.  Consider get_pointer_generic, which computes a value of
the form:

    void *base + (int x * int stride + int y) * unsigned bpp

The usual arithmetic conversions will coerce the (x*stride + y)
subexpression to unsigned.  Since stride can be negative, this is
disastrous.

Fixes at least the following piglit tests on Ironlake:

    fbo/fbo-blit-d24s8
    spec/ARB_depth_texture/fbo-clear-formats
    spec/EXT_packed_depth_stencil/fbo-clear-formats

NOTE: This is a candidate for the 7.10 branch.

Reviewed-by: Chad Versace <chad.versace@intel.com>
Signed-off-by: Adam Jackson <ajax@redhat.com>
src/mesa/main/formats.c
src/mesa/main/formats.h