mesa: Fix return type of _mesa_get_format_bytes() (#37351)

Despite that negative values aren't sensible here, making this unsigned
is dangerous.  Consider get_pointer_generic, which computes a value of
the form:

    void *base + (int x * int stride + int y) * unsigned bpp

The usual arithmetic conversions will coerce the (x*stride + y)
subexpression to unsigned.  Since stride can be negative, this is
disastrous.

Fixes at least the following piglit tests on Ironlake:

    fbo/fbo-blit-d24s8
    spec/ARB_depth_texture/fbo-clear-formats
    spec/EXT_packed_depth_stencil/fbo-clear-formats

NOTE: This is a candidate for the 7.10 branch.

Reviewed-by: Chad Versace <chad.versace@intel.com>
Signed-off-by: Adam Jackson <ajax@redhat.com>

diff --git a/src/mesa/main/formats.c b/src/mesa/main/formats.c
index 60e8ae390a5052d0fd076da9e4e9ad0e32556f6b..e88ba43971b695f17fa40fe693a89d9e3e256420 100644 (file)
--- a/src/mesa/main/formats.c
+++ b/src/mesa/main/formats.c
@@ -1118,8 +1118,10 @@ _mesa_get_format_name(gl_format format)
  * Return bytes needed to store a block of pixels in the given format.
  * Normally, a block is 1x1 (a single pixel).  But for compressed formats
  * a block may be 4x4 or 8x4, etc.
+ *
+ * Note: not GLuint, so as not to coerce math to unsigned. cf. fdo #37351
  */
-GLuint
+GLint
 _mesa_get_format_bytes(gl_format format)
 {
    const struct gl_format_info *info = _mesa_get_format_info(format);

diff --git a/src/mesa/main/formats.h b/src/mesa/main/formats.h
index df9ed70a3e0fd47b8f866145a970ee4cf9e4c402..0640bbc4af10b37fef660eb45a00d0fe5f0d2d5f 100644 (file)
--- a/src/mesa/main/formats.h
+++ b/src/mesa/main/formats.h
@@ -216,7 +216,7 @@ typedef enum
 extern const char *
 _mesa_get_format_name(gl_format format);
 
-extern GLuint
+extern GLint
 _mesa_get_format_bytes(gl_format format);
 
 extern GLint