llvmpipe: fix overflow bug in total texture size computation
authorBrian Paul <brianp@vmware.com>
Thu, 20 Sep 2012 15:13:37 +0000 (09:13 -0600)
committerBrian Paul <brianp@vmware.com>
Thu, 20 Sep 2012 15:47:09 +0000 (09:47 -0600)
commit0bcad0295543425f5ec4b44bfe6e600794e9c166
tree999da4b5dfe104465f41f65397a972cc8d27bfe2
parent7b4aefd3c9c1b04832306b8114b3f2007318f087
llvmpipe: fix overflow bug in total texture size computation

v2: use uint64_t for the total_size variable, per Jose.

Also add two earlier checks for exceeding the max texture size.
For example a 1K^3 RGBA volume would overflow the lpr->image_stride
variable.

Use simple algebra to avoid overflow in intermediate values.
So instead of "x * y > z" use "x > z / y".

This should work if we happen to be on a platform that doesn't have
64-bit types.

Reviewed-by: Jose Fonseca <jfonseca@vmware.com>
src/gallium/drivers/llvmpipe/lp_texture.c