mesa: rework texture size error checking