Prevent overflow in rl_set_screen_size
authorSaagar Jha <saagar@saagarjha.com>
Wed, 27 Feb 2019 18:48:35 +0000 (18:48 +0000)
committerPedro Alves <palves@redhat.com>
Wed, 27 Feb 2019 18:48:35 +0000 (18:48 +0000)
commit23031e319207140e76a9c18e308995fe28322b58
treee182f185d020c2572216a1b916ad8f647aeaa78d
parent6c28e44a359e9f6cf455ddff0009ca99406f7224
Prevent overflow in rl_set_screen_size

GDB calls rl_set_screen_size in readline with the current screen size,
measured in rows and columns.  To represent "infinite" sizes, GDB
passes in INT_MAX; however, since rl_set_screen_size internally
multiplies the number of rows and columns, this causes a signed
integer overflow.  To prevent this we can instead pass in the
approximate square root of INT_MAX (which is still reasonably large),
so that even when the number of rows and columns is "infinite" we
don't overflow.

gdb/ChangeLog:
2019-02-27  Saagar Jha  <saagar@saagarjha.com>
    Pedro Alves  <palves@redhat.com>

* utils.c (set_screen_size): Reduce "infinite" rows and columns
before calling rl_set_screen_size.
gdb/ChangeLog
gdb/utils.c