From 23031e319207140e76a9c18e308995fe28322b58 Mon Sep 17 00:00:00 2001 From: Saagar Jha Date: Wed, 27 Feb 2019 18:48:35 +0000 Subject: [PATCH] Prevent overflow in rl_set_screen_size GDB calls rl_set_screen_size in readline with the current screen size, measured in rows and columns. To represent "infinite" sizes, GDB passes in INT_MAX; however, since rl_set_screen_size internally multiplies the number of rows and columns, this causes a signed integer overflow. To prevent this we can instead pass in the approximate square root of INT_MAX (which is still reasonably large), so that even when the number of rows and columns is "infinite" we don't overflow. gdb/ChangeLog: 2019-02-27 Saagar Jha Pedro Alves * utils.c (set_screen_size): Reduce "infinite" rows and columns before calling rl_set_screen_size. --- gdb/ChangeLog | 6 ++++++ gdb/utils.c | 21 +++++++++++++++++---- 2 files changed, 23 insertions(+), 4 deletions(-) diff --git a/gdb/ChangeLog b/gdb/ChangeLog index de23d4b98ef..02602b2320a 100644 --- a/gdb/ChangeLog +++ b/gdb/ChangeLog @@ -1,3 +1,9 @@ +2019-02-27 Saagar Jha + Pedro Alves + + * utils.c (set_screen_size): Reduce "infinite" rows and columns + before calling rl_set_screen_size. + 2019-02-27 Tom Tromey * configure.ac (HAVE_LIBPYTHON2_4, HAVE_LIBPYTHON2_5): Never diff --git a/gdb/utils.c b/gdb/utils.c index ec2619642a1..069da23542e 100644 --- a/gdb/utils.c +++ b/gdb/utils.c @@ -1380,11 +1380,24 @@ set_screen_size (void) int rows = lines_per_page; int cols = chars_per_line; - if (rows <= 0) - rows = INT_MAX; + /* If we get 0 or negative ROWS or COLS, treat as "infinite" size. + A negative number can be seen here with the "set width/height" + commands and either: - if (cols <= 0) - cols = INT_MAX; + - the user specified "unlimited", which maps to UINT_MAX, or + - the user spedified some number between INT_MAX and UINT_MAX. + + Cap "infinity" to approximately sqrt(INT_MAX) so that we don't + overflow in rl_set_screen_size, which multiplies rows and columns + to compute the number of characters on the screen. */ + + const int sqrt_int_max = INT_MAX >> (sizeof (int) * 8 / 2); + + if (rows <= 0 || rows > sqrt_int_max) + rows = sqrt_int_max; + + if (cols <= 0 || cols > sqrt_int_max) + cols = sqrt_int_max; /* Update Readline's idea of the terminal size. */ rl_set_screen_size (rows, cols); -- 2.30.2