From: Saagar Jha <saagar@saagarjha.com>
Date: Wed, 27 Feb 2019 18:48:35 +0000 (+0000)
Subject: Prevent overflow in rl_set_screen_size
X-Git-Url: https://git.libre-soc.org/?a=commitdiff_plain;h=23031e319207140e76a9c18e308995fe28322b58;p=binutils-gdb.git

Prevent overflow in rl_set_screen_size

GDB calls rl_set_screen_size in readline with the current screen size,
measured in rows and columns.  To represent "infinite" sizes, GDB
passes in INT_MAX; however, since rl_set_screen_size internally
multiplies the number of rows and columns, this causes a signed
integer overflow.  To prevent this we can instead pass in the
approximate square root of INT_MAX (which is still reasonably large),
so that even when the number of rows and columns is "infinite" we
don't overflow.

gdb/ChangeLog:
2019-02-27  Saagar Jha  <saagar@saagarjha.com>
	    Pedro Alves  <palves@redhat.com>

	* utils.c (set_screen_size): Reduce "infinite" rows and columns
	before calling rl_set_screen_size.
---

diff --git a/gdb/ChangeLog b/gdb/ChangeLog
index de23d4b98ef..02602b2320a 100644
--- a/gdb/ChangeLog
+++ b/gdb/ChangeLog
@@ -1,3 +1,9 @@
+2019-02-27  Saagar Jha  <saagar@saagarjha.com>
+	    Pedro Alves  <palves@redhat.com>
+
+	* utils.c (set_screen_size): Reduce "infinite" rows and columns
+	before calling rl_set_screen_size.
+
 2019-02-27  Tom Tromey  <tromey@adacore.com>
 
 	* configure.ac (HAVE_LIBPYTHON2_4, HAVE_LIBPYTHON2_5): Never
diff --git a/gdb/utils.c b/gdb/utils.c
index ec2619642a1..069da23542e 100644
--- a/gdb/utils.c
+++ b/gdb/utils.c
@@ -1380,11 +1380,24 @@ set_screen_size (void)
   int rows = lines_per_page;
   int cols = chars_per_line;
 
-  if (rows <= 0)
-    rows = INT_MAX;
+  /* If we get 0 or negative ROWS or COLS, treat as "infinite" size.
+     A negative number can be seen here with the "set width/height"
+     commands and either:
 
-  if (cols <= 0)
-    cols = INT_MAX;
+     - the user specified "unlimited", which maps to UINT_MAX, or
+     - the user spedified some number between INT_MAX and UINT_MAX.
+
+     Cap "infinity" to approximately sqrt(INT_MAX) so that we don't
+     overflow in rl_set_screen_size, which multiplies rows and columns
+     to compute the number of characters on the screen.  */
+
+  const int sqrt_int_max = INT_MAX >> (sizeof (int) * 8 / 2);
+
+  if (rows <= 0 || rows > sqrt_int_max)
+    rows = sqrt_int_max;
+
+  if (cols <= 0 || cols > sqrt_int_max)
+    cols = sqrt_int_max;
 
   /* Update Readline's idea of the terminal size.  */
   rl_set_screen_size (rows, cols);