Improve ptrace-error detection on Linux targets

Improve ptrace-error detection on Linux targets

In Fedora GDB, we carry the following patch:

  https://src.fedoraproject.org/rpms/gdb/blob/8ac06474ff1e2aa4920d14e0666b083eeaca8952/f/gdb-attach-fail-reasons-5of5.patch

Its purpose is to try to detect a specific scenario where SELinux's
'deny_ptrace' option is enabled, which prevents GDB from ptrace'ing in
order to debug the inferior (PTRACE_ATTACH and PTRACE_TRACEME will
fail with EACCES in this case).

I like the idea of improving error detection and providing more
information to the user (a simple "Permission denied" can be really
frustrating), but I don't fully agree with the way the patch was
implemented: it makes GDB link against libselinux only for the sake of
consulting the 'deny_ptrace' setting, and then prints a warning if
ptrace failed and this setting is on.

My first thought (and attempt) was to make GDB print a generic warning
when a ptrace error happened; this message would just point the user
to our documentation, where she could find more information about
possible causes for the error (and try to diagnose/fix the problem).
This proved to be too simple, and I was convinced that it is actually
a good idea to go the extra kilometre and try to pinpoint the specific
problem (or problems) preventing ptrace from working, as well as
provide useful suggestions on how the user can fix things.

Here is the patch I came up with.  It implements a new function,
'linux_ptrace_restricted_fail_reason', which does a few things to
check what's wrong with ptrace:

  - It dlopen's "libselinux.so.1" and checks if the "deny_ptrace"
    option is enabled.

  - It reads the contents of "/proc/sys/kernel/yama/ptrace_scope" and
    checks if it's different than 0.

For each of these checks, if it succeeds, the user will see a message
informing about the restriction in place, and how it can be disabled.
For example, if "deny_ptrace" is enabled, the user will see:

  # gdb /usr/bin/true
  ...
  Starting program: /usr/bin/true
  warning: Could not trace the inferior process.
  warning: ptrace: Permission denied
  The SELinux 'deny_ptrace' option is enabled and preventing GDB
  from using 'ptrace'.  You can disable it by executing (as root):

    setsebool deny_ptrace off

  If you are debugging the inferior remotely, the ptrace restriction(s) need
  to be disabled in the target system (e.g., where GDBserver is running).
  During startup program exited with code 127.
  (gdb)

In case "/proc/sys/kernel/yama/ptrace_scope" is > 0:

  # gdb /usr/bin/true
  ...
  Starting program: /usr/bin/true
  warning: Could not trace the inferior process.
  warning: ptrace: Operation not permitted
  The Linux kernel's Yama ptrace scope is in effect, which can prevent
  GDB from using 'ptrace'.  You can disable it by executing (as root):

    echo 0 > /proc/sys/kernel/yama/ptrace_scope

  If you are debugging the inferior remotely, the ptrace restriction(s) need
  to be disabled in the target system (e.g., where GDBserver is running).
  During startup program exited with code 127.
  (gdb)

If both restrictions are enabled, both messages will show up.

This works for gdbserver as well, and actually fixes a latent bug I
found: when ptrace is restricted, gdbserver would hang due to an
unchecked ptrace call:

  # gdbserver :9988 /usr/bin/true
  gdbserver: linux_ptrace_test_ret_to_nx: Cannot PTRACE_TRACEME: Operation not permitted
  gdbserver: linux_ptrace_test_ret_to_nx: status 256 is not WIFSTOPPED!
  gdbserver: linux_ptrace_test_ret_to_nx: failed to kill child pid 2668100 No such process
  [ Here you would have to issue a C-c ]

Now, you will see:

  # gdbserver :9988 /usr/bin/true
  gdbserver: linux_ptrace_test_ret_to_nx: Cannot PTRACE_TRACEME: Permission denied
  gdbserver: linux_ptrace_test_ret_to_nx: status 256 is not WIFSTOPPED!
  gdbserver: linux_ptrace_test_ret_to_nx: failed to kill child pid 2766868 No such process
  gdbserver: Could not trace the inferior process.
  gdbserver: ptrace: Permission denied
  The SELinux 'deny_ptrace' option is enabled and preventing GDB
  from using 'ptrace'.  You can disable it by executing (as root):

    setsebool deny_ptrace off

  If you are debugging the inferior remotely, the ptrace restriction(s) need
  to be disabled in the target system (e.g., where GDBserver is running).
  #

(I decided to keep all the other messages, even though I find them a
bit distracting).

If GDB can't determine the cause for the failure, it will still print
the generic error message which tells the user to check our
documentation:

  There might be restrictions preventing ptrace from working.  Please see
  the appendix "Linux kernel ptrace restrictions" in the GDB documentation
  for more details.
  If you are debugging the inferior remotely, the ptrace restriction(s) need
  to be disabled in the target system (e.g., where GDBserver is running).

This means that the patch expands our documentation and creates a new
appendix section named "Linux kernel ptrace restrictions", with
sub-sections for each possible restriction that might be in place.

Notice how, on every message, we instruct the user to "do the right
thing" if gdbserver is being used.  This is because if the user
started gdbserver *before* any ptrace restriction was in place, and
then, for some reason, one or more restrictions get enabled, then the
error message will be displayed both on gdbserver *and* on the
connected GDB.  Since the user will be piloting GDB, it's important to
explicitly say that the ptrace restrictions are enabled in the target,
where gdbserver is running.

The current list of possible restrictions is:

  - SELinux's 'deny_ptrace' option (detected).

  - YAMA's /proc/sys/kernel/yama/ptrace_scope setting (detected).

  - seccomp on Docker containers (I couldn't find how to detect).

It's important to mention that all of this is Linux-specific; as far
as I know, SELinux, YAMA and seccomp are Linux-only features.

I tested this patch locally, on my Fedora 30 machine (actually, a
Fedora Rawhide VM), but I'm not proposing a testcase for it because of
the difficulty of writing one.

WDYT?

gdb/doc/ChangeLog:
2019-09-26  Sergio Durigan Junior  <sergiodj@redhat.com>

	* gdb.texinfo (Linux kernel ptrace restrictions): New appendix
	section.

gdb/ChangeLog:
2019-09-26  Sergio Durigan Junior  <sergiodj@redhat.com>
	    Jan Kratochvil  <jan.kratochvil@redhat.com>
	    Pedro Alves  <palves@redhat.com>

	* gdbsupport/gdb-dlfcn.h (gdb_dlopen): Update comment and
	mention that the function throws an error.
	* inf-ptrace.c (default_inf_ptrace_me_fail_reason): New
	function.
	(inf_ptrace_me_fail_reason): New variable.
	(inf_ptrace_me): Update call to 'trace_start_error_with_name'.
	* inf-ptrace.h (inf_ptrace_me_fail_reason): New variable.
	* linux-nat.c (attach_proc_task_lwp_callback): Call
	'linux_ptrace_attach_fail_reason_lwp'.
	(linux_nat_target::attach): Update call to
	'linux_ptrace_attach_fail_reason'.
	(_initialize_linux_nat): Set 'inf_ptrace_me_fail_reason'.
	* nat/fork-inferior.c (trace_start_error_with_name): Add
	optional 'append' argument.
	* nat/fork-inferior.h (trace_start_error_with_name): Update
	prototype.
	* nat/linux-ptrace.c: Include "gdbsupport/gdb-dlfcn.h",
	"gdbsupport/filestuff.h" and "nat/fork-inferior.h".
	(selinux_ftype): New typedef.
	(linux_ptrace_restricted_fail_reason): New function.
	(linux_ptrace_attach_fail_reason_1): New function.
	(linux_ptrace_attach_fail_reason): Change first argument type
	from 'ptid_t' to 'pid_t'.  Call
	'linux_ptrace_attach_fail_reason_1' and
	'linux_ptrace_restricted_fail_reason'.
	(linux_ptrace_attach_fail_reason_lwp): New function.
	(linux_ptrace_me_fail_reason): New function.
	(errno_pipe): New variable.
	(linux_fork_to_function): Initialize pipe before forking.
	(linux_child_function): Deal with errno-passing from child.
	Handle ptrace error.
	(linux_check_child_ptrace_errno): New function.
	(linux_check_child_ptrace_errno): Call
	'linux_check_child_ptrace_errno'.
	* nat/linux-ptrace.h (linux_ptrace_attach_fail_reason): Update
	prototype.
	(linux_ptrace_attach_fail_reason_lwp): New prototype.
	(linux_ptrace_me_fail_reason): New prototype.
	* remote.c (extended_remote_target::attach): Handle error
	message passed by the server when attach fails.

gdb/gdbserver/ChangeLog:
2019-09-26  Sergio Durigan Junior  <sergiodj@redhat.com>
	    Pedro Alves  <palves@redhat.com>

	* linux-low.c (linux_ptrace_fun): Call
	'linux_ptrace_me_fail_reason'.
	(attach_proc_task_lwp_callback): Call
	'linux_ptrace_attach_fail_reason_lwp'.
	(linux_attach): Call 'linux_ptrace_attach_fail_reason'.
	* server.c (handle_v_attach): Use try..catch when calling
	'attach_inferior', and send an error message to the client
	when needed.
	* thread-db.c (attach_thread): Call
	'linux_ptrace_attach_fail_reason_lwp'.