gdb/amdgpu: Fix debugging multiple inferiors using the ROCm runtime
When debugging a multi-process application where a parent spawns
multiple child processes using the ROCm runtime, I see the following
assertion failure:
../../gdb/amd-dbgapi-target.c:1071: internal-error: process_one_event: Assertion `runtime_state == AMD_DBGAPI_RUNTIME_STATE_UNLOADED' failed.
A problem internal to GDB has been detected,
further debugging may prove unreliable.
----- Backtrace -----
0x556e9a318540 gdb_internal_backtrace_1
../../gdb/bt-utils.c:122
0x556e9a318540 _Z22gdb_internal_backtracev
../../gdb/bt-utils.c:168
0x556e9a730224 internal_vproblem
../../gdb/utils.c:396
0x556e9a7304e0 _Z15internal_verrorPKciS0_P13__va_list_tag
../../gdb/utils.c:476
0x556e9a87aeb4 _Z18internal_error_locPKciS0_z
../../gdbsupport/errors.cc:58
0x556e9a29f446 process_one_event
../../gdb/amd-dbgapi-target.c:1071
0x556e9a29f446 process_event_queue
../../gdb/amd-dbgapi-target.c:1156
0x556e9a29faf2 _ZN17amd_dbgapi_target4waitE6ptid_tP17target_waitstatus10enum_flagsI16target_wait_flagE
../../gdb/amd-dbgapi-target.c:1262
0x556e9a6b0965 _Z11target_wait6ptid_tP17target_waitstatus10enum_flagsI16target_wait_flagE
../../gdb/target.c:2586
0x556e9a4c221f do_target_wait_1
../../gdb/infrun.c:3876
0x556e9a4d8489 operator()
../../gdb/infrun.c:3935
0x556e9a4d8489 do_target_wait
../../gdb/infrun.c:3964
0x556e9a4d8489 _Z20fetch_inferior_eventv
../../gdb/infrun.c:4365
0x556e9a87b915 gdb_wait_for_event
../../gdbsupport/event-loop.cc:694
0x556e9a87c3a9 gdb_wait_for_event
../../gdbsupport/event-loop.cc:593
0x556e9a87c3a9 _Z16gdb_do_one_eventi
../../gdbsupport/event-loop.cc:217
0x556e9a521689 start_event_loop
../../gdb/main.c:412
0x556e9a521689 captured_command_loop
../../gdb/main.c:476
0x556e9a523c04 captured_main
../../gdb/main.c:1320
0x556e9a523c04 _Z8gdb_mainP18captured_main_args
../../gdb/main.c:1339
0x556e9a24b1bf main
../../gdb/gdb.c:32
---------------------
../../gdb/amd-dbgapi-target.c:1071: internal-error: process_one_event: Assertion `runtime_state == AMD_DBGAPI_RUNTIME_STATE_UNLOADED' failed.
A problem internal to GDB has been detected,
Before diving into why this error appears, let's explore how things are
expected to work in normal circumstances. When a process being debugged
starts using the ROCm runtime, the following happens:
- The runtime registers itself to the driver.
- The driver creates a "runtime loaded" event and notifies the debugger
that a new event is available by writing to a file descriptor which is
registered in GDB's main event loop.
- GDB core calls the callback associated with this file descriptor
(dbgapi_notifier_handler). Because the amd-dbgapi-target is not
pushed at this point, the handler pulls the "runtime loaded" event
from the driver (this is the only event which can be available at this
point) and eventually pushes the amd-dbgapi-target on the inferior's
target stack.
In a nutshell, this is the expected AMDGPU runtime activation process.
From there, when new events are available regarding the GPU threads, the
same file descriptor is written to. The callback sees that the
amd-dbgapi-target is pushed so marks the amd_dbgapi_async_event_handler.
This will later cause amd_dbgapi_target::wait to be called. The wait
method pulls all the available events from the driver and handles them.
The wait method returns the information conveyed by the first event, the
other events are cached for later calls of the wait method.
Note that because we are under the wait method, we know that the
amd-dbgapi-target is pushed on the inferior target stack. This implies
that the runtime activation event has been seen already. As a
consequence, we cannot receive another event indicating that the runtime
gets activated. This is what the failing assertion checks.
In the case when we have multiple inferiors however, there is a flaw in
what have been described above. If one inferior (let's call it inferior
1) already has the amd-dbgapi-target pushed to its target stack and
another inferior (inferior 2) activates the ROCm runtime, here is what
can happen:
- The driver creates the runtime activation for inferior 2 and writes to
the associated file descriptor.
- GDB has inferior 1 selected and calls target_wait for some reason.
- This prompts amd_dbgapi_target::wait to be called. The method pulls
all events from the driver, including the runtime activation event for
inferior 2, leading to the assertion failure.
The fix for this problem is simple. To avoid such problem, we need to
make sure that amd_dbgapi_target::wait only pulls events for the current
inferior from the driver. This is what this patch implements.
This patch also includes a testcase which could fail before this patch.
This patch has been tested on a system with multiple GPUs which had more
chances to reproduce the original bug. It has also been tested on top
of the downstream ROCgdb port which has more AMDGPU related tests. The
testcase has been tested with `make check check-read1 check-readmore`.
Approved-By: Pedro Alves <pedro@palves.net>