From: Pedro Alves Date: Wed, 11 Jun 2014 10:04:31 +0000 (+0100) Subject: PR remote/17028: GDB+GDBserver hangs on Windows X-Git-Url: https://git.libre-soc.org/?a=commitdiff_plain;h=364fe1f72265eb54bce08511233d06ed48e9c41a;p=binutils-gdb.git PR remote/17028: GDB+GDBserver hangs on Windows Since target-async was turned on by default, debugging on Windows using GDB+GDBserver sometimes hangs while waiting for a RSP reply. The problem is a race in the gdb_select machinery. This is what we see for a faulty next on the GDB side: (gdb) n infrun: clear_proceed_status_thread (Thread 4424) infrun: proceed (addr=0xffffffff, signal=GDB_SIGNAL_DEFAULT, step=1) (...) infrun: resume (step=1, signal=GDB_SIGNAL_0), ... Sending packet: $vCont;s:1148;c#5e... *hang* At this point, attaching a debugger to the hanging GDB confirms that it is blocked, waiting for a socket event: #6 0x757841d8 in WaitForMultipleObjects () from C:\Windows\syswow64\kernel32.dll #7 0x004708e7 in gdb_select (n=469, readfds=0x88ca50 , writefds=0x88cb54 , exceptfds=0x88cc58 , timeout=0x0) at /[...]/gdb/mingw-hdep.c:172 #8 0x00527926 in gdb_wait_for_event (block=1) at /[...]/gdb/event-loop.c:831 #9 0x00526ff1 in gdb_do_one_event () at /[...]/gdb/event-loop.c:403 However, on the GDBserver side, we see that GDBserver already sent a T05 packet reply: gdbserver: kernel event EXCEPTION_DEBUG_EVENT for pid=4968 tid=1148 EXCEPTION_SINGLE_STEP Child Stopped with signal = 5 Writing resume reply for LWP 4968.4424:1 DEBUG: write_prim ($T0505:c8fe2800;04:a0fe2800;08:38164000;thread:1148;#f0) -> 55 To recap, on Windows, 'select' only works with sockets, so we have a wrapper, gdb_select, that uses the GDB serial abstraction to handle sockets, consoles, pipes, and serial ports. Each serial descriptor has a thread associated (we call those the select threads), and those threads communicate with the main thread by means of standard Windows events. It basically goes like this: gdb_select first loops through all fds of interest, calling their wait_handle hooks, which returns an event that WaitForMultipleObjects can wait on. gdb_select then blocks in WaitForMultipleObjects with all those event handles. The wait_handle hook is responsible for arranging for the returned event to become set once data is available. This is done by setting the descriptor's helper thread running, which itself knows how to wait for data from the type of handle it manages (sockets, pipes, consoles, files, etc.). Once data arrives, the select thread sets the corresponding event which unblocks WaitForMultipleObjects within gdb_select. However, the wait_handle hook can also apply an optimization: if data is already pending, then there's no need to set the thread running, and the descriptors event can be set immediately. It's around this latter aspect that lies the bug/race. Adding some ad hoc debug logs to ser-mingw.c and mingw-hdep.c, we see the following sequence of events, right after sending "$vCont;s:1148;c#5e". Thread 1 is the main thread, and thread 2 is the socket's helper/select thread. gdb_select was only passed one descriptor to wait on, the remote target's socket. net_windows_select_thread is the entry point of the select threads for sockets. #1 - thread 1: gdb_select: enter #2 - thread 2: net_windows_select_thread: WaitForMultipleObjects blocking gdb_select walked over the wait_handle hooks, and woke up the socket's helper thread. The helper thread is now blocked waiting for socket events. #3 - thread 1: gdb_select: WaitForMultipleObjects polling (timeout=0ms) #4 - thread 1: gdb_select: WaitForMultipleObjects returned 102 (WAIT_TIMEOUT) There was no pending data available yet, and gdb_select was passed timeout==0ms, and so WaitForMultipleObjects times out immediately. #5 - thread 2: net_windows_select_thread: WaitForMultipleObjects returned 1 Just afterwards, socket data arrives, and thread 2 wakes up. Thread 2 calls WSAEnumNetworkEvents, which clears state->sock_event, and marks the serial's read_event event, telling the main thread that data is available. #6 - thread 1: gdb_select: call serial_done_wait_handle on each serial gdb_select stops all the helper/select threads. #7 - thread 1: gdb_select: return 0 (WAIT_TIMEOUT) gdb_select in the main thread returns to the caller. Note that at this point, data is pending on the socket, the serial's read_event is set, but the socket's sock_event event is not set, until _further_ data arrives. Now GDB does its thing and goes back to the event loop. That calls gdb_select, but with timeout==INFINITE. Again, gdb_select calls the socket serial's wait_handle hook. It first clears its events, starting from a clean slate: ResetEvent (state->base.read_event); ResetEvent (state->base.except_event); ResetEvent (state->base.stop_select); That cleared read_event, which was previously set in #5 above. And then it checks for pending events, in the sock_event event: /* Check any pending events. This both avoids starting the thread unnecessarily, and handles stray FD_READ events (see below). */ if (WaitForSingleObject (state->sock_event, 0) == WAIT_OBJECT_0) { That also fails because state->sock_event was cleared in #5 too... So the wait_handle hook erroneously decides that it needs to start the helper thread to wait for input: #8 - thread 2: net_windows_select_thread: WaitForMultipleObjects blocking #9 - thread 1: gdb_select: WaitForMultipleObjects blocking (INFINITE) But, GDBserver already sent all it had to send, so both threads waits forever... At first I thought that net_windows_wait_handle shouldn't be resetting state->base.read_event or state->base.except_event, but looking deeper, the pipe and console wait_handle hooks reset all events too. It actually makes sense that way -- consuming an event from different threads is bad practice, and, we should always be able to query pending state without looking at the state->sock_event from within net_windows_wait_handle. The end result is much simpler, and makes net_windows_select_thread look a lot like console_select_thread, actually. gdb/ 2014-06-11 Pedro Alves PR remote/17028 * ser-mingw.c (net_windows_socket_check_pending): New function. (net_windows_select_thread): Ignore spurious wakeups. Use net_windows_socket_check_pending. (net_windows_wait_handle): Check for pending events with ioctlsocket, through net_windows_socket_check_pending, instead of checking the socket's event. --- diff --git a/gdb/ChangeLog b/gdb/ChangeLog index 9fd45e0c3d1..6077c419ef1 100644 --- a/gdb/ChangeLog +++ b/gdb/ChangeLog @@ -1,3 +1,13 @@ +2014-06-11 Pedro Alves + + PR remote/17028 + * ser-mingw.c (net_windows_socket_check_pending): New function. + (net_windows_select_thread): Ignore spurious wakeups. Use + net_windows_socket_check_pending. + (net_windows_wait_handle): Check for pending events with + ioctlsocket, through net_windows_socket_check_pending, instead of + checking the socket's event. + 2014-06-10 Siva Chandra Reddy * python/python-internal.h (gdb_PyObject_GetAttrString) diff --git a/gdb/ser-mingw.c b/gdb/ser-mingw.c index db049991e87..a6469ea8cfb 100644 --- a/gdb/ser-mingw.c +++ b/gdb/ser-mingw.c @@ -1046,6 +1046,32 @@ struct net_windows_state HANDLE sock_event; }; +/* Check whether the socket has any pending data to be read. If so, + set the select thread's read event. On error, set the select + thread's except event. If any event was set, return true, + otherwise return false. */ + +static int +net_windows_socket_check_pending (struct serial *scb) +{ + struct net_windows_state *state = scb->state; + unsigned long available; + + if (ioctlsocket (scb->fd, FIONREAD, &available) != 0) + { + /* The socket closed, or some other error. */ + SetEvent (state->base.except_event); + return 1; + } + else if (available > 0) + { + SetEvent (state->base.read_event); + return 1; + } + + return 0; +} + static DWORD WINAPI net_windows_select_thread (void *arg) { @@ -1065,33 +1091,54 @@ net_windows_select_thread (void *arg) wait_events[0] = state->base.stop_select; wait_events[1] = state->sock_event; - event_index = WaitForMultipleObjects (2, wait_events, FALSE, INFINITE); - - if (event_index == WAIT_OBJECT_0 - || WaitForSingleObject (state->base.stop_select, 0) == WAIT_OBJECT_0) - /* We have been requested to stop. */ - ; - else if (event_index != WAIT_OBJECT_0 + 1) - /* Some error has occured. Assume that this is an error - condition. */ - SetEvent (state->base.except_event); - else + /* Wait for something to happen on the socket. */ + while (1) { + event_index = WaitForMultipleObjects (2, wait_events, FALSE, INFINITE); + + if (event_index == WAIT_OBJECT_0 + || WaitForSingleObject (state->base.stop_select, 0) == WAIT_OBJECT_0) + { + /* We have been requested to stop. */ + break; + } + + if (event_index != WAIT_OBJECT_0 + 1) + { + /* Some error has occured. Assume that this is an error + condition. */ + SetEvent (state->base.except_event); + break; + } + /* Enumerate the internal network events, and reset the object that signalled us to catch the next event. */ - WSAEnumNetworkEvents (scb->fd, state->sock_event, &events); - - gdb_assert (events.lNetworkEvents & (FD_READ | FD_CLOSE)); - + if (WSAEnumNetworkEvents (scb->fd, state->sock_event, &events) != 0) + { + /* Something went wrong. Maybe the socket is gone. */ + SetEvent (state->base.except_event); + break; + } + if (events.lNetworkEvents & FD_READ) - SetEvent (state->base.read_event); - + { + if (net_windows_socket_check_pending (scb)) + break; + + /* Spurious wakeup. That is, the socket's event was + signalled before we last called recv. */ + } + if (events.lNetworkEvents & FD_CLOSE) - SetEvent (state->base.except_event); + { + SetEvent (state->base.except_event); + break; + } } SetEvent (state->base.have_stopped); } + return 0; } static void @@ -1107,60 +1154,10 @@ net_windows_wait_handle (struct serial *scb, HANDLE *read, HANDLE *except) *read = state->base.read_event; *except = state->base.except_event; - /* Check any pending events. This both avoids starting the thread - unnecessarily, and handles stray FD_READ events (see below). */ - if (WaitForSingleObject (state->sock_event, 0) == WAIT_OBJECT_0) - { - WSANETWORKEVENTS events; - int any = 0; - - /* Enumerate the internal network events, and reset the object that - signalled us to catch the next event. */ - WSAEnumNetworkEvents (scb->fd, state->sock_event, &events); - - /* You'd think that FD_READ or FD_CLOSE would be set here. But, - sometimes, neither is. I suspect that the FD_READ is set and - the corresponding event signalled while recv is running, and - the FD_READ is then lowered when recv consumes all the data, - but there's no way to un-signal the event. This isn't a - problem for the call in net_select_thread, since any new - events after this point will not have been drained by recv. - It just means that we can't have the obvious assert here. */ - - /* If there is a read event, it might be still valid, or it might - not be - it may have been signalled before we last called - recv. Double-check that there is data. */ - if (events.lNetworkEvents & FD_READ) - { - unsigned long available; - - if (ioctlsocket (scb->fd, FIONREAD, &available) == 0 - && available > 0) - { - SetEvent (state->base.read_event); - any = 1; - } - else - /* Oops, no data. This call to recv will cause future - data to retrigger the event, e.g. while we are - in net_select_thread. */ - recv (scb->fd, NULL, 0, 0); - } - - /* If there's a close event, then record it - it is obviously - still valid, and it will not be resignalled. */ - if (events.lNetworkEvents & FD_CLOSE) - { - SetEvent (state->base.except_event); - any = 1; - } - - /* If we set either handle, there's no need to wake the thread. */ - if (any) - return; - } - - start_select_thread (&state->base); + /* Check any pending events. Otherwise, start the select + thread. */ + if (!net_windows_socket_check_pending (scb)) + start_select_thread (&state->base); } static void