watch_thread_num.exp and targets with fairer event reporting

watch_thread_num.exp and targets with fairer event reporting

This patch fixes the watch_thread_num.exp test to work when the target
is better at making event handling be fair among threads.

I wrote patches that make GDB native and GDBserver event handling
fairer between threads.  That is, if threads A and B both
simultaneously trigger some debug event, GDB will pick either A or B
at random, rather than always handling the event of A first.  There's
code for that in the Linux backends (gdb and gdbserver) already, but
it can be improved, and only works in all-stop mode.

With those fixes in place, I found that the watch_thread_num.exp would
often time out.  The problem is that the test only works _because_
event handling isn't as fair as intended.  With the fairness fixes,
the test falls victim of PR10116 (gdb drops watchpoints on
multi-threaded apps) quite often.

To expand on the PR10116 reference, consider that stop events are
serialized to GDB core, through target_wait.  Say a thread-specific
watchpoint as set on thread A.  When the "right" thread and some other
"wrong" thread both trigger a watchpoint simultaneously, the target
may report the "wrong" thread's hit to GDB first (thread B).  When
handling that event, GDB notices the watchpoint is for another thread,
and so shouldn't cause a user-visible stop.  On resume, GDB saves the
now current value of the watched expression.  Afterwards, the "right"
thread (thread A) reports its watchpoint trigger.  But the watched
value hasn't changed since GDB last saved it, and so GDB doesn't
report the watchpoint hit to the user.

The way the test is written, the watchpoint is associated with the
first thread that happens to report an event.  It happens that GDB is
processing events much more often for one of the threads, which
usually will be that same first thread.

Hacking the test with "set debug infrun 1", we see exactly that:

$ grep  "infrun.*\[Thread.*," testsuite/gdb.log | sort | uniq -c | sort -nr
     70 infrun:   8798 [Thread 8798],
     37 infrun:   8798 [Thread 8802],
     36 infrun:   8798 [Thread 8804],
     36 infrun:   8798 [Thread 8803],
     35 infrun:   8798 [Thread 8805],
     34 infrun:   8798 [Thread 8806],

The first column shows the number of times the target reported an
event for that thread, from:

 infrun: target_wait (-1, status) =
 infrun:   8798 [Thread 8798],
 infrun:   status->kind = stopped, signal = GDB_SIGNAL_TRAP

This masks out the PR10116 issue.

However, if the target is better at giving equal priority to all
threads, the PR10116 issue happens often, so it may take quite a while
for the right thread to be the first to report its watchpoint event
just after the memory being watched really changed, resulting in test
time outs.

Here's the number of events handled for each thread on a gdbserver run
with the event fairness patches:

$ grep  "infrun.*\[Thread.*," gdb.log | sort | uniq -c
   2961 infrun:   13591 [Thread 13591],
   2956 infrun:   13591 [Thread 13595],
   2941 infrun:   13591 [Thread 13596],
   2932 infrun:   13591 [Thread 13597],
   2905 infrun:   13591 [Thread 13598],
   2891 infrun:   13591 [Thread 13599],

Note how the number of events is much higher.  The test routinely
takes over 10 seconds to finish on my machine rather than under a
second as with unpatched gdbserver, when it succeeds, but often it'll
fail with timeouts too.

So to make the test robust, this patch switches the tests to using
"awatch" instead of "watch", as access watchpoints don't care about
the watchpoint's "old value".  With this, the test always finishes
quickly, and we can even bump the number of threads concurrently
writting to the shared variable, to have better assurance we're really
testing the case of the "wrong" thread triggering a watchpoint.

Here's the number of events I see for each thread on a run on my
machine, with a gdbserver patched with the event fairness series:

$ grep  "infrun.*\[Thread.*," testsuite/gdb.log | sort | uniq -c
      5 infrun:   5298 [Thread 5302],
      4 infrun:   5298 [Thread 5303],
      4 infrun:   5298 [Thread 5304],
      4 infrun:   5298 [Thread 5305],
      4 infrun:   5298 [Thread 5306],
      4 infrun:   5298 [Thread 5307],
      4 infrun:   5298 [Thread 5308],
      4 infrun:   5298 [Thread 5309],
      4 infrun:   5298 [Thread 5310],
      4 infrun:   5298 [Thread 5311],
      4 infrun:   5298 [Thread 5312],
      4 infrun:   5298 [Thread 5313],
      4 infrun:   5298 [Thread 5314],
      4 infrun:   5298 [Thread 5315],
      4 infrun:   5298 [Thread 5316],

gdb/testsuite/
2015-01-09  Pedro Alves  <palves@redhat.com>

	* gdb.base/annota1.exp (thread_test): Use srcfile and binfile from
	the global scope.  Set a breakpoint after all threads are started
	rather than stepping over two source lines.  Expect the prompt.
	* gdb.base/watch_thread_num.c (threads_started_barrier): New
	global.
	(NUM): Now 15.
	(main): Use threads_started_barrier to wait for all threads to
	start.  Main thread no longer calls thread_function.  Exit after
	180 seconds.
	(loop): New function.
	(thread_function): Wait on threads_started_barrier barrier.  Call
	'loop' at each iteration.
	* gdb.base/watch_thread_num.exp: Continue to breakpoint after all
	threads have started, instead of hardcoding number of "next"
	steps.  Use an access watchpoint instead of a write watchpoint.