libstdc++-v3/doc/html/manual/test.html

   1 <html><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"><title>Test</title><meta name="generator" content="DocBook XSL-NS Stylesheets V1.76.1"><meta name="keywords" content="
   2       ISO C++
   3     ,
   4       test
   5     ,
   6       testsuite
   7     ,
   8       performance
   9     ,
  10       conformance
  11     ,
  12       ABI
  13     ,
  14       exception safety
  15     "><meta name="keywords" content="
  16       ISO C++
  17     ,
  18       library
  19     "><meta name="keywords" content="
  20       ISO C++
  21     ,
  22       runtime
  23     ,
  24       library
  25     "><link rel="home" href="../index.html" title="The GNU C++ Library"><link rel="up" href="appendix_porting.html" title="Appendix B.  Porting and Maintenance"><link rel="prev" href="internals.html" title="Porting to New Hardware or Operating Systems"><link rel="next" href="abi.html" title="ABI Policy and Guidelines"></head><body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">Test</th></tr><tr><td width="20%" align="left"><a accesskey="p" href="internals.html">Prev</a> </td><th width="60%" align="center">Appendix B.
  26   Porting and Maintenance
  27
  28 </th><td width="20%" align="right"> <a accesskey="n" href="abi.html">Next</a></td></tr></table><hr></div><div class="section" title="Test"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="manual.intro.setup.test"></a>Test</h2></div></div></div><p>
  29 The libstdc++ testsuite includes testing for standard conformance,
  30 regressions, ABI, and performance.
  31 </p><div class="section" title="Organization"><div class="titlepage"><div><div><h3 class="title"><a name="test.organization"></a>Organization</h3></div></div></div><div class="section" title="Directory Layout"><div class="titlepage"><div><div><h4 class="title"><a name="test.organization.layout"></a>Directory Layout</h4></div></div></div><p>
  32   The directory <span class="emphasis"><em>libsrcdir/testsuite</em></span> contains the
  33   individual test cases organized in sub-directories corresponding to
  34   chapters of the C++ standard (detailed below), the dejagnu test
  35   harness support files, and sources to various testsuite utilities
  36   that are packaged in a separate testing library.
  37 </p><p>
  38   All test cases for functionality required by the runtime components
  39   of the C++ standard (ISO 14882) are files within the following
  40   directories.
  41 </p><pre class="programlisting">
  42 17_intro
  43 18_support
  44 19_diagnostics
  45 20_util
  46 21_strings
  47 22_locale
  48 23_containers
  49 25_algorithms
  50 26_numerics
  51 27_io
  52 28_regex
  53 29_atomics
  54 30_threads
  55    </pre><p>
  56       In addition, the following directories include test files:
  57    </p><pre class="programlisting">
  58 tr1               Tests for components as described by the Technical Report on Standard Library Extensions (TR1).
  59 backward          Tests for backwards compatibility and deprecated features.
  60 demangle          Tests for __cxa_demangle, the IA 64 C++ ABI demangler
  61 ext               Tests for extensions.
  62 performance       Tests for performance analysis, and performance regressions.
  63    </pre><p>
  64       Some directories don't have test files, but instead contain
  65       auxiliary information:
  66    </p><pre class="programlisting">
  67 config            Files for the dejagnu test harness.
  68 lib               Files for the dejagnu test harness.
  69 libstdc++*        Files for the dejagnu test harness.
  70 data              Sample text files for testing input and output.
  71 util              Files for libtestc++, utilities and testing routines.
  72    </pre><p>
  73       Within a directory that includes test files, there may be
  74       additional subdirectories, or files.  Originally, test cases
  75       were appended to one file that represented a particular section
  76       of the chapter under test, and was named accordingly. For
  77       instance, to test items related to <code class="code"> 21.3.6.1 -
  78       basic_string::find [lib.string::find]</code> in the standard,
  79       the following was used:
  80    </p><pre class="programlisting">
  81 21_strings/find.cc
  82    </pre><p>
  83       However, that practice soon became a liability as the test cases
  84       became huge and unwieldy, and testing new or extended
  85       functionality (like wide characters or named locales) became
  86       frustrating, leading to aggressive pruning of test cases on some
  87       platforms that covered up implementation errors. Now, the test
  88       suite has a policy of one file, one test case, which solves the
  89       above issues and gives finer grained results and more manageable
  90       error debugging. As an example, the test case quoted above
  91       becomes:
  92    </p><pre class="programlisting">
  93 21_strings/basic_string/find/char/1.cc
  94 21_strings/basic_string/find/char/2.cc
  95 21_strings/basic_string/find/char/3.cc
  96 21_strings/basic_string/find/wchar_t/1.cc
  97 21_strings/basic_string/find/wchar_t/2.cc
  98 21_strings/basic_string/find/wchar_t/3.cc
  99    </pre><p>
 100       All new tests should be written with the policy of one test
 101       case, one file in mind.
 102    </p></div><div class="section" title="Naming Conventions"><div class="titlepage"><div><div><h4 class="title"><a name="test.organization.naming"></a>Naming Conventions</h4></div></div></div><p>
 103       In addition, there are some special names and suffixes that are
 104       used within the testsuite to designate particular kinds of
 105       tests.
 106    </p><div class="itemizedlist"><ul class="itemizedlist" type="disc"><li class="listitem"><p>
 107    <span class="emphasis"><em>_xin.cc</em></span>
 108   </p><p>
 109       This test case expects some kind of interactive input in order
 110       to finish or pass. At the moment, the interactive tests are not
 111       run by default. Instead, they are run by hand, like:
 112    </p><pre class="programlisting">
 113 g++ 27_io/objects/char/3_xin.cc
 114 cat 27_io/objects/char/3_xin.in | a.out
 115      </pre></li><li class="listitem"><p>
 116      <span class="emphasis"><em>.in</em></span>
 117    </p><p>
 118       This file contains the expected input for the corresponding <span class="emphasis"><em>
 119       _xin.cc</em></span> test case.
 120    </p></li><li class="listitem"><p>
 121    <span class="emphasis"><em>_neg.cc</em></span>
 122   </p><p>
 123       This test case is expected to fail: it's a negative test. At the
 124       moment, these are almost always compile time errors.
 125    </p></li><li class="listitem"><p>
 126    <span class="emphasis"><em>char</em></span>
 127   </p><p>
 128       This can either be a directory name or part of a longer file
 129       name, and indicates that this file, or the files within this
 130       directory are testing the <code class="code">char</code> instantiation of a
 131       template.
 132    </p></li><li class="listitem"><p>
 133    <span class="emphasis"><em>wchar_t</em></span>
 134   </p><p>
 135       This can either be a directory name or part of a longer file
 136       name, and indicates that this file, or the files within this
 137       directory are testing the <code class="code">wchar_t</code> instantiation of
 138       a template. Some hosts do not support <code class="code">wchar_t</code>
 139       functionality, so for these targets, all of these tests will not
 140       be run.
 141    </p></li><li class="listitem"><p>
 142    <span class="emphasis"><em>thread</em></span>
 143   </p><p>
 144       This can either be a directory name or part of a longer file
 145       name, and indicates that this file, or the files within this
 146       directory are testing situations where multiple threads are
 147       being used.
 148    </p></li><li class="listitem"><p>
 149    <span class="emphasis"><em>performance</em></span>
 150   </p><p>
 151       This can either be an enclosing directory name or part of a
 152       specific file name. This indicates a test that is used to
 153       analyze runtime performance, for performance regression testing,
 154       or for other optimization related analysis. At the moment, these
 155       test cases are not run by default.
 156    </p></li></ul></div></div></div><div class="section" title="Running the Testsuite"><div class="titlepage"><div><div><h3 class="title"><a name="test.run"></a>Running the Testsuite</h3></div></div></div><div class="section" title="Basic"><div class="titlepage"><div><div><h4 class="title"><a name="test.run.basic"></a>Basic</h4></div></div></div><p>
 157       You can check the status of the build without installing it
 158       using the dejagnu harness, much like the rest of the gcc
 159       tools.</p><pre class="programlisting"> make check</pre><p>in the <span class="emphasis"><em>libbuilddir</em></span> directory.</p><p>or</p><pre class="programlisting"> make check-target-libstdc++-v3</pre><p>in the <span class="emphasis"><em>gccbuilddir</em></span> directory.
 160      </p><p>
 161        These commands are functionally equivalent and will create a
 162        'testsuite' directory underneath
 163        <span class="emphasis"><em>libbuilddir</em></span> containing the results of the
 164        tests. Two results files will be generated: <span class="emphasis"><em>
 165        libstdc++.sum</em></span>, which is a PASS/FAIL summary for each
 166        test, and <span class="emphasis"><em>libstdc++.log</em></span> which is a log of
 167        the exact command line passed to the compiler, the compiler
 168        output, and the executable output (if any).
 169      </p><p>
 170        Archives of test results for various versions and platforms are
 171        available on the GCC website in the <a class="link" href="http://gcc.gnu.org/gcc-4.3/buildstat.html" target="_top">build
 172        status</a> section of each individual release, and are also
 173        archived on a daily basis on the <a class="link" href="http://gcc.gnu.org/ml/gcc-testresults/current" target="_top">gcc-testresults</a>
 174        mailing list. Please check either of these places for a similar
 175        combination of source version, operating system, and host CPU.
 176      </p></div><div class="section" title="Variations"><div class="titlepage"><div><div><h4 class="title"><a name="test.run.variations"></a>Variations</h4></div></div></div><p>
 177       There are several options for running tests, including testing
 178       the regression tests, testing a subset of the regression tests,
 179       testing the performance tests, testing just compilation, testing
 180       installed tools, etc. In addition, there is a special rule for
 181       checking the exported symbols of the shared library.
 182     </p><p>
 183       To debug the dejagnu test harness during runs, try invoking with a
 184       specific argument to the variable RUNTESTFLAGS, as below.
 185     </p><pre class="programlisting">
 186 make check-target-libstdc++-v3 RUNTESTFLAGS="-v"
 187 </pre><p>
 188       or
 189     </p><pre class="programlisting">
 190 make check-target-libstdc++-v3 RUNTESTFLAGS="-v -v"
 191 </pre><p>
 192       To run a subset of the library tests, you will need to generate
 193       the <span class="emphasis"><em>testsuite_files</em></span> file by running
 194       <span class="command"><strong>make testsuite_files</strong></span> in the
 195       <span class="emphasis"><em>libbuilddir/testsuite</em></span> directory, described
 196       below.  Edit the file to remove the tests you don't want and
 197       then run the testsuite as normal.
 198     </p><p>
 199       There are two ways to run on a simulator: set up DEJAGNU to point to a
 200       specially crafted site.exp, or pass down --target_board flags.
 201     </p><p>
 202     Example flags to pass down for various embedded builds are as follows:
 203     </p><pre class="programlisting">
 204       --target=powerpc-eabism (libgloss/sim)
 205 make check-target-libstdc++-v3 RUNTESTFLAGS="--target_board=powerpc-sim"
 206
 207 --target=calmrisc32 (libgloss/sid)
 208 make check-target-libstdc++-v3 RUNTESTFLAGS="--target_board=calmrisc32-sid"
 209
 210 --target=xscale-elf (newlib/sim)
 211 make check-target-libstdc++-v3 RUNTESTFLAGS="--target_board=arm-sim"
 212 </pre><p>
 213       Also, here is an example of how to run the libstdc++ testsuite
 214       for a multilibed build directory with different ABI settings:
 215     </p><pre class="programlisting">
 216 make check-target-libstdc++-v3 RUNTESTFLAGS='--target_board \"unix{-mabi=32,,-mabi=64}\"'
 217 </pre><p>
 218       You can run the tests with a compiler and library that have
 219       already been installed.  Make sure that the compiler (e.g.,
 220       <code class="code">g++</code>) is in your <code class="code">PATH</code>.  If you are
 221       using shared libraries, then you must also ensure that the
 222       directory containing the shared version of libstdc++ is in your
 223       <code class="code">LD_LIBRARY_PATH</code>, or equivalent.  If your GCC source
 224       tree is at <code class="code">/path/to/gcc</code>, then you can run the tests
 225       as follows:
 226     </p><pre class="programlisting">
 227 runtest --tool libstdc++ --srcdir=/path/to/gcc/libstdc++-v3/testsuite
 228 </pre><p>
 229       The testsuite will create a number of files in the directory in
 230       which you run this command,.  Some of those files might use the
 231       same name as files created by other testsuites (like the ones
 232       for GCC and G++), so you should not try to run all the
 233       testsuites in parallel from the same directory.
 234     </p><p>
 235       In addition, there are some testing options that are mostly of
 236       interest to library maintainers and system integrators. As such,
 237       these tests may not work on all cpu and host combinations, and
 238       may need to be executed in the
 239       <span class="emphasis"><em>libbuilddir/testsuite</em></span> directory.  These
 240       options include, but are not necessarily limited to, the
 241       following:
 242    </p><pre class="programlisting">
 243    make testsuite_files
 244    </pre><p>
 245     Five files are generated that determine what test files
 246     are run. These files are:
 247   </p><div class="itemizedlist"><ul class="itemizedlist" type="disc"><li class="listitem"><p>
 248          <span class="emphasis"><em>testsuite_files</em></span>
 249        </p><p>
 250          This is a list of all the test cases that will be run. Each
 251          test case is on a separate line, given with an absolute path
 252          from the <span class="emphasis"><em>libsrcdir/testsuite</em></span> directory.
 253        </p></li><li class="listitem"><p>
 254          <span class="emphasis"><em>testsuite_files_interactive</em></span>
 255        </p><p>
 256          This is a list of all the interactive test cases, using the
 257          same format as the file list above. These tests are not run
 258          by default.
 259      </p></li><li class="listitem"><p>
 260          <span class="emphasis"><em>testsuite_files_performance</em></span>
 261        </p><p>
 262          This is a list of all the performance test cases, using the
 263          same format as the file list above. These tests are not run
 264          by default.
 265      </p></li><li class="listitem"><p>
 266          <span class="emphasis"><em>testsuite_thread</em></span>
 267        </p><p>
 268          This file indicates that the host system can run tests which
 269          involved multiple threads.
 270        </p></li><li class="listitem"><p>
 271          <span class="emphasis"><em>testsuite_wchar_t</em></span>
 272        </p><p>
 273          This file indicates that the host system can run the wchar_t
 274          tests, and corresponds to the macro definition <code class="code">
 275          _GLIBCXX_USE_WCHAR_T</code> in the file c++config.h.
 276        </p></li></ul></div><pre class="programlisting">
 277    make check-abi
 278    </pre><p>
 279      The library ABI can be tested. This involves testing the shared
 280      library against an ABI-defining previous version of symbol
 281      exports.
 282    </p><pre class="programlisting">
 283    make check-compile
 284   </pre><p>
 285      This rule compiles, but does not link or execute, the
 286      <span class="emphasis"><em>testsuite_files</em></span> test cases and displays the
 287      output on stdout.
 288    </p><pre class="programlisting">
 289    make check-performance
 290    </pre><p>
 291      This rule runs through the
 292      <span class="emphasis"><em>testsuite_files_performance</em></span> test cases and
 293      collects information for performance analysis and can be used to
 294      spot performance regressions. Various timing information is
 295      collected, as well as number of hard page faults, and memory
 296      used. This is not run by default, and the implementation is in
 297      flux.
 298    </p><p>
 299       We are interested in any strange failures of the testsuite;
 300       please email the main libstdc++ mailing list if you see
 301       something odd or have questions.
 302    </p></div><div class="section" title="Permutations"><div class="titlepage"><div><div><h4 class="title"><a name="test.run.permutations"></a>Permutations</h4></div></div></div><p>
 303       To run the libstdc++ test suite under the <a class="link" href="debug_mode.html" title="Chapter 17. Debug Mode">debug mode</a>, edit
 304       <code class="filename">libstdc++-v3/scripts/testsuite_flags</code> to add the
 305       compile-time flag <code class="constant">-D_GLIBCXX_DEBUG</code> to the
 306       result printed by the <code class="literal">--build-cxx</code>
 307       option. Additionally, add the
 308       <code class="constant">-D_GLIBCXX_DEBUG_PEDANTIC</code> flag to turn on
 309       pedantic checking. The libstdc++ test suite should produce
 310       precisely the same results under debug mode that it does under
 311       release mode: any deviation indicates an error in either the
 312       library or the test suite.
 313     </p><p>
 314       The <a class="link" href="parallel_mode.html" title="Chapter 18. Parallel Mode">parallel
 315       mode</a> can be tested in much the same manner, substituting
 316       <code class="constant">-D_GLIBCXX_PARALLEL</code> for
 317       <code class="constant">-D_GLIBCXX_DEBUG</code> in the previous paragraph.
 318     </p><p>
 319       Or, just run the testsuites with <code class="constant">CXXFLAGS</code>
 320       set to <code class="constant">-D_GLIBCXX_DEBUG</code> or
 321       <code class="constant">-D_GLIBCXX_PARALLEL</code>.
 322     </p></div></div><div class="section" title="Writing a new test case"><div class="titlepage"><div><div><h3 class="title"><a name="test.new_tests"></a>Writing a new test case</h3></div></div></div><p>
 323     The first step in making a new test case is to choose the correct
 324     directory and file name, given the organization as previously
 325     described.
 326    </p><p>
 327     All files are copyright the FSF, and GPL'd: this is very
 328     important.  The first copyright year should correspond to the date
 329     the file was checked in to SVN.
 330    </p><p>
 331      As per the dejagnu instructions, always return 0 from main to
 332      indicate success.
 333    </p><p>
 334    A bunch of utility functions and classes have already been
 335    abstracted out into the testsuite utility library, <code class="code">
 336    libtestc++</code>. To use this functionality, just include the
 337    appropriate header file: the library or specific object files will
 338    automatically be linked in as part of the testsuite run.
 339    </p><p>
 340    For a test that needs to take advantage of the dejagnu test
 341    harness, what follows below is a list of special keyword that
 342    harness uses. Basically, a test case contains dg-keywords (see
 343    dg.exp) indicating what to do and what kinds of behavior are to be
 344    expected.  New test cases should be written with the new style
 345    DejaGnu framework in mind.
 346    </p><p>
 347     To ease transition, here is the list of dg-keyword documentation
 348     lifted from dg.exp.
 349    </p><pre class="programlisting">
 350 # The currently supported options are:
 351 #
 352 # dg-prms-id N
 353 #       set prms_id to N
 354 #
 355 # dg-options "options ..." [{ target selector }]
 356 #       specify special options to pass to the tool (eg: compiler)
 357 #
 358 # dg-do do-what-keyword [{ target/xfail selector }]
 359 #       `do-what-keyword' is tool specific and is passed unchanged to
 360 #       ${tool}-dg-test.  An example is gcc where `keyword' can be any of:
 361 #       preprocess|compile|assemble|link|run
 362 #       and will do one of: produce a .i, produce a .s, produce a .o,
 363 #       produce an a.out, or produce an a.out and run it (the default is
 364 #       compile).
 365 #
 366 # dg-error regexp comment [{ target/xfail selector } [{.|0|linenum}]]
 367 #       indicate an error message &lt;regexp&gt; is expected on this line
 368 #       (the test fails if it doesn't occur)
 369 #       Linenum=0 for general tool messages (eg: -V arg missing).
 370 #       "." means the current line.
 371 #
 372 # dg-warning regexp comment [{ target/xfail selector } [{.|0|linenum}]]
 373 #       indicate a warning message &lt;regexp&gt; is expected on this line
 374 #       (the test fails if it doesn't occur)
 375 #
 376 # dg-bogus regexp comment [{ target/xfail selector } [{.|0|linenum}]]
 377 #       indicate a bogus error message &lt;regexp&gt; use to occur here
 378 #       (the test fails if it does occur)
 379 #
 380 # dg-build regexp comment [{ target/xfail selector }]
 381 #       indicate the build use to fail for some reason
 382 #       (errors covered here include bad assembler generated, tool crashes,
 383 #       and link failures)
 384 #       (the test fails if it does occur)
 385 #
 386 # dg-excess-errors comment [{ target/xfail selector }]
 387 #       indicate excess errors are expected (any line)
 388 #       (this should only be used sparingly and temporarily)
 389 #
 390 # dg-output regexp [{ target selector }]
 391 #       indicate the expected output of the program is &lt;regexp&gt;
 392 #       (there may be multiple occurrences of this, they are concatenated)
 393 #
 394 # dg-final { tcl code }
 395 #       add some tcl code to be run at the end
 396 #       (there may be multiple occurrences of this, they are concatenated)
 397 #       (unbalanced braces must be \-escaped)
 398 #
 399 # "{ target selector }" is a list of expressions that determine whether the
 400 # test succeeds or fails for a particular target, or in some cases whether the
 401 # option applies for a particular target.  If the case of `dg-do' it specifies
 402 # whether the test case is even attempted on the specified target.
 403 #
 404 # The target selector is always optional.  The format is one of:
 405 #
 406 # { xfail *-*-* ... } - the test is expected to fail for the given targets
 407 # { target *-*-* ... } - the option only applies to the given targets
 408 #
 409 # At least one target must be specified, use *-*-* for "all targets".
 410 # At present it is not possible to specify both `xfail' and `target'.
 411 # "native" may be used in place of "*-*-*".
 412
 413 Example 1: Testing compilation only
 414 // { dg-do compile }
 415
 416 Example 2: Testing for expected warnings on line 36, which all targets fail
 417 // { dg-warning "string literals" "" { xfail *-*-* } 36 }
 418
 419 Example 3: Testing for expected warnings on line 36
 420 // { dg-warning "string literals" "" { target *-*-* } 36 }
 421
 422 Example 4: Testing for compilation errors on line 41
 423 // { dg-do compile }
 424 // { dg-error "no match for" "" { target *-*-* } 41 }
 425
 426 Example 5: Testing with special command line settings, or without the
 427 use of pre-compiled headers, in particular the stdc++.h.gch file. Any
 428 options here will override the DEFAULT_CXXFLAGS and PCH_CXXFLAGS set
 429 up in the normal.exp file.
 430 // { dg-options "-O0" { target *-*-* } }
 431 </pre><p>
 432     More examples can be found in the libstdc++-v3/testsuite/*/*.cc files.
 433    </p></div><div class="section" title="Test Harness and Utilities"><div class="titlepage"><div><div><h3 class="title"><a name="test.harness"></a>Test Harness and Utilities</h3></div></div></div><div class="section" title="Dejagnu Harness Details"><div class="titlepage"><div><div><h4 class="title"><a name="test.harness.dejagnu"></a>Dejagnu Harness Details</h4></div></div></div><p>
 434     Underlying details of testing for conformance and regressions are
 435     abstracted via the GNU Dejagnu package. This is similar to the
 436     rest of GCC.
 437   </p><p>This is information for those looking at making changes to the testsuite
 438 structure, and/or needing to trace dejagnu's actions with --verbose.  This
 439 will not be useful to people who are "merely" adding new tests to the existing
 440 structure.
 441 </p><p>The first key point when working with dejagnu is the idea of a "tool".
 442 Files, directories, and functions are all implicitly used when they are
 443 named after the tool in use.  Here, the tool will always be "libstdc++".
 444 </p><p>The <code class="code">lib</code> subdir contains support routines.  The
 445 <code class="code">lib/libstdc++.exp</code> file ("support library") is loaded
 446 automagically, and must explicitly load the others.  For example, files can
 447 be copied from the core compiler's support directory into <code class="code">lib</code>.
 448 </p><p>Some routines in <code class="code">lib/libstdc++.exp</code> are callbacks, some are
 449 our own.  Callbacks must be prefixed with the name of the tool.  To easily
 450 distinguish the others, by convention our own routines are named "v3-*".
 451 </p><p>The next key point when working with dejagnu is "test files".  Any
 452 directory whose name starts with the tool name will be searched for test files.
 453 (We have only one.)  In those directories, any <code class="code">.exp</code> file is
 454 considered a test file, and will be run in turn.  Our main test file is called
 455 <code class="code">normal.exp</code>; it runs all the tests in testsuite_files using the
 456 callbacks loaded from the support library.
 457 </p><p>The <code class="code">config</code> directory is searched for any particular "target
 458 board" information unique to this library.  This is currently unused and sets
 459 only default variables.
 460 </p></div><div class="section" title="Utilities"><div class="titlepage"><div><div><h4 class="title"><a name="test.harness.utils"></a>Utilities</h4></div></div></div><p>
 461   </p><p>
 462    The testsuite directory also contains some files that implement
 463    functionality that is intended to make writing test cases easier,
 464    or to avoid duplication, or to provide error checking in a way that
 465    is consistent across platforms and test harnesses. A stand-alone
 466    executable, called <span class="emphasis"><em>abi_check</em></span>, and a static
 467    library called <span class="emphasis"><em>libtestc++</em></span> are
 468    constructed. Both of these items are not installed, and only used
 469    during testing.
 470   </p><p>
 471   These files include the following functionality:
 472   </p><div class="itemizedlist"><ul class="itemizedlist" type="disc"><li class="listitem"><p>
 473        <span class="emphasis"><em>testsuite_abi.h</em></span>,
 474        <span class="emphasis"><em>testsuite_abi.cc</em></span>,
 475        <span class="emphasis"><em>testsuite_abi_check.cc</em></span>
 476        </p><p>
 477         Creates the executable <span class="emphasis"><em>abi_check</em></span>.
 478         Used to check correctness of symbol versioning, visibility of
 479         exported symbols, and compatibility on symbols in the shared
 480         library, for hosts that support this feature. More information
 481         can be found in the ABI documentation <a class="link" href="abi.html" title="ABI Policy and Guidelines">here</a>
 482        </p></li><li class="listitem"><p>
 483        <span class="emphasis"><em>testsuite_allocator.h</em></span>,
 484        <span class="emphasis"><em>testsuite_allocator.cc</em></span>
 485        </p><p>
 486         Contains specialized allocators that keep track of construction
 487         and destruction. Also, support for overriding global new and
 488         delete operators, including verification that new and delete
 489         are called during execution, and that allocation over max_size
 490         fails.
 491        </p></li><li class="listitem"><p>
 492        <span class="emphasis"><em>testsuite_character.h</em></span>
 493        </p><p>
 494         Contains <code class="code">std::char_traits</code> and
 495         <code class="code">std::codecvt</code> specializations for a user-defined
 496         POD.
 497        </p></li><li class="listitem"><p>
 498        <span class="emphasis"><em>testsuite_hooks.h</em></span>,
 499        <span class="emphasis"><em>testsuite_hooks.cc</em></span>
 500        </p><p>
 501        A large number of utilities, including:
 502        </p><div class="itemizedlist"><ul class="itemizedlist" type="circle"><li class="listitem"><p>VERIFY</p></li><li class="listitem"><p>set_memory_limits</p></li><li class="listitem"><p>verify_demangle</p></li><li class="listitem"><p>run_tests_wrapped_locale</p></li><li class="listitem"><p>run_tests_wrapped_env</p></li><li class="listitem"><p>try_named_locale</p></li><li class="listitem"><p>try_mkfifo</p></li><li class="listitem"><p>func_callback</p></li><li class="listitem"><p>counter</p></li><li class="listitem"><p>copy_tracker</p></li><li class="listitem"><p>copy_constructor</p></li><li class="listitem"><p>assignment_operator</p></li><li class="listitem"><p>destructor</p></li><li class="listitem"><p>pod_char, pod_int and associated char_traits specializations</p></li></ul></div></li><li class="listitem"><p>
 503          <span class="emphasis"><em>testsuite_io.h</em></span>
 504        </p><p>
 505        Error, exception, and constraint checking for
 506        <code class="code">std::streambuf, std::basic_stringbuf, std::basic_filebuf</code>.
 507        </p></li><li class="listitem"><p>
 508          <span class="emphasis"><em>testsuite_iterators.h</em></span>
 509        </p><p>
 510          Wrappers for various iterators.
 511        </p></li><li class="listitem"><p>
 512          <span class="emphasis"><em>testsuite_performance.h</em></span>
 513        </p><p>
 514        A number of class abstractions for performance counters, and
 515        reporting functions including:
 516        </p><div class="itemizedlist"><ul class="itemizedlist" type="circle"><li class="listitem"><p>time_counter</p></li><li class="listitem"><p>resource_counter</p></li><li class="listitem"><p>report_performance</p></li></ul></div></li></ul></div></div></div><div class="section" title="Special Topics"><div class="titlepage"><div><div><h3 class="title"><a name="test.special"></a>Special Topics</h3></div></div></div><div class="section" title="Qualifying Exception Safety Guarantees"><div class="titlepage"><div><div><h4 class="title"><a name="test.exception.safety"></a>
 517   Qualifying Exception Safety Guarantees
 518   <a class="indexterm" name="id681679"></a>
 519 </h4></div></div></div><div class="section" title="Overview"><div class="titlepage"><div><div><h5 class="title"><a name="test.exception.safety.overview"></a>Overview</h5></div></div></div><p>
 520          Testing is composed of running a particular test sequence,
 521          and looking at what happens to the surrounding code when
 522          exceptions are thrown. Each test is composed of measuring
 523          initial state, executing a particular sequence of code under
 524          some instrumented conditions, measuring a final state, and
 525          then examining the differences between the two states.
 526        </p><p>
 527          Test sequences are composed of constructed code sequences
 528          that exercise a particular function or member function, and
 529          either confirm no exceptions were generated, or confirm the
 530          consistency/coherency of the test subject in the event of a
 531          thrown exception.
 532        </p><p>
 533          Random code paths can be constructed using the basic test
 534          sequences and instrumentation as above, only combined in a
 535          random or pseudo-random way.
 536        </p><p> To compute the code paths that throw, test instruments
 537          are used that throw on allocation events
 538          (<code class="classname">__gnu_cxx::throw_allocator_random</code>
 539          and <code class="classname">__gnu_cxx::throw_allocator_limit</code>)
 540          and copy, assignment, comparison, increment, swap, and
 541          various operators
 542          (<code class="classname">__gnu_cxx::throw_type_random</code>
 543          and <code class="classname">__gnu_cxx::throw_type_limit</code>). Looping
 544          through a given test sequence and conditionally throwing in
 545          all instrumented places.  Then, when the test sequence
 546          completes without an exception being thrown, assume all
 547          potential error paths have been exercised in a sequential
 548          manner.
 549        </p></div><div class="section" title="Existing tests"><div class="titlepage"><div><div><h5 class="title"><a name="test.exception.safety.status"></a>
 550     Existing tests
 551 </h5></div></div></div><div class="itemizedlist"><ul class="itemizedlist" type="disc"><li class="listitem"><p>
 552          Ad Hoc
 553        </p><p>
 554          For example,
 555          <code class="filename">testsuite/23_containers/list/modifiers/3.cc</code>.
 556        </p></li><li class="listitem"><p>
 557          Policy Based Data Structures
 558        </p><p>
 559          For example, take the test
 560          functor <code class="classname">rand_reg_test</code> in
 561          in <code class="filename">testsuite/ext/pb_ds/regression/tree_no_data_map_rand.cc</code>. This uses <code class="classname">container_rand_regression_test</code> in
 562 <code class="filename">testsuite/util/regression/rand/assoc/container_rand_regression_test.h</code>.
 563
 564        </p><p>
 565          Which has several tests for container member functions,
 566 Includes control and test container objects. Configuration includes
 567 random seed, iterations, number of distinct values, and the
 568 probability that an exception will be thrown. Assumes instantiating
 569 container uses an extension
 570 allocator, <code class="classname">__gnu_cxx::throw_allocator_random</code>,
 571 as the allocator type.
 572        </p></li><li class="listitem"><p>
 573          C++11 Container Requirements.
 574        </p><p>
 575          Coverage is currently limited to testing container
 576          requirements for exception safety,
 577          although <code class="classname">__gnu_cxx::throw_type</code> meets
 578          the additional type requirements for testing numeric data
 579          structures and instantiating algorithms.
 580        </p><p>
 581          Of particular interest is extending testing to algorithms and
 582          then to parallel algorithms. Also io and locales.
 583        </p><p>
 584          The test instrumentation should also be extended to add
 585          instrumentation to <code class="classname">iterator</code>
 586          and <code class="classname">const_iterator</code> types that throw
 587          conditionally on iterator operations.
 588        </p></li></ul></div></div><div class="section" title="C++11 Requirements Test Sequence Descriptions"><div class="titlepage"><div><div><h5 class="title"><a name="test.exception.safety.containers"></a>
 589 C++11 Requirements Test Sequence Descriptions
 590 </h5></div></div></div><div class="itemizedlist"><ul class="itemizedlist" type="disc"><li class="listitem"><p>
 591          Basic
 592        </p><p>
 593          Basic consistency on exception propagation tests. For
 594          each container, an object of that container is constructed,
 595          a specific member function is exercised in
 596          a <code class="literal">try</code> block, and then any thrown
 597          exceptions lead to error checking in the appropriate
 598          <code class="literal">catch</code> block. The container's use of
 599          resources is compared to the container's use prior to the
 600          test block. Resource monitoring is limited to allocations
 601          made through the container's <span class="type">allocator_type</span>,
 602          which should be sufficient for container data
 603          structures. Included in these tests are member functions
 604          are <span class="type">iterator</span> and <span class="type">const_iterator</span>
 605          operations, <code class="function">pop_front</code>, <code class="function">pop_back</code>, <code class="function">push_front</code>, <code class="function">push_back</code>, <code class="function">insert</code>, <code class="function">erase</code>, <code class="function">swap</code>, <code class="function">clear</code>,
 606          and <code class="function">rehash</code>. The container in question is
 607          instantiated with two instrumented template arguments,
 608          with <code class="classname">__gnu_cxx::throw_allocator_limit</code>
 609          as the allocator type, and
 610          with <code class="classname">__gnu_cxx::throw_type_limit</code> as
 611          the value type. This allows the test to loop through
 612          conditional throw points.
 613        </p><p>
 614          The general form is demonstrated in
 615          <code class="filename">testsuite/23_containers/list/requirements/exception/basic.cc
 616          </code>. The instantiating test object is <code class="classname">__gnu_test::basic_safety</code> and is detailed in <code class="filename">testsuite/util/exception/safety.h</code>.
 617        </p></li><li class="listitem"><p>
 618          Generation Prohibited
 619        </p><p>
 620          Exception generation tests. For each container, an object of
 621          that container is constructed and all member functions
 622          required to not throw exceptions are exercised. Included in
 623          these tests are member functions
 624          are <span class="type">iterator</span> and <span class="type">const_iterator</span> operations, <code class="function">erase</code>, <code class="function">pop_front</code>, <code class="function">pop_back</code>, <code class="function">swap</code>,
 625          and <code class="function">clear</code>. The container in question is
 626          instantiated with two instrumented template arguments,
 627          with <code class="classname">__gnu_cxx::throw_allocator_random</code>
 628          as the allocator type, and
 629          with <code class="classname">__gnu_cxx::throw_type_random</code> as
 630          the value type. This test does not loop, an instead is sudden
 631          death: first error fails.
 632        </p><p>
 633          The general form is demonstrated in
 634          <code class="filename">testsuite/23_containers/list/requirements/exception/generation_prohibited.cc
 635          </code>. The instantiating test object is <code class="classname">__gnu_test::generation_prohibited</code> and is detailed in <code class="filename">testsuite/util/exception/safety.h</code>.
 636        </p></li><li class="listitem"><p>
 637          Propagation Consistent
 638        </p><p>
 639          Container rollback on exception propagation tests. For
 640          each container, an object of that container is constructed,
 641          a specific member function that requires rollback to a previous
 642          known good state is exercised in
 643          a <code class="literal">try</code> block, and then any thrown
 644          exceptions lead to error checking in the appropriate
 645          <code class="literal">catch</code> block. The container is compared to
 646          the container's last known good state using such parameters
 647          as size, contents, and iterator references. Included in these
 648          tests are member functions
 649          are <code class="function">push_front</code>, <code class="function">push_back</code>, <code class="function">insert</code>,
 650          and <code class="function">rehash</code>. The container in question is
 651          instantiated with two instrumented template arguments,
 652          with <code class="classname">__gnu_cxx::throw_allocator_limit</code>
 653          as the allocator type, and
 654          with <code class="classname">__gnu_cxx::throw_type_limit</code> as
 655          the value type. This allows the test to loop through
 656          conditional throw points.
 657        </p><p>
 658          The general form demonstrated in
 659          <code class="filename">testsuite/23_containers/list/requirements/exception/propagation_coherent.cc
 660          </code>. The instantiating test object is <code class="classname">__gnu_test::propagation_coherent</code> and is detailed in <code class="filename">testsuite/util/exception/safety.h</code>.
 661        </p></li></ul></div></div></div></div></div><div class="navfooter"><hr><table width="100%" summary="Navigation footer"><tr><td width="40%" align="left"><a accesskey="p" href="internals.html">Prev</a> </td><td width="20%" align="center"><a accesskey="u" href="appendix_porting.html">Up</a></td><td width="40%" align="right"> <a accesskey="n" href="abi.html">Next</a></td></tr><tr><td width="40%" align="left" valign="top">Porting to New Hardware or Operating Systems </td><td width="20%" align="center"><a accesskey="h" href="../index.html">Home</a></td><td width="40%" align="right" valign="top"> ABI Policy and Guidelines</td></tr></table></div></body></html>