libstdc++-v3/doc/html/manual/facets.html

   1 <html><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"><title>Facets</title><meta name="generator" content="DocBook XSL-NS Stylesheets V1.76.1"><meta name="keywords" content="
   2       ISO C++
   3     ,
   4       library
   5     "><meta name="keywords" content="
   6       ISO C++
   7     ,
   8       runtime
   9     ,
  10       library
  11     "><link rel="home" href="../index.html" title="The GNU C++ Library"><link rel="up" href="localization.html" title="Chapter 8.  Localization"><link rel="prev" href="localization.html" title="Chapter 8.  Localization"><link rel="next" href="containers.html" title="Chapter 9.  Containers"></head><body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">Facets</th></tr><tr><td width="20%" align="left"><a accesskey="p" href="localization.html">Prev</a> </td><th width="60%" align="center">Chapter 8.
  12   Localization
  13
  14 </th><td width="20%" align="right"> <a accesskey="n" href="containers.html">Next</a></td></tr></table><hr></div><div class="section" title="Facets"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="std.localization.facet"></a>Facets</h2></div></div></div><div class="section" title="ctype"><div class="titlepage"><div><div><h3 class="title"><a name="std.localization.facet.ctype"></a>ctype</h3></div></div></div><div class="section" title="Implementation"><div class="titlepage"><div><div><h4 class="title"><a name="facet.ctype.impl"></a>Implementation</h4></div></div></div><div class="section" title="Specializations"><div class="titlepage"><div><div><h5 class="title"><a name="id625164"></a>Specializations</h5></div></div></div><p>
  15 For the required specialization codecvt&lt;wchar_t, char, mbstate_t&gt; ,
  16 conversions are made between the internal character set (always UCS4
  17 on GNU/Linux) and whatever the currently selected locale for the
  18 LC_CTYPE category implements.
  19 </p><p>
  20 The two required specializations are implemented as follows:
  21 </p><p>
  22 <code class="code">
  23 ctype&lt;char&gt;
  24 </code>
  25 </p><p>
  26 This is simple specialization. Implementing this was a piece of cake.
  27 </p><p>
  28 <code class="code">
  29 ctype&lt;wchar_t&gt;
  30 </code>
  31 </p><p>
  32 This specialization, by specifying all the template parameters, pretty
  33 much ties the hands of implementors. As such, the implementation is
  34 straightforward, involving mcsrtombs for the conversions between char
  35 to wchar_t and wcsrtombs for conversions between wchar_t and char.
  36 </p><p>
  37 Neither of these two required specializations deals with Unicode
  38 characters.
  39 </p></div></div><div class="section" title="Future"><div class="titlepage"><div><div><h4 class="title"><a name="facet.ctype.future"></a>Future</h4></div></div></div><div class="itemizedlist"><ul class="itemizedlist" type="disc"><li class="listitem"><p>
  40    How to deal with the global locale issue?
  41    </p></li><li class="listitem"><p>
  42    How to deal with different types than char, wchar_t? </p></li><li class="listitem"><p>
  43    Overlap between codecvt/ctype: narrow/widen
  44    </p></li><li class="listitem"><p>
  45        Mask typedef in codecvt_base, argument types in codecvt.  what
  46        is know about this type?
  47    </p></li><li class="listitem"><p>
  48    Why mask* argument in codecvt?
  49    </p></li><li class="listitem"><p>
  50        Can this be made (more) generic? is there a simple way to
  51        straighten out the configure-time mess that is a by-product of
  52        this class?
  53    </p></li><li class="listitem"><p>
  54        Get the ctype&lt;wchar_t&gt;::mask stuff under control. Need to
  55        make some kind of static table, and not do lookup every time
  56        somebody hits the do_is... functions. Too bad we can't just
  57        redefine mask for ctype&lt;wchar_t&gt;
  58    </p></li><li class="listitem"><p>
  59        Rename abstract base class. See if just smash-overriding is a
  60        better approach. Clarify, add sanity to naming.
  61      </p></li></ul></div></div><div class="bibliography" title="Bibliography"><div class="titlepage"><div><div><h4 class="title"><a name="facet.ctype.biblio"></a>Bibliography</h4></div></div></div><div class="biblioentry"><a name="id625288"></a><p><span class="citetitle"><em class="citetitle">
  62       The GNU C Library
  63     </em>. </span><span class="author"><span class="firstname">Roland</span> <span class="surname">McGrath</span>. </span><span class="author"><span class="firstname">Ulrich</span> <span class="surname">Drepper</span>. </span><span class="copyright">Copyright © 2007 FSF. </span><span class="pagenums">Chapters 6  Character Set Handling and 7 Locales and Internationalization. </span></p></div><div class="biblioentry"><a name="id625328"></a><p><span class="citetitle"><em class="citetitle">
  64       Correspondence
  65     </em>. </span><span class="author"><span class="firstname">Ulrich</span> <span class="surname">Drepper</span>. </span><span class="copyright">Copyright © 2002 . </span></p></div><div class="biblioentry"><a name="id625354"></a><p><span class="citetitle"><em class="citetitle">
  66       ISO/IEC 14882:1998 Programming languages - C++
  67     </em>. </span><span class="copyright">Copyright © 1998 ISO. </span></p></div><div class="biblioentry"><a name="id625373"></a><p><span class="citetitle"><em class="citetitle">
  68       ISO/IEC 9899:1999 Programming languages - C
  69     </em>. </span><span class="copyright">Copyright © 1999 ISO. </span></p></div><div class="biblioentry" title="The Open Group Base Specifications, Issue 6 (IEEE Std. 1003.1-2004)"><a name="id625392"></a><p><span class="title"><i>
  70         <a class="link" href="http://www.unix.org/version3/ieee_std.html" target="_top">
  71         The Open Group Base Specifications, Issue 6 (IEEE Std. 1003.1-2004)
  72         </a>
  73       </i>. </span><span class="copyright">Copyright © 1999
  74       The Open Group/The Institute of Electrical and Electronics Engineers, Inc.. </span></p></div><div class="biblioentry"><a name="id625419"></a><p><span class="citetitle"><em class="citetitle">
  75       The C++ Programming Language, Special Edition
  76     </em>. </span><span class="author"><span class="firstname">Bjarne</span> <span class="surname">Stroustrup</span>. </span><span class="copyright">Copyright © 2000 Addison Wesley, Inc.. </span><span class="pagenums">Appendix D. </span><span class="publisher"><span class="publishername">
  77         Addison Wesley
  78       . </span></span></p></div><div class="biblioentry"><a name="id625457"></a><p><span class="citetitle"><em class="citetitle">
  79       Standard C++ IOStreams and Locales
  80     </em>. </span><span class="subtitle">
  81       Advanced Programmer's Guide and Reference
  82     . </span><span class="author"><span class="firstname">Angelika</span> <span class="surname">Langer</span>. </span><span class="author"><span class="firstname">Klaus</span> <span class="surname">Kreft</span>. </span><span class="copyright">Copyright © 2000 Addison Wesley Longman, Inc.. </span><span class="publisher"><span class="publishername">
  83         Addison Wesley Longman
  84       . </span></span></p></div></div></div><div class="section" title="codecvt"><div class="titlepage"><div><div><h3 class="title"><a name="std.localization.facet.codecvt"></a>codecvt</h3></div></div></div><p>
  85 The standard class codecvt attempts to address conversions between
  86 different character encoding schemes. In particular, the standard
  87 attempts to detail conversions between the implementation-defined wide
  88 characters (hereafter referred to as wchar_t) and the standard type
  89 char that is so beloved in classic <span class="quote">“<span class="quote">C</span>”</span> (which can now be
  90 referred to as narrow characters.)  This document attempts to describe
  91 how the GNU libstdc++ implementation deals with the conversion between
  92 wide and narrow characters, and also presents a framework for dealing
  93 with the huge number of other encodings that iconv can convert,
  94 including Unicode and UTF8. Design issues and requirements are
  95 addressed, and examples of correct usage for both the required
  96 specializations for wide and narrow characters and the
  97 implementation-provided extended functionality are given.
  98 </p><div class="section" title="Requirements"><div class="titlepage"><div><div><h4 class="title"><a name="facet.codecvt.req"></a>Requirements</h4></div></div></div><p>
  99 Around page 425 of the C++ Standard, this charming heading comes into view:
 100 </p><div class="blockquote"><blockquote class="blockquote"><p>
 101 22.2.1.5 - Template class codecvt
 102 </p></blockquote></div><p>
 103 The text around the codecvt definition gives some clues:
 104 </p><div class="blockquote"><blockquote class="blockquote"><p>
 105 <span class="emphasis"><em>
 106 -1- The class codecvt&lt;internT,externT,stateT&gt; is for use when
 107 converting from one codeset to another, such as from wide characters
 108 to multibyte characters, between wide character encodings such as
 109 Unicode and EUC.
 110 </em></span>
 111 </p></blockquote></div><p>
 112 Hmm. So, in some unspecified way, Unicode encodings and
 113 translations between other character sets should be handled by this
 114 class.
 115 </p><div class="blockquote"><blockquote class="blockquote"><p>
 116 <span class="emphasis"><em>
 117 -2- The stateT argument selects the pair of codesets being mapped between.
 118 </em></span>
 119 </p></blockquote></div><p>
 120 Ah ha! Another clue...
 121 </p><div class="blockquote"><blockquote class="blockquote"><p>
 122 <span class="emphasis"><em>
 123 -3- The instantiations required in the Table ??
 124 (lib.locale.category), namely codecvt&lt;wchar_t,char,mbstate_t&gt; and
 125 codecvt&lt;char,char,mbstate_t&gt;, convert the implementation-defined
 126 native character set. codecvt&lt;char,char,mbstate_t&gt; implements a
 127 degenerate conversion; it does not convert at
 128 all. codecvt&lt;wchar_t,char,mbstate_t&gt; converts between the native
 129 character sets for tiny and wide characters. Instantiations on
 130 mbstate_t perform conversion between encodings known to the library
 131 implementor.  Other encodings can be converted by specializing on a
 132 user-defined stateT type. The stateT object can contain any state that
 133 is useful to communicate to or from the specialized do_convert member.
 134 </em></span>
 135 </p></blockquote></div><p>
 136 At this point, a couple points become clear:
 137 </p><p>
 138 One: The standard clearly implies that attempts to add non-required
 139 (yet useful and widely used) conversions need to do so through the
 140 third template parameter, stateT.</p><p>
 141 Two: The required conversions, by specifying mbstate_t as the third
 142 template parameter, imply an implementation strategy that is mostly
 143 (or wholly) based on the underlying C library, and the functions
 144 mcsrtombs and wcsrtombs in particular.</p></div><div class="section" title="Design"><div class="titlepage"><div><div><h4 class="title"><a name="facet.codecvt.design"></a>Design</h4></div></div></div><div class="section" title="wchar_t Size"><div class="titlepage"><div><div><h5 class="title"><a name="codecvt.design.wchar_t_size"></a><span class="type">wchar_t</span> Size</h5></div></div></div><p>
 145       The simple implementation detail of wchar_t's size seems to
 146       repeatedly confound people. Many systems use a two byte,
 147       unsigned integral type to represent wide characters, and use an
 148       internal encoding of Unicode or UCS2. (See AIX, Microsoft NT,
 149       Java, others.) Other systems, use a four byte, unsigned integral
 150       type to represent wide characters, and use an internal encoding
 151       of UCS4. (GNU/Linux systems using glibc, in particular.) The C
 152       programming language (and thus C++) does not specify a specific
 153       size for the type wchar_t.
 154     </p><p>
 155       Thus, portable C++ code cannot assume a byte size (or endianness) either.
 156     </p></div><div class="section" title="Support for Unicode"><div class="titlepage"><div><div><h5 class="title"><a name="codecvt.design.unicode"></a>Support for Unicode</h5></div></div></div><p>
 157     Probably the most frequently asked question about code conversion
 158     is: "So dudes, what's the deal with Unicode strings?"
 159     The dude part is optional, but apparently the usefulness of
 160     Unicode strings is pretty widely appreciated. Sadly, this specific
 161     encoding (And other useful encodings like UTF8, UCS4, ISO 8859-10,
 162     etc etc etc) are not mentioned in the C++ standard.
 163   </p><p>
 164     A couple of comments:
 165   </p><p>
 166     The thought that all one needs to convert between two arbitrary
 167     codesets is two types and some kind of state argument is
 168     unfortunate. In particular, encodings may be stateless. The naming
 169     of the third parameter as stateT is unfortunate, as what is really
 170     needed is some kind of generalized type that accounts for the
 171     issues that abstract encodings will need. The minimum information
 172     that is required includes:
 173   </p><div class="itemizedlist"><ul class="itemizedlist" type="disc"><li class="listitem"><p>
 174         Identifiers for each of the codesets involved in the
 175         conversion. For example, using the iconv family of functions
 176         from the Single Unix Specification (what used to be called
 177         X/Open) hosted on the GNU/Linux operating system allows
 178         bi-directional mapping between far more than the following
 179         tantalizing possibilities:
 180       </p><p>
 181         (An edited list taken from <code class="code">`iconv --list`</code> on a
 182         Red Hat 6.2/Intel system:
 183       </p><div class="blockquote"><blockquote class="blockquote"><pre class="programlisting">
 184 8859_1, 8859_9, 10646-1:1993, 10646-1:1993/UCS4, ARABIC, ARABIC7,
 185 ASCII, EUC-CN, EUC-JP, EUC-KR, EUC-TW, GREEK-CCIcode, GREEK, GREEK7-OLD,
 186 GREEK7, GREEK8, HEBREW, ISO-8859-1, ISO-8859-2, ISO-8859-3,
 187 ISO-8859-4, ISO-8859-5, ISO-8859-6, ISO-8859-7, ISO-8859-8,
 188 ISO-8859-9, ISO-8859-10, ISO-8859-11, ISO-8859-13, ISO-8859-14,
 189 ISO-8859-15, ISO-10646, ISO-10646/UCS2, ISO-10646/UCS4,
 190 ISO-10646/UTF-8, ISO-10646/UTF8, SHIFT-JIS, SHIFT_JIS, UCS-2, UCS-4,
 191 UCS2, UCS4, UNICODE, UNICODEBIG, UNICODELIcodeLE, US-ASCII, US, UTF-8,
 192 UTF-16, UTF8, UTF16).
 193 </pre></blockquote></div><p>
 194 For iconv-based implementations, string literals for each of the
 195 encodings (i.e. "UCS-2" and "UTF-8") are necessary,
 196 although for other,
 197 non-iconv implementations a table of enumerated values or some other
 198 mechanism may be required.
 199 </p></li><li class="listitem"><p>
 200  Maximum length of the identifying string literal.
 201 </p></li><li class="listitem"><p>
 202  Some encodings require explicit endian-ness. As such, some kind
 203   of endian marker or other byte-order marker will be necessary. See
 204   "Footnotes for C/C++ developers" in Haible for more information on
 205   UCS-2/Unicode endian issues. (Summary: big endian seems most likely,
 206   however implementations, most notably Microsoft, vary.)
 207 </p></li><li class="listitem"><p>
 208  Types representing the conversion state, for conversions involving
 209   the machinery in the "C" library, or the conversion descriptor, for
 210   conversions using iconv (such as the type iconv_t.)  Note that the
 211   conversion descriptor encodes more information than a simple encoding
 212   state type.
 213 </p></li><li class="listitem"><p>
 214  Conversion descriptors for both directions of encoding. (i.e., both
 215   UCS-2 to UTF-8 and UTF-8 to UCS-2.)
 216 </p></li><li class="listitem"><p>
 217  Something to indicate if the conversion requested if valid.
 218 </p></li><li class="listitem"><p>
 219  Something to represent if the conversion descriptors are valid.
 220 </p></li><li class="listitem"><p>
 221  Some way to enforce strict type checking on the internal and
 222   external types. As part of this, the size of the internal and
 223   external types will need to be known.
 224 </p></li></ul></div></div><div class="section" title="Other Issues"><div class="titlepage"><div><div><h5 class="title"><a name="codecvt.design.issues"></a>Other Issues</h5></div></div></div><p>
 225 In addition, multi-threaded and multi-locale environments also impact
 226 the design and requirements for code conversions. In particular, they
 227 affect the required specialization codecvt&lt;wchar_t, char, mbstate_t&gt;
 228 when implemented using standard "C" functions.
 229 </p><p>
 230 Three problems arise, one big, one of medium importance, and one small.
 231 </p><p>
 232 First, the small: mcsrtombs and wcsrtombs may not be multithread-safe
 233 on all systems required by the GNU tools. For GNU/Linux and glibc,
 234 this is not an issue.
 235 </p><p>
 236 Of medium concern, in the grand scope of things, is that the functions
 237 used to implement this specialization work on null-terminated
 238 strings. Buffers, especially file buffers, may not be null-terminated,
 239 thus giving conversions that end prematurely or are otherwise
 240 incorrect. Yikes!
 241 </p><p>
 242 The last, and fundamental problem, is the assumption of a global
 243 locale for all the "C" functions referenced above. For something like
 244 C++ iostreams (where codecvt is explicitly used) the notion of
 245 multiple locales is fundamental. In practice, most users may not run
 246 into this limitation. However, as a quality of implementation issue,
 247 the GNU C++ library would like to offer a solution that allows
 248 multiple locales and or simultaneous usage with computationally
 249 correct results. In short, libstdc++ is trying to offer, as an
 250 option, a high-quality implementation, damn the additional complexity!
 251 </p><p>
 252 For the required specialization codecvt&lt;wchar_t, char, mbstate_t&gt; ,
 253 conversions are made between the internal character set (always UCS4
 254 on GNU/Linux) and whatever the currently selected locale for the
 255 LC_CTYPE category implements.
 256 </p></div></div><div class="section" title="Implementation"><div class="titlepage"><div><div><h4 class="title"><a name="facet.codecvt.impl"></a>Implementation</h4></div></div></div><p>
 257 The two required specializations are implemented as follows:
 258 </p><p>
 259 <code class="code">
 260 codecvt&lt;char, char, mbstate_t&gt;
 261 </code>
 262 </p><p>
 263 This is a degenerate (i.e., does nothing) specialization. Implementing
 264 this was a piece of cake.
 265 </p><p>
 266 <code class="code">
 267 codecvt&lt;char, wchar_t, mbstate_t&gt;
 268 </code>
 269 </p><p>
 270 This specialization, by specifying all the template parameters, pretty
 271 much ties the hands of implementors. As such, the implementation is
 272 straightforward, involving mcsrtombs for the conversions between char
 273 to wchar_t and wcsrtombs for conversions between wchar_t and char.
 274 </p><p>
 275 Neither of these two required specializations deals with Unicode
 276 characters. As such, libstdc++ implements a partial specialization
 277 of the codecvt class with and iconv wrapper class, encoding_state as the
 278 third template parameter.
 279 </p><p>
 280 This implementation should be standards conformant. First of all, the
 281 standard explicitly points out that instantiations on the third
 282 template parameter, stateT, are the proper way to implement
 283 non-required conversions. Second of all, the standard says (in Chapter
 284 17) that partial specializations of required classes are a-ok. Third
 285 of all, the requirements for the stateT type elsewhere in the standard
 286 (see 21.1.2 traits typedefs) only indicate that this type be copy
 287 constructible.
 288 </p><p>
 289 As such, the type encoding_state is defined as a non-templatized, POD
 290 type to be used as the third type of a codecvt instantiation. This
 291 type is just a wrapper class for iconv, and provides an easy interface
 292 to iconv functionality.
 293 </p><p>
 294 There are two constructors for encoding_state:
 295 </p><p>
 296 <code class="code">
 297 encoding_state() : __in_desc(0), __out_desc(0)
 298 </code>
 299 </p><p>
 300 This default constructor sets the internal encoding to some default
 301 (currently UCS4) and the external encoding to whatever is returned by
 302 nl_langinfo(CODESET).
 303 </p><p>
 304 <code class="code">
 305 encoding_state(const char* __int, const char* __ext)
 306 </code>
 307 </p><p>
 308 This constructor takes as parameters string literals that indicate the
 309 desired internal and external encoding. There are no defaults for
 310 either argument.
 311 </p><p>
 312 One of the issues with iconv is that the string literals identifying
 313 conversions are not standardized. Because of this, the thought of
 314 mandating and or enforcing some set of pre-determined valid
 315 identifiers seems iffy: thus, a more practical (and non-migraine
 316 inducing) strategy was implemented: end-users can specify any string
 317 (subject to a pre-determined length qualifier, currently 32 bytes) for
 318 encodings. It is up to the user to make sure that these strings are
 319 valid on the target system.
 320 </p><p>
 321 <code class="code">
 322 void
 323 _M_init()
 324 </code>
 325 </p><p>
 326 Strangely enough, this member function attempts to open conversion
 327 descriptors for a given encoding_state object. If the conversion
 328 descriptors are not valid, the conversion descriptors returned will
 329 not be valid and the resulting calls to the codecvt conversion
 330 functions will return error.
 331 </p><p>
 332 <code class="code">
 333 bool
 334 _M_good()
 335 </code>
 336 </p><p>
 337 Provides a way to see if the given encoding_state object has been
 338 properly initialized. If the string literals describing the desired
 339 internal and external encoding are not valid, initialization will
 340 fail, and this will return false. If the internal and external
 341 encodings are valid, but iconv_open could not allocate conversion
 342 descriptors, this will also return false. Otherwise, the object is
 343 ready to convert and will return true.
 344 </p><p>
 345 <code class="code">
 346 encoding_state(const encoding_state&amp;)
 347 </code>
 348 </p><p>
 349 As iconv allocates memory and sets up conversion descriptors, the copy
 350 constructor can only copy the member data pertaining to the internal
 351 and external code conversions, and not the conversion descriptors
 352 themselves.
 353 </p><p>
 354 Definitions for all the required codecvt member functions are provided
 355 for this specialization, and usage of codecvt&lt;internal character type,
 356 external character type, encoding_state&gt; is consistent with other
 357 codecvt usage.
 358 </p></div><div class="section" title="Use"><div class="titlepage"><div><div><h4 class="title"><a name="facet.codecvt.use"></a>Use</h4></div></div></div><p>A conversions involving string literal.</p><pre class="programlisting">
 359   typedef codecvt_base::result                  result;
 360   typedef unsigned short                        unicode_t;
 361   typedef unicode_t                             int_type;
 362   typedef char                                  ext_type;
 363   typedef encoding_state                          state_type;
 364   typedef codecvt&lt;int_type, ext_type, state_type&gt; unicode_codecvt;
 365
 366   const ext_type*       e_lit = "black pearl jasmine tea";
 367   int                   size = strlen(e_lit);
 368   int_type              i_lit_base[24] =
 369   { 25088, 27648, 24832, 25344, 27392, 8192, 28672, 25856, 24832, 29184,
 370     27648, 8192, 27136, 24832, 29440, 27904, 26880, 28160, 25856, 8192, 29696,
 371     25856, 24832, 2560
 372   };
 373   const int_type*       i_lit = i_lit_base;
 374   const ext_type*       efrom_next;
 375   const int_type*       ifrom_next;
 376   ext_type*             e_arr = new ext_type[size + 1];
 377   ext_type*             eto_next;
 378   int_type*             i_arr = new int_type[size + 1];
 379   int_type*             ito_next;
 380
 381   // construct a locale object with the specialized facet.
 382   locale                loc(locale::classic(), new unicode_codecvt);
 383   // sanity check the constructed locale has the specialized facet.
 384   VERIFY( has_facet&lt;unicode_codecvt&gt;(loc) );
 385   const unicode_codecvt&amp; cvt = use_facet&lt;unicode_codecvt&gt;(loc);
 386   // convert between const char* and unicode strings
 387   unicode_codecvt::state_type state01("UNICODE", "ISO_8859-1");
 388   initialize_state(state01);
 389   result r1 = cvt.in(state01, e_lit, e_lit + size, efrom_next,
 390                      i_arr, i_arr + size, ito_next);
 391   VERIFY( r1 == codecvt_base::ok );
 392   VERIFY( !int_traits::compare(i_arr, i_lit, size) );
 393   VERIFY( efrom_next == e_lit + size );
 394   VERIFY( ito_next == i_arr + size );
 395 </pre></div><div class="section" title="Future"><div class="titlepage"><div><div><h4 class="title"><a name="facet.codecvt.future"></a>Future</h4></div></div></div><div class="itemizedlist"><ul class="itemizedlist" type="disc"><li class="listitem"><p>
 396    a. things that are sketchy, or remain unimplemented:
 397       do_encoding, max_length and length member functions
 398       are only weakly implemented. I have no idea how to do
 399       this correctly, and in a generic manner.  Nathan?
 400 </p></li><li class="listitem"><p>
 401    b. conversions involving std::string
 402   </p><div class="itemizedlist"><ul class="itemizedlist" type="circle"><li class="listitem"><p>
 403       how should operators != and == work for string of
 404       different/same encoding?
 405       </p></li><li class="listitem"><p>
 406       what is equal? A byte by byte comparison or an
 407       encoding then byte comparison?
 408       </p></li><li class="listitem"><p>
 409       conversions between narrow, wide, and unicode strings
 410       </p></li></ul></div></li><li class="listitem"><p>
 411    c. conversions involving std::filebuf and std::ostream
 412 </p><div class="itemizedlist"><ul class="itemizedlist" type="circle"><li class="listitem"><p>
 413       how to initialize the state object in a
 414       standards-conformant manner?
 415       </p></li><li class="listitem"><p>
 416       how to synchronize the "C" and "C++"
 417       conversion information?
 418       </p></li><li class="listitem"><p>
 419       wchar_t/char internal buffers and conversions between
 420       internal/external buffers?
 421       </p></li></ul></div></li></ul></div></div><div class="bibliography" title="Bibliography"><div class="titlepage"><div><div><h4 class="title"><a name="facet.codecvt.biblio"></a>Bibliography</h4></div></div></div><div class="biblioentry"><a name="id626108"></a><p><span class="citetitle"><em class="citetitle">
 422       The GNU C Library
 423     </em>. </span><span class="author"><span class="firstname">Roland</span> <span class="surname">McGrath</span>. </span><span class="author"><span class="firstname">Ulrich</span> <span class="surname">Drepper</span>. </span><span class="copyright">Copyright © 2007 FSF. </span><span class="pagenums">
 424       Chapters 6 Character Set Handling and 7 Locales and Internationalization
 425     . </span></p></div><div class="biblioentry"><a name="id626147"></a><p><span class="citetitle"><em class="citetitle">
 426       Correspondence
 427     </em>. </span><span class="author"><span class="firstname">Ulrich</span> <span class="surname">Drepper</span>. </span><span class="copyright">Copyright © 2002 . </span></p></div><div class="biblioentry"><a name="id626173"></a><p><span class="citetitle"><em class="citetitle">
 428       ISO/IEC 14882:1998 Programming languages - C++
 429     </em>. </span><span class="copyright">Copyright © 1998 ISO. </span></p></div><div class="biblioentry"><a name="id626192"></a><p><span class="citetitle"><em class="citetitle">
 430       ISO/IEC 9899:1999 Programming languages - C
 431     </em>. </span><span class="copyright">Copyright © 1999 ISO. </span></p></div><div class="biblioentry" title="System Interface Definitions, Issue 7 (IEEE Std. 1003.1-2008)"><a name="id626211"></a><p><span class="title"><i>
 432         <a class="link" href="http://www.opengroup.org/austin" target="_top">
 433       System Interface Definitions, Issue 7 (IEEE Std. 1003.1-2008)
 434         </a>
 435       </i>. </span><span class="copyright">Copyright © 2008
 436         The Open Group/The Institute of Electrical and Electronics
 437         Engineers, Inc.
 438       . </span></p></div><div class="biblioentry"><a name="id626238"></a><p><span class="citetitle"><em class="citetitle">
 439       The C++ Programming Language, Special Edition
 440     </em>. </span><span class="author"><span class="firstname">Bjarne</span> <span class="surname">Stroustrup</span>. </span><span class="copyright">Copyright © 2000 Addison Wesley, Inc.. </span><span class="pagenums">Appendix D. </span><span class="publisher"><span class="publishername">
 441         Addison Wesley
 442       . </span></span></p></div><div class="biblioentry"><a name="id626276"></a><p><span class="citetitle"><em class="citetitle">
 443       Standard C++ IOStreams and Locales
 444     </em>. </span><span class="subtitle">
 445       Advanced Programmer's Guide and Reference
 446     . </span><span class="author"><span class="firstname">Angelika</span> <span class="surname">Langer</span>. </span><span class="author"><span class="firstname">Klaus</span> <span class="surname">Kreft</span>. </span><span class="copyright">Copyright © 2000 Addison Wesley Longman, Inc.. </span><span class="publisher"><span class="publishername">
 447         Addison Wesley Longman
 448       . </span></span></p></div><div class="biblioentry" title="A brief description of Normative Addendum 1"><a name="id626324"></a><p><span class="title"><i>
 449         <a class="link" href="http://www.lysator.liu.se/c/na1.html" target="_top">
 450       A brief description of Normative Addendum 1
 451         </a>
 452       </i>. </span><span class="author"><span class="firstname">Clive</span> <span class="surname">Feather</span>. </span><span class="pagenums">Extended Character Sets. </span></p></div><div class="biblioentry" title="The Unicode HOWTO"><a name="id626351"></a><p><span class="title"><i>
 453         <a class="link" href="http://tldp.org/HOWTO/Unicode-HOWTO.html" target="_top">
 454           The Unicode HOWTO
 455         </a>
 456       </i>. </span><span class="author"><span class="firstname">Bruno</span> <span class="surname">Haible</span>. </span></p></div><div class="biblioentry" title="UTF-8 and Unicode FAQ for Unix/Linux"><a name="id626374"></a><p><span class="title"><i>
 457         <a class="link" href="http://www.cl.cam.ac.uk/~mgk25/unicode.html" target="_top">
 458       UTF-8 and Unicode FAQ for Unix/Linux
 459         </a>
 460       </i>. </span><span class="author"><span class="firstname">Markus</span> <span class="surname">Khun</span>. </span></p></div></div></div><div class="section" title="messages"><div class="titlepage"><div><div><h3 class="title"><a name="manual.localization.facet.messages"></a>messages</h3></div></div></div><p>
 461 The std::messages facet implements message retrieval functionality
 462 equivalent to Java's java.text.MessageFormat .using either GNU gettext
 463 or IEEE 1003.1-200 functions.
 464 </p><div class="section" title="Requirements"><div class="titlepage"><div><div><h4 class="title"><a name="facet.messages.req"></a>Requirements</h4></div></div></div><p>
 465 The std::messages facet is probably the most vaguely defined facet in
 466 the standard library. It's assumed that this facility was built into
 467 the standard library in order to convert string literals from one
 468 locale to the other. For instance, converting the "C" locale's
 469 <code class="code">const char* c = "please"</code> to a German-localized <code class="code">"bitte"</code>
 470 during program execution.
 471 </p><div class="blockquote"><blockquote class="blockquote"><p>
 472 22.2.7.1 - Template class messages [lib.locale.messages]
 473 </p></blockquote></div><p>
 474 This class has three public member functions, which directly
 475 correspond to three protected virtual member functions.
 476 </p><p>
 477 The public member functions are:
 478 </p><p>
 479 <code class="code">catalog open(const string&amp;, const locale&amp;) const</code>
 480 </p><p>
 481 <code class="code">string_type get(catalog, int, int, const string_type&amp;) const</code>
 482 </p><p>
 483 <code class="code">void close(catalog) const</code>
 484 </p><p>
 485 While the virtual functions are:
 486 </p><p>
 487 <code class="code">catalog do_open(const string&amp;, const locale&amp;) const</code>
 488 </p><div class="blockquote"><blockquote class="blockquote"><p>
 489 <span class="emphasis"><em>
 490 -1- Returns: A value that may be passed to get() to retrieve a
 491 message, from the message catalog identified by the string name
 492 according to an implementation-defined mapping. The result can be used
 493 until it is passed to close().  Returns a value less than 0 if no such
 494 catalog can be opened.
 495 </em></span>
 496 </p></blockquote></div><p>
 497 <code class="code">string_type do_get(catalog, int, int, const string_type&amp;) const</code>
 498 </p><div class="blockquote"><blockquote class="blockquote"><p>
 499 <span class="emphasis"><em>
 500 -3- Requires: A catalog cat obtained from open() and not yet closed.
 501 -4- Returns: A message identified by arguments set, msgid, and dfault,
 502 according to an implementation-defined mapping. If no such message can
 503 be found, returns dfault.
 504 </em></span>
 505 </p></blockquote></div><p>
 506 <code class="code">void do_close(catalog) const</code>
 507 </p><div class="blockquote"><blockquote class="blockquote"><p>
 508 <span class="emphasis"><em>
 509 -5- Requires: A catalog cat obtained from open() and not yet closed.
 510 -6- Effects: Releases unspecified resources associated with cat.
 511 -7- Notes: The limit on such resources, if any, is implementation-defined.
 512 </em></span>
 513 </p></blockquote></div></div><div class="section" title="Design"><div class="titlepage"><div><div><h4 class="title"><a name="facet.messages.design"></a>Design</h4></div></div></div><p>
 514 A couple of notes on the standard.
 515 </p><p>
 516 First, why is <code class="code">messages_base::catalog</code> specified as a typedef
 517 to int? This makes sense for implementations that use
 518 <code class="code">catopen</code> and define <code class="code">nl_catd</code> as int, but not for
 519 others. Fortunately, it's not heavily used and so only a minor irritant.
 520 This has been reported as a possible defect in the standard (LWG 2028).
 521 </p><p>
 522 Second, by making the member functions <code class="code">const</code>, it is
 523 impossible to save state in them. Thus, storing away information used
 524 in the 'open' member function for use in 'get' is impossible. This is
 525 unfortunate.
 526 </p><p>
 527 The 'open' member function in particular seems to be oddly
 528 designed. The signature seems quite peculiar. Why specify a <code class="code">const
 529 string&amp; </code> argument, for instance, instead of just <code class="code">const
 530 char*</code>? Or, why specify a <code class="code">const locale&amp;</code> argument that is
 531 to be used in the 'get' member function? How, exactly, is this locale
 532 argument useful? What was the intent? It might make sense if a locale
 533 argument was associated with a given default message string in the
 534 'open' member function, for instance. Quite murky and unclear, on
 535 reflection.
 536 </p><p>
 537 Lastly, it seems odd that messages, which explicitly require code
 538 conversion, don't use the codecvt facet. Because the messages facet
 539 has only one template parameter, it is assumed that ctype, and not
 540 codecvt, is to be used to convert between character sets.
 541 </p><p>
 542 It is implicitly assumed that the locale for the default message
 543 string in 'get' is in the "C" locale. Thus, all source code is assumed
 544 to be written in English, so translations are always from "en_US" to
 545 other, explicitly named locales.
 546 </p></div><div class="section" title="Implementation"><div class="titlepage"><div><div><h4 class="title"><a name="facet.messages.impl"></a>Implementation</h4></div></div></div><div class="section" title="Models"><div class="titlepage"><div><div><h5 class="title"><a name="messages.impl.models"></a>Models</h5></div></div></div><p>
 547     This is a relatively simple class, on the face of it. The standard
 548     specifies very little in concrete terms, so generic
 549     implementations that are conforming yet do very little are the
 550     norm. Adding functionality that would be useful to programmers and
 551     comparable to Java's java.text.MessageFormat takes a bit of work,
 552     and is highly dependent on the capabilities of the underlying
 553     operating system.
 554   </p><p>
 555     Three different mechanisms have been provided, selectable via
 556     configure flags:
 557   </p><div class="itemizedlist"><ul class="itemizedlist" type="disc"><li class="listitem"><p>
 558        generic
 559      </p><p>
 560        This model does very little, and is what is used by default.
 561      </p></li><li class="listitem"><p>
 562        gnu
 563      </p><p>
 564        The gnu model is complete and fully tested. It's based on the
 565        GNU gettext package, which is part of glibc. It uses the
 566        functions <code class="code">textdomain, bindtextdomain, gettext</code> to
 567        implement full functionality. Creating message catalogs is a
 568        relatively straight-forward process and is lightly documented
 569        below, and fully documented in gettext's distributed
 570        documentation.
 571      </p></li><li class="listitem"><p>
 572        ieee_1003.1-200x
 573      </p><p>
 574        This is a complete, though untested, implementation based on
 575        the IEEE standard. The functions <code class="code">catopen, catgets,
 576        catclose</code> are used to retrieve locale-specific messages
 577        given the appropriate message catalogs that have been
 578        constructed for their use. Note, the script <code class="code">
 579        po2msg.sed</code> that is part of the gettext distribution can
 580        convert gettext catalogs into catalogs that
 581        <code class="code">catopen</code> can use.
 582    </p></li></ul></div><p>
 583 A new, standards-conformant non-virtual member function signature was
 584 added for 'open' so that a directory could be specified with a given
 585 message catalog. This simplifies calling conventions for the gnu
 586 model.
 587 </p></div><div class="section" title="The GNU Model"><div class="titlepage"><div><div><h5 class="title"><a name="messages.impl.gnu"></a>The GNU Model</h5></div></div></div><p>
 588     The messages facet, because it is retrieving and converting
 589     between characters sets, depends on the ctype and perhaps the
 590     codecvt facet in a given locale. In addition, underlying "C"
 591     library locale support is necessary for more than just the
 592     <code class="code">LC_MESSAGES</code> mask: <code class="code">LC_CTYPE</code> is also
 593     necessary. To avoid any unpleasantness, all bits of the "C" mask
 594     (i.e. <code class="code">LC_ALL</code>) are set before retrieving messages.
 595   </p><p>
 596     Making the message catalogs can be initially tricky, but become
 597     quite simple with practice. For complete info, see the gettext
 598     documentation. Here's an idea of what is required:
 599   </p><div class="itemizedlist"><ul class="itemizedlist" type="disc"><li class="listitem"><p>
 600        Make a source file with the required string literals that need
 601        to be translated. See <code class="code">intl/string_literals.cc</code> for
 602        an example.
 603      </p></li><li class="listitem"><p>
 604        Make initial catalog (see "4 Making the PO Template File" from
 605        the gettext docs).</p><p>
 606    <code class="code"> xgettext --c++ --debug string_literals.cc -o libstdc++.pot </code>
 607    </p></li><li class="listitem"><p>Make language and country-specific locale catalogs.</p><p>
 608    <code class="code">cp libstdc++.pot fr_FR.po</code>
 609    </p><p>
 610    <code class="code">cp libstdc++.pot de_DE.po</code>
 611    </p></li><li class="listitem"><p>
 612        Edit localized catalogs in emacs so that strings are
 613        translated.
 614      </p><p>
 615    <code class="code">emacs fr_FR.po</code>
 616    </p></li><li class="listitem"><p>Make the binary mo files.</p><p>
 617    <code class="code">msgfmt fr_FR.po -o fr_FR.mo</code>
 618    </p><p>
 619    <code class="code">msgfmt de_DE.po -o de_DE.mo</code>
 620    </p></li><li class="listitem"><p>Copy the binary files into the correct directory structure.</p><p>
 621    <code class="code">cp fr_FR.mo (dir)/fr_FR/LC_MESSAGES/libstdc++.mo</code>
 622    </p><p>
 623    <code class="code">cp de_DE.mo (dir)/de_DE/LC_MESSAGES/libstdc++.mo</code>
 624    </p></li><li class="listitem"><p>Use the new message catalogs.</p><p>
 625    <code class="code">locale loc_de("de_DE");</code>
 626    </p><p>
 627    <code class="code">
 628    use_facet&lt;messages&lt;char&gt; &gt;(loc_de).open("libstdc++", locale(), dir);
 629    </code>
 630    </p></li></ul></div></div></div><div class="section" title="Use"><div class="titlepage"><div><div><h4 class="title"><a name="facet.messages.use"></a>Use</h4></div></div></div><p>
 631    A simple example using the GNU model of message conversion.
 632  </p><pre class="programlisting">
 633 #include &lt;iostream&gt;
 634 #include &lt;locale&gt;
 635 using namespace std;
 636
 637 void test01()
 638 {
 639   typedef messages&lt;char&gt;::catalog catalog;
 640   const char* dir =
 641   "/mnt/egcs/build/i686-pc-linux-gnu/libstdc++/po/share/locale";
 642   const locale loc_de("de_DE");
 643   const messages&lt;char&gt;&amp; mssg_de = use_facet&lt;messages&lt;char&gt; &gt;(loc_de);
 644
 645   catalog cat_de = mssg_de.open("libstdc++", loc_de, dir);
 646   string s01 = mssg_de.get(cat_de, 0, 0, "please");
 647   string s02 = mssg_de.get(cat_de, 0, 0, "thank you");
 648   cout &lt;&lt; "please in german:" &lt;&lt; s01 &lt;&lt; '\n';
 649   cout &lt;&lt; "thank you in german:" &lt;&lt; s02 &lt;&lt; '\n';
 650   mssg_de.close(cat_de);
 651 }
 652 </pre></div><div class="section" title="Future"><div class="titlepage"><div><div><h4 class="title"><a name="facet.messages.future"></a>Future</h4></div></div></div><div class="itemizedlist"><ul class="itemizedlist" type="disc"><li class="listitem"><p>
 653     Things that are sketchy, or remain unimplemented:
 654   </p><div class="itemizedlist"><ul class="itemizedlist" type="circle"><li class="listitem"><p>
 655           _M_convert_from_char, _M_convert_to_char are in flux,
 656           depending on how the library ends up doing character set
 657           conversions. It might not be possible to do a real character
 658           set based conversion, due to the fact that the template
 659           parameter for messages is not enough to instantiate the
 660           codecvt facet (1 supplied, need at least 2 but would prefer
 661           3).
 662         </p></li><li class="listitem"><p>
 663           There are issues with gettext needing the global locale set
 664           to extract a message. This dependence on the global locale
 665           makes the current "gnu" model non MT-safe. Future versions
 666           of glibc, i.e. glibc 2.3.x will fix this, and the C++ library
 667           bits are already in place.
 668         </p></li></ul></div></li><li class="listitem"><p>
 669     Development versions of the GNU "C" library, glibc 2.3 will allow
 670     a more efficient, MT implementation of std::messages, and will
 671     allow the removal of the _M_name_messages data member. If this is
 672     done, it will change the library ABI. The C++ parts to support
 673     glibc 2.3 have already been coded, but are not in use: once this
 674     version of the "C" library is released, the marked parts of the
 675     messages implementation can be switched over to the new "C"
 676     library functionality.
 677   </p></li><li class="listitem"><p>
 678     At some point in the near future, std::numpunct will probably use
 679     std::messages facilities to implement truename/falsename
 680     correctly. This is currently not done, but entries in
 681     libstdc++.pot have already been made for "true" and "false" string
 682     literals, so all that remains is the std::numpunct coding and the
 683     configure/make hassles to make the installed library search its
 684     own catalog. Currently the libstdc++.mo catalog is only searched
 685     for the testsuite cases involving messages members.
 686   </p></li><li class="listitem"><p> The following member functions:</p><p>
 687    <code class="code">
 688         catalog
 689         open(const basic_string&lt;char&gt;&amp; __s, const locale&amp; __loc) const
 690    </code>
 691    </p><p>
 692    <code class="code">
 693    catalog
 694    open(const basic_string&lt;char&gt;&amp;, const locale&amp;, const char*) const;
 695    </code>
 696    </p><p>
 697    Don't actually return a "value less than 0 if no such catalog
 698    can be opened" as required by the standard in the "gnu"
 699    model. As of this writing, it is unknown how to query to see
 700    if a specified message catalog exists using the gettext
 701    package.
 702    </p></li></ul></div></div><div class="bibliography" title="Bibliography"><div class="titlepage"><div><div><h4 class="title"><a name="facet.messages.biblio"></a>Bibliography</h4></div></div></div><div class="biblioentry"><a name="id627050"></a><p><span class="citetitle"><em class="citetitle">
 703       The GNU C Library
 704     </em>. </span><span class="author"><span class="firstname">Roland</span> <span class="surname">McGrath</span>. </span><span class="author"><span class="firstname">Ulrich</span> <span class="surname">Drepper</span>. </span><span class="copyright">Copyright © 2007 FSF. </span><span class="pagenums">Chapters 6 Character Set Handling, and 7 Locales and Internationalization
 705     . </span></p></div><div class="biblioentry"><a name="id627090"></a><p><span class="citetitle"><em class="citetitle">
 706       Correspondence
 707     </em>. </span><span class="author"><span class="firstname">Ulrich</span> <span class="surname">Drepper</span>. </span><span class="copyright">Copyright © 2002 . </span></p></div><div class="biblioentry"><a name="id627115"></a><p><span class="citetitle"><em class="citetitle">
 708       ISO/IEC 14882:1998 Programming languages - C++
 709     </em>. </span><span class="copyright">Copyright © 1998 ISO. </span></p></div><div class="biblioentry"><a name="id627134"></a><p><span class="citetitle"><em class="citetitle">
 710       ISO/IEC 9899:1999 Programming languages - C
 711     </em>. </span><span class="copyright">Copyright © 1999 ISO. </span></p></div><div class="biblioentry" title="System Interface Definitions, Issue 7 (IEEE Std. 1003.1-2008)"><a name="id627153"></a><p><span class="title"><i>
 712         <a class="link" href="http://www.opengroup.org/austin" target="_top">
 713       System Interface Definitions, Issue 7 (IEEE Std. 1003.1-2008)
 714         </a>
 715       </i>. </span><span class="copyright">Copyright © 2008
 716         The Open Group/The Institute of Electrical and Electronics
 717         Engineers, Inc.
 718       . </span></p></div><div class="biblioentry"><a name="id627180"></a><p><span class="citetitle"><em class="citetitle">
 719       The C++ Programming Language, Special Edition
 720     </em>. </span><span class="author"><span class="firstname">Bjarne</span> <span class="surname">Stroustrup</span>. </span><span class="copyright">Copyright © 2000 Addison Wesley, Inc.. </span><span class="pagenums">Appendix D. </span><span class="publisher"><span class="publishername">
 721         Addison Wesley
 722       . </span></span></p></div><div class="biblioentry"><a name="id627219"></a><p><span class="citetitle"><em class="citetitle">
 723       Standard C++ IOStreams and Locales
 724     </em>. </span><span class="subtitle">
 725       Advanced Programmer's Guide and Reference
 726     . </span><span class="author"><span class="firstname">Angelika</span> <span class="surname">Langer</span>. </span><span class="author"><span class="firstname">Klaus</span> <span class="surname">Kreft</span>. </span><span class="copyright">Copyright © 2000 Addison Wesley Longman, Inc.. </span><span class="publisher"><span class="publishername">
 727         Addison Wesley Longman
 728       . </span></span></p></div><div class="biblioentry" title="API Specifications, Java Platform"><a name="id627266"></a><p><span class="title"><i>
 729         <a class="link" href="http://java.sun.com/reference/api/index.html" target="_top">
 730         API Specifications, Java Platform
 731         </a>
 732       </i>. </span><span class="pagenums">java.util.Properties, java.text.MessageFormat,
 733 java.util.Locale, java.util.ResourceBundle
 734     . </span></p></div><div class="biblioentry" title="GNU gettext tools, version 0.10.38, Native Language Support Library and Tools."><a name="id627286"></a><p><span class="title"><i>
 735         <a class="link" href="http://www.gnu.org/software/gettext" target="_top">
 736       GNU gettext tools, version 0.10.38, Native Language Support
 737       Library and Tools.
 738         </a>
 739       </i>. </span></p></div></div></div></div><div class="navfooter"><hr><table width="100%" summary="Navigation footer"><tr><td width="40%" align="left"><a accesskey="p" href="localization.html">Prev</a> </td><td width="20%" align="center"><a accesskey="u" href="localization.html">Up</a></td><td width="40%" align="right"> <a accesskey="n" href="containers.html">Next</a></td></tr><tr><td width="40%" align="left" valign="top">Chapter 8.
 740   Localization
 741
 742  </td><td width="20%" align="center"><a accesskey="h" href="../index.html">Home</a></td><td width="40%" align="right" valign="top"> Chapter 9.
 743   Containers
 744
 745 </td></tr></table></div></body></html>