ext/pybind11/docs/faq.rst

   1 Frequently asked questions
   2 ##########################
   3
   4 "ImportError: dynamic module does not define init function"
   5 ===========================================================
   6
   7 You are likely using an incompatible version of Python (for instance, the
   8 extension library was compiled against Python 2, while the interpreter is
   9 running on top of some version of Python 3, or vice versa).
  10
  11 "Symbol not found: ``__Py_ZeroStruct`` / ``_PyInstanceMethod_Type``"
  12 ========================================================================
  13
  14 See the first answer.
  15
  16 "SystemError: dynamic module not initialized properly"
  17 ======================================================
  18
  19 See the first answer.
  20
  21 The Python interpreter immediately crashes when importing my module
  22 ===================================================================
  23
  24 See the first answer.
  25
  26 CMake doesn't detect the right Python version
  27 =============================================
  28
  29 The CMake-based build system will try to automatically detect the installed
  30 version of Python and link against that. When this fails, or when there are
  31 multiple versions of Python and it finds the wrong one, delete
  32 ``CMakeCache.txt`` and then invoke CMake as follows:
  33
  34 .. code-block:: bash
  35
  36     cmake -DPYTHON_EXECUTABLE:FILEPATH=<path-to-python-executable> .
  37
  38 Limitations involving reference arguments
  39 =========================================
  40
  41 In C++, it's fairly common to pass arguments using mutable references or
  42 mutable pointers, which allows both read and write access to the value
  43 supplied by the caller. This is sometimes done for efficiency reasons, or to
  44 realize functions that have multiple return values. Here are two very basic
  45 examples:
  46
  47 .. code-block:: cpp
  48
  49     void increment(int &i) { i++; }
  50     void increment_ptr(int *i) { (*i)++; }
  51
  52 In Python, all arguments are passed by reference, so there is no general
  53 issue in binding such code from Python.
  54
  55 However, certain basic Python types (like ``str``, ``int``, ``bool``,
  56 ``float``, etc.) are **immutable**. This means that the following attempt
  57 to port the function to Python doesn't have the same effect on the value
  58 provided by the caller -- in fact, it does nothing at all.
  59
  60 .. code-block:: python
  61
  62     def increment(i):
  63         i += 1 # nope..
  64
  65 pybind11 is also affected by such language-level conventions, which means that
  66 binding ``increment`` or ``increment_ptr`` will also create Python functions
  67 that don't modify their arguments.
  68
  69 Although inconvenient, one workaround is to encapsulate the immutable types in
  70 a custom type that does allow modifications.
  71
  72 An other alternative involves binding a small wrapper lambda function that
  73 returns a tuple with all output arguments (see the remainder of the
  74 documentation for examples on binding lambda functions). An example:
  75
  76 .. code-block:: cpp
  77
  78     int foo(int &i) { i++; return 123; }
  79
  80 and the binding code
  81
  82 .. code-block:: cpp
  83
  84    m.def("foo", [](int i) { int rv = foo(i); return std::make_tuple(rv, i); });
  85
  86
  87 How can I reduce the build time?
  88 ================================
  89
  90 It's good practice to split binding code over multiple files, as in the
  91 following example:
  92
  93 :file:`example.cpp`:
  94
  95 .. code-block:: cpp
  96
  97     void init_ex1(py::module &);
  98     void init_ex2(py::module &);
  99     /* ... */
 100
 101     PYBIND11_MODULE(example, m) {
 102         init_ex1(m);
 103         init_ex2(m);
 104         /* ... */
 105     }
 106
 107 :file:`ex1.cpp`:
 108
 109 .. code-block:: cpp
 110
 111     void init_ex1(py::module &m) {
 112         m.def("add", [](int a, int b) { return a + b; });
 113     }
 114
 115 :file:`ex2.cpp`:
 116
 117 .. code-block:: cpp
 118
 119     void init_ex1(py::module &m) {
 120         m.def("sub", [](int a, int b) { return a - b; });
 121     }
 122
 123 :command:`python`:
 124
 125 .. code-block:: pycon
 126
 127     >>> import example
 128     >>> example.add(1, 2)
 129     3
 130     >>> example.sub(1, 1)
 131     0
 132
 133 As shown above, the various ``init_ex`` functions should be contained in
 134 separate files that can be compiled independently from one another, and then
 135 linked together into the same final shared object.  Following this approach
 136 will:
 137
 138 1. reduce memory requirements per compilation unit.
 139
 140 2. enable parallel builds (if desired).
 141
 142 3. allow for faster incremental builds. For instance, when a single class
 143    definition is changed, only a subset of the binding code will generally need
 144    to be recompiled.
 145
 146 "recursive template instantiation exceeded maximum depth of 256"
 147 ================================================================
 148
 149 If you receive an error about excessive recursive template evaluation, try
 150 specifying a larger value, e.g. ``-ftemplate-depth=1024`` on GCC/Clang. The
 151 culprit is generally the generation of function signatures at compile time
 152 using C++14 template metaprogramming.
 153
 154 .. _`faq:hidden_visibility`:
 155
 156 "‘SomeClass’ declared with greater visibility than the type of its field ‘SomeClass::member’ [-Wattributes]"
 157 ============================================================================================================
 158
 159 This error typically indicates that you are compiling without the required
 160 ``-fvisibility`` flag.  pybind11 code internally forces hidden visibility on
 161 all internal code, but if non-hidden (and thus *exported*) code attempts to
 162 include a pybind type (for example, ``py::object`` or ``py::list``) you can run
 163 into this warning.
 164
 165 To avoid it, make sure you are specifying ``-fvisibility=hidden`` when
 166 compiling pybind code.
 167
 168 As to why ``-fvisibility=hidden`` is necessary, because pybind modules could
 169 have been compiled under different versions of pybind itself, it is also
 170 important that the symbols defined in one module do not clash with the
 171 potentially-incompatible symbols defined in another.  While Python extension
 172 modules are usually loaded with localized symbols (under POSIX systems
 173 typically using ``dlopen`` with the ``RTLD_LOCAL`` flag), this Python default
 174 can be changed, but even if it isn't it is not always enough to guarantee
 175 complete independence of the symbols involved when not using
 176 ``-fvisibility=hidden``.
 177
 178 Additionally, ``-fvisiblity=hidden`` can deliver considerably binary size
 179 savings.  (See the following section for more details).
 180
 181
 182 .. _`faq:symhidden`:
 183
 184 How can I create smaller binaries?
 185 ==================================
 186
 187 To do its job, pybind11 extensively relies on a programming technique known as
 188 *template metaprogramming*, which is a way of performing computation at compile
 189 time using type information. Template metaprogamming usually instantiates code
 190 involving significant numbers of deeply nested types that are either completely
 191 removed or reduced to just a few instructions during the compiler's optimization
 192 phase. However, due to the nested nature of these types, the resulting symbol
 193 names in the compiled extension library can be extremely long. For instance,
 194 the included test suite contains the following symbol:
 195
 196 .. only:: html
 197
 198     .. code-block:: none
 199
 200         __ZN8pybind1112cpp_functionC1Iv8Example2JRNSt3__16vectorINS3_12basic_stringIwNS3_11char_traitsIwEENS3_9allocatorIwEEEENS8_ISA_EEEEEJNS_4nameENS_7siblingENS_9is_methodEA28_cEEEMT0_FT_DpT1_EDpRKT2_
 201
 202 .. only:: not html
 203
 204     .. code-block:: cpp
 205
 206         __ZN8pybind1112cpp_functionC1Iv8Example2JRNSt3__16vectorINS3_12basic_stringIwNS3_11char_traitsIwEENS3_9allocatorIwEEEENS8_ISA_EEEEEJNS_4nameENS_7siblingENS_9is_methodEA28_cEEEMT0_FT_DpT1_EDpRKT2_
 207
 208 which is the mangled form of the following function type:
 209
 210 .. code-block:: cpp
 211
 212     pybind11::cpp_function::cpp_function<void, Example2, std::__1::vector<std::__1::basic_string<wchar_t, std::__1::char_traits<wchar_t>, std::__1::allocator<wchar_t> >, std::__1::allocator<std::__1::basic_string<wchar_t, std::__1::char_traits<wchar_t>, std::__1::allocator<wchar_t> > > >&, pybind11::name, pybind11::sibling, pybind11::is_method, char [28]>(void (Example2::*)(std::__1::vector<std::__1::basic_string<wchar_t, std::__1::char_traits<wchar_t>, std::__1::allocator<wchar_t> >, std::__1::allocator<std::__1::basic_string<wchar_t, std::__1::char_traits<wchar_t>, std::__1::allocator<wchar_t> > > >&), pybind11::name const&, pybind11::sibling const&, pybind11::is_method const&, char const (&) [28])
 213
 214 The memory needed to store just the mangled name of this function (196 bytes)
 215 is larger than the actual piece of code (111 bytes) it represents! On the other
 216 hand, it's silly to even give this function a name -- after all, it's just a
 217 tiny cog in a bigger piece of machinery that is not exposed to the outside
 218 world. So we'll generally only want to export symbols for those functions which
 219 are actually called from the outside.
 220
 221 This can be achieved by specifying the parameter ``-fvisibility=hidden`` to GCC
 222 and Clang, which sets the default symbol visibility to *hidden*, which has a
 223 tremendous impact on the final binary size of the resulting extension library.
 224 (On Visual Studio, symbols are already hidden by default, so nothing needs to
 225 be done there.)
 226
 227 In addition to decreasing binary size, ``-fvisibility=hidden`` also avoids
 228 potential serious issues when loading multiple modules and is required for
 229 proper pybind operation.  See the previous FAQ entry for more details.
 230
 231 Another aspect that can require a fair bit of code are function signature
 232 descriptions. pybind11 automatically generates human-readable function
 233 signatures for docstrings, e.g.:
 234
 235 .. code-block:: none
 236
 237      |  __init__(...)
 238      |      __init__(*args, **kwargs)
 239      |      Overloaded function.
 240      |
 241      |      1. __init__(example.Example1) -> NoneType
 242      |
 243      |      Docstring for overload #1 goes here
 244      |
 245      |      2. __init__(example.Example1, int) -> NoneType
 246      |
 247      |      Docstring for overload #2 goes here
 248      |
 249      |      3. __init__(example.Example1, example.Example1) -> NoneType
 250      |
 251      |      Docstring for overload #3 goes here
 252
 253
 254 In C++11 mode, these are generated at run time using string concatenation,
 255 which can amount to 10-20% of the size of the resulting binary. If you can,
 256 enable C++14 language features (using ``-std=c++14`` for GCC/Clang), in which
 257 case signatures are efficiently pre-generated at compile time. Unfortunately,
 258 Visual Studio's C++14 support (``constexpr``) is not good enough as of April
 259 2016, so it always uses the more expensive run-time approach.
 260
 261 Working with ancient Visual Studio 2009 builds on Windows
 262 =========================================================
 263
 264 The official Windows distributions of Python are compiled using truly
 265 ancient versions of Visual Studio that lack good C++11 support. Some users
 266 implicitly assume that it would be impossible to load a plugin built with
 267 Visual Studio 2015 into a Python distribution that was compiled using Visual
 268 Studio 2009. However, no such issue exists: it's perfectly legitimate to
 269 interface DLLs that are built with different compilers and/or C libraries.
 270 Common gotchas to watch out for involve not ``free()``-ing memory region
 271 that that were ``malloc()``-ed in another shared library, using data
 272 structures with incompatible ABIs, and so on. pybind11 is very careful not
 273 to make these types of mistakes.