--- /dev/null
+# Packaging for PyPi
+
+Generally, PyPi packages with binary components are packaged as
+[wheels](https://packaging.python.org/en/latest/glossary/#term-Wheel).
+
+
+## Linux wheels
+
+Python wheels with C/C++ extensions need to be built on an old version of Linux
+so that they are supported on many platforms. The standard approach is to use
+the [manylinux](https://github.com/pypa/manylinux) Docker images as proposed in
+[PEP-513](https://www.python.org/dev/peps/pep-0513/).
+
+Most of the process for building wheels is automated by scripts in
+`contrib/packaging_python/` and is usually done for releases automatically by
+our CI.
+**PyPi does not allow reuploading any wheels with the same version number.**
+This means there is
+only one shot per version number (and per Python version), so it should work and
+be a release version.
+
+To build the python wheel from you current build directory, simply run
+`mk_wheel.py bdist_wheel -d dist` from the build directory.
+
+### Building wheels with manylinux2014
+
+The main script for building and uploading wheels is implemented in the
+`package-python-wheel` action. It builds a Docker container that is a slightly
+extended version of `manylinux2014` (see `manylinux2014/Dockerfile`), runs the `mk_clean_wheel.sh` script for every supported python version and finally collects and uploads the wheels.
+
+The `mk_clean_wheel.sh`:
+
+1. prepares the environment by creating and activating a proper python venv and installing some packages;
+2. configures cvc5 appropriately using the `mk_build_dir.py` script, which makes sure that the venv python version is used;
+3. builds cvc5 and the python extension via the `mk_wheel.py` script;
+4. postprocesses the wheel (with `auditwheel`) and moves it out of the build folder.
+
+
+## Uploading to PyPi
+
+To upload a wheel to test PyPi,
+
+ twine upload --repository testpypi -u $USERNAME -p $PASSWORD <path to wheel>
+
+Note that you will need a TestPyPi login. Once it has been uploaded, you can
+test (from anywhere, not just the container) that the wheel works by installing
+it from the TestPyPi repository.
+
+```
+python3 -m pip install --index-url https://test.pypi.org/simple/ pycvc5==<cvc5 version number>
+python3 -c "import pycvc5; solver = pycvc5.Solver(); print(solver.getIntegerSort())"
+```
+
+You can remove the repository argument from the upload command to upload to
+real PyPi.
+
+## What goes on within the Docker container
+
+### Setup
+
+Compared to `manylinux2014`, the container comes with `ccache` and some cvc5
+dependencies preinstalled. Additionally, we set up appropriate symlinks so that
+`ccache` is used with the standard C and C++ compilers (`cc`, `c++`, `gcc`, `g++`).
+
+The idea is to mount a cvc5 checkout into the Docker container at `/home/pycvc5`.
+This way, we don't need to care about cloning cvc5 within Docker and the `ccache` cache is automatically stored persistently outside of the container.
+
+### Building
+
+To build the wheel for a specific python version, simply run `mk_clean_wheel.sh` within the docker container, and pass it the python binary it shall use (`/opt/python/cp<something>/bin/python`) and options to the cvc5 configure script (usually something like `production --auto-download`).
+The script will create a new build folder (`build_wheel`), configure and build cvc5 accordingly and store the wheel within a `wheel-<version>` folder.
+
+### Testing
+
+From within Docker, you can test a wheel as follows: to ensure that the `cvc5`
+library is linked correctly (and not looking at a local path). To do this,
+start from the top-level `cvc5` directory and run the following:
+```
+# delete build directory to get rid of local library
+rm -r build
+# start virtualenv
+source ./env<Python version>/bin/activate
+# install the wheel
+pip install <path to wheel for this Python version>
+# run a small example to make sure it works
+python3 -c "import pycvc5; solver = pycvc5.Solver(); print(solver.getIntegerSort())"
+```
+
+### Test PyPi
+
+In addition to the local test described above, you can do a test upload to
+[TestPyPi](https://packaging.python.org/guides/using-testpypi/). To do this
+run the following after building the wheel:
+```
+# start the virtualenv for this version (twine was installed here)
+source ./env<Python version>/bin/activate
+# upload to Test PyPi
+twine upload --repository testpypi <path to wheel>
+```
+
+Note that you will need a TestPyPi login. Once it has been uploaded, you can
+test (from anywhere, not just the container) that the wheel works by installing
+it from the TestPyPi repository.
+
+```
+python3 -m pip install --index-url https://test.pypi.org/simple/ pycvc5==<cvc5 version number>
+python3 -c "import pycvc5; solver = pycvc5.Solver(); print(solver.getIntegerSort())"
+```
+
+### Upload to PyPi
+
+Once you are certain this wheel file is ready, you can upload it to PyPi. Note,
+you will need a login for PyPi (this is separate from the TestPyPi login). The
+steps are almost identical to the TestPyPi upload:
+```
+# start virtualenv where twine was installed for this Python version
+source ./env<Python version>
+# upload to PyPi with twine (be sure this ready!!)
+twine upload <wheel file>
+```
+
+Once the `.whl` has been built, you should be able to upload it from anywhere.
--- /dev/null
+FROM quay.io/pypa/manylinux2014_x86_64
+WORKDIR /home/pycvc5
+RUN yum -q -y install ccache jre libffi-devel
+RUN ccache --set-config=cache_dir=/home/pycvc5/ccache-dir
+
+ENV PATH="/usr/lib64/ccache:${PATH}"
+
+RUN ln -s /usr/bin/ccache /usr/lib64/ccache/g++ && \
+ ln -s /usr/bin/ccache /usr/lib64/ccache/gcc && \
+ ln -s /usr/bin/ccache /usr/lib64/ccache/c++ && \
+ ln -s /usr/bin/ccache /usr/lib64/ccache/cc
--- /dev/null
+###############################################################################
+# Top contributors (to current version):
+# Gereon Kremer
+#
+# This file is part of the cvc5 project.
+#
+# Copyright (c) 2009-2021 by the authors listed in the file AUTHORS
+# in the top-level source directory and their institutional affiliations.
+# All rights reserved. See the file COPYING in the top-level source
+# directory for licensing information.
+# ############################################################################
+#
+# Run configure.sh and make cmake pick up whatever python interpreter this
+# script is run with. Additionally, all arguments passed to this script are
+# forwarded as well.
+##
+
+import subprocess
+import sys
+
+from skbuild.cmaker import CMaker
+
+python_version = CMaker.get_python_version()
+args = [
+ '-DPYTHON_VERSION_STRING:STRING=' + sys.version.split(' ')[0],
+ '-DPYTHON_INCLUDE_DIR:PATH=' +
+ CMaker.get_python_include_dir(python_version),
+ '-DPYTHON_LIBRARY:FILEPATH=' +
+ CMaker.get_python_library(python_version),
+]
+
+subprocess.check_call(['./configure.sh', *sys.argv[1:], *args])
--- /dev/null
+#!/bin/bash
+###############################################################################
+# Top contributors (to current version):
+# Makai Mann, Alex Ozdemir, Gereon Kremer
+#
+# This file is part of the cvc5 project.
+#
+# Copyright (c) 2009-2021 by the authors listed in the file AUTHORS
+# in the top-level source directory and their institutional affiliations.
+# All rights reserved. See the file COPYING in the top-level source
+# directory for licensing information.
+# ############################################################################
+#
+# Build the python wheel for cvc5 with a given python interpreter and additional
+# configuration arguments in a clean build folder.
+# Call as follows:
+# contrib/packaging_python/mk_clean_wheel.sh /usr/bin/python3 "--cadical"
+#
+# The script first sets up a python venv with the given interpreter and installs
+# some requirements in this venv. It then uses the mk_build_dir.py script to
+# prepare a fresh build folder build_wheel/ and builds the wheel in there using
+# mk_wheel.py. The wheel is fixes (using auditwheel or delocate-wheel) and
+# copied out of the build folder.
+##
+
+PYTHONBIN=$1
+CONFIG="$2"
+PYVERSION=$($PYTHONBIN -c "import sys; print(sys.implementation.name + sys.version.split()[0])")
+
+# setup and activate venv
+echo "Making venv with $PYTHONBIN"
+ENVDIR=env$PYVERSION
+$PYTHONBIN -m venv ./$ENVDIR
+source ./$ENVDIR/bin/activate
+
+# install packages
+pip install -q --upgrade pip setuptools auditwheel
+pip install -q Cython pytest toml scikit-build
+if [ "$(uname)" == "Darwin" ]; then
+ # Mac version of auditwheel
+ pip install -q delocate
+fi
+
+# configure cvc5
+echo "Configuring"
+rm -rf build_wheel/
+python contrib/packaging_python/mk_build_dir.py $CONFIG --python-bindings --name=build_wheel
+
+# building wheel
+echo "Building pycvc5 wheel"
+
+pushd build_wheel
+python ../contrib/packaging_python/mk_wheel.py bdist_wheel -d dist
+
+cd dist
+
+# resolve the links and bundle the library with auditwheel
+if [ "$(uname)" == "Linux" ]; then
+ auditwheel show ./*.whl
+ auditwheel repair ./*.whl
+elif [ "$(uname)" == "Darwin" ]; then
+ delocate-wheel -w wheelhouse ./*.whl
+else
+ echo "Unhandled system $(uname) for packing libraries with wheel."
+fi
+
+popd
+
+mv build_wheel/dist/wheelhouse/*.whl .
--- /dev/null
+###############################################################################
+# Top contributors (to current version):
+# Makai Mann, Gereon Kremer
+#
+# This file is part of the cvc5 project.
+#
+# Copyright (c) 2009-2021 by the authors listed in the file AUTHORS
+# in the top-level source directory and their institutional affiliations.
+# All rights reserved. See the file COPYING in the top-level source
+# directory for licensing information.
+# ############################################################################
+#
+# Build script for python wheels for cvc5.
+# It provides two main functions:
+# - get_cvc5_version() uses the version.cmake script to obtain the proper cvc5
+# version string. It is extracted from the versioninfo.cpp file and passed on
+# to the setuptools config.
+# - build_extension() builds the cvc5 python api(s) using the cvc5_python_api
+# target and copying it to wherever setuptools expects it to be.
+#
+# Example usage (from build directory):
+# python3 ../contrib/packaging_python/mk_wheel.py bdist_wheel -d dist
+##
+
+import os
+import re
+import platform
+import subprocess
+import shutil
+
+from setuptools import setup, Extension
+from setuptools.command.build_ext import build_ext
+from distutils.version import LooseVersion
+
+
+def get_project_src_path():
+ # expecting this script to be in contrib/packaging_python/
+ # The project source directory is two directories up
+ name = __file__
+ for i in range(3):
+ name = os.path.dirname(name)
+ name = os.path.abspath(name)
+ if not os.path.isfile(os.path.join(name, 'configure.sh')):
+ raise RuntimeError('Finding the project source path failed. We guessed ' + name)
+ return name
+
+
+def get_cvc5_version():
+ # run the version detection in cmake
+ subprocess.check_call([
+ 'cmake', '-DPROJECT_SOURCE_DIR=' + get_project_src_path(),
+ '-DCMAKE_BINARY_DIR=.', '-P',
+ get_project_src_path() + '/cmake/version.cmake'
+ ])
+
+ # read versioninfo.cpp to get version number
+ with open(os.path.join('src', 'base', 'versioninfo.cpp'), 'r') as f:
+ m = re.search('CVC5_FULL_VERSION = "([0-9a-z.-]+)"', f.read())
+ if m:
+ version = m.group(1)
+ version = re.sub('-dev\.([0-9]+)\.[0-9a-f]+', '.dev\\1', version)
+ return version
+ else:
+ raise Exception('Could not find version')
+
+
+class CMakeExtension(Extension):
+ def __init__(self, name, sourcedir=''):
+ Extension.__init__(self, name, sources=[])
+ self.sourcedir = os.path.abspath(sourcedir)
+
+
+class CMakeBuild(build_ext):
+ def run(self):
+ try:
+ out = subprocess.check_output(['cmake', '--version'])
+ except OSError:
+ raise RuntimeError(
+ "CMake must be installed to build the following extensions: " +
+ ", ".join(e.name for e in self.extensions))
+
+ if self.is_windows():
+ cmake_version = LooseVersion(
+ re.search(r'version\s*([\d.]+)', out.decode()).group(1))
+ if cmake_version < '3.1.0':
+ raise RuntimeError("CMake >= 3.1.0 is required on Windows")
+
+ for ext in self.extensions:
+ self.build_extension(ext)
+
+ @staticmethod
+ def is_windows():
+ tag = platform.system().lower()
+ return tag == "windows"
+
+ def build_extension(self, ext):
+ # build the python api
+ subprocess.check_call(['cmake', '--build', '.', '--target', 'cvc5_python_api'])
+
+ # copy the library over. we need to consider other users that are not on linux
+ # module is a directory called cvc5_python_base_module
+ extdir = os.path.abspath(
+ os.path.dirname(self.get_ext_fullpath(ext.name)))
+ cvc5_python_base_module = os.path.join("src", "api", "python", "cvc5")
+ dst_name = os.path.join(extdir, "cvc5")
+
+ shutil.rmtree(dst_name, ignore_errors=True)
+ shutil.copytree(cvc5_python_base_module, dst_name)
+
+
+setup(
+ name='cvc5',
+ version=get_cvc5_version(),
+ long_description='Python bindings for cvc5',
+ url='https://github.com/cvc5/cvc5',
+ license='BSD-3-Clause',
+ zip_safe=False,
+ ext_modules=[CMakeExtension('cvc5')],
+ cmdclass=dict(build_ext=CMakeBuild),
+ tests_require=['pytest']
+)
+++ /dev/null
-###############################################################################
-# Top contributors (to current version):
-# Makai Mann
-#
-# This file is part of the cvc5 project.
-#
-# Copyright (c) 2009-2021 by the authors listed in the file AUTHORS
-# in the top-level source directory and their institutional affiliations.
-# All rights reserved. See the file COPYING in the top-level source
-# directory for licensing information.
-# ############################################################################
-#
-# Script for building wheels distribution for cvc5
-#
-# Example usage (from top-level directory):
-# python3 ./src/api/python/wheels/build_wheel.py bdist_wheel
-# Configures and builds in directory ./build
-# Creates wheel in ./dist
-#
-# Note: takes an *optional* environment variable VERSION_SUFFIX. If set, this
-# suffix will be appended to the pypi package version.
-# Example:
-# VERSION_SUFFIX=rc1 python3 ./src/api/python/wheels/build_wheel.py bdist_wheel
-# would create versions X.Y.Zrc1
-#
-# The suffix should start with a letter, and end with a number
-##
-
-import os
-import re
-import sys
-import platform
-import string
-import subprocess
-import multiprocessing
-import shutil
-
-from setuptools import setup, Extension
-from setuptools.command.build_ext import build_ext
-from skbuild.cmaker import CMaker
-from distutils.version import LooseVersion
-
-WORKING_DIR = "build"
-
-
-def get_project_src_path():
- # expecting this script to be in src/api/python/wheels
- # The project source directory is 4 directories up
- # (5 dirnames because start with file)
- project_src_path = __file__
- for i in range(5):
- project_src_path = os.path.dirname(project_src_path)
- return os.path.abspath(project_src_path)
-
-
-def get_cvc5_version():
- project_src_path = get_project_src_path()
-
- # read CMakeLists.txt to get version number
- version = ''
- str_pattern = 'set\(CVC5_LAST_RELEASE\s*"([^"]+)"\)'
- pattern = re.compile(str_pattern)
- with open(os.path.join(project_src_path, 'cmake', 'version-base.cmake'), 'r') as f:
- m = pattern.search(f.read())
- if m:
- version = m.group(1)
- return version
- else:
- raise Exception('Could not find version')
-
-
-class CMakeExtension(Extension):
- def __init__(self, name, sourcedir=''):
- Extension.__init__(self, name, sources=[])
- self.sourcedir = os.path.abspath(sourcedir)
-
-
-class CMakeBuild(build_ext):
- def run(self):
- try:
- out = subprocess.check_output(['cmake', '--version'])
- except OSError:
- raise RuntimeError(
- "CMake must be installed to build the following extensions: " +
- ", ".join(e.name for e in self.extensions))
-
- if self.is_windows():
- cmake_version = LooseVersion(
- re.search(r'version\s*([\d.]+)', out.decode()).group(1))
- if cmake_version < '3.1.0':
- raise RuntimeError("CMake >= 3.1.0 is required on Windows")
-
- for ext in self.extensions:
- self.build_extension(ext)
-
- @staticmethod
- def is_windows():
- tag = platform.system().lower()
- return tag == "windows"
-
- def build_extension(self, ext):
- extdir = os.path.abspath(
- os.path.dirname(self.get_ext_fullpath(ext.name)))
-
- cfg = 'Production'
- build_args = ['--config', cfg]
-
- cpu_count = max(2, multiprocessing.cpu_count() // 2)
- build_args += ['--', '-j{0}'.format(cpu_count)]
-
- project_src_path = get_project_src_path()
- build_dir = os.path.join(project_src_path, WORKING_DIR)
-
- # configure with the working directory python-build-wheel
- args = ['--python-bindings',
- '--auto-download',
- '--name='+WORKING_DIR]
- # find correct Python include directory and library
- # works even for nonstandard Python installations
- # (e.g., on pypa/manylinux)
- args.append('-DPYTHON_VERSION_STRING:STRING=' +
- sys.version.split(' ')[0])
- python_version = CMaker.get_python_version()
- args.append('-DPYTHON_INCLUDE_DIR:PATH=' +
- CMaker.get_python_include_dir(python_version))
- args.append('-DPYTHON_LIBRARY:FILEPATH=' +
- CMaker.get_python_library(python_version))
-
- config_filename = os.path.join(project_src_path, "configure.sh")
- subprocess.check_call([config_filename] + args)
-
- # build the main library
- subprocess.check_call(
- ['cmake', '--build', '.', "--target", "cvc5"] + build_args, cwd=build_dir)
-
- # build the python binding
- python_build_dir = os.path.join(build_dir, "src", "api", "python")
- subprocess.check_call(["make"], cwd=python_build_dir)
- # the build folder gets cleaned during the config, need to create it again
- # this is necessary since "build" is a python dist folder
- if not os.path.isdir(extdir):
- os.mkdir(extdir)
-
- # copy the library over. we need to consider other users that are not on linux
- # module is a directory called cvc5_python_base_module
- cvc5_python_base_module = os.path.join(python_build_dir, "cvc5")
- dst_name = os.path.join(extdir, "cvc5")
- if os.path.isdir(dst_name):
- # remove, then replace
- shutil.rmtree(dst_name)
-
- shutil.copytree(cvc5_python_base_module, dst_name)
-
-
-version_suffix = os.getenv('VERSION_SUFFIX', '')
-if len(version_suffix) > 0:
- assert all(c in string.ascii_letters + string.digits for c in version_suffix)
- assert version_suffix[0] in string.ascii_letters
- assert version_suffix[-1] in string.ascii_digits
- print("Setting version suffix to", version_suffix)
-
-
-setup(
- name='cvc5',
- version=get_cvc5_version() + version_suffix,
- long_description='Python bindings for cvc5',
- url='https://github.com/cvc5/cvc5',
- license='BSD-3-Clause',
- zip_safe=False,
- ext_modules=[CMakeExtension('cvc5')],
- cmdclass=dict(build_ext=CMakeBuild),
- tests_require=['pytest']
-)
projects / cvc5.git / commitdiff