From: Gereon Kremer Date: Mon, 28 Feb 2022 18:47:28 +0000 (+0100) Subject: Add scripts to build python wheels (#8132) X-Git-Tag: cvc5-1.0.0~366 X-Git-Url: https://git.libre-soc.org/?a=commitdiff_plain;h=86aefbc07acff372225852ce8ac9a5d734046058;p=cvc5.git Add scripts to build python wheels (#8132) This adds a new set of scripts to build python wheels for cvc5. It includes mk_wheel.py which builds the python wheel within an existing build folder mk_build_dir.py which runs configure.sh and directs cmake to a custom python interpreter mk_clean_wheel.sh which builds the python wheel in a clean environment (a python venv and a fresh build folder) a dockerfile that slightly customizes manylinux2014 to our use case a readme giving a highlevel overview of what we are doing --- diff --git a/contrib/packaging_python/Readme.md b/contrib/packaging_python/Readme.md new file mode 100644 index 000000000..d75a201fe --- /dev/null +++ b/contrib/packaging_python/Readme.md @@ -0,0 +1,122 @@ +# Packaging for PyPi + +Generally, PyPi packages with binary components are packaged as +[wheels](https://packaging.python.org/en/latest/glossary/#term-Wheel). + + +## Linux wheels + +Python wheels with C/C++ extensions need to be built on an old version of Linux +so that they are supported on many platforms. The standard approach is to use +the [manylinux](https://github.com/pypa/manylinux) Docker images as proposed in +[PEP-513](https://www.python.org/dev/peps/pep-0513/). + +Most of the process for building wheels is automated by scripts in +`contrib/packaging_python/` and is usually done for releases automatically by +our CI. +**PyPi does not allow reuploading any wheels with the same version number.** +This means there is +only one shot per version number (and per Python version), so it should work and +be a release version. + +To build the python wheel from you current build directory, simply run +`mk_wheel.py bdist_wheel -d dist` from the build directory. + +### Building wheels with manylinux2014 + +The main script for building and uploading wheels is implemented in the +`package-python-wheel` action. It builds a Docker container that is a slightly +extended version of `manylinux2014` (see `manylinux2014/Dockerfile`), runs the `mk_clean_wheel.sh` script for every supported python version and finally collects and uploads the wheels. + +The `mk_clean_wheel.sh`: + +1. prepares the environment by creating and activating a proper python venv and installing some packages; +2. configures cvc5 appropriately using the `mk_build_dir.py` script, which makes sure that the venv python version is used; +3. builds cvc5 and the python extension via the `mk_wheel.py` script; +4. postprocesses the wheel (with `auditwheel`) and moves it out of the build folder. + + +## Uploading to PyPi + +To upload a wheel to test PyPi, + + twine upload --repository testpypi -u $USERNAME -p $PASSWORD + +Note that you will need a TestPyPi login. Once it has been uploaded, you can +test (from anywhere, not just the container) that the wheel works by installing +it from the TestPyPi repository. + +``` +python3 -m pip install --index-url https://test.pypi.org/simple/ pycvc5== +python3 -c "import pycvc5; solver = pycvc5.Solver(); print(solver.getIntegerSort())" +``` + +You can remove the repository argument from the upload command to upload to +real PyPi. + +## What goes on within the Docker container + +### Setup + +Compared to `manylinux2014`, the container comes with `ccache` and some cvc5 +dependencies preinstalled. Additionally, we set up appropriate symlinks so that +`ccache` is used with the standard C and C++ compilers (`cc`, `c++`, `gcc`, `g++`). + +The idea is to mount a cvc5 checkout into the Docker container at `/home/pycvc5`. +This way, we don't need to care about cloning cvc5 within Docker and the `ccache` cache is automatically stored persistently outside of the container. + +### Building + +To build the wheel for a specific python version, simply run `mk_clean_wheel.sh` within the docker container, and pass it the python binary it shall use (`/opt/python/cp/bin/python`) and options to the cvc5 configure script (usually something like `production --auto-download`). +The script will create a new build folder (`build_wheel`), configure and build cvc5 accordingly and store the wheel within a `wheel-` folder. + +### Testing + +From within Docker, you can test a wheel as follows: to ensure that the `cvc5` +library is linked correctly (and not looking at a local path). To do this, +start from the top-level `cvc5` directory and run the following: +``` +# delete build directory to get rid of local library +rm -r build +# start virtualenv +source ./env/bin/activate +# install the wheel +pip install +# run a small example to make sure it works +python3 -c "import pycvc5; solver = pycvc5.Solver(); print(solver.getIntegerSort())" +``` + +### Test PyPi + +In addition to the local test described above, you can do a test upload to +[TestPyPi](https://packaging.python.org/guides/using-testpypi/). To do this +run the following after building the wheel: +``` +# start the virtualenv for this version (twine was installed here) +source ./env/bin/activate +# upload to Test PyPi +twine upload --repository testpypi +``` + +Note that you will need a TestPyPi login. Once it has been uploaded, you can +test (from anywhere, not just the container) that the wheel works by installing +it from the TestPyPi repository. + +``` +python3 -m pip install --index-url https://test.pypi.org/simple/ pycvc5== +python3 -c "import pycvc5; solver = pycvc5.Solver(); print(solver.getIntegerSort())" +``` + +### Upload to PyPi + +Once you are certain this wheel file is ready, you can upload it to PyPi. Note, +you will need a login for PyPi (this is separate from the TestPyPi login). The +steps are almost identical to the TestPyPi upload: +``` +# start virtualenv where twine was installed for this Python version +source ./env +# upload to PyPi with twine (be sure this ready!!) +twine upload +``` + +Once the `.whl` has been built, you should be able to upload it from anywhere. diff --git a/contrib/packaging_python/manylinux2014/Dockerfile b/contrib/packaging_python/manylinux2014/Dockerfile new file mode 100644 index 000000000..522f4f982 --- /dev/null +++ b/contrib/packaging_python/manylinux2014/Dockerfile @@ -0,0 +1,11 @@ +FROM quay.io/pypa/manylinux2014_x86_64 +WORKDIR /home/pycvc5 +RUN yum -q -y install ccache jre libffi-devel +RUN ccache --set-config=cache_dir=/home/pycvc5/ccache-dir + +ENV PATH="/usr/lib64/ccache:${PATH}" + +RUN ln -s /usr/bin/ccache /usr/lib64/ccache/g++ && \ + ln -s /usr/bin/ccache /usr/lib64/ccache/gcc && \ + ln -s /usr/bin/ccache /usr/lib64/ccache/c++ && \ + ln -s /usr/bin/ccache /usr/lib64/ccache/cc diff --git a/contrib/packaging_python/mk_build_dir.py b/contrib/packaging_python/mk_build_dir.py new file mode 100644 index 000000000..ae41ebb6c --- /dev/null +++ b/contrib/packaging_python/mk_build_dir.py @@ -0,0 +1,32 @@ +############################################################################### +# Top contributors (to current version): +# Gereon Kremer +# +# This file is part of the cvc5 project. +# +# Copyright (c) 2009-2021 by the authors listed in the file AUTHORS +# in the top-level source directory and their institutional affiliations. +# All rights reserved. See the file COPYING in the top-level source +# directory for licensing information. +# ############################################################################ +# +# Run configure.sh and make cmake pick up whatever python interpreter this +# script is run with. Additionally, all arguments passed to this script are +# forwarded as well. +## + +import subprocess +import sys + +from skbuild.cmaker import CMaker + +python_version = CMaker.get_python_version() +args = [ + '-DPYTHON_VERSION_STRING:STRING=' + sys.version.split(' ')[0], + '-DPYTHON_INCLUDE_DIR:PATH=' + + CMaker.get_python_include_dir(python_version), + '-DPYTHON_LIBRARY:FILEPATH=' + + CMaker.get_python_library(python_version), +] + +subprocess.check_call(['./configure.sh', *sys.argv[1:], *args]) diff --git a/contrib/packaging_python/mk_clean_wheel.sh b/contrib/packaging_python/mk_clean_wheel.sh new file mode 100755 index 000000000..4b19e1146 --- /dev/null +++ b/contrib/packaging_python/mk_clean_wheel.sh @@ -0,0 +1,69 @@ +#!/bin/bash +############################################################################### +# Top contributors (to current version): +# Makai Mann, Alex Ozdemir, Gereon Kremer +# +# This file is part of the cvc5 project. +# +# Copyright (c) 2009-2021 by the authors listed in the file AUTHORS +# in the top-level source directory and their institutional affiliations. +# All rights reserved. See the file COPYING in the top-level source +# directory for licensing information. +# ############################################################################ +# +# Build the python wheel for cvc5 with a given python interpreter and additional +# configuration arguments in a clean build folder. +# Call as follows: +# contrib/packaging_python/mk_clean_wheel.sh /usr/bin/python3 "--cadical" +# +# The script first sets up a python venv with the given interpreter and installs +# some requirements in this venv. It then uses the mk_build_dir.py script to +# prepare a fresh build folder build_wheel/ and builds the wheel in there using +# mk_wheel.py. The wheel is fixes (using auditwheel or delocate-wheel) and +# copied out of the build folder. +## + +PYTHONBIN=$1 +CONFIG="$2" +PYVERSION=$($PYTHONBIN -c "import sys; print(sys.implementation.name + sys.version.split()[0])") + +# setup and activate venv +echo "Making venv with $PYTHONBIN" +ENVDIR=env$PYVERSION +$PYTHONBIN -m venv ./$ENVDIR +source ./$ENVDIR/bin/activate + +# install packages +pip install -q --upgrade pip setuptools auditwheel +pip install -q Cython pytest toml scikit-build +if [ "$(uname)" == "Darwin" ]; then + # Mac version of auditwheel + pip install -q delocate +fi + +# configure cvc5 +echo "Configuring" +rm -rf build_wheel/ +python contrib/packaging_python/mk_build_dir.py $CONFIG --python-bindings --name=build_wheel + +# building wheel +echo "Building pycvc5 wheel" + +pushd build_wheel +python ../contrib/packaging_python/mk_wheel.py bdist_wheel -d dist + +cd dist + +# resolve the links and bundle the library with auditwheel +if [ "$(uname)" == "Linux" ]; then + auditwheel show ./*.whl + auditwheel repair ./*.whl +elif [ "$(uname)" == "Darwin" ]; then + delocate-wheel -w wheelhouse ./*.whl +else + echo "Unhandled system $(uname) for packing libraries with wheel." +fi + +popd + +mv build_wheel/dist/wheelhouse/*.whl . diff --git a/contrib/packaging_python/mk_wheel.py b/contrib/packaging_python/mk_wheel.py new file mode 100644 index 000000000..297cca98d --- /dev/null +++ b/contrib/packaging_python/mk_wheel.py @@ -0,0 +1,121 @@ +############################################################################### +# Top contributors (to current version): +# Makai Mann, Gereon Kremer +# +# This file is part of the cvc5 project. +# +# Copyright (c) 2009-2021 by the authors listed in the file AUTHORS +# in the top-level source directory and their institutional affiliations. +# All rights reserved. See the file COPYING in the top-level source +# directory for licensing information. +# ############################################################################ +# +# Build script for python wheels for cvc5. +# It provides two main functions: +# - get_cvc5_version() uses the version.cmake script to obtain the proper cvc5 +# version string. It is extracted from the versioninfo.cpp file and passed on +# to the setuptools config. +# - build_extension() builds the cvc5 python api(s) using the cvc5_python_api +# target and copying it to wherever setuptools expects it to be. +# +# Example usage (from build directory): +# python3 ../contrib/packaging_python/mk_wheel.py bdist_wheel -d dist +## + +import os +import re +import platform +import subprocess +import shutil + +from setuptools import setup, Extension +from setuptools.command.build_ext import build_ext +from distutils.version import LooseVersion + + +def get_project_src_path(): + # expecting this script to be in contrib/packaging_python/ + # The project source directory is two directories up + name = __file__ + for i in range(3): + name = os.path.dirname(name) + name = os.path.abspath(name) + if not os.path.isfile(os.path.join(name, 'configure.sh')): + raise RuntimeError('Finding the project source path failed. We guessed ' + name) + return name + + +def get_cvc5_version(): + # run the version detection in cmake + subprocess.check_call([ + 'cmake', '-DPROJECT_SOURCE_DIR=' + get_project_src_path(), + '-DCMAKE_BINARY_DIR=.', '-P', + get_project_src_path() + '/cmake/version.cmake' + ]) + + # read versioninfo.cpp to get version number + with open(os.path.join('src', 'base', 'versioninfo.cpp'), 'r') as f: + m = re.search('CVC5_FULL_VERSION = "([0-9a-z.-]+)"', f.read()) + if m: + version = m.group(1) + version = re.sub('-dev\.([0-9]+)\.[0-9a-f]+', '.dev\\1', version) + return version + else: + raise Exception('Could not find version') + + +class CMakeExtension(Extension): + def __init__(self, name, sourcedir=''): + Extension.__init__(self, name, sources=[]) + self.sourcedir = os.path.abspath(sourcedir) + + +class CMakeBuild(build_ext): + def run(self): + try: + out = subprocess.check_output(['cmake', '--version']) + except OSError: + raise RuntimeError( + "CMake must be installed to build the following extensions: " + + ", ".join(e.name for e in self.extensions)) + + if self.is_windows(): + cmake_version = LooseVersion( + re.search(r'version\s*([\d.]+)', out.decode()).group(1)) + if cmake_version < '3.1.0': + raise RuntimeError("CMake >= 3.1.0 is required on Windows") + + for ext in self.extensions: + self.build_extension(ext) + + @staticmethod + def is_windows(): + tag = platform.system().lower() + return tag == "windows" + + def build_extension(self, ext): + # build the python api + subprocess.check_call(['cmake', '--build', '.', '--target', 'cvc5_python_api']) + + # copy the library over. we need to consider other users that are not on linux + # module is a directory called cvc5_python_base_module + extdir = os.path.abspath( + os.path.dirname(self.get_ext_fullpath(ext.name))) + cvc5_python_base_module = os.path.join("src", "api", "python", "cvc5") + dst_name = os.path.join(extdir, "cvc5") + + shutil.rmtree(dst_name, ignore_errors=True) + shutil.copytree(cvc5_python_base_module, dst_name) + + +setup( + name='cvc5', + version=get_cvc5_version(), + long_description='Python bindings for cvc5', + url='https://github.com/cvc5/cvc5', + license='BSD-3-Clause', + zip_safe=False, + ext_modules=[CMakeExtension('cvc5')], + cmdclass=dict(build_ext=CMakeBuild), + tests_require=['pytest'] +) diff --git a/src/api/python/wheels/build_wheel.py b/src/api/python/wheels/build_wheel.py deleted file mode 100644 index a47b2ac2c..000000000 --- a/src/api/python/wheels/build_wheel.py +++ /dev/null @@ -1,173 +0,0 @@ -############################################################################### -# Top contributors (to current version): -# Makai Mann -# -# This file is part of the cvc5 project. -# -# Copyright (c) 2009-2021 by the authors listed in the file AUTHORS -# in the top-level source directory and their institutional affiliations. -# All rights reserved. See the file COPYING in the top-level source -# directory for licensing information. -# ############################################################################ -# -# Script for building wheels distribution for cvc5 -# -# Example usage (from top-level directory): -# python3 ./src/api/python/wheels/build_wheel.py bdist_wheel -# Configures and builds in directory ./build -# Creates wheel in ./dist -# -# Note: takes an *optional* environment variable VERSION_SUFFIX. If set, this -# suffix will be appended to the pypi package version. -# Example: -# VERSION_SUFFIX=rc1 python3 ./src/api/python/wheels/build_wheel.py bdist_wheel -# would create versions X.Y.Zrc1 -# -# The suffix should start with a letter, and end with a number -## - -import os -import re -import sys -import platform -import string -import subprocess -import multiprocessing -import shutil - -from setuptools import setup, Extension -from setuptools.command.build_ext import build_ext -from skbuild.cmaker import CMaker -from distutils.version import LooseVersion - -WORKING_DIR = "build" - - -def get_project_src_path(): - # expecting this script to be in src/api/python/wheels - # The project source directory is 4 directories up - # (5 dirnames because start with file) - project_src_path = __file__ - for i in range(5): - project_src_path = os.path.dirname(project_src_path) - return os.path.abspath(project_src_path) - - -def get_cvc5_version(): - project_src_path = get_project_src_path() - - # read CMakeLists.txt to get version number - version = '' - str_pattern = 'set\(CVC5_LAST_RELEASE\s*"([^"]+)"\)' - pattern = re.compile(str_pattern) - with open(os.path.join(project_src_path, 'cmake', 'version-base.cmake'), 'r') as f: - m = pattern.search(f.read()) - if m: - version = m.group(1) - return version - else: - raise Exception('Could not find version') - - -class CMakeExtension(Extension): - def __init__(self, name, sourcedir=''): - Extension.__init__(self, name, sources=[]) - self.sourcedir = os.path.abspath(sourcedir) - - -class CMakeBuild(build_ext): - def run(self): - try: - out = subprocess.check_output(['cmake', '--version']) - except OSError: - raise RuntimeError( - "CMake must be installed to build the following extensions: " + - ", ".join(e.name for e in self.extensions)) - - if self.is_windows(): - cmake_version = LooseVersion( - re.search(r'version\s*([\d.]+)', out.decode()).group(1)) - if cmake_version < '3.1.0': - raise RuntimeError("CMake >= 3.1.0 is required on Windows") - - for ext in self.extensions: - self.build_extension(ext) - - @staticmethod - def is_windows(): - tag = platform.system().lower() - return tag == "windows" - - def build_extension(self, ext): - extdir = os.path.abspath( - os.path.dirname(self.get_ext_fullpath(ext.name))) - - cfg = 'Production' - build_args = ['--config', cfg] - - cpu_count = max(2, multiprocessing.cpu_count() // 2) - build_args += ['--', '-j{0}'.format(cpu_count)] - - project_src_path = get_project_src_path() - build_dir = os.path.join(project_src_path, WORKING_DIR) - - # configure with the working directory python-build-wheel - args = ['--python-bindings', - '--auto-download', - '--name='+WORKING_DIR] - # find correct Python include directory and library - # works even for nonstandard Python installations - # (e.g., on pypa/manylinux) - args.append('-DPYTHON_VERSION_STRING:STRING=' + - sys.version.split(' ')[0]) - python_version = CMaker.get_python_version() - args.append('-DPYTHON_INCLUDE_DIR:PATH=' + - CMaker.get_python_include_dir(python_version)) - args.append('-DPYTHON_LIBRARY:FILEPATH=' + - CMaker.get_python_library(python_version)) - - config_filename = os.path.join(project_src_path, "configure.sh") - subprocess.check_call([config_filename] + args) - - # build the main library - subprocess.check_call( - ['cmake', '--build', '.', "--target", "cvc5"] + build_args, cwd=build_dir) - - # build the python binding - python_build_dir = os.path.join(build_dir, "src", "api", "python") - subprocess.check_call(["make"], cwd=python_build_dir) - # the build folder gets cleaned during the config, need to create it again - # this is necessary since "build" is a python dist folder - if not os.path.isdir(extdir): - os.mkdir(extdir) - - # copy the library over. we need to consider other users that are not on linux - # module is a directory called cvc5_python_base_module - cvc5_python_base_module = os.path.join(python_build_dir, "cvc5") - dst_name = os.path.join(extdir, "cvc5") - if os.path.isdir(dst_name): - # remove, then replace - shutil.rmtree(dst_name) - - shutil.copytree(cvc5_python_base_module, dst_name) - - -version_suffix = os.getenv('VERSION_SUFFIX', '') -if len(version_suffix) > 0: - assert all(c in string.ascii_letters + string.digits for c in version_suffix) - assert version_suffix[0] in string.ascii_letters - assert version_suffix[-1] in string.ascii_digits - print("Setting version suffix to", version_suffix) - - -setup( - name='cvc5', - version=get_cvc5_version() + version_suffix, - long_description='Python bindings for cvc5', - url='https://github.com/cvc5/cvc5', - license='BSD-3-Clause', - zip_safe=False, - ext_modules=[CMakeExtension('cvc5')], - cmdclass=dict(build_ext=CMakeBuild), - tests_require=['pytest'] -)