Testing, building, and releasing¶
This section explains our testing, building, and release process.
We use tox to orchestrate our maintenance
tasks. To use tox, all you need to do is install it (
pip install tox).
All other dependencies will be managed by tox.
Linting and testing¶
To run linting and testing across all supported Python versions, run
To lint, run
$ tox -e lint
To run tests for a specific Python version (e.g., 2.7), run
$ tox -e py27-unpinned
We generally follow PEP 8, with a
few exceptions which are listed in
Our CI pipeline (run in Bitbucket Pipelines right inside the repo) runs tox under all our test environments for every commit.
We pin one specific set of dependency versions in
requirements.txt. This is
the set of pinned versions used for our Docker image builds as well as special
pyXX-pinned test environments in tox. The
environments do not pin any package versions; therefore, they closely resemble
the experience of a new user attempting to run
pip install lib5c.
pyXX-pinned test environments are skipped on Windows, since our pinned
package versions in
requirements.txt include some packages which are not
installable on Windows.
The git repository for
lib5c is https://bitbucket.org/creminslab/lib5c/.
We roughly try to follow GitHub Flow
when committing, trying to never allow the
master branch to contain breaking
We build Python wheels and source distributions, as well as Docker images.
Before building, you can optionally tag the release with a version number. We
try to follow semantic versioning whenever possible.
Our version tags do not include a leading “v” (e.g., we would use
0.5.3 as a
tag rather than
v0.5.3). To tag, run
$ git tag 0.5.1 $ git push --tags
We use setuptools-scm to obtain information about the current version directly from git.
Python wheel and source distribution¶
To build the wheel for
$ python setup.py bdist_wheel
To build the source distribution for
$ python setup.py sdist
To build both normal and slim docker images, run
$ tox -e docker build
To promote the current images (normal and slim) to the
tags, respectively, run
$ tox -e docker promote
To build the
lib5c Docker images, we pass the version name into the
Dockerfile as a
VERSION. To get the version in a
cross-platform way, we provide a utility script
when run, simply prints the current version. Running this script requires that
either lib5c or setuptools-scm
pip install setuptools_scm) be installed. setuptools-scm is likely to be
the easier option since it is much lighter to install (it has no dependencies),
but developers who have already installed lib5c (e.g., in dev mode) do not need
to install setuptools-scm. The “docker” tox testenv installs setuptools-scm in
its own isolated environment.
We supply two images for each tag: one based on
python:2.7 (about 1.3 GB
total) and a second with a
-slim suffix based on
600 MB total). The main Dockerfile is located in the root directory of the
project and is recommended if you already use the
python:2.7 base image
anywhere else on your machine. The
-slim Dockerfile is located in
docker-slim/ and is recommended if you don’t plan to use
python:2.7 as a
base image for any other work on your machine.
At the current time, we only supply these Python 2.7 based images. In the future, we may also supply Python 3 based images.
Note that we tag the Docker image with the direct output of
since Docker image tags can’t contain plus signs.
To build both the wheel and source distribution and upload them to PyPI, run
$ python setup.py sdist bdist_wheel upload
To push the Docker images to Docker Hub, run
$ tox -e docker push $ tox -e docker pushlatest # pushes the latest and slim tags
These commands assume you’ve logged in with Docker by running
To build docs, run
$ tox -e docs
Documentation pages are stored in
docs/ and are built using Sphinx.
We use the sphinxcontrib.apidoc
extension to run
sphinx-apidoc on every doc build. This means that the
lib5c*.rst files should not be checked into git. The tox
testenv deletes these files for you after building the docs.
Docs are built automatically on every commit to
dev, publishing the results
Tagged versions can be added to the list of stable versions via the readthedocs
website. The latest tagged version on
master will be published to
To test the tutorials, run
$ tox -e tutorials
Tutorial source notebooks (containing no outputs) are stored in the
tutorials/ directory under the project root.
The tutorials are run by Bitbucket Pipelines but need to be triggered manually, which results in the creation of new notebooks that include outputs. These resulting notebooks are pushed to a separate repository.
To strip outputs from the notebooks before committing them back to the lib5c repo, run:
$ python tutorials/clean.py
The commands explained above are collected all in one place in the cheat sheet below:
# lint and test tox # git commit git commit -m 'commit message' git push # git tag (optional) git tag 0.5.1 git push --tags # build wheel python setup.py bdist_wheel # build and upload to pypi python setup.py sdist bdist_wheel upload # local Docker build and tag tox -e docker build tox -e docker promote # push Docker images tox -e docker push tox -e docker pushlatest # build docs tox -e docs # run tutorials tox -e tutorials # cleanup tutorials python tutorials/clean.py