Testing, building, and releasing¶

This section explains our testing, building, and release process.

Maintenance automation¶

We use tox to orchestrate our maintenance tasks. To use tox, all you need to do is install it (pip install tox). All other dependencies will be managed by tox.

Linting and testing¶

To run linting and testing across all supported Python versions, run

$ tox

To lint, run

$ tox -e lint

To run tests for a specific Python version (e.g., 2.7), run

$ tox -e py27-unpinned

We lint with flake8. and run tests with nosetests.

We generally follow PEP 8, with a few exceptions which are listed in setup.cfg.

Tests are primarily set up via doctest, though a few are set up using unittest and live in lib5c.<some_module>.tests.test_<some_feature>.py.

Our CI pipeline (run in Bitbucket Pipelines right inside the repo) runs tox under all our test environments for every commit.

We pin one specific set of dependency versions in requirements.txt. This is the set of pinned versions used for our Docker image builds as well as special pyXX-pinned test environments in tox. The pyXX-unpinned test environments do not pin any package versions; therefore, they closely resemble the experience of a new user attempting to run pip install lib5c.

The pyXX-pinned test environments are skipped on Windows, since our pinned package versions in requirements.txt include some packages which are not installable on Windows.

Committing¶

The git repository for lib5c is https://bitbucket.org/creminslab/lib5c/.

We roughly try to follow GitHub Flow when committing, trying to never allow the master branch to contain breaking code.

Building¶

We build Python wheels and source distributions, as well as Docker images.

Versioning¶

Before building, you can optionally tag the release with a version number. We try to follow semantic versioning whenever possible. Our version tags do not include a leading “v” (e.g., we would use 0.5.3 as a tag rather than v0.5.3). To tag, run

$ git tag 0.5.1
$ git push --tags

We use setuptools-scm to obtain information about the current version directly from git.

Python wheel and source distribution¶

To build the wheel for lib5c, run

$ python setup.py bdist_wheel

To build the source distribution for lib5c, run

$ python setup.py sdist

Docker image¶

To build both normal and slim docker images, run

$ tox -e docker build

To promote the current images (normal and slim) to the latest and slim tags, respectively, run

$ tox -e docker promote

To build the lib5c Docker images, we pass the version name into the Dockerfile as a build-arg called VERSION. To get the version in a cross-platform way, we provide a utility script lib5c/_version.py which, when run, simply prints the current version. Running this script requires that either lib5c or setuptools-scm (pip install setuptools_scm) be installed. setuptools-scm is likely to be the easier option since it is much lighter to install (it has no dependencies), but developers who have already installed lib5c (e.g., in dev mode) do not need to install setuptools-scm. The “docker” tox testenv installs setuptools-scm in its own isolated environment.

We supply two images for each tag: one based on python:2.7 (about 1.3 GB total) and a second with a -slim suffix based on python:2.7-slim (about 600 MB total). The main Dockerfile is located in the root directory of the project and is recommended if you already use the python:2.7 base image anywhere else on your machine. The -slim Dockerfile is located in docker-slim/ and is recommended if you don’t plan to use python:2.7 as a base image for any other work on your machine.

At the current time, we only supply these Python 2.7 based images. In the future, we may also supply Python 3 based images.

Note that we tag the Docker image with the direct output of git describe, since Docker image tags can’t contain plus signs.

Releasing¶

We release wheel and source distributions to PyPI and the Docker images to Docker Hub.

PyPI¶

To build both the wheel and source distribution and upload them to PyPI, run

$ python setup.py sdist bdist_wheel upload

Docker Hub¶

To push the Docker images to Docker Hub, run

$ tox -e docker push
$ tox -e docker pushlatest  # pushes the latest and slim tags

These commands assume you’ve logged in with Docker by running docker login.

Documentation¶

To build docs, run

$ tox -e docs

Documentation pages are stored in docs/ and are built using Sphinx.

We use the sphinxcontrib.apidoc extension to run sphinx-apidoc on every doc build. This means that the apidoc-generated lib5c*.rst files should not be checked into git. The tox testenv deletes these files for you after building the docs.

Docs are built automatically on every commit to dev, publishing the results to https://lib5c.readthedocs.io/en/latest/.

Tagged versions can be added to the list of stable versions via the readthedocs website. The latest tagged version on master will be published to https://lib5c.readthedocs.io/en/stable/.

Tutorials¶

To test the tutorials, run

$ tox -e tutorials

Tutorial source notebooks (containing no outputs) are stored in the tutorials/ directory under the project root.

The tutorials are run by Bitbucket Pipelines but need to be triggered manually, which results in the creation of new notebooks that include outputs. These resulting notebooks are pushed to a separate repository.

To strip outputs from the notebooks before committing them back to the lib5c repo, run:

$ python tutorials/clean.py

Cheat sheet¶

The commands explained above are collected all in one place in the cheat sheet below:

# lint and test
tox

# git commit
git commit -m 'commit message'
git push

# git tag (optional)
git tag 0.5.1
git push --tags

# build wheel
python setup.py bdist_wheel

# build and upload to pypi
python setup.py sdist bdist_wheel upload

# local Docker build and tag
tox -e docker build
tox -e docker promote

# push Docker images
tox -e docker push
tox -e docker pushlatest

# build docs
tox -e docs

# run tutorials
tox -e tutorials

# cleanup tutorials
python tutorials/clean.py