Testing, building, and releasing¶
This section explains our testing, building, and release process.
Maintenance automation¶
We use tox to orchestrate our maintenance
tasks. To use tox, all you need to do is install it (pip install tox
).
All other dependencies will be managed by tox.
Linting and testing¶
To run linting and testing across all supported Python versions, run
$ tox
To lint, run
$ tox -e lint
To run tests for a specific Python version (e.g., 2.7), run
$ tox -e py27-unpinned
We lint with flake8. and run tests with nosetests.
We generally follow PEP 8, with a
few exceptions which are listed in setup.cfg
.
Tests are primarily set up via doctest,
though a few are set up using unittest
and live in lib5c.<some_module>.tests.test_<some_feature>.py
.
Our CI pipeline (run in Bitbucket Pipelines right inside the repo) runs tox under all our test environments for every commit.
We pin one specific set of dependency versions in requirements.txt
. This is
the set of pinned versions used for our Docker image builds as well as special
pyXX-pinned
test environments in tox. The pyXX-unpinned
test
environments do not pin any package versions; therefore, they closely resemble
the experience of a new user attempting to run pip install lib5c
.
The pyXX-pinned
test environments are skipped on Windows, since our pinned
package versions in requirements.txt
include some packages which are not
installable on Windows.
Committing¶
The git repository for lib5c
is https://bitbucket.org/creminslab/lib5c/.
We roughly try to follow GitHub Flow
when committing, trying to never allow the master
branch to contain breaking
code.
Building¶
We build Python wheels and source distributions, as well as Docker images.
Versioning¶
Before building, you can optionally tag the release with a version number. We
try to follow semantic versioning whenever possible.
Our version tags do not include a leading “v” (e.g., we would use 0.5.3
as a
tag rather than v0.5.3
). To tag, run
$ git tag 0.5.1
$ git push --tags
We use setuptools-scm to obtain information about the current version directly from git.
Python wheel and source distribution¶
To build the wheel for lib5c
, run
$ python setup.py bdist_wheel
To build the source distribution for lib5c
, run
$ python setup.py sdist
Docker image¶
To build both normal and slim docker images, run
$ tox -e docker build
To promote the current images (normal and slim) to the latest
and slim
tags, respectively, run
$ tox -e docker promote
To build the lib5c
Docker images, we pass the version name into the
Dockerfile
as a build-arg
called VERSION
. To get the version in a
cross-platform way, we provide a utility script lib5c/_version.py
which,
when run, simply prints the current version. Running this script requires that
either lib5c or setuptools-scm
(pip install setuptools_scm
) be installed. setuptools-scm is likely to be
the easier option since it is much lighter to install (it has no dependencies),
but developers who have already installed lib5c (e.g., in dev mode) do not need
to install setuptools-scm. The “docker” tox testenv installs setuptools-scm in
its own isolated environment.
We supply two images for each tag: one based on python:2.7
(about 1.3 GB
total) and a second with a -slim
suffix based on python:2.7-slim
(about
600 MB total). The main Dockerfile is located in the root directory of the
project and is recommended if you already use the python:2.7
base image
anywhere else on your machine. The -slim
Dockerfile is located in
docker-slim/
and is recommended if you don’t plan to use python:2.7
as a
base image for any other work on your machine.
At the current time, we only supply these Python 2.7 based images. In the future, we may also supply Python 3 based images.
Note that we tag the Docker image with the direct output of git describe
,
since Docker image tags can’t contain plus signs.
Releasing¶
We release wheel and source distributions to PyPI and the Docker images to Docker Hub.
PyPI¶
To build both the wheel and source distribution and upload them to PyPI, run
$ python setup.py sdist bdist_wheel upload
Docker Hub¶
To push the Docker images to Docker Hub, run
$ tox -e docker push
$ tox -e docker pushlatest # pushes the latest and slim tags
These commands assume you’ve logged in with Docker by running docker login
.
Documentation¶
To build docs, run
$ tox -e docs
Documentation pages are stored in docs/
and are built using Sphinx.
We use the sphinxcontrib.apidoc
extension to run sphinx-apidoc
on every doc build. This means that the
apidoc-generated lib5c*.rst
files should not be checked into git. The tox
testenv deletes these files for you after building the docs.
Docs are built automatically on every commit to dev
, publishing the results
to https://lib5c.readthedocs.io/en/latest/.
Tagged versions can be added to the list of stable versions via the readthedocs
website. The latest tagged version on master
will be published to
https://lib5c.readthedocs.io/en/stable/.
Tutorials¶
To test the tutorials, run
$ tox -e tutorials
Tutorial source notebooks (containing no outputs) are stored in the
tutorials/
directory under the project root.
The tutorials are run by Bitbucket Pipelines but need to be triggered manually, which results in the creation of new notebooks that include outputs. These resulting notebooks are pushed to a separate repository.
To strip outputs from the notebooks before committing them back to the lib5c repo, run:
$ python tutorials/clean.py
Cheat sheet¶
The commands explained above are collected all in one place in the cheat sheet below:
# lint and test
tox
# git commit
git commit -m 'commit message'
git push
# git tag (optional)
git tag 0.5.1
git push --tags
# build wheel
python setup.py bdist_wheel
# build and upload to pypi
python setup.py sdist bdist_wheel upload
# local Docker build and tag
tox -e docker build
tox -e docker promote
# push Docker images
tox -e docker push
tox -e docker pushlatest
# build docs
tox -e docs
# run tutorials
tox -e tutorials
# cleanup tutorials
python tutorials/clean.py