Testing, building, and releasing

This section explains our testing, building, and release process.

Linting and testing

We lint with flake8. and run tests with nosetests.

We generally follow PEP 8, with a few exceptions which are listed in setup.cfg.

Tests are primarily set up via doctest, though a few are set up using unittest and live in lib5c.<some_module>.tests.test_<some_feature>.py.

Extra dependencies needed for linting and testing are listed in extras_require['test'] in setup.py. These include:

  • nose>=1.3.7
  • nose-exclude>=0.5.0
  • flake8>=3.4.1

To lint, run

$ flake8 lib5c

To run tests, run

$ nosetests

Our CI pipeline (run in Bitbucket Pipelines right inside the repo) installs the package (using the frozen package versions from requirements.txt) and runs these two commands on every commit to the repo.

Committing

The git repository for lib5c is https://bitbucket.org/creminslab/lib5c/.

We roughly try to follow GitHub Flow when committing, trying to never allow the master branch to contain breaking code.

Building

We build Python wheels and source distributions, as well as Docker images.

Versioning

Before building, you can optionally tag the release with a version number. We try to follow semantic versioning whenever possible. Our version tags do not include a leading “v” (e.g., we would use 0.5.3 as a tag rather than v0.5.3). To tag, run

$ git tag 0.5.1
$ git push --tags

If you don’t tag the release, versioneer will automatically fill in a PEP 440 compliant version for you.

Python wheel and source distribution

To build the wheel for lib5c, run

$ python setup.py bdist_wheel

To build the source distribution for lib5c, run

$ python setup.py sdist

Docker image

First, ensure you’ve built the Python wheel and that it exists as dist/lib5c-<version>-py2-none-any.whl.

To build the lib5c Docker images, we pass the version name into the Dockerfile as a build-arg.

We supply two images for each tag: one based on python:2.7 (about 1.4 GB total) and a second with a -slim suffix based on python:2.7-slim (about 700 MB total). The main Dockerfile is located in the root directory of the project and is recommended if you already use the python:2.7 base image anywhere else on your machine. The -slim Dockerfile is located in docker-slim/ and is recommended if you don’t plan to use python:2.7 as a base image for any other work on your machine.

To build, run

$ docker build --build-arg VERSION=$(python lib5c/_version.py) -t creminslab/lib5c:$(git describe --tags --dirty --always) .
$ docker build --build-arg VERSION=$(python lib5c/_version.py) -t creminslab/lib5c:$(git describe --tags --dirty --always)-slim -f docker-slim/Dockerfile .

We set the latest tag to point to the latest normal (non-slim) image, and the slim tag to point to the latest slim image:

$ docker tag creminslab/lib5c:$(git describe --tags --dirty --always) creminslab/lib5c:latest
$ docker tag creminslab/lib5c:$(git describe --tags --dirty --always)-slim creminslab/lib5c:slim

Note that we tag the Docker image with the direct output of git describe, since Docker image tags can’t contain plus signs.

Releasing

We release wheel and source distributions to PyPI and the Docker images to Docker Hub.

PyPI

To build both the wheel and source distribution and upload them to PyPI, run

$ python setup.py sdist bdist_wheel upload

Docker Hub

To push the Docker images to Docker Hub, run

$ docker push creminslab/lib5c:$(git describe --tags --dirty --always)
$ docker push creminslab/lib5c:$(git describe --tags --dirty --always)-slim
$ docker push creminslab/lib5c:latest
$ docker push creminslab/lib5c:slim

Documentation

Docs are built using Sphinx.

Extra dependencies needed for building docs are listed in extras_require['docs'] in setup.py. These include:

  • Sphinx>=1.7.2
  • sphinx-rtd-theme>=0.3.0
  • mock>=3.0.5

We also require Pandoc.

Documentation pages are stored in docs/.

The documentation is built by running

$ sphinx-apidoc -o docs/ -f -T -e lib5c/ "**/tests" "**/*legacy*.py"
$ sphinx-build -b dirhtml docs/ docs_build/

When new modules are added to the lib5c library, sphinx-apidoc needs to be manually re-run to generate autodoc .rst files for the new modules. These files should be checked into git.

The sphinx-build step gets run automatically on every commit to dev, publishing the results to https://lib5c.readthedocs.io/en/latest/.

Tagged versions can be added to the list of stable versions via the readthedocs website. The latest tagged version on master will be published to https://lib5c.readthedocs.io/en/stable/.

Tutorials

Tutorial source notebooks (containing no outputs) are stored in the tutorials/ directory under the project root.

Extra dependencies needed for running the tutorials are listed in extras_require['tutorials'] in setup.py. These include:

  • ipykernel>=4.10.0,<6.0
  • nbconvert>=5.4.0
  • nbformat>=4.4.0
  • nbstripout>=0.3.5

The tutorials are run by Bitbucket Pipelines but need to be triggered manually, which results in the creation of new notebooks that include outputs. These resulting notebooks are pushed to a a separate repository

The tutorials can be run programmatically by running:

$ python tutorials/run.py

To strip outputs from the notebooks before committing them back to the lib5c repo, run:

$ python tutorials/clean.py

Cheat sheet

The commands explained above are collected all in one place in the cheat sheet below:

# test (check extras_require['test'] in setup.py for requirements)
flake8 lib5c
nosetests

# git commit
git commit -m 'commit message'
git push

# git tag (optional)
git tag 0.5.1
git push --tags

# build wheel
python setup.py bdist_wheel

# build and upload to pypi
python setup.py sdist bdist_wheel upload

# local Docker build and tag
docker build --build-arg VERSION=$(python lib5c/_version.py) -t creminslab/lib5c:$(git describe --tags --dirty --always) .
docker build --build-arg VERSION=$(python lib5c/_version.py) -t creminslab/lib5c:$(git describe --tags --dirty --always)-slim -f docker-slim/Dockerfile .
docker tag creminslab/lib5c:$(git describe --tags --dirty --always) creminslab/lib5c:latest
docker tag creminslab/lib5c:$(git describe --tags --dirty --always)-slim creminslab/lib5c:slim

# push Docker images
docker push creminslab/lib5c:$(git describe --tags --dirty --always)
docker push creminslab/lib5c:$(git describe --tags --dirty --always)-slim
docker push creminslab/lib5c:latest
docker push creminslab/lib5c:slim

# build docs
sphinx-apidoc -o docs/ -f -T -e lib5c/ "**/tests" "**/*legacy*.py"
sphinx-build -b dirhtml docs/ docs_build/

# run tutorials
python tutorials/run.py

# cleanup tutorials
python tutorials/clean.py