Testing, building, and releasing¶
This section explains our testing, building, and release process.
Linting and testing¶
We lint with flake8. and run tests with nosetests.
We generally follow PEP 8, with a few
exceptions which are listed in setup.cfg
.
Tests are primarily set up via doctest,
though a few are set up using unittest
and live in lib5c.<some_module>.tests.test_<some_feature>.py
.
Extra dependencies needed for linting and testing are listed in
extras_require['test']
in setup.py
. These include:
nose>=1.3.7
nose-exclude>=0.5.0
flake8>=3.4.1
To lint, run
$ flake8 lib5c
To run tests, run
$ nosetests
Our CI pipeline (run in Bitbucket Pipelines right inside the repo) installs the
package (using the frozen package versions from requirements.txt
) and runs
these two commands on every commit to the repo.
Committing¶
The git repository for lib5c
is https://bitbucket.org/creminslab/lib5c/.
We roughly try to follow GitHub Flow
when committing, trying to never allow the master
branch to contain breaking
code.
Building¶
We build Python wheels and source distributions, as well as Docker images.
Versioning¶
Before building, you can optionally tag the release with a version number. We
try to follow semantic versioning whenever possible.
Our version tags do not include a leading “v” (e.g., we would use 0.5.3
as a
tag rather than v0.5.3
). To tag, run
$ git tag 0.5.1
$ git push --tags
If you don’t tag the release, versioneer will automatically fill in a PEP 440 compliant version for you.
Python wheel and source distribution¶
To build the wheel for lib5c
, run
$ python setup.py bdist_wheel
To build the source distribution for lib5c
, run
$ python setup.py sdist
Docker image¶
First, ensure you’ve built the Python wheel and that it exists as
dist/lib5c-<version>-py2-none-any.whl
.
To build the lib5c
Docker images, we pass the version name into the
Dockerfile
as a build-arg
.
We supply two images for each tag: one based on python:2.7
(about 1.4 GB
total) and a second with a -slim
suffix based on python:2.7-slim
(about
700 MB total). The main Dockerfile is located in the root directory of the
project and is recommended if you already use the python:2.7
base image
anywhere else on your machine. The -slim
Dockerfile is located in
docker-slim/
and is recommended if you don’t plan to use python:2.7
as a
base image for any other work on your machine.
To build, run
$ docker build --build-arg VERSION=$(python lib5c/_version.py) -t creminslab/lib5c:$(git describe --tags --dirty --always) .
$ docker build --build-arg VERSION=$(python lib5c/_version.py) -t creminslab/lib5c:$(git describe --tags --dirty --always)-slim -f docker-slim/Dockerfile .
We set the latest
tag to point to the latest normal (non-slim) image, and
the slim
tag to point to the latest slim image:
$ docker tag creminslab/lib5c:$(git describe --tags --dirty --always) creminslab/lib5c:latest
$ docker tag creminslab/lib5c:$(git describe --tags --dirty --always)-slim creminslab/lib5c:slim
Note that we tag the Docker image with the direct output of git describe
,
since Docker image tags can’t contain plus signs.
Releasing¶
We release wheel and source distributions to PyPI and the Docker images to Docker Hub.
PyPI¶
To build both the wheel and source distribution and upload them to PyPI, run
$ python setup.py sdist bdist_wheel upload
Docker Hub¶
To push the Docker images to Docker Hub, run
$ docker push creminslab/lib5c:$(git describe --tags --dirty --always)
$ docker push creminslab/lib5c:$(git describe --tags --dirty --always)-slim
$ docker push creminslab/lib5c:latest
$ docker push creminslab/lib5c:slim
Documentation¶
Docs are built using Sphinx.
Extra dependencies needed for building docs are listed in
extras_require['docs']
in setup.py
. These include:
Sphinx>=1.7.2
sphinx-rtd-theme>=0.3.0
mock>=3.0.5
We also require Pandoc.
Documentation pages are stored in docs/
.
The documentation is built by running
$ sphinx-apidoc -o docs/ -f -T -e lib5c/ "**/tests" "**/*legacy*.py"
$ sphinx-build -b dirhtml docs/ docs_build/
When new modules are added to the lib5c
library, sphinx-apidoc
needs to
be manually re-run to generate autodoc .rst files for the new modules. These
files should be checked into git.
The sphinx-build
step gets run automatically on every commit to dev
,
publishing the results to https://lib5c.readthedocs.io/en/latest/.
Tagged versions can be added to the list of stable versions via the readthedocs
website. The latest tagged version on master
will be published to
https://lib5c.readthedocs.io/en/stable/.
Tutorials¶
Tutorial source notebooks (containing no outputs) are stored in the
tutorials/
directory under the project root.
Extra dependencies needed for running the tutorials are listed in
extras_require['tutorials']
in setup.py
. These include:
ipykernel>=4.10.0,<6.0
nbconvert>=5.4.0
nbformat>=4.4.0
nbstripout>=0.3.5
The tutorials are run by Bitbucket Pipelines but need to be triggered manually, which results in the creation of new notebooks that include outputs. These resulting notebooks are pushed to a a separate repository
The tutorials can be run programmatically by running:
$ python tutorials/run.py
To strip outputs from the notebooks before committing them back to the lib5c repo, run:
$ python tutorials/clean.py
Cheat sheet¶
The commands explained above are collected all in one place in the cheat sheet below:
# test (check extras_require['test'] in setup.py for requirements)
flake8 lib5c
nosetests
# git commit
git commit -m 'commit message'
git push
# git tag (optional)
git tag 0.5.1
git push --tags
# build wheel
python setup.py bdist_wheel
# build and upload to pypi
python setup.py sdist bdist_wheel upload
# local Docker build and tag
docker build --build-arg VERSION=$(python lib5c/_version.py) -t creminslab/lib5c:$(git describe --tags --dirty --always) .
docker build --build-arg VERSION=$(python lib5c/_version.py) -t creminslab/lib5c:$(git describe --tags --dirty --always)-slim -f docker-slim/Dockerfile .
docker tag creminslab/lib5c:$(git describe --tags --dirty --always) creminslab/lib5c:latest
docker tag creminslab/lib5c:$(git describe --tags --dirty --always)-slim creminslab/lib5c:slim
# push Docker images
docker push creminslab/lib5c:$(git describe --tags --dirty --always)
docker push creminslab/lib5c:$(git describe --tags --dirty --always)-slim
docker push creminslab/lib5c:latest
docker push creminslab/lib5c:slim
# build docs
sphinx-apidoc -o docs/ -f -T -e lib5c/ "**/tests" "**/*legacy*.py"
sphinx-build -b dirhtml docs/ docs_build/
# run tutorials
python tutorials/run.py
# cleanup tutorials
python tutorials/clean.py