Workflows (Nox and Poetry)

Following sections provide information on how to use the excellent Hypermodern-Python project foundation proposed by Claudio Jolowicz

Contents

Documentation
Testing
Committing
Realeasing and Publishing
- Latest deveolpment publishes on TestPyPI
- Stable releases and publishes on PyPI
Managing Dependencies

Documentation 

This project uses Sphinx relying on docstrings in NumPy Style which get inforced by flake8-docstrings and darglint. Use Nox to conveniently build the documentaiton inside the docs/_build folder:

To tweak or add nox sessions, alter the noxfile.py inside this project’s root directory.

nox -s docs – Statically build whats new 

Build the documentation while only acutally rebuilding those files that changed:

nox -s docs

nox -s docs_rebuild – Statically build everything from scratch 

Rebuild the entire documentation from scratch:

nox -s docs_rebuild

nox -s docs_live – Dynamically build from scratch (once)

Builts the documentation from scratch, servs it locally on port 8000, opens your default browser on the main page (docs/_build/index.html) and rebuilts any pages live, that are subject to change (when saved to disk).

Invaluable when creating the documentation!

nox -s docs_live

Testing 

This project uses Nox to conveniently run both:

unittests via the defacto standard pytest

doctests via xdoctest

To tweak or add nox testing sessions, alter the noxfile.py inside this project’s root directory.

nox -s tests – Unittests 

Unittests reside in tests/ inside the root directory of this project. Make sure to provide docstrings (since they are enforced, heh!) and add new test modules to docs/source/unittests.rst.

Run all unittests using nox:

nox -s tests

nox -r tests -- -m MARKER – Run specificly marked tests, excluded by default 

Unittests can be marked by adding a @pytest.mark.MARKER decorator as for example in tests/test_connectivity.py:

@pytest.mark.con
def test_wikipedia_connectivity(request_random_wiki_article):
  """Try reaching the wikipedia site to get a random article."""
  answer = request_random_wiki_article
  print(answer)
  assert "Error" not in answer

These markers can be explicitly run by passsing the -m MARKER option to the nox session as in:

nox -s tests -- -m MARKER

This templates supports following markers by default:

con – Marks interernet connection attempts

e2e – Marks end 2 end tests

slow - Marks particularly slow tests

These markers are excluded from the default nox -s test session (which also gets invoked by just calling nox). These are thus also excluded from the Tests CI-Workflow in .github/workflows/tests.yml. To modify this behavior or exclude additional markers modify the "not e2e and not con and not slow", line inside the noxfile.py:

@nox.session(python="3.10")
def tests(session):
    """Run test suite."""
    args = session.posargs or [
        "--cov",
        "-m",
        "not e2e and not con and not slow",
        # append exlcuded markers as "and not ..."
    ]
    session.run("poetry", "install", "--no-dev", external=True)
    install_with_constraints(
        session,
        "coverage[toml]",
        "pytest",
        "pytest-cov",
        "pytest-mock",
    )
    session.run("pytest", *args)

So to test one of them run e.g.:

nox -s tests ---m con

nox -s xdoctests – Doctests 

Me personally, I love doctests. I thinks they are the most natural form of testing. Since archiev both with them: enforced tests and pretty, copy-pastable examples inside your documentation.

Run all doctests using nox:

nox -s xdoctests

Committing 

After new code is added and all tests are passed, following is the usua workflow:

Run Black to format your code nox -s black
Stage your changes using git add
Run the pre-commit session to test lint and format your package using nox -s pre-commit
Stage again to reflect changes done by pre-commit git add
Commit your changes using git commit -m "MY MESSAGE"

Realeasing and Publishing 

This project template provides two major forms of automated publishing

Development ‘release’ publishes on TestPyPI

Stable release publishes on PyPI

Latest deveolpment publishes on TestPyPI 

Pseudo release a (potentially unstable) development version of your package by Pushing or Merging a Pull-Request to your remote develop branch. This automatically triggers the TestPyPI Workflow in .github/workflows/test-pypi, which publishes a development version on TestPyPI.

To enable your repo interacting with your TestPyPI account you need to create an API-Token named TEST_PYPI_TOKEN in your TestPyPI account settings and declare it a Secret in your remote Github repo.

Assuming you’ve successfully generated and declared your Secret TestPyPI Api-Token, following workflow is proposed for creating a new (unstable) development release:

Add all changes to your local develop branch

Run the full test and lint suite using nox.

Commit and Push your changes to the remote develop branch.

The TestPyPI Workflow in .github/workflows/test-pypi.yml automatically publishes the package using Poetry using a dev versioning scheme.

Stable releases and publishes on PyPI 

Release a stable version of your package by creating a Release of your main/ master branch via the Github website. This triggers the github Workflow called PyPI residing in .github/workflows/pypi.yml, which automatically creates a release on PyPI.

To enable your repo interacting with your PyPI account you need to create an API-Token named PYPI_TOKEN in your PyPI account settings and declare it a Secret in your remote Github repo.

Assuming you’ve successfully generated and declared your Secret PyPI Api-Token, following workflow is proposed for creating a new release:

Bump the package version on your local develop branch using poetry version major|minor|patch| following the Semantic-Versioning.

Run the full test and lint suite using nox.

Commit and Push your changes to the remote develop branch.

Create a Pull-Request from your remote develop branch to the remote main / master branch via your remote repo’s github webpage.

Merge the Pull-Request on your remote repo using the github webpage

Create a Release using the remote repos webpage.

Note that the Release Drafter Workflow in .github/workflows/release-drafter.yml automatically creates a release draft listing all your changes.

The PyPI Workflow in .github/workflows/pypi.yml automatically publishes the package using Poetry

Managing Dependencies 

Project dependencies are managed using Poetry.

Adding 

Adding third party dependencies is done by using the poetry add command.

poetry add PACKAGE – Adding required dependencies 

Add a required third party package to your package by using poetry:

poetry add PACKAGE

poetry add --dev PACKAGE ^^ Adding additional developer dependencies 

Add additional developer dependencies by using one of the following poetry commands:

poetry add --dev PACKAGE

poetry add package^1.0
poetry add "package>=1.0"
poetry add strutils@latest
poetry add git+https://github.com/tZ3ma/strutils.git
poetry add git+https://github.com/tZ3ma/strutils.git#develop
poetry add ./my-package/

Adding local dependencies in editable mode 

Modify the pyproject.toml file inside this project’s root directory:

[tool.poetry.dependencies]
my-package = {path = "../my/path", develop = true}

Adding extras/optional dependencies 

If the package(s) you want to install provide extras, you can specify them when adding the package by using one of the following lines:

poetry add requests[security,socks]
poetry add "requests[security,socks]~=2.22.0"
poetry add "git+https://github.com/pallets/flask.git@1.1.1[dotenv,dev]"

Updating 

Updating third party dependencies is done by using the poetry add command.

nox update – Updating all dependencies 

Update all project dependencies by using:

poetry update

nox update package1 package 2 – Updating explicit dependencies 

Update specific dependencies by using:

poetry update package1 pakage2

Versioning 

Bumping your package’s verion is done by using the poetry version semver command. Where semver is one of poetry’s supported Semantic-Versioning specifiers.

poetry version – Bump yout package version 

To bump your package’s version use one of the following poetry commands:

poetry add patch
poetry add minor
poetry add major
poetry add prepatch
poetry add preminor
poetry add premajor
poetry add prerelease

Removing 

Removing third party dependencies is done by using the poetry remove command.

poetry remove – Remove third party dependencies 

Remove a required third party package from your package by using poetry:

poetry remove PACKAGE