Python pyproject.toml with Poetry adn a Gitlab build pipeline

Package a dist

At the simplest point we want some source code to get made into a package we can then distribute. 

flowchart LR
    source
    pyproject.toml --> poetry.lock
    poetry.lock --> libs
    source ----> dist[dist\nwhl - 'compiled'\ntar.gz - source]
    libs --> dist
    meta[Other assets\nREADME.md\nfiles, images\nicons etc.] ----> dist

Simple layout:

.
├── ./dist
├── ./.env
├── ./.flake8
├── ./kmpycore
│   └── ./kmpycore/__init__.py
├── ./Makefile
├── ./pip-requirements.txt
├── ./poetry.lock
├── ./pyproject.toml
├── ./README.md
├── ./tests
│   ├── ./tests/assets
│   │   └── ./tests/assets/file1.xml
│   └── ./tests/__init__.py
└── ./.vscode
    └── ./.vscode/launch.json

pyproject.toml

At the most simple we need a name for the package and a version.

[tool.poetry]
name = "kmpycore" # matches folder we are in (can use - in name?)
version = "0.1.0"
description = "Keith's generally useful bits n bobs"
authors = ["Keith Marston <keith@sneconsulting.co.uk>"]
license = "MIT"
readme = "README.md"

[tool.poetry.dependencies]
python = ">=3.8,<3.12"

[tool.poetry.group.dev.dependencies]
black = "^23.3.0"
flake8 = "^4.0.1"
mypy = "^1.3.0"
isort = "^5"
pytest = "^7"
coverage = "^7"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

If we are in a pipeline that is going to publish the distribution we need to bump the minor version with poetry before making the dists. This is a problem for later though.

From this we can use poetry to create a lock file based on the required versions - this then also goes in version control even though it is a generated file it means you need to make a consouse decision to move forward a version of a dependant libary.

Use a Makefile

But wait how do we install python, what version of pip are we using and how do we install poetry. This is all required before we even make the distribution. First lets do some nice helpful bits in our make file...

#!make
include .env
export $(shell sed 's/=.*//' .env)

# Get first name = from pyproject
NAME := $(shell grep '^name = ' pyproject.toml | head -1 | cut -d \" -f2)

INSTALL_STAMP := .install.stamp
POETRY := $(shell command -v poetry 2> /dev/null)

.DEFAULT_GOAL := help

.PHONY: help
help:  ## Display this help
    @awk 'BEGIN {FS = ":.*##"; printf "\nUsage:\n  make \033[36m<target>\033[0m\n"} /^[a-zA-Z_-]+:.*?##/ { printf "  \033[36m%-20s\033[0m %s\n", $$1, $$2 } /^##@/ { printf "\n\033[1m%s\033[0m\n", substr($$0, 5) } ' $(MAKEFILE_LIST)

And then move on to an install step followed by a package step.

At this stage we are potentially just quickly knocking up a local package - so we won't even worry about the full lintter process running as that can be annoying. Maybe put format then linit in the pre push hooks, or format as a minium.

To keep our workflow consistent we will use a Makefile so the same process is used to create the package in the pipeline as on our local machine.

Makefile advantages

Basline for a basic Python project and Makefile

https://python-poetry.org/docs/basic-usage/

Now lets consider formating and linters

Formatting is good for merge compares as general search of your code base. Before a merge a format should be performed probably in a pre commit hook:
make format
and then we can lint before the merge / down vote the merge if it is not onto a protected branch.
The point here been a prtected branch should be "protected" from code that fails a lint but a feature branch can have trial code merged but a thread to resolve raised if it does not lint.

pre-commit checks

Lets make sure we always have formated code going into our repo.

How to post the merge issue / auto review coment

non-protected branch

make lint

So now we have a "thing" that needs a level of testing before been publish on any debug/development packaging server as if it does not pass our unit tests then what is the point of doing any further testing? Utimatly from a coding team point of view the process is?...??

flowchart LR
    newCode --> MergeRequest

    MergeRequest --> linter -->|comments| MergeRequest

    MergeRequest --> unitTest[Unit Tests\npytest]
    unitTest -->|comments| MergeRequest

    MergeRequest --> intTest[Intergration tests\nHow much can we automate?]
    intTest -->|comments| MergeRequest

    MergeRequest --->|fail at tests if protected branch\nwhy waste human time?| codeReview

protected branch

versions on publishing

Look at this to manage the publish workflow logic
https://github.com/jedie/python-code-snippets/blob/master/CodeSnippets/setup_publish.py
This also uses twine to publish then which gives PGP siging support. Can we carry that though to PGP checking on download in poetry?

making the mono repo

walk you repo

#!/bin/sh
BASEDIR=$PWD
PKG=${PWD##*/}
DEPS=`grep -o '".*libs.*"' $PWD/poetry.lock | sed -e 's/"//g' | cut -d/ -f4`

mkdir -p $BASEDIR/dist/deps

for dep in $DEPS; do
  cd ../../libs/$dep
  rm -rf ./dist/*
  poetry build
  cp dist/*.whl $BASEDIR/dist/deps/.
  cd $BASEDIR
done

cd $BASEDIR
poetry build

https://github.com/dermidgen/python-monorepo/blob/master/tools/ci-build.sh

Refs:

vscode mermaid plugin for markdown preview Ctrl-P then ext install bierner.markdown-mermaid

links

social