Nikhil Kumar Thota

This guide serves as a short summary of some popular guides and documentation (referenced below) I have read when learning how to build a python package. Any views expressed in this article are my own.

Python packages provide an easy way for users to import the necessary methods and classes in your code through the python import function. A built python package is stored as a python wheel (.whl) which is a ZIP-format archive, containing all the files necessary to install your python package. Wheels are the latest standard for storing python packages, replacing eggs. Once a wheel is generated it can be distributed locally or pushed to the Python Package Index (PyPI) which is global server hosting python software. If you have ever done pip install X, then pip would most likely be downloading from PyPI.

Build Frontend vs Build Backend

Those familiar using pip should be aware that it cannot directly convert your code into a python package. That is the job of a build backend. pip and build are examples of what the community calls as build frontends, tools that help install python packages. In the background these build frontend tools identify the dependencies of the package you requested and attempt to resolve any dependency conflicts among the dependencies. They do so by downloading the source distribution for each dependency and calling the build backend on each to install the dependency and check for conflicts. Tools like build create an environment with the appropriate python packages that will invoked by the build backend tools (called setup dependencies), during the installation of your python code (called runtime dependencies) and finally during testing (called test dependencies).

At this point I should probably mention the subtle distinction between a source and built distribution. A source distribution is an archive file (stored as .tar.gz) that contains your source code and metadata. It is not platform specific (ex:Windows, Linux) distribution. While a built distribution is an archive file (stored as .whl) that is specific to your hardware, OS and python version and can be run directly without having to install python.

Build backends are responsible for converting your source tree - the directory containing all files and folders relevant to your source code - to either a source or build distribution. Examples of build backends are flit-core, hatchling, maturin, steuptools and poetry. In this tutorial we will be focusing on using setuptools.

Step 1 : pip install build

While we can invoke setuptools directly, it is easier and preferred to invoke it through the python build module. To start make sure you have build installed in your python environment type pip install --upgrade build in your terminal.

Step 2 : Structure your codebase

Now for the purpose of this demo we create a test_package directory with following layout.

test_package/
├── pyproject.toml
├── src/
│   └── test_package/
│       ├── __init__.py
│       └── math_operations.py
├── tests/
│   ├── test_add.py
│   ├── test_multiply.py
│   └── test_exp.py
├── LICENSE
├── README.md
└── test_package.yml

In this tutorial we will be focusing on pyproject.toml and the src directory. In math_operations.py lets define some simple math operations :

# math_operations.py
import numpy

def add(x, y):
    return x+y

def multiply(x, y):
    return x*y

def exp(x):
    return np.exp(x)

The __init__.py is a file that is required for the test_package directory to be registered as a module which allows us to write commands like : from test_package.math_operations import add. It can also can store information related to your source code such as authors, license and package version. Here is a sample __init__.py layout:

# __init__.py
__author__ = "your_name"
__license__= "MIT'"
__version__= "0.0.1"
__all__ = ["add", "multiply", "exp"] 

The __all__ entry is used to specify what all modules should be imported when a user does a wildcard import.

Step 3 : Write the pyproject.toml file

This file tells the build frontend tools, what build backend to use which in this case is setuptools. As a side note, for those familiar with using setup.py or setup.cfg for configuring the build of their source tree should note that this practice is slowly getting depreciated. Please refer to this blog post for more details. Below shows a very simple pyproject.toml file that covers the basic required items. For more advanced utilities I would refer you to the setuptools documentation. For instance this does not cover how to include tools like pytest unit testing and how to incorporate linting tools. When including these tools make sure to include a [project.optional-dependencies]section.

# pyproject.toml
[build-system]
requires=["setuptools"]
build-backend="setuptools.build_meta"

[project]
name="test_package"
authors=[
    {name = "your_name", email = "your_email@xyz.com"},
]
description = "your_project_description"
readme="README.md"
license= {text = "MIT"}
requires-python=">=3.8"
keywords=["some_keywords", "separated_by_commas"]
dependencies = [
          "numpy",
]
dynamic=["version"]

[project.urls]
Homepage = "https://your_website.com"
Documentation = "https://readthedocs.org"
Repository = "https://github.com/me/your_git_repo.git"

[tool.setuptools.dynamic]
version={attr="test_package.__version__"}

Step 4 : Building the wheel and source distribution using build

Now go back to the directory containing pyproject.toml and in your terminal type python -m build --wheel to create a built distribution or type python -m build --sdist to create a source distribution. Both the built and source distributions are located under the dist directory. The wheel will be listed with the package version number and python versions as test_package-0.0.1-py3-none-any.whl.

Once we have the built distribution we can use pip to install the package and its dependencies (which in this case is numpy). To do this go into the dist directory and type pip install test_package-0.0.1-py3-none-any.whl in your terminal.

Voilá ! you have now created your very own (and maybe first!) python package. You can now type python in your terminal and import the different math functions we have defined.

user@computer python
>> from test_package import math_operations
>> math_operations.add(2, 3)
5
>> math_operations.exp(2)
7.38905609893065

References :
[1] https://packaging.python.org/en/latest/tutorials/packaging-projects/
[2] https://setuptools.pypa.io/en/latest/userguide/quickstart.html
[3] Some GitHub repos you can refer to : https://github.com/joreiff/rodeo, https://github.com/rxhernandez/ReLMM