Managing Python dependencies¶
- All pip dependency files stored in
.config
- All dependency file names must match this
requirements.(txt|in)
. Needed for dependabot compatibility.
Recommended filenames:¶
.config/requirements.in
- runtime deps.config/requirements-test.in
- test requirements.config/requirements-docs.in
- docs requirements.config/requirements-lock.txt
- locked (pinned) runtime requirements for projects havinglock
extra..config/constraints.txt
- unified testing constraint file to use asPIP_CONSTRAINTS
. Is named like this for Dependabot compatibility. It also pins all extras.
Upgrading dependencies¶
To upgrade dependencies, it's recommended to use pip-tools
as part of the pre-commit
hook and invoke manually via a tox profile named deps
.
Example .pre-commit-config.yaml
- repo: https://github.com/jazzband/pip-tools
rev: 7.3.0
hooks:
- id: pip-compile
name: deps
alias: deps
stages: [manual]
entry: pip-compile .config/requirements.in --upgrade --all-extras --no-annotate --strip-extras --output-file=.config/constraints.txt pyproject.toml
files: ^.config\/.*requirements.*$
language_version: "3.10" # minimal we support officially
Example tox.ini
[testenv:deps]
description = Bump all dependencies
base_python = python3.10
skip_install = true
deps =
{[testenv:lint]deps}
extras =
set_env =
PIP_CONSTRAINT = /dev/null
commands_pre =
commands =
-pre-commit run --all-files --show-diff-on-failure --hook-stage manual deps
-pre-commit autoupdate
git diff --exit-code
env_dir = {toxworkdir}/lint
To upgrade dependencies, execute tox -e deps
in the local project.
Dependabot Github configuration¶
To minimise the amount of PRs Dependabot would create, it is recommended to group all dependencies updates together. This can be accomplish with the following config file:
Example .github/dependabot.yml
---
version: 2
updates:
- package-ecosystem: pip
directory: /.config/
schedule:
day: sunday
interval: weekly
labels:
- dependabot-deps-updates
- skip-changelog
groups:
dependencies:
patterns:
- "*"
Dependabot quirks¶
As Dependabot has very limited configurability, filenames matter and we can only make it work well if they match.
If you have a pair of requirements.in
and requirements.txt
in a folder, dependabot always rewrite the .txt
file as being the lock for the .in
file. This means that if we use the txt as a constraint and we also have some extras, the lock file will not be correct. Dependabot will attempt to mess the file.
Dependabot parses requirements files and tries executing the same command specified in the header as a comment. Please take a look at an example header below.
#
# This file is autogenerated by pip-compile with Python 3.11
# by the following command:
#
# pip-compile --no-annotate --output-file=.config/requirements.txt --strip-extras .config/requirements.in pyproject.toml
#
The actual pip-compile
command executed by Dependbadot won't be the same as it parses arguments and only uses known ones, giving possible different results.
Dependabot does not support --extra
when running pip-compile based on requirements files. See dependabot/dependabot-core#6406