Skip to content

Managing Python dependencies

  1. All pip dependency files stored in .config
  2. All dependency file names must match this requirements.(txt|in). Needed for dependabot compatibility.
  • .config/requirements.in - runtime deps
  • .config/requirements-test.in - test requirements
  • .config/requirements-docs.in - docs requirements
  • .config/requirements-lock.txt - locked (pinned) runtime requirements for projects having lock extra.
  • .config/constraints.txt - unified testing constraint file to use as PIP_CONSTRAINTS. Is named like this for Dependabot compatibility. It also pins all extras.

Upgrading dependencies

To upgrade dependencies, it's recommended to use pip-tools as part of the pre-commit hook and invoke manually via a tox profile named deps.

Example .pre-commit-config.yaml

- repo: https://github.com/jazzband/pip-tools
    rev: 7.3.0
    hooks:
      - id: pip-compile
        name: deps
        alias: deps
        stages: [manual]
        entry: pip-compile .config/requirements.in --upgrade --all-extras --no-annotate --strip-extras --output-file=.config/constraints.txt pyproject.toml
        files: ^.config\/.*requirements.*$
        language_version: "3.10" # minimal we support officially

Example tox.ini

[testenv:deps]
description = Bump all dependencies
base_python = python3.10
skip_install = true
deps =
    {[testenv:lint]deps}
extras =
set_env =
    PIP_CONSTRAINT = /dev/null
commands_pre =
commands =
    -pre-commit run --all-files --show-diff-on-failure --hook-stage manual deps
    -pre-commit autoupdate
    git diff --exit-code
env_dir = {toxworkdir}/lint

To upgrade dependencies, execute tox -e deps in the local project.

Dependabot Github configuration

To minimise the amount of PRs Dependabot would create, it is recommended to group all dependencies updates together. This can be accomplish with the following config file:

Example .github/dependabot.yml

---
version: 2
updates:
  - package-ecosystem: pip
    directory: /.config/
    schedule:
      day: sunday
      interval: weekly
    labels:
      - dependabot-deps-updates
      - skip-changelog
    groups:
      dependencies:
        patterns:
          - "*"

Dependabot quirks

As Dependabot has very limited configurability, filenames matter and we can only make it work well if they match.

If you have a pair of requirements.in and requirements.txt in a folder, dependabot always rewrite the .txt file as being the lock for the .in file. This means that if we use the txt as a constraint and we also have some extras, the lock file will not be correct. Dependabot will attempt to mess the file.

Dependabot parses requirements files and tries executing the same command specified in the header as a comment. Please take a look at an example header below.

#
# This file is autogenerated by pip-compile with Python 3.11
# by the following command:
#
#    pip-compile --no-annotate --output-file=.config/requirements.txt --strip-extras .config/requirements.in pyproject.toml
#

The actual pip-compile command executed by Dependbadot won't be the same as it parses arguments and only uses known ones, giving possible different results. Dependabot does not support --extra when running pip-compile based on requirements files. See dependabot/dependabot-core#6406