Fix your Python requirements with pip-tools

Your hand-written requirements.txt(or setup.py/pyproject.toml) doesn’t work anymore after some time? That’s because you are doing it wrong: continue reading to understand how to do it the right way.

TL;DR

Python dependencies may break after some time. Use pip-tools to lock all the dependencies and sub-dependencies in a requirements.txt, given a requirements.in (or setup.py/pyproject.toml) as input.

Why requirements break

Let’s suppose you wrote the following requirements file some years ago, while using Python 3.6.0 and pip 9.0:

# This could be either in requirements.txt, setup.py or pyproject.toml
tensorflow==1.3

All was working fine at that time.

Today you need to modify the project, but your old environment is no longer on your machine. No problem! You have a requirements file, so you just create a new environment with Python 3.6.0 and pip 9.0 and you install the requirements in it. You have locked the tensorflow version, what could go wrong?

Surprise, surprise: the pip install fails:

Collecting protobuf>=3.3.0 (from tensorflow==1.3->-r requirements.txt (line 1))
  Downloading https://files.pythonhosted.org/packages/6c/be/4e32d02bf08b8f76bf6e59f2a531690c1e4264530404501f3489ca975d9a/protobuf-4.21.0-py2.py3-none-any.whl (164kB)
    100% |████████████████████████████████| 174kB 10.6MB/s
protobuf requires Python '>=3.7' but the running Python is 3.6.0

This is happening because tensorflow==1.3 requires protobuf>=3.3.0, which was working years ago when you started the project. But after years, protobuf has been updated, and the latest version no longer supports Python 3.6.

To solve this problem, you should specify in the requirements the exact protobuf version that you were using years ago. The problem is that now you have no idea which one it was!

Prevent the problem

To prevent this from happening, years ago you should have specified all the dependencies and sub-dependencies in your requirements, not only the top-level one that you needed. To do this, you can use pip-tools.

You must specify your top-level dependencies in a requirements.in file. It has the same syntax as the requirements.txt, so you can just rename it. If you have a setup.py/pyproject.toml instead, pip-tools will read it as it is and you don’t need to do nothing.

# This could be either in requirements.in, setup.py or pyproject.toml
tensorflow==1.3

Then run:

pip install pip-tools
pip-compile

This will generate a requirements.txt that looks like this:

#
# This file is autogenerated by pip-compile with python 3.6
# To update, run:
#
#    pip-compile
#
bleach==1.5.0
    # via tensorflow-tensorboard
dataclasses==0.8
    # via werkzeug
html5lib==0.9999999
    # via
    #   bleach
    #   tensorflow-tensorboard
importlib-metadata==4.8.3
    # via markdown
markdown==3.3.7
    # via tensorflow-tensorboard
numpy==1.19.5
    # via
    #   tensorflow
    #   tensorflow-tensorboard
protobuf==3.19.6
    # via
    #   tensorflow
    #   tensorflow-tensorboard
six==1.16.0
    # via
    #   bleach
    #   html5lib
    #   tensorflow
    #   tensorflow-tensorboard
tensorflow==1.3
    # via -r requirements.in
tensorflow-tensorboard==0.1.8
    # via tensorflow
typing-extensions==4.1.1
    # via importlib-metadata
werkzeug==2.0.3
    # via tensorflow-tensorboard
wheel==0.37.1
    # via
    #   tensorflow
    #   tensorflow-tensorboard
zipp==3.6.0
    # via importlib-metadata

As you can see, all your dependencies and sub-dependencies are listed now, so you won’t face the previous problem anymore, because you can do the following to sync your environment with the dependencies in requirements.txt:

# Do this if you are in a clean environment (e.g. in a Dockerfile)
pip install -r requirements.txt
# Do this if your environment is possibly dirty (e.g. a local environment)
pip-sync

Note: someone refers to the top-level dependencies as abstract requirements, while the full list of dependencies and sub-dependencies as concrete requirements.

Apply requirements.txt to setup.py/pyproject.toml

If you are using a setup.py/pyproject.toml, probably you are installing your package in your environment like this:

pip install -e .

Now that pip-tools has generated a requirements.txt, you should first install them and then install your package:

# Do this if you are in a clean environment (e.g. in a Dockerfile)
pip install -r requirements.txt -e .
# Do this if your environment is possibly dirty (e.g. a local environment)
pip-sync
pip install -e .

Upgrade requirements

By default, pip-compile and pip-sync will use the locked versions in the requirements.txt, and will not upgrade them. This is good, because uncontrolled upgrades may break our environment when we least expect it (e.g. in our CI pipeline).

But we don’t want to stay with old Python 3.6 libraries forever, right? No problem, because we can upgrade our requirements.txt with an explicit command:

# Use this to upgrade all the packages
pip-compile --upgrade
# Use this to upgrade only tensorflow
pip-compile --upgrade-package tensorflow

Alternatives to pip-tools

pip-tools is the simplest utility that is capable of locking dependencies, but there are more complex and feature-rich alternatives.

The most notable one is Pipenv, but if your project is a package (i.e. you have a setup.py or a pyproject.toml), then probably you want to take a look at tools that tackle also other package-related problems, like Poetry or PDM.

Conclusion

Python dependencies may break after some time if you don’t prevent this from happening. Make sure to lock all your dependencies and sub-dependencies to avoid unpleasant surprises in the future.

In other words, you should always translate your abstract requirements into concrete requirements, and install the concrete requirements in your environment.