In this post, I want to have some understanding of dependency management in Python. Python dependency management is a completely different world. I've been developing code for the JVM for over 20 years, first Java, and then Kotlin.

In this post, I want to have some understanding of dependency management in Python. Python dependency management is a completely different world.

20 For over the years, I have been developing code for the JVM, first Java, and then Kotlin. However, JVM is not a panacea, such as , in scripts:

1. virtual machines require extra memory
2. In many cases, the script does not run long enough to get any benefits in terms of performance. The bytecode is interpreted and never compiled into native code.

For these reasons, I am now writing scripts in Python. One of them collects social media metrics from different sources and stores them in BigQuery for analysis.

I'm not a Python developer, but I'm learning - it's hard. In this post, I want to have some understanding of dependency management in Python.

Python enough dependency management

On the JVM, dependency management seems to be a solved issue. First, you choose your build tool, preferably Maven or another tool that I shouldn't name. You then declare your direct dependency and the tool manages indirect dependencies. This doesn't mean there are no pitfalls, but you can solve them more or less quickly.

Python Dependency Management is a completely different world. First, in Python, the runtime and its dependencies are system-wide. A system has only one runtime, and dependencies are shared among all projects on that system. Because it is not feasible, the first thing to do when starting a new project is to create a virtual environment. The solution to this problem is to create a virtual environment, a self-contained directory tree that contains a Python installation of a specific version of Python, and some additional packages.

Then different applications can use different virtual environments. To resolve the previous conflicting requirements example, Application A can have its own virtual environment with version 1.0 installed, while Application B has another virtual environment with version 2.0 installed. If Application B needs to upgrade the library to version 3.0, this will not affect Application A's environment.

--The virtual environment and package

Once it is completed, things will start seriously.

pipPython provides an out-of-the-box dependency management tool:

You can install, upgrade and delete packages using a program called pip.

--Use pip management package

workflow is as follows:

1. Install required dependencies in a virtual environment: pip install flask
2. After installing all required dependencies, save them in a file named by requirements.txt by convention: pip freeze requirements.txt

   This file should be saved with regular code in one person's VCS.

3. pip Other project developers can install the same dependencies by pointing to requirements.txt: pip install -r requirements.txt

The following is the result of the above command for requirements.txt:

click==8.1.3
Flask==2.2.2
itsdangerous==2.1.2
Jinja2==3.1.2
MarkupSafe==2.1.1
Werkzeug==2.2.2

dependencies and transitive dependencies

Before describing this problem, we need to explain what is passing dependencies. Transitive dependencies are dependencies that the project does not directly need, but dependencies that one of the project's dependencies or dependencies of the dependencies are always needed. In the example above, I added the flask dependency, but pip has installed a total of 6 dependencies.

We can install the deptree dependency to check the dependency tree.

pip install deptree

output as follows:

Flask==2.2.2 # flask
Werkzeug==2.2.2 # Werkzeug=2.2.2
MarkupSafe==2.1.1 # MarkupSafe=2.1.1 # MarkupSafe=2.1.1 Jinja2==3.1.2 # Jinja2=3.0
MarkupSafe==2.1.1 # MarkupSafe=2.0
itsdangerous==2.1.2 # itsdangerous=2.0
click==8.1.3 # click=8.0

#deptree and pip trees

Its content is as follows: Flaskrequires Werkzeug, which in turn requires MarkupSafe. Werkzeug and MarkupSafe are eligible as a transitive dependency for my project.The

version part is also very interesting. The first part mentions the installed version, while the comment part refers to the compatible version range. For example, Jinja requires version 3.0 or above, and the installed version is 3.1.2.

The installed version is the latest compatible pip version found during installation. pip and deptree to understand the compatibility of the files distributed along each library of setup.py: the

installation script is the center of all activities that build, distribute and install modules using Distutils. The main purpose of setting up scripts is to describe your module distribution to Distutils so that the various commands that operate on your module perform the correct operations.

--Writing the installation script

flask here:

from setuptools import setupup

setup(
name="Flask",
install_requires=[
"Werkzeug = 2.2.2",
"Jinja2 = 3.0",
"itsdangerous = 2.0",
"click = 8.0",
"importlib-metadata = 3.6.0; python_version '3.10'",
],
],
extras_require={
"async": ["asgiref = 3.2"],
"dotenv": ["python-dotenv"],
},
)

points and transitive dependency

has problems because I want my dependencies to be up to date. To do this, I have configured Dependabot to monitor requirements.txt. When such an event occurs, it opens a PR in my repo. Most of the time, PR is like a charm, but in a few cases, an error occurs when I run the script after the merge. It looks like this:

plain text 1 error: libfoo 1.0.0 requires libbar2.5,=2.0, but you will have incompatible libbar2.5. The problem with

is that Dependabot opens a PR for each library listed in . But a new library version can be released, which is beyond the scope of compatibility.

Imagine the following situation. My project requires libfoo dependencies. In turn, libfoo requires libbar dependencies. When installing, pip uses the latest version of libfoo and the latest compatible with version of libbar. The result requirements.txt is:

plain text 1libfoo==1.0.02 library==2.0

Everything works as expected. After a while, Dependabot ran and found that libbar had released a new version of such as 2.5. Faithfully, it opened a PR to merge the following changes:

plain text 1libfoo==1.0.02 library==2.5

Whether the above problem occurs depends only on how libfoo 1.0.0 is in setup.py. If 2.5 is within the scope of compatibility, it works; if not, it won't.

pip-compile saves

problem pip is that it lists transitive dependencies and direct dependencies. Dependabot then gets the latest version of all dependencies, but does not verify that the transitive dependency version update is within that range. It may check, but the requirements.txt file format is not structured: it does not distinguish between direct dependencies and transitive dependencies. The obvious solution is to list only direct dependencies.

The good news is that pip only allows listing of direct dependencies; it automatically installs transitive dependencies. The bad news is that we now have two requirements.txt options that cannot distinguish them: some list only direct dependencies, while others list all dependencies.

It requires an alternative.pip-tools has a:

1. A list of their direct dependencies in a file, the requirements.in file format is in the same format as requirements.txt
2. The pip-compile tool requirements.txt from requirements.in.

For example, given our Flask example:

#
# This file is autogenerated by pip-compile with python 3.10
# To update, run:
#
# pip-compile requirements.in
#
click==8.1.3
# via flask
flask==2.2.2
# via -r requirements.in
itsdangerous==2.1.2
# via flask
jinja2==3.1.2
# via flask
markupsafe==2.1.1
# via flask
markupsafe==2.1.1
# via jinja2
# werkzeught
werkzeug==2.2.2
# via flask

pip install -r requirements.txt

It has the following benefits and consequences:

• generated requirements.txt contains comments to understand the dependency tree
• Because pip-compile generated files, you should not save them in VCS
• The project is compatible with legacy tools that depend on requirements.txt
• Last but not least, it changes the installation workflow. Instead of installing packages and then saving them, listing the packages first and then installing them.

In addition, Dependabot can manage pip-compile.

Conclusion

This article describes the default Python dependency management system and how it breaks automatic version upgrades. We continue to describe alternatives to pip-compile's problem solving.

Please note that Python has dependency management specifications, PEP 621 - storing project metadata in pyproject.toml. It's similar to Maven's POM, but in different formats. This is redundant in my script context because I don't need to distribute the project. But you should do this, knowing that pip-compile is compatible.