Anoduck's Das Wiki
Pages (Latest 10 updated) :
Menu (Edit):
Posts (Latest 10 updated) : Read all
Link List (Edit):
Contents:
  1. Python Package Management
    1. The evolution of virtualenvs in python.
    2. Python Virtual Environments Managers
    3. Python Project Managers
#  ____        _   _                   ____            _
# |  _ \ _   _| |_| |__   ___  _ __   |  _ \ __ _  ___| | ____ _  __ _  ___
# | |_) | | | | __| '_ \ / _ \| '_ \  | |_) / _` |/ __| |/ / _` |/ _` |/ _ \
# |  __/| |_| | |_| | | | (_) | | | | |  __/ (_| | (__|   < (_| | (_| |  __/
# |_|    \__, |\__|_| |_|\___/|_| |_| |_|   \__,_|\___|_|\_\__,_|\__, |\___|
#        |___/                                                   |___/
#  __  __                                                   _
# |  \/  | __ _ _ __   __ _  __ _  ___ _ __ ___   ___ _ __ | |_
# | |\/| |/ _` | '_ \ / _` |/ _` |/ _ \ '_ ` _ \ / _ \ '_ \| __|
# | |  | | (_| | | | | (_| | (_| |  __/ | | | | |  __/ | | | |_
# |_|  |_|\__,_|_| |_|\__,_|\__, |\___|_| |_| |_|\___|_| |_|\__|
#                           |___/
#

Python Package Management

Perhaps unwise to nest this topic in the “Programming” category, as mastering this is as important to a system administrator as managing the package management system of one’s preferred “*nix” distribution. Nevertheless, let us dive into the realm of python package management.

The evolution of virtualenvs in python.

Python has evolved rather quickly in comparison to other programming languages. Yes, there are other newer, more trendy programming languages of our time, but by comparison of it’s predecessors, Python exploded into the mainstream of computational usage. The point of this being, because of python’s popularity and rapid development it encountered problems that other programming languages have not had to endure. Mainly, by this we are referring to the issue of maintaining version compatilibility.

The issue with python and maintaining version compatilibility with a module, is simply one module will be written using one version, while another module will be written using a version that is either a few versions behind or a few versions forward. In which case, in a basic installation of python, only one version can remain, thus it is a matter of breaking the dependencies of one package in order to satisfy the requirements of another. Now, take this example of two packages with conflicting requirements and muiltiply it times 158, and what you will then have is a fucking mess.

To resolve this, one dude came up with the clever idea to contain the dependencies of a module within a virtual environment. Who this brilliant dude was, I have no idea, but he was clever, and pretty soon use of virtual environments became extremely common. This unfortunately did not completely resolve the problem, but in actuality created another issue. The process to create those virtual environments was rather tedious and time consuming, it all required a large amount of space to store an entire functioning implementation of python in it’s own environment. So, another clever little fellow came up with a way to make creation of those environments faster, and then later on more fellers jumped in and created their own implementations of short cutting the virtualenv create process.

To summarize, what is now a long winded dribble, Python now has finally decided that creation of virtualenvs is obviously the path of the future, and have implemented this process in their main package management application, pip.

TODO: Finish this damn essay later.

Python Virtual Environments Managers

As mentioned above, the following are applications used to manage python virtual environments on your system.

Pipenv

One of the original shortcuts to management virtual environments.

To install: pip3 install --user -U pipenv

To use, you will need to clone the repository of the module/application you want to use, and then cd inside of it.

  1. pipenv --pythonX.X will install an environment for the module with python version x.x.
  2. pipenv install will create a virtual environment and install all dependencies listed in requirements.txt.
  3. pipenv run $application will run the $application in the virtual env.
  4. pipenv shell will open a shell in the virtual environment.

Pipx

Pipx takes a stab at seamlessly managing all user installed python applications, to make life so much easier. If you can’t tell, I am a little partial to this tool.

To install: pip3 install --user -U pipx

Unlike with pipenv, you will not need to clone the repository, or be within it. Pipx can be executed from anywhere as long as it is in your $PATH. This is because, pipx uses python’s official repository of packages, but it can be used to install from repositories as well.

  1. pipx install $PACKAGE will install the $PACKAGE, and your done.
  2. pipx self update will update pipx.
  3. pipx upgrade-all will upgrade all of your pipx installed packages and their environments.
  4. pipx upgrade $PACKAGE will upgrade $PACKAGE
  5. pipx inject $PARENT_PACKAGE $CHILD_PACKAGE will inject the child package into the same virtual environment as the parent package, which is necessary if the child package needs to access the parent package’s libraries.
  6. pipx remove $PACKAGE will uninstall the package and remove it’s virtual environment.

You see how much easier it is.

Pyenv

Pyenv does not provide the frontend interface as with the other virtual env managers, but provides a solid backend that is used by several frontend virtual env managers, despite being developed seperately.

More TDL. TODO: Finish Later.

Python Project Managers

Below are a few tools who different slightly than their brothers above, as they do provide virtual environment and package management, but are targeted more towards developing python applications than just using them. Nevertheless they provide features end users will enjoy using.

One of the key differences between package managers and project managers is project managers manage language versioning as well as package versioning. Meaning, if a particular version of python is required for the development of a project, the project manager will allow the user to install that version of python into a virtual environment seperate from the system libraries and python distrobution.

Poetry

Poetry is more aimed at the development side of things than pipx, and does take a few more steps to setup than pipx as well. To setup a project is very similar to pipenv, although the version managment and environment configuration is exponentially more robust. If you want to develop python applications or modules, this is undoubtedly your best choice of those listed here.

To install: pip3 install --user -U poetry

Just like pipenv, you will need to be located within the project repository in order to execute successfully.

  1. poetry init will initialize poetry for the project and generate your pyproject.toml file. This file is where you can customize numerous details of your development environtment.
  2. poetry add $PACKAGE will add the $PACKAGE as a dependency to the project.
  3. poetry del $PACKAGE will remove the $PACKAGE from the dependency list.
  4. poetry install installs all the dependencies with you just gave to poetry in the previous command, and will crete your virtual env.
  5. poetry run will allow you to run and test project.
  6. poetry shell will allow you to open a shell in the environment of your project.
  7. poetry export will export the requirements.txt file of all the required dependencies for the project, this will allow users to use other tools, such as pipenv, to install the required dependencies.
  8. poetry update will update all the packages of dependencies in the virtual env.

Uv

Uv, much like poetry, is targeted more towards development of python projects, but unlike poetry it provides features regular end users would be interested in. The acclaimed benefit of uv over other project managers is it’s speed. Uv is actually written in the rust language, and is compiled into a single binary file. Rust is faster than python, because rust does not have to load the python interpreter and is binary compiled.

Installation

For Linux and Windon’t systems, uv can be installed from a precompiled binary with a single shell command.

curl -LsSf https://astral.sh/uv/install.sh | sh

For the rest of us cool folks, uv will need to be compiled from rust, also with a single command. The only downside to this is keeping uv up to date will need to be manually performed.

sudo cargo install --git https://github.com/astral-sh/uv uv
  1. uv init --no-source $PATH will create a new template project framework for creation of python projects.
  2. uv add will add packages to the project dependency list.
  3. uv remove will remove packages from the project dependency list.
  4. uv export will export the dependency list to another format.
  5. uv venv will create a virtual environment for the project located in ./.venv
  6. uv sync will update the virtual environment with changes, if any.

While experience has left us still mystified as to how some tasks are performed with uv, it still is clear uv is a viable developement environment to employ and test out.

Hatch

Hatch is even more targeted at developing python programs than any of the other project managers mentioned more on this page, hatch is also written completely in python, which means installation is much easier. The benefits provided by hatch are more customization in the developement workflow, and is focused more on testing than any of the other project managers listed here. Hatch also provide’s it’s own package building engine, which allows for more consistent builds of projects.

TDL.