Python Packaging & Dependency Management
Python packaging has a reputation for confusion: pip, virtualenv, setup.py, requirements.txt, Poetry, Pipenv, conda, and now uv all overlap. The good news is that the ecosystem has converged. pyproject.toml is the single project config file, virtual environments isolate dependencies per project, and lockfiles make installs reproducible. Pick a modern tool and you rarely have to think about the rest.
This page covers the concepts behind every tool (environments, specifiers, locking, wheels) and the practical workflow for applications and libraries.
TL;DR
- Always use a virtual environment per project, never the system Python.
pyproject.tomldeclares the project's metadata, dependencies, and build backend.setup.pyis legacy.- Lock exact versions (
uv.lock,poetry.lock, or pinnedrequirements.txtwith hashes) for reproducible installs. - uv is the fast, all-in-one default for new projects: Python versions, venvs, dependencies, lockfile, and tool runs.
- Applications pin everything; libraries declare compatible ranges and let the app resolve.
- Build wheels with
uv buildorpython -m build, and publish with trusted publishing from CI.
Quick Example
A new application with uv:
A teammate clones the repo and runs uv sync. They get the exact same versions from uv.lock, in seconds.
Core Concepts
Virtual Environments
A virtual environment is a directory containing a Python interpreter link and its own site-packages. Packages installed there don't affect other projects or the OS Python.
Tools like uv and Poetry create and manage the venv for you. Never sudo pip install into the system interpreter; modern Linux distributions block it for good reason (PEP 668, "externally managed environment").
pyproject.toml
The standard project file (PEP 517/518/621) holds:
[project]: name, version,requires-python,dependencies,optional-dependencies(extras), and entry points ([project.scripts]).[build-system]: which backend builds the package (hatchling, setuptools, flit-core, pdm-backend, uv_build).[dependency-groups]: dev and test dependencies that aren't published (PEP 735).[tool.*]: configuration for ruff, mypy, pytest, and more.
Version Specifiers
Lockfiles and Reproducibility
pyproject.toml says what you want ("sqlalchemy 2.x"). The lockfile records exactly what you got, every transitive dependency with versions and hashes, so CI, production, and teammates install identical environments. Commit the lockfile for applications. PEP 751 standardizes a pylock.toml format that tools are adopting.
Wheels and Source Distributions
A wheel (.whl) is a pre-built package: installing it is just unzipping, with compiled extensions included per platform. An sdist (.tar.gz) is source code that must be built on install. It's slower and may need compilers. PyPI serves wheels whenever they exist for your platform.
Choosing a Tool
Publishing a Library
- Choose a build backend in
[build-system]and keep dependency ranges broad (>=2.0,<3). - Use a
src/layout (src/mylib/__init__.py) so tests run against the installed package, not the working directory. - Build with
uv buildorpython -m build, which produces both a wheel and an sdist indist/. - Publish with trusted publishing from GitHub Actions: PyPI verifies the workflow's OIDC token, so no API token is stored anywhere.
Best Practices
Commit the Lockfile, Not the venv
Add .venv/ to .gitignore. Commit pyproject.toml and the lockfile. Anyone can recreate the environment in seconds.
Pin Python Too
Set requires-python and a .python-version file so every environment, including Docker images and CI, uses the same interpreter version. uv installs missing versions automatically.
Separate Runtime and Dev Dependencies
Test runners, linters, and type checkers belong in a dev dependency group. Production images install only runtime dependencies (uv sync --no-dev), which keeps them smaller with less attack surface.
Update Deliberately and Often
Let Renovate or Dependabot open PRs that update the lockfile, and let CI run the test suite against them. Small, frequent updates are far easier than a big-bang upgrade after two years.
Common Mistakes
Unpinned requirements.txt for Deployments
Pinning Exact Versions in a Library
A library that requires requests==2.32.3 conflicts with every app that needs a different patch version. Libraries declare ranges; applications pin.
Installing Into the Wrong Interpreter
pip install x followed by ModuleNotFoundError usually means pip and python point at different interpreters. Use python -m pip, or uv run, which always targets the project environment.
FAQ
Should I switch to uv?
For most projects, yes. It's a drop-in replacement for pip/pip-tools (uv pip install) and a full project manager (uv add, uv sync, uv run), and it's dramatically faster in CI and Docker builds. Poetry projects can migrate incrementally; the pyproject.toml [project] table is standard across tools.
What's the difference between requirements.txt and pyproject.toml?
pyproject.toml declares a project's metadata and abstract dependency ranges. requirements.txt is a flat install list, often fully pinned, used as a poor man's lockfile. Modern tools generate lockfiles from pyproject.toml; requirements.txt survives mainly for simple scripts and as an export format.
Do I need conda?
Only if you depend on non-Python binaries that aren't available as wheels, such as specific CUDA toolkits, geospatial libraries, or mixed R/Python stacks. PyPI wheels now cover NumPy, PyTorch, and most scientific packages, so plain venvs are often enough even for ML.
How do I install command-line tools globally?
Use uv tool install ruff or pipx install ruff. Each tool gets its own isolated environment, available on your PATH, without polluting any project or the system Python. uvx ruff check runs a tool without installing it permanently.
Related Topics
- Python — The language overview
- Package Managers — Dependency management across ecosystems
- Python Type Hints — Configuring type checkers in pyproject.toml
- Docker Multi-Stage Builds — Shipping Python apps in small images
- pytest — The testing tool in your dev group