Why do Python projects use virtual environments, and what are the modern tools for dependency management?

How to think about it

Picture two projects on one machine: one needs requests==2.28, the other requests==2.31. The system Python has a single site-packages, so only one version can win — and whichever loses, that project breaks. A virtual environment fixes this by giving each project its own isolated site-packages, so the two never collide.

Without a venv                  With venvs
─────────────────────────       ──────────────────────────────────
System Python                   project-a/.venv/site-packages/
└── site-packages/                 └── requests 2.28
    ├── requests 2.28           project-b/.venv/site-packages/
    └── (one version wins)         └── requests 2.31
                                each project isolated — no conflict

Creating and using one

python -m venv .venv            # create the environment (just a directory)
source .venv/bin/activate       # activate — macOS / Linux
.venv\Scripts\activate          # activate — Windows
pip install requests==2.31      # installs only into .venv
deactivate                      # back to the system Python

Because it’s only a directory, you can delete .venv and rebuild it without touching anything system-wide. Always add .venv/ to .gitignore.

requirements.txt vs a lock file

A requirements.txt usually lists direct dependencies with loose constraints (requests>=2.28). Run pip install from that on two machines and they can resolve different transitive trees — reproducibility quietly gone. A lock file instead pins every package, direct and transitive, to an exact version and hash:

requests==2.31.0 --hash=sha256:58cd2187...
certifi==2024.2.2 --hash=sha256:abc123...

Now pip install -r requirements.lock gives everyone byte-for-byte the same environment.

The modern tools

Tool	Lock file	Speed	Notes
pip-tools	compiled requirements.txt	moderate	minimal, widely used
Poetry	poetry.lock	moderate	full project manager: build + publish
uv	uv.lock	very fast (Rust)	drop-in pip replacement, PEP 517/518
conda	environment.yml	moderate	handles non-Python deps; common in ML

# uv — the fastest modern workflow
pip install uv
uv venv
uv pip install pandas scikit-learn

For shipping, pex bundles a whole environment into one executable zip — handy for a CLI tool or a Spark job that can’t manage a venv on every worker — and Docker images do the same job for server workloads.